Apr 16 00:00:30 rdma06 systemd[1]: Starting Rotate log files...
Apr 16 00:00:30 rdma06 systemd[1]: rsyslog.service: Sent signal SIGHUP to main process 1041 (rsyslogd) on client request.
Apr 16 00:00:30 rdma06 systemd[1]: logrotate.service: Deactivated successfully.
Apr 16 00:00:30 rdma06 systemd[1]: Finished Rotate log files.
Apr 16 00:00:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:00:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:00:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:00:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:00:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:00:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:00:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:00:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:00:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:00:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:00:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:00:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:00:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:01:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:01:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:01:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:01:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:01:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:01:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:01:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:01:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:01:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:01:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:01:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:01:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:01:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:01:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:01:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:01:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:01:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:01:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:01:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:01:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:01:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:01:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:01:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:01:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:01:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:01:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:01:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:01:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:02:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:02:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:02:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:02:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:02:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:02:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:02:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:02:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:02:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:02:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:02:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:02:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:02:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:02:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:02:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:02:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:02:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:02:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:02:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:02:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:02:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:02:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:02:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:02:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:02:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:02:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:02:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:03:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:03:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:03:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:03:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:03:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:03:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:03:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:03:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:03:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:03:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:03:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:03:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:03:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:03:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:03:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:03:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:03:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:03:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:03:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:03:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:03:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:03:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:03:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:03:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:03:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:03:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:03:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:04:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:04:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:04:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:04:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:04:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:04:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:04:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:04:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:04:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:04:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:04:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:04:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:04:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:04:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:04:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:04:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:04:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:04:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:04:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:04:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:04:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:04:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:04:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:04:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:04:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:04:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:04:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:04:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:05:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:05:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:05:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:05:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:05:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:05:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:05:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:05:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:05:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:05:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:05:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:05:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:05:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:05:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:05:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:05:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:05:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:05:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:05:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:05:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:05:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:05:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:05:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:05:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:05:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:05:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:05:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:06:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:06:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:06:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:06:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:06:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:06:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:06:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:06:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:06:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:06:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:06:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:06:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:06:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:06:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:06:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:06:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:06:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:06:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:06:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:06:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:06:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:06:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:06:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:06:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:06:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:06:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:06:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:07:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:07:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:07:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:07:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:07:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:07:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:07:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:07:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:07:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:07:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:07:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:07:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:07:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:07:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:07:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:07:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:07:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:07:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:07:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:07:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:07:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:07:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:07:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:07:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:07:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:07:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:07:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:07:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:08:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:08:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:08:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:08:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:08:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:08:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:08:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:08:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:08:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:08:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:08:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:08:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:08:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:08:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:08:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:08:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:08:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:08:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:08:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:08:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:08:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:08:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:08:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:08:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:08:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:08:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:08:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:09:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:09:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:09:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:09:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:09:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:09:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:09:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:09:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:09:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:09:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:09:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:09:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:09:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:09:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:09:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:09:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:09:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:09:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:09:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:09:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:09:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:09:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:09:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:09:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:09:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:09:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:09:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:10:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:10:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:10:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:10:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:10:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:10:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:10:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:10:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:10:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:10:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:10:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:10:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:10:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:10:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:10:30 rdma06 rsyslogd[1041]: [origin software="rsyslogd" swVersion="8.2102.0-113.el9_2" x-pid="1041" x-info="https://www.rsyslog.com"] rsyslogd was HUPed
Apr 16 00:10:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:10:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:10:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:10:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:10:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:10:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:10:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:10:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:10:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:10:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:10:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:10:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:10:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:10:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:11:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:11:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:11:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:11:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:11:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:11:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:11:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:11:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:11:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:11:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:11:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:11:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:11:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:11:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:11:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:11:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:11:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:11:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:11:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:11:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:11:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:11:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:11:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:11:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:11:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:11:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:11:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:12:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:12:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:12:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:12:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:12:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:12:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:12:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:12:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:12:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:12:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:12:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:12:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:12:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:12:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:12:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:12:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:12:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:12:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:12:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:12:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:12:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:12:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:12:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:12:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:12:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:12:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:12:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:13:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:13:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:13:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:13:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:13:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:13:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:13:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:13:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:13:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:13:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:13:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:13:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:13:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:13:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:13:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:13:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:13:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:13:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:13:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:13:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:13:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:13:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:13:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:13:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:13:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:13:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:13:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:14:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:14:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:14:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:14:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:14:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:14:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:14:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:14:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:14:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:14:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:14:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:14:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:14:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:14:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:14:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:14:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:14:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:14:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:14:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:14:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:14:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:14:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:14:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:14:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:14:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:14:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:14:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:14:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:15:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:15:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:15:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:15:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:15:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:15:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:15:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:15:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:15:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:15:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:15:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:15:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:15:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:15:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:15:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:15:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:15:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:15:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:15:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:15:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:15:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:15:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:15:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:15:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:15:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:15:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:15:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:16:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:16:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:16:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:16:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:16:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:16:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:16:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:16:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:16:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:16:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:16:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:16:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:16:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:16:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:16:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:16:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:16:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:16:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:16:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:16:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:16:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:16:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:16:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:16:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:16:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:16:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:16:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:17:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:17:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:17:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:17:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:17:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:17:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:17:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:17:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:17:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:17:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:17:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:17:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:17:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:17:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:17:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:17:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:17:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:17:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:17:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:17:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:17:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:17:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:17:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:17:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:17:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:17:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:17:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:17:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:18:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:18:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:18:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:18:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:18:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:18:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:18:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:18:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:18:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:18:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:18:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:18:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:18:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:18:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:18:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:18:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:18:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:18:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:18:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:18:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:18:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:18:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:18:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:18:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:18:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:18:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:18:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:19:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:19:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:19:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:19:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:19:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:19:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:19:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:19:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:19:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:19:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:19:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:19:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:19:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:19:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:19:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:19:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:19:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:19:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:19:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:19:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:19:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:19:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:19:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:19:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:19:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:19:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:19:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:20:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:20:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:20:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:20:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:20:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:20:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:20:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:20:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:20:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:20:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:20:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:20:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:20:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:20:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:20:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:20:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:20:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:20:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:20:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:20:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:20:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:20:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:20:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:20:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:20:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:20:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:20:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:20:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:21:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:21:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:21:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:21:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:21:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:21:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:21:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:21:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:21:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:21:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:21:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:21:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:21:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:21:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:21:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:21:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:21:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:21:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:21:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:21:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:21:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:21:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:21:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:21:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:21:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:21:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:21:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:22:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:22:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:22:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:22:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:22:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:22:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:22:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:22:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:22:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:22:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:22:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:22:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:22:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:22:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:22:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:22:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:22:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:22:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:22:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:22:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:22:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:22:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:22:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:22:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:22:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:22:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:22:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:23:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:23:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:23:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:23:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:23:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:23:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:23:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:23:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:23:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:23:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:23:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:23:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:23:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:23:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:23:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:23:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:23:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:23:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:23:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:23:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:23:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:23:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:23:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:23:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:23:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:23:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:23:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:23:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:24:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:24:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:24:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:24:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:24:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:24:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:24:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:24:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:24:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:24:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:24:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:24:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:24:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:24:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:24:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:24:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:24:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:24:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:24:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:24:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:24:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:24:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:24:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:24:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:24:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:24:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:24:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:25:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:25:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:25:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:25:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:25:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:25:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:25:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:25:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:25:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:25:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:25:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:25:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:25:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:25:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:25:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:25:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:25:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:25:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:25:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:25:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:25:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:25:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:25:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:25:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:25:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:25:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:25:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:26:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:26:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:26:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:26:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:26:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:26:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:26:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:26:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:26:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:26:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:26:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:26:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:26:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:26:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:26:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:26:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:26:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:26:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:26:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:26:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:26:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:26:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:26:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:26:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:26:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:26:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:26:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:26:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:27:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:27:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:27:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:27:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:27:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:27:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:27:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:27:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:27:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:27:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:27:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:27:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:27:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:27:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:27:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:27:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:27:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:27:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:27:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:27:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:27:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:27:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:27:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:27:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:27:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:27:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:27:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:28:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:28:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:28:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:28:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:28:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:28:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:28:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:28:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:28:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:28:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:28:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:28:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:28:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:28:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:28:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:28:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:28:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:28:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:28:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:28:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:28:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:28:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:28:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:28:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:28:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:28:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:28:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:29:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:29:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:29:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:29:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:29:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:29:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:29:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:29:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:29:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:29:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:29:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:29:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:29:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:29:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:29:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:29:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:29:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:29:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:29:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:29:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:29:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:29:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:29:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:29:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:29:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:29:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:29:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:29:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:30:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:30:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:30:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:30:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:30:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:30:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:30:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:30:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:30:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:30:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:30:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:30:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:30:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:30:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:30:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:30:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:30:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:30:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:30:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:30:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:30:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:30:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:30:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:30:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:30:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:30:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:30:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:31:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:31:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:31:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:31:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:31:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:31:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:31:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:31:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:31:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:31:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:31:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:31:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:31:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:31:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:31:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:31:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:31:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:31:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:31:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:31:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:31:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:31:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:31:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:31:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:31:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:31:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:31:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:32:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:32:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:32:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:32:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:32:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:32:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:32:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:32:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:32:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:32:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:32:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:32:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:32:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:32:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:32:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:32:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:32:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:32:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:32:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:32:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:32:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:32:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:32:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:32:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:32:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:32:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:32:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:33:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:33:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:33:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:33:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:33:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:33:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:33:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:33:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:33:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:33:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:33:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:33:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:33:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:33:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:33:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:33:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:33:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:33:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:33:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:33:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:33:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:33:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:33:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:33:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:33:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:33:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:33:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:33:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:34:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:34:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:34:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:34:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:34:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:34:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:34:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:34:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:34:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:34:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:34:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:34:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:34:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:34:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:34:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:34:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:34:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:34:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:34:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:34:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:34:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:34:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:34:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:34:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:34:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:34:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:34:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:35:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:35:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:35:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:35:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:35:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:35:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:35:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:35:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:35:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:35:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:35:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:35:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:35:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:35:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:35:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:35:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:35:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:35:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:35:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:35:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:35:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:35:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:35:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:35:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:35:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:35:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:35:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:36:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:36:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:36:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:36:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:36:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:36:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:36:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:36:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:36:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:36:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:36:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:36:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:36:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:36:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:36:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:36:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:36:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:36:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:36:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:36:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:36:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:36:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:36:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:36:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:36:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:36:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:36:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:36:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:37:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:37:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:37:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:37:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:37:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:37:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:37:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:37:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:37:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:37:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:37:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:37:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:37:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:37:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:37:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:37:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:37:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:37:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:37:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:37:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:37:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:37:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:37:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:37:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:37:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:37:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:37:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:38:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:38:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:38:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:38:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:38:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:38:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:38:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:38:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:38:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:38:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:38:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:38:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:38:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:38:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:38:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:38:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:38:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:38:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:38:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:38:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:38:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:38:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:38:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:38:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:38:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:38:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:38:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:39:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:39:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:39:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:39:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:39:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:39:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:39:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:39:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:39:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:39:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:39:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:39:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:39:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:39:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:39:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:39:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:39:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:39:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:39:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:39:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:39:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:39:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:39:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:39:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:39:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:39:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:39:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:39:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:40:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:40:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:40:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:40:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:40:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:40:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:40:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:40:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:40:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:40:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:40:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:40:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:40:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:40:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:40:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:40:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:40:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:40:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:40:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:40:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:40:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:40:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:40:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:40:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:40:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:40:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:40:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:41:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:41:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:41:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:41:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:41:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:41:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:41:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:41:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:41:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:41:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:41:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:41:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:41:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:41:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:41:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:41:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:41:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:41:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:41:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:41:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:41:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:41:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:41:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:41:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:41:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:41:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:41:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:42:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:42:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:42:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:42:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:42:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:42:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:42:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:42:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:42:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:42:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:42:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:42:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:42:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:42:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:42:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:42:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:42:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:42:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:42:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:42:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:42:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:42:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:42:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:42:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:42:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:42:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:42:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:42:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:43:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:43:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:43:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:43:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:43:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:43:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:43:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:43:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:43:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:43:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:43:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:43:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:43:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:43:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:43:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:43:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:43:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:43:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:43:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:43:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:43:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:43:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:43:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:43:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:43:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:43:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:43:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:44:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:44:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:44:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:44:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:44:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:44:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:44:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:44:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:44:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:44:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:44:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:44:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:44:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:44:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:44:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:44:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:44:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:44:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:44:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:44:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:44:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:44:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:44:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:44:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:44:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:44:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:44:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:45:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:45:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:45:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:45:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:45:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:45:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:45:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:45:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:45:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:45:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:45:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:45:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:45:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:45:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:45:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:45:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:45:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:45:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:45:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:45:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:45:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:45:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:45:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:45:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:45:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:45:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:45:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:45:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:46:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:46:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:46:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:46:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:46:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:46:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:46:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:46:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:46:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:46:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:46:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:46:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:46:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:46:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:46:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:46:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:46:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:46:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:46:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:46:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:46:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:46:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:46:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:46:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:46:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:46:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:46:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:47:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:47:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:47:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:47:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:47:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:47:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:47:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:47:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:47:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:47:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:47:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:47:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:47:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:47:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:47:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:47:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:47:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:47:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:47:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:47:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:47:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:47:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:47:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:47:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:47:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:47:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:47:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:48:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:48:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:48:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:48:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:48:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:48:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:48:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:48:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:48:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:48:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:48:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:48:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:48:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:48:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:48:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:48:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:48:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:48:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:48:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:48:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:48:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:48:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:48:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:48:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:48:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:48:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:48:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:49:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:49:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:49:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:49:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:49:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:49:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:49:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:49:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:49:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:49:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:49:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:49:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:49:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:49:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:49:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:49:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:49:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:49:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:49:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:49:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:49:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:49:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:49:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:49:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:49:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:49:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:49:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:49:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:50:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:50:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:50:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:50:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:50:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:50:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:50:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:50:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:50:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:50:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:50:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:50:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:50:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:50:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:50:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:50:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:50:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:50:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:50:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:50:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:50:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:50:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:50:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:50:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:50:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:50:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:50:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:51:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:51:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:51:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:51:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:51:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:51:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:51:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:51:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:51:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:51:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:51:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:51:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:51:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:51:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:51:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:51:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:51:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:51:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:51:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:51:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:51:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:51:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:51:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:51:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:51:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:51:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:51:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:52:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:52:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:52:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:52:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:52:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:52:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:52:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:52:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:52:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:52:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:52:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:52:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:52:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:52:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:52:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:52:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:52:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:52:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:52:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:52:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:52:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:52:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:52:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:52:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:52:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:52:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:52:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:52:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:53:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:53:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:53:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:53:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:53:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:53:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:53:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:53:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:53:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:53:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:53:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:53:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:53:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:53:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:53:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:53:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:53:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:53:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:53:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:53:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:53:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:53:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:53:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:53:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:53:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:53:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:53:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:54:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:54:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:54:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:54:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:54:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:54:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:54:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:54:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:54:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:54:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:54:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:54:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:54:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:54:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:54:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:54:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:54:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:54:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:54:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:54:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:54:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:54:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:54:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:54:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:54:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:54:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:54:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:55:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:55:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:55:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:55:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:55:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:55:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:55:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:55:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:55:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:55:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:55:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:55:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:55:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:55:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:55:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:55:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:55:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:55:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:55:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:55:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:55:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:55:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:55:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:55:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:55:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:55:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:55:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:55:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:56:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:56:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:56:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:56:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:56:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:56:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:56:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:56:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:56:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:56:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:56:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:56:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:56:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:56:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:56:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:56:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:56:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:56:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:56:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:56:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:56:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:56:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:56:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:56:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:56:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:56:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:56:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:57:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:57:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:57:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:57:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:57:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:57:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:57:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:57:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:57:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:57:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:57:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:57:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:57:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:57:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:57:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:57:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:57:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:57:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:57:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:57:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:57:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:57:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:57:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:57:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:57:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:57:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:57:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:58:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:58:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:58:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:58:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:58:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:58:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:58:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:58:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:58:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:58:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:58:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:58:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:58:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:58:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:58:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:58:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:58:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:58:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:58:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:58:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:58:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:58:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:58:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:58:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:58:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:58:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:58:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:58:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:59:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:59:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:59:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:59:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:59:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:59:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:59:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:59:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:59:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:59:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:59:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:59:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:59:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:59:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:59:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:59:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:59:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:59:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:59:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:59:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:59:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:59:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 00:59:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 00:59:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 00:59:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 00:59:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 00:59:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:00:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:00:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:00:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:00:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:00:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:00:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:00:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:00:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:00:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:00:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:00:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:00:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:00:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:00:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:00:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:00:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:00:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:00:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:00:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:00:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:00:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:00:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:00:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:00:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:00:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:00:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:00:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:01:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:01:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:01:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:01:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:01:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:01:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:01:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:01:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:01:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:01:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:01:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:01:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:01:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:01:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:01:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:01:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:01:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:01:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:01:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:01:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:01:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:01:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:01:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:01:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:01:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:01:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:01:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:01:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:02:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:02:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:02:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:02:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:02:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:02:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:02:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:02:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:02:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:02:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:02:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:02:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:02:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:02:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:02:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:02:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:02:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:02:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:02:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:02:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:02:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:02:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:02:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:02:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:02:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:02:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:02:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:03:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:03:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:03:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:03:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:03:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:03:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:03:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:03:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:03:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:03:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:03:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:03:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:03:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:03:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:03:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:03:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:03:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:03:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:03:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:03:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:03:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:03:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:03:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:03:47 rdma06 systemd[1]: Starting dnf makecache...
Apr 16 01:03:48 rdma06 dnf[1077680]: Updating Subscription Management repositories.
Apr 16 01:03:48 rdma06 dnf[1077680]: Unable to read consumer identity
Apr 16 01:03:48 rdma06 dnf[1077680]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 16 01:03:48 rdma06 dnf[1077680]: Metadata cache refreshed recently.
Apr 16 01:03:48 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 16 01:03:48 rdma06 systemd[1]: Finished dnf makecache.
Apr 16 01:03:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:03:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:03:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:03:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:04:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:04:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:04:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:04:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:04:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:04:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:04:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:04:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:04:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:04:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:04:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:04:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:04:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:04:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:04:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:04:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:04:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:04:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:04:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:04:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:04:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:04:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:04:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:04:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:04:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:04:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:04:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:05:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:05:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:05:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:05:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:05:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:05:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:05:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:05:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:05:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:05:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:05:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:05:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:05:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:05:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:05:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:05:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:05:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:05:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:05:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:05:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:05:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:05:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:05:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:05:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:05:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:05:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:05:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:05:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:06:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:06:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:06:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:06:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:06:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:06:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:06:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:06:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:06:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:06:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:06:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:06:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:06:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:06:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:06:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:06:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:06:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:06:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:06:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:06:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:06:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:06:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:06:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:06:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:06:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:06:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:06:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:07:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:07:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:07:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:07:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:07:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:07:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:07:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:07:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:07:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:07:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:07:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:07:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:07:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:07:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:07:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:07:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:07:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:07:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:07:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:07:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:07:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:07:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:07:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:07:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:07:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:07:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:07:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:08:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:08:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:08:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:08:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:08:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:08:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:08:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:08:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:08:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:08:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:08:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:08:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:08:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:08:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:08:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:08:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:08:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:08:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:08:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:08:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:08:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:08:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:08:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:08:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:08:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:08:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:08:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:08:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:09:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:09:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:09:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:09:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:09:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:09:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:09:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:09:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:09:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:09:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:09:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:09:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:09:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:09:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:09:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:09:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:09:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:09:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:09:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:09:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:09:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:09:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:09:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:09:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:09:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:09:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:09:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:10:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:10:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:10:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:10:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:10:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:10:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:10:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:10:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:10:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:10:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:10:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:10:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:10:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:10:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:10:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:10:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:10:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:10:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:10:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:10:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:10:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:10:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:10:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:10:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:10:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:10:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:10:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:11:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:11:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:11:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:11:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:11:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:11:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:11:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:11:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:11:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:11:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:11:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:11:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:11:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:11:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:11:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:11:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:11:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:11:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:11:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:11:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:11:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:11:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:11:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:11:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:11:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:11:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:11:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:11:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:12:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:12:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:12:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:12:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:12:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:12:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:12:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:12:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:12:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:12:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:12:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:12:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:12:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:12:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:12:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:12:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:12:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:12:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:12:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:12:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:12:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:12:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:12:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:12:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:12:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:12:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:12:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:13:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:13:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:13:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:13:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:13:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:13:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:13:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:13:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:13:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:13:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:13:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:13:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:13:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:13:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:13:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:13:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:13:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:13:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:13:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:13:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:13:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:13:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:13:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:13:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:13:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:13:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:13:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:14:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:14:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:14:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:14:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:14:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:14:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:14:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:14:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:14:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:14:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:14:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:14:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:14:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:14:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:14:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:14:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:14:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:14:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:14:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:14:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:14:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:14:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:14:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:14:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:14:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:14:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:14:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:14:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:15:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:15:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:15:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:15:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:15:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:15:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:15:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:15:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:15:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:15:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:15:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:15:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:15:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:15:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:15:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:15:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:15:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:15:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:15:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:15:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:15:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:15:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:15:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:15:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:15:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:15:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:15:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:16:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:16:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:16:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:16:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:16:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:16:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:16:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:16:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:16:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:16:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:16:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:16:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:16:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:16:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:16:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:16:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:16:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:16:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:16:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:16:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:16:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:16:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:16:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:16:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:16:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:16:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:16:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:17:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:17:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:17:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:17:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:17:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:17:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:17:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:17:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:17:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:17:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:17:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:17:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:17:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:17:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:17:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:17:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:17:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:17:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:17:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:17:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:17:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:17:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:17:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:17:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:17:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:17:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:17:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:17:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:18:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:18:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:18:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:18:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:18:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:18:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:18:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:18:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:18:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:18:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:18:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:18:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:18:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:18:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:18:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:18:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:18:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:18:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:18:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:18:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:18:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:18:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:18:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:18:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:18:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:18:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:18:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:19:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:19:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:19:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:19:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:19:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:19:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:19:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:19:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:19:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:19:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:19:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:19:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:19:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:19:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:19:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:19:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:19:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:19:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:19:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:19:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:19:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:19:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:19:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:19:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:19:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:19:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:19:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:20:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:20:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:20:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:20:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:20:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:20:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:20:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:20:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:20:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:20:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:20:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:20:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:20:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:20:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:20:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:20:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:20:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:20:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:20:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:20:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:20:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:20:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:20:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:20:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:20:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:20:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:20:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:21:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:21:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:21:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:21:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:21:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:21:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:21:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:21:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:21:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:21:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:21:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:21:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:21:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:21:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:21:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:21:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:21:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:21:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:21:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:21:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:21:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:21:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:21:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:21:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:21:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:21:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:21:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:21:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:22:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:22:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:22:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:22:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:22:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:22:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:22:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:22:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:22:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:22:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:22:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:22:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:22:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:22:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:22:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:22:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:22:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:22:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:22:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:22:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:22:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:22:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:22:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:22:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:22:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:22:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:22:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:23:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:23:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:23:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:23:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:23:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:23:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:23:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:23:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:23:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:23:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:23:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:23:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:23:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:23:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:23:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:23:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:23:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:23:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:23:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:23:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:23:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:23:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:23:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:23:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:23:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:23:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:23:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:24:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:24:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:24:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:24:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:24:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:24:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:24:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:24:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:24:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:24:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:24:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:24:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:24:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:24:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:24:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:24:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:24:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:24:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:24:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:24:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:24:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:24:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:24:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:24:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:24:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:24:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:24:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:24:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:25:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:25:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:25:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:25:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:25:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:25:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:25:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:25:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:25:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:25:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:25:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:25:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:25:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:25:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:25:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:25:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:25:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:25:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:25:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:25:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:25:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:25:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:25:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:25:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:25:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:25:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:25:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:26:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:26:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:26:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:26:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:26:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:26:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:26:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:26:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:26:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:26:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:26:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:26:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:26:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:26:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:26:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:26:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:26:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:26:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:26:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:26:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:26:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:26:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:26:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:26:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:26:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:26:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:26:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:27:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:27:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:27:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:27:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:27:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:27:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:27:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:27:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:27:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:27:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:27:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:27:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:27:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:27:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:27:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:27:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:27:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:27:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:27:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:27:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:27:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:27:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:27:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:27:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:27:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:27:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:27:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:27:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:28:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:28:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:28:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:28:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:28:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:28:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:28:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:28:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:28:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:28:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:28:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:28:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:28:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:28:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:28:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:28:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:28:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:28:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:28:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:28:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:28:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:28:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:28:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:28:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:28:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:28:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:28:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:29:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:29:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:29:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:29:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:29:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:29:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:29:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:29:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:29:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:29:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:29:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:29:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:29:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:29:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:29:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:29:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:29:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:29:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:29:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:29:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:29:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:29:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:29:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:29:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:29:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:29:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:29:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:30:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:30:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:30:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:30:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:30:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:30:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:30:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:30:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:30:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:30:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:30:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:30:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:30:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:30:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:30:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:30:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:30:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:30:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:30:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:30:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:30:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:30:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:30:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:30:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:30:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:30:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:30:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:30:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:31:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:31:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:31:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:31:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:31:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:31:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:31:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:31:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:31:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:31:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:31:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:31:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:31:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:31:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:31:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:31:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:31:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:31:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:31:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:31:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:31:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:31:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:31:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:31:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:31:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:31:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:31:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:32:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:32:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:32:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:32:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:32:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:32:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:32:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:32:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:32:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:32:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:32:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:32:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:32:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:32:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:32:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:32:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:32:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:32:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:32:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:32:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:32:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:32:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:32:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:32:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:32:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:32:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:32:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:33:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:33:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:33:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:33:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:33:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:33:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:33:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:33:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:33:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:33:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:33:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:33:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:33:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:33:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:33:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:33:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:33:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:33:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:33:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:33:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:33:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:33:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:33:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:33:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:33:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:33:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:33:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:33:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:34:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:34:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:34:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:34:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:34:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:34:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:34:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:34:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:34:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:34:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:34:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:34:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:34:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:34:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:34:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:34:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:34:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:34:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:34:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:34:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:34:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:34:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:34:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:34:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:34:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:34:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:34:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:35:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:35:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:35:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:35:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:35:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:35:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:35:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:35:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:35:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:35:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:35:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:35:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:35:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:35:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:35:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:35:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:35:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:35:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:35:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:35:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:35:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:35:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:35:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:35:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:35:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:35:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:35:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:36:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:36:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:36:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:36:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:36:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:36:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:36:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:36:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:36:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:36:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:36:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:36:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:36:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:36:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:36:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:36:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:36:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:36:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:36:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:36:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:36:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:36:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:36:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:36:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:36:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:36:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:36:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:36:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:37:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:37:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:37:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:37:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:37:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:37:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:37:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:37:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:37:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:37:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:37:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:37:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:37:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:37:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:37:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:37:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:37:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:37:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:37:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:37:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:37:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:37:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:37:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:37:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:37:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:37:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:37:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:38:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:38:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:38:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:38:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:38:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:38:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:38:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:38:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:38:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:38:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:38:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:38:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:38:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:38:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:38:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:38:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:38:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:38:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:38:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:38:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:38:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:38:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:38:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:38:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:38:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:38:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:38:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:39:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:39:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:39:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:39:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:39:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:39:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:39:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:39:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:39:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:39:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:39:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:39:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:39:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:39:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:39:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:39:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:39:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:39:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:39:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:39:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:39:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:39:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:39:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:39:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:39:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:39:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:39:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:40:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:40:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:40:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:40:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:40:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:40:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:40:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:40:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:40:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:40:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:40:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:40:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:40:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:40:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:40:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:40:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:40:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:40:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:40:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:40:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:40:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:40:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:40:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:40:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:40:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:40:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:40:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:40:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:41:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:41:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:41:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:41:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:41:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:41:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:41:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:41:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:41:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:41:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:41:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:41:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:41:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:41:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:41:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:41:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:41:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:41:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:41:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:41:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:41:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:41:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:41:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:41:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:41:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:41:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:41:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:42:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:42:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:42:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:42:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:42:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:42:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:42:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:42:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:42:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:42:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:42:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:42:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:42:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:42:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:42:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:42:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:42:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:42:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:42:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:42:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:42:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:42:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:42:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:42:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:42:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:42:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:42:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:43:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:43:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:43:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:43:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:43:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:43:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:43:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:43:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:43:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:43:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:43:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:43:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:43:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:43:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:43:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:43:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:43:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:43:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:43:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:43:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:43:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:43:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:43:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:43:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:43:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:43:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:43:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:43:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:44:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:44:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:44:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:44:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:44:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:44:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:44:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:44:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:44:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:44:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:44:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:44:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:44:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:44:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:44:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:44:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:44:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:44:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:44:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:44:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:44:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:44:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:44:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:44:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:44:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:44:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:44:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:45:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:45:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:45:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:45:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:45:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:45:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:45:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:45:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:45:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:45:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:45:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:45:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:45:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:45:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:45:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:45:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:45:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:45:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:45:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:45:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:45:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:45:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:45:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:45:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:45:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:45:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:45:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:46:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:46:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:46:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:46:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:46:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:46:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:46:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:46:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:46:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:46:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:46:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:46:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:46:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:46:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:46:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:46:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:46:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:46:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:46:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:46:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:46:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:46:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:46:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:46:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:46:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:46:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:46:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:46:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:47:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:47:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:47:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:47:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:47:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:47:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:47:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:47:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:47:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:47:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:47:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:47:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:47:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:47:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:47:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:47:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:47:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:47:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:47:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:47:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:47:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:47:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:47:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:47:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:47:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:47:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:47:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:48:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:48:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:48:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:48:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:48:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:48:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:48:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:48:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:48:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:48:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:48:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:48:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:48:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:48:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:48:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:48:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:48:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:48:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:48:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:48:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:48:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:48:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:48:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:48:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:48:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:48:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:48:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:49:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:49:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:49:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:49:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:49:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:49:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:49:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:49:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:49:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:49:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:49:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:49:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:49:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:49:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:49:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:49:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:49:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:49:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:49:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:49:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:49:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:49:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:49:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:49:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:49:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:49:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:49:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:49:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:50:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:50:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:50:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:50:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:50:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:50:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:50:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:50:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:50:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:50:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:50:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:50:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:50:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:50:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:50:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:50:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:50:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:50:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:50:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:50:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:50:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:50:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:50:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:50:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:50:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:50:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:50:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:51:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:51:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:51:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:51:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:51:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:51:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:51:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:51:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:51:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:51:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:51:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:51:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:51:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:51:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:51:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:51:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:51:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:51:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:51:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:51:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:51:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:51:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:51:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:51:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:51:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:51:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:51:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:52:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:52:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:52:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:52:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:52:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:52:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:52:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:52:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:52:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:52:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:52:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:52:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:52:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:52:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:52:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:52:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:52:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:52:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:52:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:52:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:52:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:52:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:52:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:52:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:52:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:52:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:52:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:52:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:53:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:53:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:53:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:53:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:53:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:53:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:53:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:53:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:53:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:53:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:53:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:53:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:53:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:53:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:53:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:53:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:53:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:53:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:53:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:53:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:53:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:53:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:53:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:53:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:53:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:53:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:53:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:54:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:54:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:54:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:54:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:54:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:54:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:54:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:54:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:54:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:54:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:54:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:54:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:54:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:54:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:54:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:54:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:54:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:54:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:54:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:54:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:54:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:54:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:54:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:54:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:54:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:54:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:54:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:55:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:55:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:55:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:55:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:55:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:55:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:55:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:55:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:55:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:55:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:55:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:55:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:55:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:55:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:55:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:55:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:55:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:55:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:55:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:55:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:55:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:55:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:55:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:55:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:55:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:55:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:55:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:56:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:56:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:56:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:56:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:56:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:56:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:56:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:56:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:56:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:56:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:56:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:56:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:56:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:56:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:56:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:56:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:56:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:56:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:56:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:56:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:56:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:56:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:56:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:56:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:56:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:56:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:56:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:56:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:57:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:57:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:57:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:57:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:57:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:57:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:57:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:57:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:57:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:57:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:57:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:57:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:57:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:57:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:57:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:57:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:57:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:57:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:57:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:57:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:57:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:57:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:57:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:57:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:57:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:57:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:57:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:58:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:58:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:58:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:58:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:58:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:58:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:58:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:58:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:58:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:58:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:58:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:58:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:58:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:58:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:58:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:58:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:58:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:58:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:58:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:58:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:58:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:58:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:58:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:58:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:58:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:58:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:58:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:59:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:59:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:59:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:59:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:59:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:59:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:59:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:59:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:59:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:59:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:59:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:59:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:59:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:59:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:59:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:59:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:59:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:59:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:59:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 01:59:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:59:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:59:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:59:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:59:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 01:59:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 01:59:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 01:59:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 01:59:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:00:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:00:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:00:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:00:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:00:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:00:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:00:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:00:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:00:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:00:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:00:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:00:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:00:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:00:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:00:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:00:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:00:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:00:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:00:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:00:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:00:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:00:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:00:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:00:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:00:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:00:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:00:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:01:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:01:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:01:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:01:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:01:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:01:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:01:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:01:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:01:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:01:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:01:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:01:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:01:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:01:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:01:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:01:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:01:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:01:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:01:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:01:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:01:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:01:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:01:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:01:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:01:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:01:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:01:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:02:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:02:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:02:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:02:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:02:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:02:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:02:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:02:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:02:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:02:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:02:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:02:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:02:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:02:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:02:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:02:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:02:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:02:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:02:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:02:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:02:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:02:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:02:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:02:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:02:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:02:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:02:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:02:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:03:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:03:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:03:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:03:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:03:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:03:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:03:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:03:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:03:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:03:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:03:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:03:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:03:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:03:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:03:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:03:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:03:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:03:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:03:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:03:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:03:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:03:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:03:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:03:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:03:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:03:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:03:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:04:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:04:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:04:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:04:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:04:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:04:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:04:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:04:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:04:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:04:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:04:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:04:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:04:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:04:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:04:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:04:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:04:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:04:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:04:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:04:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:04:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:04:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:04:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:04:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:04:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:04:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:04:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:05:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:05:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:05:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:05:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:05:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:05:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:05:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:05:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:05:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:05:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:05:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:05:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:05:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:05:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:05:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:05:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:05:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:05:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:05:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:05:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:05:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:05:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:05:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:05:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:05:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:05:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:05:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:05:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:06:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:06:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:06:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:06:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:06:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:06:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:06:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:06:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:06:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:06:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:06:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:06:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:06:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:06:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:06:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:06:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:06:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:06:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:06:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:06:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:06:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:06:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:06:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:06:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:06:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:06:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:06:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:07:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:07:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:07:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:07:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:07:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:07:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:07:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:07:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:07:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:07:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:07:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:07:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:07:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:07:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:07:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:07:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:07:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:07:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:07:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:07:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:07:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:07:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:07:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:07:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:07:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:07:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:07:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:08:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:08:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:08:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:08:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:08:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:08:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:08:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:08:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:08:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:08:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:08:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:08:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:08:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:08:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:08:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:08:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:08:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:08:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:08:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:08:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:08:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:08:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:08:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:08:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:08:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:08:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:08:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:08:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:09:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:09:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:09:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:09:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:09:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:09:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:09:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:09:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:09:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:09:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:09:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:09:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:09:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:09:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:09:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:09:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:09:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:09:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:09:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:09:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:09:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:09:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:09:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:09:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:09:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:09:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:09:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:10:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:10:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:10:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:10:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:10:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:10:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:10:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:10:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:10:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:10:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:10:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:10:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:10:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:10:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:10:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:10:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:10:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:10:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:10:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:10:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:10:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:10:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:10:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:10:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:10:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:10:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:10:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:11:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:11:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:11:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:11:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:11:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:11:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:11:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:11:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:11:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:11:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:11:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:11:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:11:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:11:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:11:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:11:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:11:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:11:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:11:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:11:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:11:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:11:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:11:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:11:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:11:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:11:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:11:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:12:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:12:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:12:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:12:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:12:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:12:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:12:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:12:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:12:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:12:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:12:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:12:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:12:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:12:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:12:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:12:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:12:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:12:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:12:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:12:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:12:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:12:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:12:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:12:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:12:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:12:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:12:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:12:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:13:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:13:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:13:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:13:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:13:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:13:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:13:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:13:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:13:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:13:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:13:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:13:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:13:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:13:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:13:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:13:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:13:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:13:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:13:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:13:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:13:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:13:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:13:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:13:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:13:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:13:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:13:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:14:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:14:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:14:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:14:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:14:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:14:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:14:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:14:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:14:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:14:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:14:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:14:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:14:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:14:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:14:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:14:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:14:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:14:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:14:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:14:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:14:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:14:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:14:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:14:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:14:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:14:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:14:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:15:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:15:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:15:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:15:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:15:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:15:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:15:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:15:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:15:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:15:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:15:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:15:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:15:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:15:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:15:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:15:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:15:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:15:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:15:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:15:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:15:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:15:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:15:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:15:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:15:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:15:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:15:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:15:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:16:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:16:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:16:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:16:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:16:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:16:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:16:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:16:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:16:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:16:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:16:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:16:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:16:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:16:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:16:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:16:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:16:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:16:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:16:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:16:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:16:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:16:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:16:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:16:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:16:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:16:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:16:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:17:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:17:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:17:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:17:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:17:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:17:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:17:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:17:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:17:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:17:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:17:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:17:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:17:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:17:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:17:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:17:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:17:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:17:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:17:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:17:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:17:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:17:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:17:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:17:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:17:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:17:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:17:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:18:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:18:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:18:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:18:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:18:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:18:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:18:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:18:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:18:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:18:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:18:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:18:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:18:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:18:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:18:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:18:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:18:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:18:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:18:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:18:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:18:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:18:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:18:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:18:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:18:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:18:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:18:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:18:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:19:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:19:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:19:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:19:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:19:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:19:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:19:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:19:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:19:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:19:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:19:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:19:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:19:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:19:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:19:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:19:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:19:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:19:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:19:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:19:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:19:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:19:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:19:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:19:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:19:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:19:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:19:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:20:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:20:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:20:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:20:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:20:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:20:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:20:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:20:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:20:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:20:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:20:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:20:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:20:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:20:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:20:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:20:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:20:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:20:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:20:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:20:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:20:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:20:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:20:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:20:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:20:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:20:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:20:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:21:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:21:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:21:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:21:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:21:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:21:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:21:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:21:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:21:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:21:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:21:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:21:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:21:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:21:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:21:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:21:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:21:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:21:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:21:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:21:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:21:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:21:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:21:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:21:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:21:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:21:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:21:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:21:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:22:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:22:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:22:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:22:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:22:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:22:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:22:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:22:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:22:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:22:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:22:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:22:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:22:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:22:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:22:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:22:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:22:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:22:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:22:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:22:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:22:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:22:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:22:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:22:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:22:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:22:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:22:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:23:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:23:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:23:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:23:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:23:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:23:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:23:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:23:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:23:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:23:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:23:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:23:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:23:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:23:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:23:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:23:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:23:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:23:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:23:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:23:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:23:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:23:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:23:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:23:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:23:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:23:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:23:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:24:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:24:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:24:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:24:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:24:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:24:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:24:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:24:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:24:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:24:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:24:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:24:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:24:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:24:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:24:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:24:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:24:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:24:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:24:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:24:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:24:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:24:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:24:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:24:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:24:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:24:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:24:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:24:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:25:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:25:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:25:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:25:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:25:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:25:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:25:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:25:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:25:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:25:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:25:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:25:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:25:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:25:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:25:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:25:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:25:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:25:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:25:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:25:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:25:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:25:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:25:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:25:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:25:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:25:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:25:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:26:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:26:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:26:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:26:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:26:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:26:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:26:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:26:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:26:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:26:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:26:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:26:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:26:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:26:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:26:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:26:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:26:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:26:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:26:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:26:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:26:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:26:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:26:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:26:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:26:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:26:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:26:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:27:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:27:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:27:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:27:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:27:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:27:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:27:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:27:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:27:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:27:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:27:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:27:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:27:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:27:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:27:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:27:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:27:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:27:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:27:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:27:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:27:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:27:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:27:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:27:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:27:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:27:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:27:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:27:53 rdma06 systemd[1]: Starting dnf makecache...
Apr 16 02:27:53 rdma06 dnf[1081128]: Updating Subscription Management repositories.
Apr 16 02:27:53 rdma06 dnf[1081128]: Unable to read consumer identity
Apr 16 02:27:53 rdma06 dnf[1081128]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 16 02:27:53 rdma06 dnf[1081128]: Metadata cache refreshed recently.
Apr 16 02:27:53 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 16 02:27:53 rdma06 systemd[1]: Finished dnf makecache.
Apr 16 02:28:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:28:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:28:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:28:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:28:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:28:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:28:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:28:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:28:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:28:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:28:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:28:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:28:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:28:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:28:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:28:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:28:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:28:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:28:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:28:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:28:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:28:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:28:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:28:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:28:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:28:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:28:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:28:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:29:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:29:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:29:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:29:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:29:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:29:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:29:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:29:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:29:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:29:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:29:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:29:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:29:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:29:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:29:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:29:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:29:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:29:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:29:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:29:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:29:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:29:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:29:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:29:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:29:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:29:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:29:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:30:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:30:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:30:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:30:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:30:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:30:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:30:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:30:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:30:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:30:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:30:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:30:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:30:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:30:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:30:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:30:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:30:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:30:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:30:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:30:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:30:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:30:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:30:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:30:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:30:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:30:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:30:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:31:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:31:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:31:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:31:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:31:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:31:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:31:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:31:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:31:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:31:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:31:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:31:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:31:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:31:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:31:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:31:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:31:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:31:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:31:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:31:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:31:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:31:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:31:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:31:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:31:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:31:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:31:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:31:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:32:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:32:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:32:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:32:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:32:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:32:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:32:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:32:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:32:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:32:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:32:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:32:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:32:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:32:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:32:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:32:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:32:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:32:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:32:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:32:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:32:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:32:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:32:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:32:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:32:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:32:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:32:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:33:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:33:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:33:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:33:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:33:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:33:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:33:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:33:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:33:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:33:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:33:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:33:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:33:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:33:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:33:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:33:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:33:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:33:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:33:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:33:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:33:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:33:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:33:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:33:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:33:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:33:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:33:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:34:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:34:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:34:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:34:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:34:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:34:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:34:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:34:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:34:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:34:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:34:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:34:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:34:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:34:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:34:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:34:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:34:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:34:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:34:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:34:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:34:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:34:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:34:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:34:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:34:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:34:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:34:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:34:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:35:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:35:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:35:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:35:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:35:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:35:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:35:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:35:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:35:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:35:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:35:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:35:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:35:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:35:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:35:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:35:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:35:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:35:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:35:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:35:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:35:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:35:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:35:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:35:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:35:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:35:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:35:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:36:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:36:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:36:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:36:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:36:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:36:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:36:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:36:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:36:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:36:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:36:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:36:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:36:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:36:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:36:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:36:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:36:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:36:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:36:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:36:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:36:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:36:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:36:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:36:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:36:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:36:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:36:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:37:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:37:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:37:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:37:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:37:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:37:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:37:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:37:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:37:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:37:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:37:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:37:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:37:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:37:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:37:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:37:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:37:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:37:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:37:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:37:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:37:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:37:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:37:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:37:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:37:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:37:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:37:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:37:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:38:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:38:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:38:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:38:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:38:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:38:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:38:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:38:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:38:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:38:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:38:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:38:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:38:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:38:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:38:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:38:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:38:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:38:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:38:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:38:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:38:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:38:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:38:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:38:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:38:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:38:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:38:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:39:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:39:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:39:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:39:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:39:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:39:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:39:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:39:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:39:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:39:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:39:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:39:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:39:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:39:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:39:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:39:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:39:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:39:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:39:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:39:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:39:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:39:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:39:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:39:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:39:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:39:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:39:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:40:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:40:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:40:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:40:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:40:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:40:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:40:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:40:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:40:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:40:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:40:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:40:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:40:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:40:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:40:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:40:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:40:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:40:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:40:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:40:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:40:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:40:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:40:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:40:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:40:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:40:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:40:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:40:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:41:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:41:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:41:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:41:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:41:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:41:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:41:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:41:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:41:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:41:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:41:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:41:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:41:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:41:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:41:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:41:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:41:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:41:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:41:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:41:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:41:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:41:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:41:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:41:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:41:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:41:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:41:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:42:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:42:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:42:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:42:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:42:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:42:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:42:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:42:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:42:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:42:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:42:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:42:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:42:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:42:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:42:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:42:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:42:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:42:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:42:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:42:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:42:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:42:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:42:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:42:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:42:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:42:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:42:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:43:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:43:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:43:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:43:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:43:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:43:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:43:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:43:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:43:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:43:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:43:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:43:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:43:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:43:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:43:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:43:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:43:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:43:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:43:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:43:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:43:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:43:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:43:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:43:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:43:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:43:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:43:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:43:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:44:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:44:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:44:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:44:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:44:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:44:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:44:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:44:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:44:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:44:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:44:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:44:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:44:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:44:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:44:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:44:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:44:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:44:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:44:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:44:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:44:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:44:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:44:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:44:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:44:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:44:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:44:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:45:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:45:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:45:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:45:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:45:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:45:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:45:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:45:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:45:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:45:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:45:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:45:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:45:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:45:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:45:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:45:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:45:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:45:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:45:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:45:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:45:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:45:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:45:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:45:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:45:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:45:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:45:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:46:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:46:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:46:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:46:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:46:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:46:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:46:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:46:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:46:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:46:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:46:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:46:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:46:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:46:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:46:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:46:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:46:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:46:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:46:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:46:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:46:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:46:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:46:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:46:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:46:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:46:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:46:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:47:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:47:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:47:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:47:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:47:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:47:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:47:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:47:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:47:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:47:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:47:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:47:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:47:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:47:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:47:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:47:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:47:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:47:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:47:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:47:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:47:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:47:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:47:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:47:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:47:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:47:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:47:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:47:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:48:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:48:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:48:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:48:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:48:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:48:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:48:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:48:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:48:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:48:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:48:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:48:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:48:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:48:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:48:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:48:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:48:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:48:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:48:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:48:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:48:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:48:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:48:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:48:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:48:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:48:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:48:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:49:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:49:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:49:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:49:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:49:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:49:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:49:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:49:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:49:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:49:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:49:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:49:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:49:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:49:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:49:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:49:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:49:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:49:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:49:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:49:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:49:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:49:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:49:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:49:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:49:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:49:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:49:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:50:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:50:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:50:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:50:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:50:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:50:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:50:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:50:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:50:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:50:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:50:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:50:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:50:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:50:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:50:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:50:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:50:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:50:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:50:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:50:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:50:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:50:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:50:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:50:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:50:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:50:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:50:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:50:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:51:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:51:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:51:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:51:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:51:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:51:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:51:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:51:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:51:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:51:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:51:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:51:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:51:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:51:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:51:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:51:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:51:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:51:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:51:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:51:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:51:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:51:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:51:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:51:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:51:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:51:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:51:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:52:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:52:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:52:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:52:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:52:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:52:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:52:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:52:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:52:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:52:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:52:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:52:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:52:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:52:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:52:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:52:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:52:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:52:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:52:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:52:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:52:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:52:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:52:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:52:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:52:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:52:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:52:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:53:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:53:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:53:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:53:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:53:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:53:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:53:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:53:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:53:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:53:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:53:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:53:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:53:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:53:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:53:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:53:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:53:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:53:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:53:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:53:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:53:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:53:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:53:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:53:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:53:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:53:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:53:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:53:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:54:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:54:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:54:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:54:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:54:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:54:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:54:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:54:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:54:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:54:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:54:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:54:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:54:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:54:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:54:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:54:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:54:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:54:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:54:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:54:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:54:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:54:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:54:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:54:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:54:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:54:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:54:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:55:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:55:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:55:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:55:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:55:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:55:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:55:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:55:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:55:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:55:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:55:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:55:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:55:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:55:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:55:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:55:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:55:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:55:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:55:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:55:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:55:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:55:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:55:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:55:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:55:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:55:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:55:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:56:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:56:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:56:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:56:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:56:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:56:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:56:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:56:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:56:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:56:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:56:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:56:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:56:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:56:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:56:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:56:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:56:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:56:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:56:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:56:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:56:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:56:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:56:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:56:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:56:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:56:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:56:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:56:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:57:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:57:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:57:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:57:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:57:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:57:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:57:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:57:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:57:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:57:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:57:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:57:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:57:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:57:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:57:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:57:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:57:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:57:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:57:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:57:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:57:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:57:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:57:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:57:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:57:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:57:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:57:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:58:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:58:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:58:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:58:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:58:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:58:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:58:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:58:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:58:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:58:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:58:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:58:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:58:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:58:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:58:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:58:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:58:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:58:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:58:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:58:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:58:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:58:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:58:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:58:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:58:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:58:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:58:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:59:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:59:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:59:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:59:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:59:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:59:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:59:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:59:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:59:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:59:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:59:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:59:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:59:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:59:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:59:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:59:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:59:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:59:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:59:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 02:59:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:59:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:59:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:59:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:59:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 02:59:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 02:59:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 02:59:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 02:59:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:00:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:00:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:00:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:00:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:00:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:00:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:00:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:00:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:00:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:00:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:00:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:00:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:00:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:00:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:00:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:00:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:00:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:00:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:00:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:00:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:00:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:00:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:00:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:00:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:00:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:00:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:00:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:01:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:01:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:01:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:01:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:01:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:01:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:01:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:01:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:01:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:01:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:01:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:01:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:01:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:01:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:01:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:01:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:01:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:01:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:01:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:01:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:01:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:01:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:01:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:01:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:01:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:01:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:01:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:02:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:02:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:02:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:02:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:02:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:02:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:02:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:02:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:02:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:02:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:02:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:02:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:02:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:02:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:02:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:02:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:02:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:02:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:02:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:02:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:02:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:02:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:02:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:02:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:02:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:02:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:02:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:03:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:03:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:03:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:03:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:03:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:03:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:03:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:03:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:03:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:03:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:03:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:03:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:03:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:03:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:03:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:03:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:03:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:03:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:03:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:03:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:03:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:03:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:03:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:03:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:03:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:03:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:03:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:03:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:04:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:04:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:04:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:04:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:04:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:04:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:04:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:04:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:04:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:04:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:04:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:04:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:04:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:04:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:04:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:04:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:04:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:04:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:04:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:04:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:04:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:04:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:04:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:04:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:04:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:04:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:04:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:05:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:05:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:05:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:05:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:05:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:05:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:05:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:05:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:05:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:05:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:05:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:05:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:05:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:05:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:05:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:05:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:05:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:05:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:05:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:05:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:05:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:05:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:05:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:05:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:05:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:05:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:05:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:06:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:06:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:06:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:06:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:06:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:06:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:06:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:06:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:06:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:06:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:06:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:06:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:06:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:06:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:06:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:06:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:06:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:06:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:06:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:06:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:06:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:06:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:06:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:06:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:06:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:06:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:06:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:06:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:07:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:07:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:07:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:07:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:07:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:07:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:07:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:07:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:07:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:07:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:07:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:07:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:07:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:07:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:07:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:07:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:07:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:07:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:07:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:07:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:07:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:07:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:07:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:07:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:07:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:07:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:07:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:08:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:08:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:08:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:08:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:08:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:08:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:08:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:08:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:08:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:08:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:08:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:08:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:08:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:08:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:08:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:08:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:08:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:08:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:08:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:08:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:08:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:08:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:08:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:08:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:08:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:08:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:08:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:09:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:09:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:09:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:09:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:09:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:09:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:09:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:09:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:09:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:09:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:09:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:09:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:09:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:09:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:09:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:09:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:09:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:09:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:09:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:09:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:09:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:09:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:09:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:09:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:09:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:09:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:09:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:09:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:10:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:10:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:10:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:10:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:10:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:10:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:10:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:10:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:10:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:10:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:10:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:10:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:10:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:10:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:10:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:10:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:10:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:10:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:10:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:10:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:10:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:10:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:10:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:10:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:10:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:10:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:10:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:11:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:11:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:11:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:11:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:11:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:11:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:11:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:11:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:11:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:11:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:11:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:11:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:11:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:11:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:11:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:11:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:11:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:11:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:11:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:11:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:11:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:11:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:11:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:11:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:11:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:11:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:11:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:12:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:12:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:12:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:12:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:12:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:12:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:12:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:12:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:12:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:12:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:12:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:12:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:12:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:12:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:12:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:12:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:12:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:12:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:12:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:12:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:12:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:12:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:12:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:12:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:12:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:12:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:12:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:12:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:13:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:13:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:13:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:13:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:13:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:13:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:13:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:13:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:13:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:13:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:13:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:13:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:13:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:13:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:13:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:13:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:13:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:13:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:13:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:13:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:13:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:13:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:13:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:13:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:13:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:13:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:13:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:14:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:14:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:14:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:14:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:14:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:14:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:14:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:14:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:14:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:14:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:14:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:14:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:14:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:14:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:14:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:14:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:14:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:14:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:14:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:14:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:14:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:14:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:14:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:14:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:14:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:14:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:14:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:15:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:15:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:15:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:15:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:15:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:15:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:15:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:15:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:15:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:15:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:15:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:15:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:15:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:15:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:15:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:15:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:15:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:15:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:15:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:15:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:15:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:15:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:15:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:15:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:15:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:15:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:15:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:15:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:16:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:16:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:16:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:16:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:16:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:16:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:16:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:16:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:16:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:16:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:16:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:16:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:16:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:16:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:16:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:16:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:16:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:16:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:16:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:16:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:16:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:16:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:16:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:16:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:16:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:16:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:16:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:17:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:17:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:17:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:17:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:17:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:17:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:17:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:17:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:17:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:17:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:17:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:17:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:17:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:17:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:17:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:17:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:17:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:17:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:17:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:17:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:17:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:17:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:17:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:17:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:17:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:17:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:17:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:18:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:18:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:18:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:18:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:18:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:18:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:18:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:18:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:18:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:18:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:18:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:18:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:18:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:18:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:18:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:18:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:18:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:18:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:18:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:18:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:18:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:18:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:18:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:18:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:18:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:18:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:18:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:19:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:19:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:19:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:19:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:19:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:19:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:19:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:19:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:19:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:19:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:19:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:19:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:19:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:19:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:19:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:19:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:19:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:19:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:19:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:19:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:19:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:19:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:19:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:19:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:19:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:19:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:19:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:19:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:20:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:20:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:20:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:20:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:20:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:20:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:20:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:20:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:20:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:20:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:20:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:20:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:20:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:20:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:20:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:20:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:20:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:20:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:20:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:20:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:20:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:20:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:20:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:20:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:20:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:20:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:20:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:21:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:21:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:21:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:21:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:21:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:21:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:21:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:21:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:21:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:21:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:21:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:21:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:21:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:21:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:21:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:21:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:21:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:21:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:21:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:21:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:21:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:21:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:21:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:21:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:21:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:21:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:21:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:22:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:22:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:22:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:22:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:22:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:22:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:22:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:22:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:22:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:22:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:22:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:22:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:22:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:22:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:22:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:22:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:22:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:22:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:22:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:22:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:22:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:22:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:22:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:22:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:22:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:22:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:22:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:22:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:23:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:23:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:23:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:23:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:23:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:23:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:23:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:23:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:23:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:23:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:23:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:23:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:23:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:23:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:23:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:23:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:23:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:23:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:23:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:23:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:23:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:23:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:23:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:23:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:23:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:23:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:23:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:24:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:24:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:24:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:24:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:24:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:24:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:24:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:24:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:24:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:24:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:24:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:24:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:24:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:24:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:24:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:24:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:24:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:24:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:24:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:24:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:24:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:24:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:24:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:24:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:24:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:24:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:24:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:25:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:25:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:25:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:25:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:25:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:25:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:25:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:25:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:25:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:25:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:25:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:25:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:25:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:25:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:25:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:25:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:25:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:25:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:25:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:25:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:25:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:25:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:25:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:25:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:25:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:25:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:25:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:25:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:26:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:26:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:26:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:26:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:26:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:26:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:26:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:26:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:26:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:26:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:26:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:26:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:26:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:26:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:26:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:26:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:26:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:26:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:26:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:26:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:26:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:26:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:26:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:26:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:26:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:26:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:26:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:27:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:27:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:27:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:27:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:27:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:27:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:27:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:27:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:27:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:27:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:27:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:27:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:27:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:27:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:27:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:27:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:27:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:27:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:27:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:27:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:27:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:27:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:27:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:27:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:27:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:27:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:27:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:28:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:28:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:28:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:28:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:28:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:28:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:28:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:28:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:28:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:28:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:28:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:28:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:28:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:28:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:28:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:28:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:28:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:28:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:28:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:28:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:28:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:28:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:28:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:28:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:28:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:28:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:28:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:28:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:29:00 rdma06 systemd[1]: Starting dnf makecache...
Apr 16 03:29:01 rdma06 dnf[1083609]: Updating Subscription Management repositories.
Apr 16 03:29:01 rdma06 dnf[1083609]: Unable to read consumer identity
Apr 16 03:29:01 rdma06 dnf[1083609]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 16 03:29:02 rdma06 dnf[1083609]: Copr repo for fmf owned by psss                 2.3 kB/s | 3.3 kB     00:01
Apr 16 03:29:02 rdma06 dnf[1083609]: beaker-AppStream                                 47 kB/s | 168  B     00:00
Apr 16 03:29:02 rdma06 dnf[1083609]: Errors during downloading metadata for repository 'beaker-AppStream':
Apr 16 03:29:02 rdma06 dnf[1083609]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 03:29:02 rdma06 dnf[1083609]: Error: Failed to download metadata for repo 'beaker-AppStream': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 03:29:02 rdma06 dnf[1083609]: beaker-AppStream-debuginfo                       56 kB/s | 168  B     00:00
Apr 16 03:29:02 rdma06 dnf[1083609]: Errors during downloading metadata for repository 'beaker-AppStream-debuginfo':
Apr 16 03:29:02 rdma06 dnf[1083609]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 03:29:02 rdma06 dnf[1083609]: Error: Failed to download metadata for repo 'beaker-AppStream-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 03:29:02 rdma06 dnf[1083609]: beaker-BaseOS                                    57 kB/s | 168  B     00:00
Apr 16 03:29:02 rdma06 dnf[1083609]: Errors during downloading metadata for repository 'beaker-BaseOS':
Apr 16 03:29:02 rdma06 dnf[1083609]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 03:29:02 rdma06 dnf[1083609]: Error: Failed to download metadata for repo 'beaker-BaseOS': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 03:29:02 rdma06 dnf[1083609]: beaker-BaseOS-debuginfo                          55 kB/s | 168  B     00:00
Apr 16 03:29:02 rdma06 dnf[1083609]: Errors during downloading metadata for repository 'beaker-BaseOS-debuginfo':
Apr 16 03:29:02 rdma06 dnf[1083609]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 03:29:02 rdma06 dnf[1083609]: Error: Failed to download metadata for repo 'beaker-BaseOS-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 03:29:02 rdma06 dnf[1083609]: beaker-CRB                                       57 kB/s | 168  B     00:00
Apr 16 03:29:02 rdma06 dnf[1083609]: Errors during downloading metadata for repository 'beaker-CRB':
Apr 16 03:29:02 rdma06 dnf[1083609]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 03:29:02 rdma06 dnf[1083609]: Error: Failed to download metadata for repo 'beaker-CRB': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 03:29:02 rdma06 dnf[1083609]: beaker-CRB-debuginfo                             56 kB/s | 168  B     00:00
Apr 16 03:29:02 rdma06 dnf[1083609]: Errors during downloading metadata for repository 'beaker-CRB-debuginfo':
Apr 16 03:29:02 rdma06 dnf[1083609]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 03:29:02 rdma06 dnf[1083609]: Error: Failed to download metadata for repo 'beaker-CRB-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 03:29:02 rdma06 dnf[1083609]: beaker-HighAvailability                          62 kB/s | 168  B     00:00
Apr 16 03:29:02 rdma06 dnf[1083609]: Errors during downloading metadata for repository 'beaker-HighAvailability':
Apr 16 03:29:02 rdma06 dnf[1083609]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 03:29:02 rdma06 dnf[1083609]: Error: Failed to download metadata for repo 'beaker-HighAvailability': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 03:29:02 rdma06 dnf[1083609]: beaker-HighAvailability-debuginfo                59 kB/s | 168  B     00:00
Apr 16 03:29:02 rdma06 dnf[1083609]: Errors during downloading metadata for repository 'beaker-HighAvailability-debuginfo':
Apr 16 03:29:02 rdma06 dnf[1083609]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 03:29:02 rdma06 dnf[1083609]: Error: Failed to download metadata for repo 'beaker-HighAvailability-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 03:29:02 rdma06 dnf[1083609]: beaker-NFV                                       65 kB/s | 168  B     00:00
Apr 16 03:29:02 rdma06 dnf[1083609]: Errors during downloading metadata for repository 'beaker-NFV':
Apr 16 03:29:02 rdma06 dnf[1083609]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 03:29:02 rdma06 dnf[1083609]: Error: Failed to download metadata for repo 'beaker-NFV': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 03:29:02 rdma06 dnf[1083609]: beaker-NFV-debuginfo                             60 kB/s | 168  B     00:00
Apr 16 03:29:02 rdma06 dnf[1083609]: Errors during downloading metadata for repository 'beaker-NFV-debuginfo':
Apr 16 03:29:02 rdma06 dnf[1083609]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 03:29:02 rdma06 dnf[1083609]: Error: Failed to download metadata for repo 'beaker-NFV-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 03:29:02 rdma06 dnf[1083609]: beaker-RT                                        67 kB/s | 168  B     00:00
Apr 16 03:29:02 rdma06 dnf[1083609]: Errors during downloading metadata for repository 'beaker-RT':
Apr 16 03:29:02 rdma06 dnf[1083609]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 03:29:02 rdma06 dnf[1083609]: Error: Failed to download metadata for repo 'beaker-RT': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 03:29:02 rdma06 dnf[1083609]: beaker-RT-debuginfo                              58 kB/s | 168  B     00:00
Apr 16 03:29:02 rdma06 dnf[1083609]: Errors during downloading metadata for repository 'beaker-RT-debuginfo':
Apr 16 03:29:02 rdma06 dnf[1083609]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 03:29:02 rdma06 dnf[1083609]: Error: Failed to download metadata for repo 'beaker-RT-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 03:29:02 rdma06 dnf[1083609]: beaker-ResilientStorage                          56 kB/s | 168  B     00:00
Apr 16 03:29:02 rdma06 dnf[1083609]: Errors during downloading metadata for repository 'beaker-ResilientStorage':
Apr 16 03:29:02 rdma06 dnf[1083609]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 03:29:02 rdma06 dnf[1083609]: Error: Failed to download metadata for repo 'beaker-ResilientStorage': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 03:29:02 rdma06 dnf[1083609]: beaker-ResilientStorage-debuginfo                66 kB/s | 168  B     00:00
Apr 16 03:29:02 rdma06 dnf[1083609]: Errors during downloading metadata for repository 'beaker-ResilientStorage-debuginfo':
Apr 16 03:29:02 rdma06 dnf[1083609]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 03:29:02 rdma06 dnf[1083609]: Error: Failed to download metadata for repo 'beaker-ResilientStorage-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 03:29:02 rdma06 dnf[1083609]: beaker-SAP                                       68 kB/s | 168  B     00:00
Apr 16 03:29:02 rdma06 dnf[1083609]: Errors during downloading metadata for repository 'beaker-SAP':
Apr 16 03:29:02 rdma06 dnf[1083609]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 03:29:02 rdma06 dnf[1083609]: Error: Failed to download metadata for repo 'beaker-SAP': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 03:29:02 rdma06 dnf[1083609]: beaker-SAP-debuginfo                             56 kB/s | 168  B     00:00
Apr 16 03:29:02 rdma06 dnf[1083609]: Errors during downloading metadata for repository 'beaker-SAP-debuginfo':
Apr 16 03:29:02 rdma06 dnf[1083609]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 03:29:02 rdma06 dnf[1083609]: Error: Failed to download metadata for repo 'beaker-SAP-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 03:29:02 rdma06 dnf[1083609]: beaker-SAPHANA                                   63 kB/s | 168  B     00:00
Apr 16 03:29:02 rdma06 dnf[1083609]: Errors during downloading metadata for repository 'beaker-SAPHANA':
Apr 16 03:29:02 rdma06 dnf[1083609]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 03:29:02 rdma06 dnf[1083609]: Error: Failed to download metadata for repo 'beaker-SAPHANA': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 03:29:02 rdma06 dnf[1083609]: beaker-SAPHANA-debuginfo                         71 kB/s | 168  B     00:00
Apr 16 03:29:02 rdma06 dnf[1083609]: Errors during downloading metadata for repository 'beaker-SAPHANA-debuginfo':
Apr 16 03:29:02 rdma06 dnf[1083609]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 03:29:02 rdma06 dnf[1083609]: Error: Failed to download metadata for repo 'beaker-SAPHANA-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 03:29:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:29:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:29:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:29:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:29:03 rdma06 dnf[1083609]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr 16 03:29:04 rdma06 dnf[1083609]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr 16 03:29:04 rdma06 dnf[1083609]: Ignoring repositories: beaker-AppStream, beaker-AppStream-debuginfo, beaker-BaseOS, beaker-BaseOS-debuginfo, beaker-CRB, beaker-CRB-debuginfo, beaker-HighAvailability, beaker-HighAvailability-debuginfo, beaker-NFV, beaker-NFV-debuginfo, beaker-RT, beaker-RT-debuginfo, beaker-ResilientStorage, beaker-ResilientStorage-debuginfo, beaker-SAP, beaker-SAP-debuginfo, beaker-SAPHANA, beaker-SAPHANA-debuginfo
Apr 16 03:29:04 rdma06 dnf[1083609]: Metadata cache created.
Apr 16 03:29:04 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 16 03:29:04 rdma06 systemd[1]: Finished dnf makecache.
Apr 16 03:29:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:29:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:29:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:29:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:29:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:29:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:29:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:29:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:29:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:29:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:29:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:29:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:29:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:29:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:29:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:29:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:29:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:29:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:29:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:29:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:29:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:29:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:29:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:30:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:30:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:30:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:30:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:30:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:30:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:30:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:30:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:30:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:30:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:30:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:30:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:30:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:30:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:30:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:30:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:30:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:30:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:30:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:30:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:30:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:30:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:30:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:30:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:30:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:30:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:30:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:31:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:31:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:31:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:31:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:31:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:31:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:31:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:31:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:31:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:31:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:31:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:31:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:31:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:31:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:31:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:31:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:31:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:31:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:31:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:31:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:31:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:31:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:31:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:31:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:31:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:31:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:31:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:31:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:32:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:32:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:32:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:32:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:32:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:32:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:32:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:32:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:32:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:32:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:32:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:32:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:32:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:32:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:32:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:32:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:32:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:32:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:32:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:32:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:32:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:32:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:32:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:32:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:32:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:32:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:32:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:33:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:33:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:33:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:33:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:33:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:33:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:33:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:33:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:33:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:33:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:33:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:33:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:33:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:33:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:33:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:33:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:33:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:33:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:33:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:33:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:33:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:33:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:33:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:33:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:33:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:33:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:33:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:34:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:34:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:34:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:34:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:34:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:34:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:34:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:34:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:34:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:34:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:34:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:34:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:34:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:34:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:34:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:34:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:34:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:34:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:34:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:34:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:34:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:34:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:34:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:34:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:34:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:34:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:34:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:35:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:35:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:35:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:35:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:35:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:35:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:35:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:35:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:35:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:35:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:35:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:35:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:35:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:35:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:35:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:35:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:35:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:35:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:35:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:35:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:35:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:35:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:35:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:35:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:35:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:35:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:35:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:35:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:36:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:36:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:36:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:36:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:36:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:36:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:36:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:36:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:36:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:36:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:36:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:36:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:36:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:36:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:36:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:36:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:36:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:36:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:36:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:36:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:36:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:36:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:36:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:36:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:36:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:36:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:36:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:37:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:37:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:37:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:37:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:37:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:37:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:37:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:37:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:37:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:37:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:37:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:37:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:37:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:37:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:37:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:37:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:37:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:37:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:37:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:37:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:37:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:37:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:37:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:37:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:37:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:37:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:37:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:38:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:38:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:38:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:38:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:38:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:38:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:38:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:38:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:38:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:38:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:38:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:38:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:38:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:38:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:38:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:38:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:38:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:38:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:38:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:38:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:38:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:38:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:38:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:38:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:38:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:38:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:38:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:38:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:39:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:39:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:39:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:39:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:39:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:39:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:39:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:39:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:39:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:39:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:39:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:39:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:39:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:39:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:39:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:39:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:39:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:39:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:39:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:39:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:39:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:39:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:39:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:39:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:39:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:39:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:39:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:40:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:40:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:40:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:40:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:40:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:40:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:40:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:40:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:40:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:40:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:40:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:40:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:40:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:40:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:40:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:40:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:40:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:40:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:40:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:40:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:40:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:40:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:40:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:40:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:40:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:40:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:40:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:41:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:41:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:41:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:41:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:41:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:41:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:41:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:41:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:41:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:41:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:41:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:41:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:41:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:41:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:41:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:41:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:41:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:41:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:41:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:41:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:41:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:41:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:41:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:41:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:41:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:41:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:41:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:41:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:42:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:42:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:42:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:42:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:42:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:42:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:42:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:42:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:42:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:42:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:42:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:42:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:42:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:42:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:42:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:42:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:42:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:42:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:42:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:42:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:42:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:42:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:42:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:42:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:42:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:42:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:42:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:43:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:43:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:43:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:43:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:43:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:43:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:43:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:43:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:43:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:43:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:43:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:43:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:43:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:43:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:43:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:43:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:43:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:43:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:43:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:43:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:43:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:43:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:43:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:43:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:43:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:43:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:43:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:44:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:44:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:44:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:44:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:44:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:44:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:44:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:44:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:44:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:44:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:44:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:44:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:44:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:44:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:44:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:44:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:44:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:44:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:44:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:44:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:44:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:44:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:44:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:44:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:44:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:44:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:44:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:44:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:45:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:45:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:45:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:45:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:45:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:45:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:45:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:45:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:45:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:45:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:45:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:45:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:45:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:45:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:45:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:45:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:45:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:45:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:45:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:45:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:45:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:45:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:45:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:45:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:45:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:45:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:45:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:46:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:46:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:46:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:46:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:46:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:46:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:46:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:46:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:46:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:46:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:46:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:46:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:46:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:46:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:46:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:46:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:46:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:46:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:46:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:46:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:46:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:46:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:46:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:46:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:46:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:46:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:46:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:47:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:47:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:47:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:47:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:47:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:47:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:47:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:47:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:47:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:47:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:47:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:47:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:47:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:47:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:47:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:47:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:47:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:47:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:47:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:47:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:47:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:47:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:47:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:47:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:47:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:47:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:47:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:47:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:48:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:48:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:48:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:48:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:48:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:48:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:48:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:48:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:48:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:48:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:48:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:48:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:48:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:48:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:48:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:48:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:48:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:48:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:48:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:48:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:48:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:48:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:48:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:48:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:48:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:48:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:48:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:49:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:49:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:49:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:49:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:49:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:49:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:49:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:49:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:49:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:49:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:49:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:49:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:49:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:49:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:49:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:49:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:49:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:49:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:49:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:49:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:49:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:49:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:49:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:49:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:49:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:49:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:49:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:50:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:50:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:50:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:50:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:50:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:50:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:50:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:50:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:50:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:50:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:50:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:50:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:50:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:50:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:50:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:50:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:50:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:50:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:50:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:50:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:50:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:50:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:50:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:50:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:50:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:50:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:50:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:50:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:51:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:51:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:51:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:51:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:51:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:51:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:51:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:51:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:51:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:51:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:51:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:51:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:51:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:51:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:51:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:51:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:51:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:51:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:51:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:51:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:51:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:51:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:51:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:51:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:51:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:51:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:51:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:52:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:52:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:52:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:52:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:52:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:52:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:52:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:52:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:52:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:52:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:52:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:52:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:52:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:52:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:52:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:52:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:52:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:52:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:52:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:52:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:52:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:52:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:52:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:52:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:52:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:52:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:52:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:53:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:53:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:53:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:53:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:53:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:53:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:53:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:53:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:53:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:53:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:53:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:53:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:53:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:53:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:53:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:53:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:53:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:53:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:53:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:53:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:53:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:53:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:53:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:53:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:53:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:53:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:53:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:54:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:54:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:54:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:54:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:54:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:54:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:54:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:54:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:54:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:54:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:54:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:54:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:54:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:54:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:54:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:54:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:54:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:54:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:54:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:54:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:54:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:54:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:54:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:54:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:54:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:54:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:54:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:54:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:55:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:55:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:55:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:55:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:55:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:55:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:55:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:55:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:55:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:55:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:55:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:55:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:55:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:55:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:55:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:55:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:55:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:55:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:55:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:55:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:55:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:55:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:55:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:55:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:55:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:55:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:55:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:56:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:56:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:56:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:56:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:56:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:56:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:56:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:56:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:56:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:56:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:56:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:56:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:56:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:56:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:56:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:56:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:56:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:56:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:56:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:56:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:56:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:56:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:56:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:56:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:56:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:56:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:56:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:57:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:57:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:57:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:57:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:57:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:57:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:57:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:57:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:57:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:57:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:57:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:57:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:57:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:57:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:57:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:57:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:57:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:57:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:57:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:57:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:57:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:57:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:57:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:57:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:57:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:57:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:57:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:57:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:58:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:58:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:58:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:58:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:58:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:58:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:58:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:58:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:58:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:58:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:58:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:58:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:58:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:58:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:58:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:58:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:58:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:58:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:58:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:58:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:58:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:58:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:58:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:58:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:58:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:58:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:58:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:59:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:59:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:59:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:59:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:59:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:59:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:59:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:59:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:59:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:59:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:59:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:59:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:59:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:59:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:59:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:59:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:59:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:59:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:59:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:59:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:59:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:59:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 03:59:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 03:59:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 03:59:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 03:59:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 03:59:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:00:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:00:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:00:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:00:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:00:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:00:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:00:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:00:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:00:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:00:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:00:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:00:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:00:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:00:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:00:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:00:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:00:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:00:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:00:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:00:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:00:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:00:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:00:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:00:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:00:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:00:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:00:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:00:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:01:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:01:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:01:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:01:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:01:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:01:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:01:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:01:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:01:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:01:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:01:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:01:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:01:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:01:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:01:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:01:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:01:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:01:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:01:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:01:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:01:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:01:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:01:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:01:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:01:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:01:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:01:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:02:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:02:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:02:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:02:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:02:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:02:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:02:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:02:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:02:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:02:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:02:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:02:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:02:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:02:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:02:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:02:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:02:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:02:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:02:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:02:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:02:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:02:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:02:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:02:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:02:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:02:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:02:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:03:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:03:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:03:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:03:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:03:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:03:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:03:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:03:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:03:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:03:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:03:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:03:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:03:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:03:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:03:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:03:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:03:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:03:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:03:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:03:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:03:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:03:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:03:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:03:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:03:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:03:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:03:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:03:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:04:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:04:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:04:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:04:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:04:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:04:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:04:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:04:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:04:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:04:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:04:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:04:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:04:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:04:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:04:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:04:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:04:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:04:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:04:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:04:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:04:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:04:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:04:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:04:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:04:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:04:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:04:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:05:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:05:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:05:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:05:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:05:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:05:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:05:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:05:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:05:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:05:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:05:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:05:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:05:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:05:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:05:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:05:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:05:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:05:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:05:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:05:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:05:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:05:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:05:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:05:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:05:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:05:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:05:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:06:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:06:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:06:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:06:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:06:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:06:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:06:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:06:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:06:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:06:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:06:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:06:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:06:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:06:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:06:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:06:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:06:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:06:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:06:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:06:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:06:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:06:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:06:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:06:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:06:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:06:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:06:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:06:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:07:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:07:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:07:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:07:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:07:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:07:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:07:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:07:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:07:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:07:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:07:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:07:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:07:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:07:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:07:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:07:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:07:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:07:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:07:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:07:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:07:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:07:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:07:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:07:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:07:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:07:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:07:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:08:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:08:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:08:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:08:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:08:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:08:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:08:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:08:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:08:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:08:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:08:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:08:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:08:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:08:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:08:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:08:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:08:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:08:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:08:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:08:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:08:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:08:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:08:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:08:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:08:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:08:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:08:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:09:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:09:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:09:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:09:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:09:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:09:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:09:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:09:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:09:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:09:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:09:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:09:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:09:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:09:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:09:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:09:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:09:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:09:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:09:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:09:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:09:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:09:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:09:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:09:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:09:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:09:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:09:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:10:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:10:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:10:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:10:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:10:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:10:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:10:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:10:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:10:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:10:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:10:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:10:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:10:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:10:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:10:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:10:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:10:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:10:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:10:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:10:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:10:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:10:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:10:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:10:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:10:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:10:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:10:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:10:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:11:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:11:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:11:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:11:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:11:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:11:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:11:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:11:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:11:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:11:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:11:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:11:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:11:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:11:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:11:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:11:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:11:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:11:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:11:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:11:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:11:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:11:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:11:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:11:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:11:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:11:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:11:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:12:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:12:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:12:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:12:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:12:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:12:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:12:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:12:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:12:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:12:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:12:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:12:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:12:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:12:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:12:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:12:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:12:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:12:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:12:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:12:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:12:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:12:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:12:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:12:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:12:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:12:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:12:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:13:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:13:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:13:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:13:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:13:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:13:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:13:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:13:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:13:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:13:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:13:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:13:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:13:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:13:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:13:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:13:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:13:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:13:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:13:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:13:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:13:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:13:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:13:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:13:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:13:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:13:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:13:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:13:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:14:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:14:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:14:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:14:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:14:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:14:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:14:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:14:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:14:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:14:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:14:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:14:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:14:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:14:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:14:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:14:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:14:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:14:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:14:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:14:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:14:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:14:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:14:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:14:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:14:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:14:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:14:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:15:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:15:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:15:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:15:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:15:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:15:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:15:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:15:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:15:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:15:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:15:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:15:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:15:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:15:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:15:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:15:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:15:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:15:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:15:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:15:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:15:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:15:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:15:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:15:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:15:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:15:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:15:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:16:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:16:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:16:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:16:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:16:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:16:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:16:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:16:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:16:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:16:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:16:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:16:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:16:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:16:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:16:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:16:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:16:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:16:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:16:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:16:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:16:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:16:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:16:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:16:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:16:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:16:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:16:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:16:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:17:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:17:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:17:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:17:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:17:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:17:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:17:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:17:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:17:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:17:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:17:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:17:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:17:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:17:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:17:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:17:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:17:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:17:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:17:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:17:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:17:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:17:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:17:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:17:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:17:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:17:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:17:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:18:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:18:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:18:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:18:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:18:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:18:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:18:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:18:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:18:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:18:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:18:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:18:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:18:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:18:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:18:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:18:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:18:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:18:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:18:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:18:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:18:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:18:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:18:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:18:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:18:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:18:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:18:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:19:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:19:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:19:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:19:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:19:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:19:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:19:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:19:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:19:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:19:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:19:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:19:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:19:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:19:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:19:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:19:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:19:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:19:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:19:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:19:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:19:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:19:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:19:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:19:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:19:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:19:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:19:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:19:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:20:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:20:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:20:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:20:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:20:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:20:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:20:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:20:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:20:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:20:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:20:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:20:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:20:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:20:32 rdma06 NetworkManager[1025]: <info>  [1681633232.9067] dhcp4 (eno1): state changed new lease, address=10.73.131.45
Apr 16 04:20:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:20:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:20:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:20:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:20:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:20:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:20:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:20:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:20:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:20:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:20:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:20:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:20:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:20:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:21:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:21:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:21:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:21:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:21:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:21:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:21:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:21:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:21:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:21:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:21:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:21:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:21:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:21:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:21:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:21:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:21:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:21:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:21:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:21:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:21:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:21:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:21:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:21:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:21:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:21:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:21:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:22:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:22:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:22:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:22:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:22:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:22:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:22:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:22:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:22:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:22:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:22:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:22:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:22:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:22:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:22:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:22:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:22:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:22:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:22:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:22:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:22:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:22:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:22:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:22:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:22:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:22:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:22:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:22:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:23:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:23:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:23:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:23:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:23:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:23:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:23:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:23:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:23:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:23:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:23:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:23:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:23:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:23:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:23:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:23:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:23:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:23:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:23:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:23:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:23:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:23:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:23:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:23:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:23:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:23:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:23:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:24:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:24:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:24:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:24:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:24:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:24:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:24:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:24:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:24:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:24:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:24:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:24:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:24:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:24:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:24:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:24:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:24:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:24:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:24:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:24:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:24:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:24:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:24:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:24:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:24:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:24:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:24:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:25:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:25:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:25:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:25:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:25:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:25:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:25:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:25:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:25:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:25:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:25:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:25:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:25:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:25:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:25:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:25:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:25:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:25:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:25:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:25:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:25:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:25:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:25:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:25:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:25:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:25:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:25:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:26:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:26:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:26:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:26:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:26:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:26:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:26:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:26:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:26:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:26:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:26:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:26:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:26:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:26:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:26:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:26:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:26:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:26:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:26:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:26:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:26:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:26:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:26:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:26:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:26:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:26:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:26:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:26:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:27:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:27:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:27:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:27:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:27:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:27:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:27:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:27:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:27:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:27:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:27:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:27:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:27:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:27:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:27:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:27:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:27:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:27:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:27:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:27:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:27:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:27:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:27:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:27:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:27:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:27:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:27:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:28:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:28:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:28:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:28:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:28:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:28:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:28:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:28:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:28:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:28:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:28:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:28:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:28:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:28:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:28:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:28:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:28:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:28:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:28:33 rdma06 systemd-journald[790]: Data hash table of /run/log/journal/95379d2f723847b68d9e976dee8967a9/system.journal has a fill level at 75.0 (163750 of 218332 items, 75497472 file size, 461 bytes per hash table item), suggesting rotation.
Apr 16 04:28:33 rdma06 systemd-journald[790]: /run/log/journal/95379d2f723847b68d9e976dee8967a9/system.journal: Journal header limits reached or header out-of-date, rotating.
Apr 16 04:28:33 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 16 04:28:33 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 16 04:28:33 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 16 04:28:33 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 16 04:28:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:28:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:28:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:28:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:28:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:28:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:28:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:28:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:28:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:29:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:29:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:29:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:29:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:29:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:29:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:29:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:29:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:29:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:29:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:29:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:29:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:29:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:29:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:29:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:29:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:29:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:29:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:29:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:29:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:29:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:29:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:29:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:29:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:29:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:29:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:29:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:29:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:30:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:30:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:30:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:30:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:30:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:30:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:30:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:30:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:30:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:30:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:30:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:30:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:30:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:30:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:30:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:30:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:30:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:30:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:30:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:30:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:30:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:30:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:30:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:30:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:30:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:30:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:30:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:31:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:31:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:31:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:31:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:31:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:31:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:31:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:31:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:31:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:31:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:31:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:31:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:31:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:31:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:31:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:31:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:31:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:31:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:31:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:31:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:31:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:31:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:31:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:31:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:31:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:31:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:31:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:32:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:32:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:32:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:32:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:32:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:32:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:32:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:32:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:32:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:32:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:32:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:32:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:32:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:32:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:32:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:32:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:32:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:32:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:32:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:32:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:32:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:32:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:32:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:32:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:32:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:32:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:32:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:32:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:33:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:33:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:33:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:33:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:33:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:33:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:33:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:33:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:33:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:33:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:33:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:33:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:33:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:33:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:33:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:33:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:33:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:33:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:33:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:33:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:33:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:33:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:33:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:33:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:33:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:33:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:33:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:34:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:34:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:34:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:34:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:34:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:34:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:34:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:34:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:34:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:34:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:34:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:34:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:34:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:34:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:34:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:34:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:34:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:34:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:34:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:34:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:34:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:34:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:34:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:34:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:34:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:34:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:34:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:35:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:35:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:35:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:35:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:35:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:35:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:35:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:35:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:35:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:35:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:35:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:35:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:35:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:35:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:35:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:35:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:35:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:35:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:35:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:35:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:35:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:35:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:35:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:35:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:35:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:35:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:35:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:35:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:36:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:36:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:36:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:36:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:36:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:36:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:36:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:36:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:36:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:36:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:36:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:36:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:36:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:36:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:36:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:36:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:36:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:36:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:36:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:36:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:36:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:36:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:36:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:36:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:36:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:36:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:36:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:37:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:37:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:37:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:37:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:37:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:37:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:37:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:37:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:37:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:37:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:37:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:37:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:37:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:37:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:37:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:37:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:37:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:37:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:37:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:37:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:37:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:37:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:37:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:37:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:37:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:37:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:37:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:38:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:38:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:38:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:38:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:38:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:38:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:38:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:38:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:38:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:38:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:38:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:38:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:38:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:38:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:38:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:38:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:38:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:38:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:38:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:38:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:38:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:38:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:38:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:38:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:38:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:38:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:38:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:38:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:39:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:39:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:39:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:39:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:39:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:39:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:39:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:39:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:39:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:39:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:39:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:39:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:39:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:39:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:39:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:39:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:39:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:39:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:39:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:39:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:39:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:39:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:39:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:39:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:39:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:39:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:39:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:40:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:40:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:40:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:40:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:40:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:40:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:40:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:40:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:40:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:40:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:40:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:40:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:40:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:40:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:40:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:40:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:40:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:40:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:40:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:40:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:40:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:40:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:40:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:40:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:40:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:40:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:40:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:41:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:41:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:41:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:41:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:41:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:41:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:41:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:41:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:41:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:41:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:41:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:41:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:41:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:41:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:41:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:41:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:41:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:41:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:41:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:41:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:41:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:41:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:41:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:41:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:41:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:41:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:41:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:42:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:42:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:42:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:42:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:42:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:42:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:42:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:42:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:42:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:42:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:42:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:42:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:42:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:42:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:42:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:42:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:42:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:42:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:42:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:42:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:42:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:42:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:42:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:42:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:42:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:42:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:42:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:42:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:43:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:43:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:43:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:43:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:43:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:43:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:43:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:43:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:43:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:43:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:43:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:43:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:43:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:43:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:43:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:43:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:43:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:43:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:43:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:43:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:43:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:43:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:43:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:43:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:43:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:43:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:43:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:44:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:44:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:44:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:44:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:44:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:44:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:44:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:44:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:44:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:44:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:44:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:44:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:44:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:44:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:44:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:44:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:44:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:44:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:44:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:44:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:44:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:44:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:44:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:44:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:44:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:44:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:44:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:45:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:45:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:45:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:45:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:45:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:45:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:45:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:45:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:45:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:45:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:45:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:45:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:45:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:45:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:45:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:45:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:45:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:45:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:45:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:45:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:45:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:45:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:45:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:45:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:45:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:45:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:45:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:45:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:46:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:46:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:46:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:46:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:46:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:46:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:46:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:46:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:46:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:46:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:46:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:46:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:46:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:46:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:46:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:46:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:46:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:46:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:46:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:46:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:46:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:46:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:46:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:46:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:46:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:46:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:46:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:47:00 rdma06 systemd[1]: Starting dnf makecache...
Apr 16 04:47:01 rdma06 dnf[1086791]: Updating Subscription Management repositories.
Apr 16 04:47:01 rdma06 dnf[1086791]: Unable to read consumer identity
Apr 16 04:47:01 rdma06 dnf[1086791]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 16 04:47:01 rdma06 dnf[1086791]: Metadata cache refreshed recently.
Apr 16 04:47:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 16 04:47:01 rdma06 systemd[1]: Finished dnf makecache.
Apr 16 04:47:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:47:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:47:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:47:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:47:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:47:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:47:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:47:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:47:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:47:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:47:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:47:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:47:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:47:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:47:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:47:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:47:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:47:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:47:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:47:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:47:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:47:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:47:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:47:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:47:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:47:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:47:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:48:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:48:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:48:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:48:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:48:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:48:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:48:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:48:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:48:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:48:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:48:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:48:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:48:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:48:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:48:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:48:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:48:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:48:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:48:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:48:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:48:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:48:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:48:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:48:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:48:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:48:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:48:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:48:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:49:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:49:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:49:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:49:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:49:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:49:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:49:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:49:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:49:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:49:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:49:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:49:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:49:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:49:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:49:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:49:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:49:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:49:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:49:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:49:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:49:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:49:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:49:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:49:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:49:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:49:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:49:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:50:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:50:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:50:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:50:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:50:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:50:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:50:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:50:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:50:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:50:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:50:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:50:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:50:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:50:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:50:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:50:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:50:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:50:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:50:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:50:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:50:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:50:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:50:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:50:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:50:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:50:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:50:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:51:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:51:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:51:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:51:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:51:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:51:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:51:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:51:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:51:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:51:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:51:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:51:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:51:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:51:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:51:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:51:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:51:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:51:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:51:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:51:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:51:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:51:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:51:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:51:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:51:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:51:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:51:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:51:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:52:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:52:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:52:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:52:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:52:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:52:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:52:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:52:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:52:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:52:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:52:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:52:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:52:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:52:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:52:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:52:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:52:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:52:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:52:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:52:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:52:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:52:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:52:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:52:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:52:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:52:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:52:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:53:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:53:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:53:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:53:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:53:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:53:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:53:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:53:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:53:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:53:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:53:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:53:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:53:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:53:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:53:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:53:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:53:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:53:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:53:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:53:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:53:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:53:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:53:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:53:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:53:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:53:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:53:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:54:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:54:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:54:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:54:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:54:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:54:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:54:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:54:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:54:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:54:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:54:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:54:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:54:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:54:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:54:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:54:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:54:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:54:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:54:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:54:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:54:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:54:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:54:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:54:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:54:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:54:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:54:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:54:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:55:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:55:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:55:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:55:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:55:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:55:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:55:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:55:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:55:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:55:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:55:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:55:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:55:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:55:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:55:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:55:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:55:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:55:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:55:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:55:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:55:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:55:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:55:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:55:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:55:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:55:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:55:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:56:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:56:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:56:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:56:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:56:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:56:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:56:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:56:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:56:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:56:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:56:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:56:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:56:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:56:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:56:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:56:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:56:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:56:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:56:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:56:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:56:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:56:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:56:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:56:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:56:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:56:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:56:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:57:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:57:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:57:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:57:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:57:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:57:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:57:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:57:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:57:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:57:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:57:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:57:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:57:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:57:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:57:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:57:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:57:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:57:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:57:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:57:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:57:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:57:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:57:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:57:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:57:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:57:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:57:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:57:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:58:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:58:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:58:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:58:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:58:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:58:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:58:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:58:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:58:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:58:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:58:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:58:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:58:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:58:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:58:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:58:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:58:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:58:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:58:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:58:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:58:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:58:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:58:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:58:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:58:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:58:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:58:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:59:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:59:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:59:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:59:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:59:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:59:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:59:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:59:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:59:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:59:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:59:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:59:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:59:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:59:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:59:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:59:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:59:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:59:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:59:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:59:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:59:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:59:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 04:59:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 04:59:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 04:59:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 04:59:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 04:59:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:00:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:00:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:00:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:00:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:00:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:00:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:00:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:00:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:00:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:00:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:00:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:00:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:00:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:00:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:00:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:00:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:00:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:00:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:00:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:00:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:00:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:00:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:00:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:00:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:00:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:00:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:00:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:01:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:01:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:01:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:01:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:01:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:01:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:01:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:01:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:01:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:01:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:01:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:01:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:01:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:01:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:01:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:01:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:01:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:01:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:01:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:01:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:01:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:01:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:01:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:01:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:01:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:01:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:01:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:01:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:02:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:02:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:02:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:02:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:02:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:02:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:02:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:02:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:02:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:02:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:02:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:02:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:02:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:02:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:02:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:02:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:02:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:02:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:02:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:02:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:02:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:02:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:02:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:02:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:02:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:02:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:02:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:03:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:03:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:03:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:03:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:03:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:03:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:03:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:03:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:03:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:03:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:03:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:03:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:03:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:03:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:03:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:03:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:03:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:03:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:03:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:03:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:03:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:03:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:03:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:03:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:03:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:03:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:03:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:04:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:04:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:04:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:04:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:04:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:04:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:04:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:04:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:04:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:04:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:04:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:04:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:04:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:04:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:04:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:04:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:04:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:04:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:04:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:04:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:04:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:04:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:04:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:04:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:04:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:04:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:04:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:04:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:05:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:05:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:05:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:05:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:05:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:05:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:05:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:05:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:05:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:05:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:05:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:05:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:05:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:05:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:05:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:05:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:05:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:05:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:05:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:05:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:05:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:05:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:05:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:05:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:05:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:05:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:05:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:06:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:06:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:06:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:06:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:06:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:06:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:06:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:06:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:06:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:06:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:06:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:06:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:06:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:06:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:06:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:06:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:06:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:06:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:06:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:06:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:06:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:06:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:06:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:06:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:06:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:06:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:06:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:07:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:07:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:07:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:07:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:07:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:07:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:07:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:07:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:07:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:07:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:07:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:07:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:07:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:07:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:07:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:07:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:07:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:07:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:07:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:07:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:07:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:07:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:07:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:07:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:07:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:07:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:07:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:07:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:08:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:08:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:08:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:08:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:08:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:08:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:08:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:08:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:08:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:08:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:08:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:08:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:08:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:08:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:08:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:08:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:08:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:08:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:08:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:08:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:08:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:08:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:08:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:08:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:08:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:08:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:08:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:09:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:09:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:09:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:09:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:09:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:09:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:09:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:09:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:09:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:09:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:09:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:09:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:09:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:09:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:09:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:09:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:09:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:09:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:09:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:09:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:09:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:09:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:09:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:09:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:09:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:09:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:09:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:10:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:10:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:10:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:10:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:10:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:10:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:10:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:10:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:10:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:10:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:10:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:10:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:10:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:10:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:10:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:10:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:10:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:10:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:10:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:10:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:10:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:10:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:10:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:10:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:10:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:10:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:10:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:10:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:11:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:11:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:11:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:11:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:11:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:11:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:11:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:11:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:11:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:11:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:11:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:11:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:11:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:11:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:11:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:11:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:11:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:11:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:11:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:11:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:11:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:11:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:11:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:11:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:11:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:11:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:11:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:12:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:12:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:12:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:12:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:12:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:12:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:12:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:12:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:12:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:12:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:12:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:12:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:12:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:12:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:12:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:12:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:12:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:12:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:12:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:12:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:12:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:12:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:12:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:12:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:12:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:12:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:12:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:13:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:13:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:13:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:13:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:13:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:13:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:13:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:13:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:13:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:13:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:13:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:13:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:13:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:13:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:13:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:13:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:13:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:13:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:13:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:13:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:13:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:13:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:13:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:13:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:13:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:13:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:13:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:13:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:14:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:14:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:14:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:14:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:14:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:14:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:14:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:14:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:14:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:14:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:14:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:14:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:14:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:14:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:14:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:14:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:14:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:14:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:14:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:14:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:14:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:14:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:14:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:14:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:14:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:14:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:14:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:15:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:15:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:15:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:15:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:15:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:15:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:15:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:15:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:15:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:15:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:15:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:15:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:15:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:15:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:15:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:15:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:15:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:15:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:15:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:15:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:15:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:15:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:15:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:15:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:15:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:15:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:15:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:16:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:16:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:16:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:16:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:16:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:16:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:16:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:16:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:16:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:16:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:16:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:16:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:16:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:16:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:16:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:16:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:16:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:16:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:16:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:16:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:16:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:16:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:16:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:16:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:16:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:16:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:16:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:17:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:17:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:17:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:17:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:17:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:17:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:17:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:17:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:17:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:17:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:17:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:17:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:17:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:17:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:17:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:17:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:17:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:17:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:17:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:17:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:17:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:17:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:17:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:17:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:17:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:17:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:17:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:17:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:18:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:18:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:18:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:18:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:18:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:18:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:18:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:18:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:18:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:18:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:18:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:18:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:18:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:18:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:18:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:18:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:18:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:18:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:18:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:18:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:18:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:18:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:18:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:18:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:18:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:18:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:18:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:19:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:19:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:19:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:19:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:19:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:19:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:19:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:19:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:19:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:19:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:19:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:19:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:19:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:19:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:19:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:19:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:19:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:19:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:19:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:19:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:19:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:19:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:19:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:19:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:19:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:19:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:19:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:20:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:20:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:20:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:20:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:20:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:20:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:20:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:20:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:20:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:20:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:20:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:20:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:20:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:20:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:20:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:20:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:20:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:20:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:20:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:20:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:20:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:20:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:20:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:20:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:20:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:20:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:20:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:20:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:21:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:21:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:21:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:21:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:21:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:21:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:21:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:21:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:21:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:21:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:21:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:21:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:21:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:21:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:21:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:21:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:21:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:21:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:21:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:21:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:21:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:21:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:21:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:21:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:21:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:21:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:21:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:22:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:22:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:22:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:22:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:22:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:22:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:22:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:22:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:22:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:22:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:22:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:22:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:22:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:22:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:22:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:22:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:22:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:22:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:22:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:22:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:22:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:22:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:22:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:22:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:22:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:22:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:22:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:23:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:23:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:23:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:23:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:23:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:23:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:23:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:23:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:23:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:23:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:23:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:23:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:23:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:23:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:23:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:23:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:23:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:23:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:23:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:23:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:23:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:23:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:23:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:23:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:23:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:23:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:23:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:23:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:24:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:24:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:24:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:24:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:24:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:24:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:24:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:24:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:24:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:24:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:24:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:24:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:24:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:24:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:24:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:24:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:24:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:24:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:24:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:24:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:24:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:24:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:24:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:24:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:24:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:24:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:24:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:25:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:25:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:25:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:25:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:25:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:25:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:25:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:25:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:25:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:25:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:25:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:25:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:25:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:25:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:25:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:25:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:25:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:25:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:25:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:25:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:25:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:25:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:25:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:25:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:25:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:25:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:25:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:26:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:26:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:26:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:26:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:26:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:26:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:26:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:26:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:26:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:26:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:26:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:26:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:26:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:26:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:26:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:26:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:26:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:26:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:26:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:26:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:26:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:26:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:26:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:26:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:26:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:26:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:26:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:26:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:27:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:27:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:27:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:27:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:27:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:27:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:27:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:27:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:27:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:27:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:27:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:27:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:27:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:27:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:27:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:27:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:27:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:27:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:27:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:27:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:27:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:27:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:27:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:27:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:27:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:27:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:27:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:28:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:28:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:28:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:28:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:28:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:28:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:28:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:28:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:28:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:28:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:28:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:28:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:28:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:28:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:28:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:28:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:28:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:28:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:28:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:28:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:28:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:28:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:28:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:28:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:28:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:28:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:28:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:29:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:29:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:29:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:29:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:29:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:29:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:29:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:29:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:29:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:29:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:29:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:29:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:29:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:29:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:29:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:29:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:29:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:29:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:29:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:29:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:29:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:29:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:29:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:29:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:29:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:29:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:29:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:29:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:30:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:30:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:30:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:30:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:30:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:30:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:30:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:30:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:30:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:30:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:30:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:30:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:30:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:30:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:30:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:30:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:30:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:30:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:30:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:30:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:30:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:30:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:30:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:30:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:30:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:30:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:30:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:31:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:31:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:31:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:31:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:31:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:31:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:31:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:31:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:31:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:31:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:31:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:31:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:31:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:31:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:31:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:31:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:31:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:31:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:31:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:31:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:31:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:31:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:31:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:31:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:31:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:31:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:31:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:32:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:32:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:32:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:32:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:32:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:32:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:32:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:32:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:32:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:32:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:32:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:32:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:32:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:32:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:32:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:32:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:32:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:32:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:32:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:32:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:32:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:32:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:32:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:32:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:32:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:32:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:32:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:33:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:33:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:33:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:33:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:33:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:33:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:33:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:33:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:33:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:33:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:33:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:33:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:33:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:33:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:33:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:33:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:33:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:33:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:33:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:33:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:33:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:33:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:33:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:33:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:33:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:33:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:33:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:33:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:34:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:34:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:34:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:34:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:34:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:34:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:34:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:34:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:34:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:34:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:34:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:34:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:34:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:34:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:34:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:34:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:34:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:34:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:34:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:34:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:34:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:34:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:34:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:34:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:34:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:34:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:34:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:35:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:35:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:35:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:35:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:35:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:35:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:35:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:35:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:35:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:35:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:35:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:35:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:35:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:35:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:35:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:35:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:35:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:35:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:35:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:35:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:35:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:35:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:35:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:35:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:35:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:35:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:35:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:36:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:36:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:36:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:36:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:36:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:36:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:36:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:36:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:36:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:36:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:36:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:36:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:36:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:36:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:36:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:36:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:36:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:36:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:36:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:36:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:36:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:36:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:36:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:36:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:36:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:36:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:36:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:36:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:37:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:37:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:37:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:37:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:37:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:37:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:37:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:37:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:37:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:37:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:37:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:37:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:37:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:37:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:37:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:37:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:37:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:37:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:37:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:37:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:37:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:37:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:37:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:37:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:37:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:37:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:37:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:38:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:38:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:38:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:38:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:38:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:38:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:38:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:38:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:38:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:38:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:38:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:38:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:38:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:38:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:38:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:38:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:38:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:38:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:38:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:38:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:38:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:38:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:38:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:38:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:38:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:38:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:38:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:39:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:39:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:39:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:39:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:39:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:39:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:39:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:39:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:39:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:39:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:39:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:39:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:39:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:39:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:39:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:39:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:39:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:39:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:39:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:39:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:39:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:39:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:39:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:39:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:39:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:39:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:39:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:39:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:40:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:40:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:40:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:40:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:40:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:40:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:40:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:40:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:40:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:40:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:40:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:40:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:40:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:40:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:40:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:40:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:40:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:40:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:40:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:40:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:40:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:40:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:40:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:40:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:40:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:40:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:40:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:41:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:41:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:41:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:41:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:41:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:41:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:41:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:41:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:41:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:41:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:41:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:41:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:41:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:41:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:41:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:41:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:41:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:41:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:41:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:41:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:41:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:41:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:41:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:41:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:41:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:41:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:41:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:42:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:42:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:42:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:42:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:42:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:42:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:42:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:42:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:42:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:42:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:42:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:42:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:42:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:42:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:42:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:42:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:42:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:42:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:42:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:42:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:42:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:42:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:42:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:42:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:42:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:42:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:42:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:42:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:43:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:43:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:43:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:43:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:43:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:43:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:43:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:43:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:43:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:43:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:43:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:43:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:43:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:43:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:43:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:43:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:43:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:43:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:43:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:43:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:43:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:43:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:43:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:43:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:43:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:43:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:43:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:44:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:44:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:44:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:44:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:44:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:44:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:44:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:44:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:44:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:44:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:44:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:44:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:44:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:44:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:44:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:44:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:44:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:44:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:44:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:44:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:44:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:44:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:44:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:44:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:44:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:44:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:44:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:45:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:45:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:45:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:45:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:45:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:45:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:45:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:45:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:45:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:45:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:45:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:45:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:45:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:45:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:45:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:45:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:45:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:45:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:45:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:45:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:45:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:45:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:45:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:45:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:45:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:45:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:45:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:45:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:46:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:46:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:46:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:46:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:46:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:46:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:46:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:46:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:46:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:46:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:46:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:46:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:46:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:46:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:46:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:46:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:46:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:46:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:46:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:46:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:46:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:46:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:46:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:46:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:46:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:46:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:46:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:47:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:47:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:47:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:47:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:47:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:47:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:47:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:47:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:47:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:47:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:47:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:47:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:47:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:47:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:47:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:47:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:47:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:47:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:47:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:47:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:47:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:47:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:47:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:47:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:47:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:47:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:47:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:48:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:48:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:48:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:48:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:48:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:48:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:48:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:48:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:48:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:48:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:48:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:48:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:48:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:48:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:48:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:48:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:48:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:48:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:48:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:48:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:48:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:48:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:48:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:48:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:48:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:48:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:48:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:49:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:49:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:49:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:49:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:49:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:49:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:49:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:49:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:49:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:49:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:49:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:49:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:49:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:49:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:49:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:49:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:49:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:49:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:49:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:49:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:49:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:49:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:49:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:49:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:49:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:49:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:49:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:49:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:50:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:50:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:50:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:50:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:50:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:50:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:50:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:50:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:50:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:50:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:50:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:50:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:50:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:50:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:50:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:50:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:50:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:50:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:50:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:50:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:50:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:50:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:50:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:50:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:50:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:50:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:50:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:51:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:51:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:51:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:51:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:51:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:51:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:51:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:51:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:51:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:51:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:51:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:51:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:51:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:51:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:51:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:51:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:51:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:51:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:51:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:51:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:51:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:51:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:51:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:51:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:51:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:51:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:51:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:52:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:52:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:52:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:52:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:52:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:52:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:52:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:52:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:52:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:52:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:52:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:52:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:52:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:52:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:52:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:52:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:52:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:52:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:52:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:52:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:52:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:52:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:52:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:52:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:52:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:52:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:52:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:52:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:53:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:53:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:53:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:53:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:53:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:53:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:53:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:53:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:53:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:53:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:53:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:53:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:53:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:53:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:53:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:53:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:53:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:53:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:53:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:53:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:53:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:53:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:53:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:53:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:53:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:53:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:53:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:54:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:54:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:54:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:54:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:54:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:54:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:54:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:54:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:54:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:54:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:54:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:54:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:54:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:54:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:54:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:54:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:54:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:54:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:54:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:54:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:54:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:54:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:54:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:54:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:54:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:54:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:54:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:55:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:55:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:55:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:55:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:55:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:55:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:55:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:55:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:55:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:55:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:55:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:55:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:55:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:55:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:55:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:55:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:55:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:55:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:55:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:55:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:55:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:55:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:55:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:55:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:55:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:55:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:55:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:55:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:56:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:56:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:56:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:56:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:56:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:56:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:56:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:56:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:56:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:56:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:56:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:56:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:56:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:56:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:56:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:56:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:56:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:56:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:56:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:56:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:56:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:56:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:56:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:56:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:56:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:56:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:56:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:57:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:57:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:57:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:57:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:57:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:57:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:57:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:57:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:57:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:57:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:57:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:57:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:57:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:57:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:57:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:57:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:57:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:57:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:57:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:57:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:57:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:57:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:57:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:57:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:57:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:57:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:57:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:58:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:58:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:58:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:58:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:58:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:58:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:58:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:58:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:58:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:58:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:58:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:58:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:58:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:58:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:58:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:58:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:58:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:58:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:58:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:58:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:58:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:58:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:58:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:58:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:58:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:58:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:58:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:58:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:59:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:59:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:59:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:59:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:59:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:59:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:59:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:59:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:59:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:59:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:59:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:59:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:59:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:59:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:59:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:59:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:59:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:59:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:59:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:59:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:59:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:59:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 05:59:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 05:59:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 05:59:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 05:59:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 05:59:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:00:00 rdma06 systemd[1]: Starting dnf makecache...
Apr 16 06:00:01 rdma06 dnf[1089750]: Updating Subscription Management repositories.
Apr 16 06:00:01 rdma06 dnf[1089750]: Unable to read consumer identity
Apr 16 06:00:01 rdma06 dnf[1089750]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 16 06:00:01 rdma06 dnf[1089750]: Metadata cache refreshed recently.
Apr 16 06:00:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 16 06:00:01 rdma06 systemd[1]: Finished dnf makecache.
Apr 16 06:00:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:00:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:00:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:00:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:00:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:00:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:00:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:00:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:00:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:00:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:00:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:00:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:00:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:00:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:00:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:00:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:00:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:00:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:00:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:00:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:00:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:00:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:00:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:00:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:00:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:00:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:00:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:01:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:01:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:01:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:01:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:01:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:01:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:01:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:01:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:01:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:01:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:01:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:01:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:01:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:01:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:01:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:01:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:01:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:01:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:01:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:01:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:01:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:01:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:01:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:01:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:01:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:01:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:01:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:01:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:02:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:02:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:02:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:02:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:02:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:02:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:02:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:02:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:02:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:02:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:02:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:02:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:02:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:02:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:02:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:02:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:02:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:02:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:02:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:02:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:02:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:02:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:02:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:02:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:02:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:02:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:02:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:03:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:03:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:03:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:03:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:03:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:03:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:03:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:03:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:03:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:03:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:03:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:03:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:03:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:03:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:03:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:03:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:03:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:03:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:03:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:03:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:03:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:03:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:03:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:03:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:03:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:03:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:03:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:04:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:04:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:04:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:04:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:04:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:04:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:04:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:04:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:04:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:04:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:04:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:04:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:04:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:04:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:04:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:04:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:04:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:04:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:04:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:04:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:04:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:04:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:04:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:04:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:04:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:04:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:04:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:04:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:05:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:05:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:05:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:05:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:05:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:05:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:05:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:05:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:05:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:05:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:05:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:05:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:05:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:05:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:05:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:05:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:05:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:05:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:05:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:05:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:05:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:05:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:05:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:05:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:05:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:05:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:05:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:06:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:06:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:06:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:06:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:06:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:06:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:06:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:06:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:06:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:06:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:06:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:06:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:06:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:06:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:06:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:06:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:06:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:06:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:06:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:06:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:06:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:06:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:06:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:06:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:06:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:06:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:06:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:07:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:07:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:07:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:07:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:07:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:07:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:07:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:07:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:07:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:07:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:07:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:07:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:07:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:07:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:07:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:07:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:07:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:07:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:07:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:07:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:07:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:07:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:07:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:07:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:07:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:07:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:07:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:08:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:08:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:08:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:08:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:08:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:08:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:08:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:08:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:08:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:08:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:08:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:08:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:08:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:08:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:08:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:08:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:08:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:08:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:08:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:08:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:08:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:08:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:08:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:08:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:08:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:08:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:08:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:08:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:09:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:09:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:09:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:09:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:09:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:09:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:09:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:09:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:09:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:09:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:09:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:09:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:09:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:09:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:09:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:09:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:09:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:09:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:09:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:09:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:09:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:09:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:09:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:09:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:09:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:09:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:09:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:10:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:10:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:10:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:10:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:10:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:10:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:10:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:10:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:10:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:10:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:10:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:10:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:10:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:10:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:10:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:10:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:10:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:10:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:10:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:10:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:10:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:10:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:10:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:10:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:10:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:10:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:10:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:11:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:11:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:11:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:11:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:11:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:11:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:11:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:11:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:11:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:11:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:11:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:11:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:11:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:11:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:11:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:11:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:11:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:11:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:11:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:11:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:11:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:11:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:11:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:11:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:11:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:11:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:11:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:11:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:12:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:12:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:12:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:12:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:12:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:12:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:12:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:12:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:12:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:12:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:12:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:12:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:12:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:12:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:12:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:12:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:12:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:12:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:12:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:12:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:12:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:12:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:12:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:12:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:12:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:12:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:12:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:13:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:13:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:13:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:13:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:13:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:13:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:13:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:13:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:13:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:13:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:13:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:13:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:13:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:13:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:13:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:13:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:13:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:13:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:13:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:13:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:13:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:13:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:13:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:13:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:13:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:13:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:13:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:14:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:14:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:14:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:14:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:14:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:14:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:14:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:14:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:14:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:14:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:14:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:14:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:14:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:14:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:14:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:14:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:14:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:14:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:14:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:14:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:14:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:14:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:14:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:14:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:14:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:14:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:14:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:14:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:15:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:15:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:15:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:15:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:15:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:15:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:15:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:15:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:15:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:15:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:15:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:15:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:15:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:15:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:15:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:15:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:15:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:15:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:15:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:15:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:15:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:15:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:15:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:15:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:15:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:15:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:15:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:16:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:16:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:16:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:16:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:16:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:16:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:16:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:16:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:16:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:16:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:16:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:16:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:16:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:16:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:16:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:16:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:16:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:16:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:16:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:16:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:16:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:16:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:16:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:16:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:16:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:16:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:16:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:17:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:17:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:17:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:17:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:17:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:17:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:17:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:17:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:17:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:17:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:17:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:17:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:17:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:17:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:17:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:17:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:17:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:17:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:17:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:17:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:17:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:17:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:17:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:17:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:17:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:17:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:17:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:17:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:18:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:18:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:18:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:18:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:18:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:18:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:18:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:18:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:18:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:18:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:18:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:18:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:18:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:18:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:18:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:18:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:18:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:18:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:18:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:18:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:18:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:18:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:18:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:18:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:18:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:18:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:18:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:19:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:19:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:19:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:19:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:19:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:19:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:19:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:19:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:19:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:19:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:19:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:19:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:19:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:19:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:19:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:19:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:19:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:19:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:19:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:19:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:19:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:19:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:19:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:19:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:19:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:19:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:19:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:20:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:20:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:20:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:20:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:20:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:20:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:20:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:20:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:20:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:20:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:20:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:20:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:20:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:20:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:20:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:20:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:20:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:20:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:20:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:20:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:20:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:20:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:20:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:20:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:20:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:20:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:20:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:20:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:21:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:21:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:21:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:21:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:21:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:21:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:21:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:21:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:21:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:21:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:21:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:21:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:21:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:21:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:21:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:21:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:21:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:21:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:21:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:21:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:21:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:21:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:21:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:21:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:21:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:21:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:21:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:22:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:22:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:22:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:22:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:22:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:22:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:22:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:22:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:22:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:22:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:22:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:22:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:22:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:22:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:22:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:22:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:22:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:22:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:22:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:22:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:22:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:22:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:22:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:22:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:22:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:22:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:22:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:23:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:23:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:23:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:23:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:23:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:23:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:23:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:23:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:23:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:23:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:23:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:23:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:23:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:23:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:23:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:23:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:23:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:23:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:23:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:23:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:23:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:23:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:23:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:23:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:23:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:23:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:23:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:24:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:24:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:24:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:24:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:24:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:24:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:24:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:24:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:24:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:24:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:24:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:24:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:24:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:24:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:24:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:24:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:24:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:24:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:24:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:24:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:24:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:24:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:24:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:24:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:24:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:24:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:24:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:24:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:25:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:25:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:25:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:25:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:25:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:25:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:25:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:25:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:25:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:25:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:25:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:25:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:25:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:25:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:25:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:25:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:25:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:25:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:25:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:25:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:25:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:25:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:25:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:25:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:25:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:25:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:25:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:26:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:26:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:26:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:26:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:26:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:26:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:26:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:26:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:26:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:26:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:26:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:26:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:26:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:26:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:26:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:26:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:26:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:26:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:26:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:26:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:26:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:26:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:26:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:26:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:26:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:26:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:26:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:27:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:27:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:27:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:27:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:27:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:27:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:27:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:27:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:27:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:27:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:27:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:27:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:27:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:27:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:27:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:27:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:27:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:27:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:27:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:27:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:27:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:27:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:27:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:27:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:27:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:27:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:27:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:27:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:28:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:28:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:28:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:28:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:28:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:28:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:28:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:28:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:28:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:28:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:28:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:28:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:28:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:28:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:28:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:28:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:28:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:28:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:28:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:28:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:28:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:28:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:28:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:28:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:28:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:28:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:28:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:29:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:29:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:29:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:29:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:29:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:29:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:29:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:29:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:29:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:29:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:29:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:29:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:29:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:29:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:29:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:29:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:29:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:29:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:29:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:29:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:29:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:29:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:29:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:29:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:29:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:29:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:29:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:30:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:30:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:30:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:30:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:30:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:30:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:30:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:30:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:30:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:30:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:30:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:30:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:30:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:30:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:30:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:30:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:30:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:30:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:30:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:30:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:30:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:30:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:30:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:30:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:30:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:30:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:30:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:30:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:31:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:31:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:31:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:31:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:31:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:31:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:31:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:31:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:31:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:31:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:31:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:31:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:31:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:31:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:31:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:31:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:31:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:31:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:31:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:31:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:31:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:31:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:31:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:31:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:31:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:31:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:31:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:32:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:32:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:32:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:32:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:32:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:32:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:32:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:32:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:32:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:32:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:32:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:32:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:32:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:32:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:32:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:32:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:32:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:32:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:32:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:32:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:32:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:32:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:32:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:32:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:32:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:32:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:32:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:33:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:33:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:33:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:33:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:33:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:33:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:33:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:33:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:33:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:33:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:33:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:33:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:33:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:33:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:33:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:33:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:33:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:33:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:33:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:33:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:33:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:33:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:33:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:33:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:33:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:33:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:33:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:33:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:34:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:34:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:34:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:34:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:34:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:34:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:34:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:34:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:34:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:34:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:34:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:34:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:34:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:34:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:34:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:34:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:34:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:34:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:34:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:34:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:34:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:34:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:34:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:34:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:34:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:34:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:34:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:35:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:35:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:35:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:35:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:35:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:35:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:35:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:35:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:35:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:35:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:35:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:35:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:35:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:35:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:35:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:35:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:35:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:35:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:35:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:35:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:35:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:35:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:35:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:35:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:35:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:35:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:35:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:36:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:36:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:36:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:36:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:36:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:36:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:36:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:36:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:36:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:36:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:36:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:36:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:36:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:36:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:36:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:36:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:36:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:36:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:36:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:36:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:36:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:36:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:36:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:36:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:36:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:36:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:36:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:36:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:37:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:37:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:37:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:37:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:37:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:37:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:37:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:37:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:37:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:37:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:37:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:37:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:37:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:37:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:37:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:37:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:37:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:37:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:37:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:37:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:37:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:37:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:37:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:37:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:37:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:37:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:37:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:38:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:38:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:38:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:38:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:38:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:38:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:38:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:38:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:38:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:38:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:38:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:38:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:38:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:38:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:38:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:38:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:38:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:38:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:38:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:38:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:38:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:38:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:38:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:38:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:38:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:38:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:38:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:39:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:39:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:39:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:39:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:39:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:39:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:39:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:39:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:39:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:39:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:39:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:39:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:39:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:39:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:39:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:39:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:39:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:39:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:39:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:39:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:39:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:39:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:39:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:39:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:39:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:39:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:39:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:40:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:40:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:40:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:40:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:40:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:40:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:40:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:40:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:40:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:40:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:40:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:40:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:40:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:40:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:40:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:40:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:40:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:40:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:40:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:40:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:40:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:40:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:40:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:40:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:40:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:40:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:40:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:40:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:41:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:41:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:41:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:41:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:41:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:41:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:41:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:41:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:41:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:41:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:41:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:41:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:41:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:41:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:41:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:41:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:41:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:41:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:41:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:41:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:41:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:41:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:41:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:41:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:41:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:41:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:41:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:42:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:42:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:42:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:42:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:42:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:42:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:42:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:42:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:42:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:42:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:42:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:42:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:42:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:42:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:42:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:42:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:42:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:42:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:42:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:42:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:42:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:42:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:42:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:42:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:42:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:42:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:42:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:43:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:43:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:43:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:43:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:43:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:43:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:43:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:43:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:43:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:43:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:43:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:43:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:43:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:43:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:43:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:43:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:43:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:43:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:43:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:43:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:43:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:43:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:43:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:43:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:43:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:43:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:43:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:43:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:44:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:44:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:44:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:44:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:44:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:44:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:44:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:44:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:44:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:44:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:44:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:44:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:44:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:44:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:44:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:44:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:44:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:44:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:44:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:44:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:44:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:44:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:44:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:44:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:44:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:44:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:44:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:45:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:45:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:45:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:45:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:45:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:45:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:45:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:45:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:45:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:45:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:45:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:45:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:45:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:45:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:45:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:45:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:45:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:45:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:45:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:45:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:45:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:45:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:45:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:45:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:45:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:45:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:45:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:46:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:46:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:46:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:46:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:46:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:46:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:46:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:46:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:46:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:46:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:46:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:46:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:46:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:46:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:46:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:46:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:46:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:46:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:46:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:46:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:46:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:46:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:46:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:46:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:46:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:46:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:46:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:46:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:47:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:47:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:47:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:47:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:47:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:47:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:47:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:47:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:47:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:47:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:47:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:47:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:47:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:47:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:47:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:47:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:47:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:47:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:47:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:47:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:47:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:47:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:47:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:47:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:47:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:47:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:47:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:48:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:48:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:48:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:48:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:48:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:48:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:48:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:48:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:48:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:48:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:48:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:48:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:48:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:48:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:48:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:48:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:48:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:48:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:48:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:48:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:48:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:48:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:48:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:48:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:48:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:48:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:48:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:49:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:49:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:49:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:49:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:49:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:49:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:49:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:49:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:49:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:49:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:49:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:49:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:49:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:49:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:49:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:49:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:49:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:49:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:49:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:49:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:49:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:49:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:49:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:49:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:49:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:49:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:49:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:49:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:50:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:50:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:50:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:50:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:50:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:50:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:50:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:50:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:50:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:50:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:50:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:50:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:50:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:50:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:50:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:50:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:50:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:50:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:50:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:50:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:50:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:50:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:50:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:50:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:50:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:50:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:50:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:51:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:51:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:51:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:51:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:51:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:51:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:51:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:51:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:51:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:51:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:51:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:51:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:51:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:51:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:51:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:51:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:51:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:51:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:51:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:51:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:51:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:51:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:51:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:51:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:51:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:51:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:51:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:52:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:52:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:52:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:52:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:52:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:52:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:52:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:52:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:52:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:52:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:52:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:52:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:52:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:52:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:52:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:52:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:52:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:52:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:52:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:52:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:52:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:52:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:52:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:52:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:52:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:52:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:52:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:52:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:53:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:53:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:53:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:53:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:53:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:53:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:53:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:53:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:53:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:53:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:53:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:53:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:53:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:53:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:53:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:53:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:53:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:53:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:53:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:53:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:53:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:53:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:53:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:53:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:53:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:53:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:53:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:54:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:54:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:54:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:54:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:54:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:54:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:54:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:54:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:54:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:54:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:54:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:54:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:54:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:54:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:54:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:54:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:54:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:54:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:54:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:54:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:54:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:54:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:54:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:54:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:54:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:54:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:54:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:55:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:55:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:55:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:55:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:55:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:55:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:55:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:55:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:55:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:55:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:55:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:55:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:55:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:55:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:55:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:55:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:55:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:55:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:55:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:55:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:55:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:55:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:55:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:55:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:55:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:55:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:55:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:56:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:56:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:56:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:56:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:56:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:56:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:56:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:56:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:56:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:56:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:56:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:56:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:56:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:56:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:56:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:56:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:56:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:56:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:56:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:56:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:56:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:56:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:56:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:56:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:56:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:56:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:56:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:56:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:57:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:57:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:57:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:57:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:57:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:57:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:57:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:57:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:57:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:57:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:57:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:57:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:57:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:57:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:57:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:57:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:57:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:57:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:57:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:57:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:57:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:57:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:57:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:57:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:57:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:57:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:57:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:58:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:58:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:58:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:58:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:58:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:58:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:58:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:58:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:58:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:58:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:58:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:58:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:58:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:58:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:58:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:58:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:58:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:58:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:58:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:58:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:58:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:58:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:58:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:58:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:58:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:58:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:58:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:59:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:59:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:59:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:59:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:59:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:59:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:59:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:59:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:59:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:59:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:59:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:59:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:59:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:59:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:59:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:59:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:59:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:59:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:59:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 06:59:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:59:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:59:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:59:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:59:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 06:59:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 06:59:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 06:59:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 06:59:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:00:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:00:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:00:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:00:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:00:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:00:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:00:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:00:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:00:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:00:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:00:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:00:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:00:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:00:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:00:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:00:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:00:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:00:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:00:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:00:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:00:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:00:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:00:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:00:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:00:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:00:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:00:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:01:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:01:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:01:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:01:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:01:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:01:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:01:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:01:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:01:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:01:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:01:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:01:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:01:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:01:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:01:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:01:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:01:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:01:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:01:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:01:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:01:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:01:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:01:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:01:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:01:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:01:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:01:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:02:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:02:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:02:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:02:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:02:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:02:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:02:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:02:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:02:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:02:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:02:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:02:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:02:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:02:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:02:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:02:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:02:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:02:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:02:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:02:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:02:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:02:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:02:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:02:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:02:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:02:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:02:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:02:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:03:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:03:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:03:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:03:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:03:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:03:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:03:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:03:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:03:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:03:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:03:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:03:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:03:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:03:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:03:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:03:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:03:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:03:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:03:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:03:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:03:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:03:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:03:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:03:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:03:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:03:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:03:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:04:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:04:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:04:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:04:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:04:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:04:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:04:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:04:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:04:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:04:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:04:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:04:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:04:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:04:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:04:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:04:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:04:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:04:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:04:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:04:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:04:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:04:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:04:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:04:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:04:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:04:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:04:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:05:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:05:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:05:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:05:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:05:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:05:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:05:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:05:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:05:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:05:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:05:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:05:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:05:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:05:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:05:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:05:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:05:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:05:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:05:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:05:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:05:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:05:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:05:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:05:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:05:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:05:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:05:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:05:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:06:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:06:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:06:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:06:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:06:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:06:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:06:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:06:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:06:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:06:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:06:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:06:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:06:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:06:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:06:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:06:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:06:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:06:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:06:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:06:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:06:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:06:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:06:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:06:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:06:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:06:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:06:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:07:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:07:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:07:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:07:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:07:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:07:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:07:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:07:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:07:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:07:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:07:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:07:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:07:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:07:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:07:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:07:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:07:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:07:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:07:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:07:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:07:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:07:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:07:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:07:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:07:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:07:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:07:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:08:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:08:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:08:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:08:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:08:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:08:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:08:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:08:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:08:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:08:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:08:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:08:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:08:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:08:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:08:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:08:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:08:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:08:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:08:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:08:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:08:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:08:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:08:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:08:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:08:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:08:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:08:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:08:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:09:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:09:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:09:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:09:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:09:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:09:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:09:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:09:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:09:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:09:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:09:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:09:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:09:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:09:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:09:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:09:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:09:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:09:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:09:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:09:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:09:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:09:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:09:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:09:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:09:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:09:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:09:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:10:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:10:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:10:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:10:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:10:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:10:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:10:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:10:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:10:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:10:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:10:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:10:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:10:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:10:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:10:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:10:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:10:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:10:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:10:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:10:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:10:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:10:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:10:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:10:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:10:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:10:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:10:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:11:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:11:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:11:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:11:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:11:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:11:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:11:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:11:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:11:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:11:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:11:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:11:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:11:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:11:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:11:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:11:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:11:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:11:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:11:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:11:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:11:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:11:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:11:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:11:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:11:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:11:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:11:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:11:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:12:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:12:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:12:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:12:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:12:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:12:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:12:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:12:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:12:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:12:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:12:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:12:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:12:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:12:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:12:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:12:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:12:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:12:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:12:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:12:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:12:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:12:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:12:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:12:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:12:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:12:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:12:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:13:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:13:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:13:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:13:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:13:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:13:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:13:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:13:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:13:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:13:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:13:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:13:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:13:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:13:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:13:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:13:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:13:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:13:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:13:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:13:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:13:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:13:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:13:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:13:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:13:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:13:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:13:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:14:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:14:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:14:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:14:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:14:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:14:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:14:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:14:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:14:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:14:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:14:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:14:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:14:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:14:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:14:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:14:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:14:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:14:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:14:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:14:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:14:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:14:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:14:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:14:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:14:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:14:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:14:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:15:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:15:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:15:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:15:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:15:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:15:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:15:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:15:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:15:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:15:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:15:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:15:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:15:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:15:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:15:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:15:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:15:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:15:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:15:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:15:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:15:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:15:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:15:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:15:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:15:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:15:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:15:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:15:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:16:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:16:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:16:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:16:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:16:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:16:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:16:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:16:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:16:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:16:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:16:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:16:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:16:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:16:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:16:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:16:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:16:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:16:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:16:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:16:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:16:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:16:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:16:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:16:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:16:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:16:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:16:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:17:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:17:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:17:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:17:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:17:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:17:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:17:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:17:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:17:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:17:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:17:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:17:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:17:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:17:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:17:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:17:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:17:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:17:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:17:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:17:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:17:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:17:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:17:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:17:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:17:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:17:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:17:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:18:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:18:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:18:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:18:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:18:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:18:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:18:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:18:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:18:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:18:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:18:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:18:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:18:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:18:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:18:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:18:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:18:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:18:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:18:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:18:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:18:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:18:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:18:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:18:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:18:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:18:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:18:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:18:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:19:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:19:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:19:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:19:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:19:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:19:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:19:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:19:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:19:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:19:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:19:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:19:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:19:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:19:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:19:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:19:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:19:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:19:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:19:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:19:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:19:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:19:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:19:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:19:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:19:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:19:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:19:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:20:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:20:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:20:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:20:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:20:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:20:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:20:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:20:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:20:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:20:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:20:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:20:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:20:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:20:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:20:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:20:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:20:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:20:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:20:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:20:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:20:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:20:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:20:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:20:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:20:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:20:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:20:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:21:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:21:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:21:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:21:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:21:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:21:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:21:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:21:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:21:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:21:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:21:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:21:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:21:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:21:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:21:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:21:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:21:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:21:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:21:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:21:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:21:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:21:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:21:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:21:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:21:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:21:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:21:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:21:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:22:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:22:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:22:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:22:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:22:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:22:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:22:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:22:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:22:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:22:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:22:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:22:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:22:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:22:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:22:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:22:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:22:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:22:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:22:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:22:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:22:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:22:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:22:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:22:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:22:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:22:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:22:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:23:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:23:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:23:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:23:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:23:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:23:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:23:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:23:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:23:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:23:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:23:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:23:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:23:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:23:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:23:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:23:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:23:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:23:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:23:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:23:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:23:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:23:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:23:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:23:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:23:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:23:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:23:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:24:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:24:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:24:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:24:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:24:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:24:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:24:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:24:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:24:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:24:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:24:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:24:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:24:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:24:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:24:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:24:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:24:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:24:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:24:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:24:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:24:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:24:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:24:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:24:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:24:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:24:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:24:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:24:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:25:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:25:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:25:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:25:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:25:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:25:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:25:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:25:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:25:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:25:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:25:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:25:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:25:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:25:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:25:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:25:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:25:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:25:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:25:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:25:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:25:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:25:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:25:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:25:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:25:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:25:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:25:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:26:00 rdma06 systemd[1]: Starting dnf makecache...
Apr 16 07:26:01 rdma06 dnf[1093238]: Updating Subscription Management repositories.
Apr 16 07:26:01 rdma06 dnf[1093238]: Unable to read consumer identity
Apr 16 07:26:01 rdma06 dnf[1093238]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 16 07:26:02 rdma06 dnf[1093238]: Copr repo for fmf owned by psss                 2.2 kB/s | 3.3 kB     00:01
Apr 16 07:26:02 rdma06 dnf[1093238]: beaker-AppStream                                 46 kB/s | 168  B     00:00
Apr 16 07:26:02 rdma06 dnf[1093238]: Errors during downloading metadata for repository 'beaker-AppStream':
Apr 16 07:26:02 rdma06 dnf[1093238]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 07:26:02 rdma06 dnf[1093238]: Error: Failed to download metadata for repo 'beaker-AppStream': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 07:26:02 rdma06 dnf[1093238]: beaker-AppStream-debuginfo                       59 kB/s | 168  B     00:00
Apr 16 07:26:02 rdma06 dnf[1093238]: Errors during downloading metadata for repository 'beaker-AppStream-debuginfo':
Apr 16 07:26:02 rdma06 dnf[1093238]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 07:26:02 rdma06 dnf[1093238]: Error: Failed to download metadata for repo 'beaker-AppStream-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 07:26:02 rdma06 dnf[1093238]: beaker-BaseOS                                    60 kB/s | 168  B     00:00
Apr 16 07:26:02 rdma06 dnf[1093238]: Errors during downloading metadata for repository 'beaker-BaseOS':
Apr 16 07:26:02 rdma06 dnf[1093238]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 07:26:02 rdma06 dnf[1093238]: Error: Failed to download metadata for repo 'beaker-BaseOS': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 07:26:02 rdma06 dnf[1093238]: beaker-BaseOS-debuginfo                          57 kB/s | 168  B     00:00
Apr 16 07:26:02 rdma06 dnf[1093238]: Errors during downloading metadata for repository 'beaker-BaseOS-debuginfo':
Apr 16 07:26:02 rdma06 dnf[1093238]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 07:26:02 rdma06 dnf[1093238]: Error: Failed to download metadata for repo 'beaker-BaseOS-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 07:26:02 rdma06 dnf[1093238]: beaker-CRB                                       64 kB/s | 168  B     00:00
Apr 16 07:26:02 rdma06 dnf[1093238]: Errors during downloading metadata for repository 'beaker-CRB':
Apr 16 07:26:02 rdma06 dnf[1093238]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 07:26:02 rdma06 dnf[1093238]: Error: Failed to download metadata for repo 'beaker-CRB': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 07:26:02 rdma06 dnf[1093238]: beaker-CRB-debuginfo                             59 kB/s | 168  B     00:00
Apr 16 07:26:02 rdma06 dnf[1093238]: Errors during downloading metadata for repository 'beaker-CRB-debuginfo':
Apr 16 07:26:02 rdma06 dnf[1093238]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 07:26:02 rdma06 dnf[1093238]: Error: Failed to download metadata for repo 'beaker-CRB-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 07:26:02 rdma06 dnf[1093238]: beaker-HighAvailability                          61 kB/s | 168  B     00:00
Apr 16 07:26:02 rdma06 dnf[1093238]: Errors during downloading metadata for repository 'beaker-HighAvailability':
Apr 16 07:26:02 rdma06 dnf[1093238]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 07:26:02 rdma06 dnf[1093238]: Error: Failed to download metadata for repo 'beaker-HighAvailability': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 07:26:02 rdma06 dnf[1093238]: beaker-HighAvailability-debuginfo                59 kB/s | 168  B     00:00
Apr 16 07:26:02 rdma06 dnf[1093238]: Errors during downloading metadata for repository 'beaker-HighAvailability-debuginfo':
Apr 16 07:26:02 rdma06 dnf[1093238]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 07:26:02 rdma06 dnf[1093238]: Error: Failed to download metadata for repo 'beaker-HighAvailability-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 07:26:02 rdma06 dnf[1093238]: beaker-NFV                                       71 kB/s | 168  B     00:00
Apr 16 07:26:02 rdma06 dnf[1093238]: Errors during downloading metadata for repository 'beaker-NFV':
Apr 16 07:26:02 rdma06 dnf[1093238]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 07:26:02 rdma06 dnf[1093238]: Error: Failed to download metadata for repo 'beaker-NFV': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 07:26:02 rdma06 dnf[1093238]: beaker-NFV-debuginfo                             60 kB/s | 168  B     00:00
Apr 16 07:26:02 rdma06 dnf[1093238]: Errors during downloading metadata for repository 'beaker-NFV-debuginfo':
Apr 16 07:26:02 rdma06 dnf[1093238]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 07:26:02 rdma06 dnf[1093238]: Error: Failed to download metadata for repo 'beaker-NFV-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 07:26:02 rdma06 dnf[1093238]: beaker-RT                                        63 kB/s | 168  B     00:00
Apr 16 07:26:03 rdma06 dnf[1093238]: Errors during downloading metadata for repository 'beaker-RT':
Apr 16 07:26:03 rdma06 dnf[1093238]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 07:26:03 rdma06 dnf[1093238]: Error: Failed to download metadata for repo 'beaker-RT': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 07:26:03 rdma06 dnf[1093238]: beaker-RT-debuginfo                              56 kB/s | 168  B     00:00
Apr 16 07:26:03 rdma06 dnf[1093238]: Errors during downloading metadata for repository 'beaker-RT-debuginfo':
Apr 16 07:26:03 rdma06 dnf[1093238]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 07:26:03 rdma06 dnf[1093238]: Error: Failed to download metadata for repo 'beaker-RT-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 07:26:03 rdma06 dnf[1093238]: beaker-ResilientStorage                          62 kB/s | 168  B     00:00
Apr 16 07:26:03 rdma06 dnf[1093238]: Errors during downloading metadata for repository 'beaker-ResilientStorage':
Apr 16 07:26:03 rdma06 dnf[1093238]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 07:26:03 rdma06 dnf[1093238]: Error: Failed to download metadata for repo 'beaker-ResilientStorage': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 07:26:03 rdma06 dnf[1093238]: beaker-ResilientStorage-debuginfo                53 kB/s | 168  B     00:00
Apr 16 07:26:03 rdma06 dnf[1093238]: Errors during downloading metadata for repository 'beaker-ResilientStorage-debuginfo':
Apr 16 07:26:03 rdma06 dnf[1093238]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 07:26:03 rdma06 dnf[1093238]: Error: Failed to download metadata for repo 'beaker-ResilientStorage-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 07:26:03 rdma06 dnf[1093238]: beaker-SAP                                       60 kB/s | 168  B     00:00
Apr 16 07:26:03 rdma06 dnf[1093238]: Errors during downloading metadata for repository 'beaker-SAP':
Apr 16 07:26:03 rdma06 dnf[1093238]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 07:26:03 rdma06 dnf[1093238]: Error: Failed to download metadata for repo 'beaker-SAP': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 07:26:03 rdma06 dnf[1093238]: beaker-SAP-debuginfo                             57 kB/s | 168  B     00:00
Apr 16 07:26:03 rdma06 dnf[1093238]: Errors during downloading metadata for repository 'beaker-SAP-debuginfo':
Apr 16 07:26:03 rdma06 dnf[1093238]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 07:26:03 rdma06 dnf[1093238]: Error: Failed to download metadata for repo 'beaker-SAP-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 07:26:03 rdma06 dnf[1093238]: beaker-SAPHANA                                   66 kB/s | 168  B     00:00
Apr 16 07:26:03 rdma06 dnf[1093238]: Errors during downloading metadata for repository 'beaker-SAPHANA':
Apr 16 07:26:03 rdma06 dnf[1093238]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 07:26:03 rdma06 dnf[1093238]: Error: Failed to download metadata for repo 'beaker-SAPHANA': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 07:26:03 rdma06 dnf[1093238]: beaker-SAPHANA-debuginfo                         58 kB/s | 168  B     00:00
Apr 16 07:26:03 rdma06 dnf[1093238]: Errors during downloading metadata for repository 'beaker-SAPHANA-debuginfo':
Apr 16 07:26:03 rdma06 dnf[1093238]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 07:26:03 rdma06 dnf[1093238]: Error: Failed to download metadata for repo 'beaker-SAPHANA-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 07:26:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:26:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:26:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:26:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:26:04 rdma06 dnf[1093238]: beaker-harness                                  1.2 kB/s | 1.3 kB     00:01
Apr 16 07:26:04 rdma06 dnf[1093238]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr 16 07:26:04 rdma06 dnf[1093238]: Ignoring repositories: beaker-AppStream, beaker-AppStream-debuginfo, beaker-BaseOS, beaker-BaseOS-debuginfo, beaker-CRB, beaker-CRB-debuginfo, beaker-HighAvailability, beaker-HighAvailability-debuginfo, beaker-NFV, beaker-NFV-debuginfo, beaker-RT, beaker-RT-debuginfo, beaker-ResilientStorage, beaker-ResilientStorage-debuginfo, beaker-SAP, beaker-SAP-debuginfo, beaker-SAPHANA, beaker-SAPHANA-debuginfo
Apr 16 07:26:04 rdma06 dnf[1093238]: Metadata cache created.
Apr 16 07:26:04 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 16 07:26:04 rdma06 systemd[1]: Finished dnf makecache.
Apr 16 07:26:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:26:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:26:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:26:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:26:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:26:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:26:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:26:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:26:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:26:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:26:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:26:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:26:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:26:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:26:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:26:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:26:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:26:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:26:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:26:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:26:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:26:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:26:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:27:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:27:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:27:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:27:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:27:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:27:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:27:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:27:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:27:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:27:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:27:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:27:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:27:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:27:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:27:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:27:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:27:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:27:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:27:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:27:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:27:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:27:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:27:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:27:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:27:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:27:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:27:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:27:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:28:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:28:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:28:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:28:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:28:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:28:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:28:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:28:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:28:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:28:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:28:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:28:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:28:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:28:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:28:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:28:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:28:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:28:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:28:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:28:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:28:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:28:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:28:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:28:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:28:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:28:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:28:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:29:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:29:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:29:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:29:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:29:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:29:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:29:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:29:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:29:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:29:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:29:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:29:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:29:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:29:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:29:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:29:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:29:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:29:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:29:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:29:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:29:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:29:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:29:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:29:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:29:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:29:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:29:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:30:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:30:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:30:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:30:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:30:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:30:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:30:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:30:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:30:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:30:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:30:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:30:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:30:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:30:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:30:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:30:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:30:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:30:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:30:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:30:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:30:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:30:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:30:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:30:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:30:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:30:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:30:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:31:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:31:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:31:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:31:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:31:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:31:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:31:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:31:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:31:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:31:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:31:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:31:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:31:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:31:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:31:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:31:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:31:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:31:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:31:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:31:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:31:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:31:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:31:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:31:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:31:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:31:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:31:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:31:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:32:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:32:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:32:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:32:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:32:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:32:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:32:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:32:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:32:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:32:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:32:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:32:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:32:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:32:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:32:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:32:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:32:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:32:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:32:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:32:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:32:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:32:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:32:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:32:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:32:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:32:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:32:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:33:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:33:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:33:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:33:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:33:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:33:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:33:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:33:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:33:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:33:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:33:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:33:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:33:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:33:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:33:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:33:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:33:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:33:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:33:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:33:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:33:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:33:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:33:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:33:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:33:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:33:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:33:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:34:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:34:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:34:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:34:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:34:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:34:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:34:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:34:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:34:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:34:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:34:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:34:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:34:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:34:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:34:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:34:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:34:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:34:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:34:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:34:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:34:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:34:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:34:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:34:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:34:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:34:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:34:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:34:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:35:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:35:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:35:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:35:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:35:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:35:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:35:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:35:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:35:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:35:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:35:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:35:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:35:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:35:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:35:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:35:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:35:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:35:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:35:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:35:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:35:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:35:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:35:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:35:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:35:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:35:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:35:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:36:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:36:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:36:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:36:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:36:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:36:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:36:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:36:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:36:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:36:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:36:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:36:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:36:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:36:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:36:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:36:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:36:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:36:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:36:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:36:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:36:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:36:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:36:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:36:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:36:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:36:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:36:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:37:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:37:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:37:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:37:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:37:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:37:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:37:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:37:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:37:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:37:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:37:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:37:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:37:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:37:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:37:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:37:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:37:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:37:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:37:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:37:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:37:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:37:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:37:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:37:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:37:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:37:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:37:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:37:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:38:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:38:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:38:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:38:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:38:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:38:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:38:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:38:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:38:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:38:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:38:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:38:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:38:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:38:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:38:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:38:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:38:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:38:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:38:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:38:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:38:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:38:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:38:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:38:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:38:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:38:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:38:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:39:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:39:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:39:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:39:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:39:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:39:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:39:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:39:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:39:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:39:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:39:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:39:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:39:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:39:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:39:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:39:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:39:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:39:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:39:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:39:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:39:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:39:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:39:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:39:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:39:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:39:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:39:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:40:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:40:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:40:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:40:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:40:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:40:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:40:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:40:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:40:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:40:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:40:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:40:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:40:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:40:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:40:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:40:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:40:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:40:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:40:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:40:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:40:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:40:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:40:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:40:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:40:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:40:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:40:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:40:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:41:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:41:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:41:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:41:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:41:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:41:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:41:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:41:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:41:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:41:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:41:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:41:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:41:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:41:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:41:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:41:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:41:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:41:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:41:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:41:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:41:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:41:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:41:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:41:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:41:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:41:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:41:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:42:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:42:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:42:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:42:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:42:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:42:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:42:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:42:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:42:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:42:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:42:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:42:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:42:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:42:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:42:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:42:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:42:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:42:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:42:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:42:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:42:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:42:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:42:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:42:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:42:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:42:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:42:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:43:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:43:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:43:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:43:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:43:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:43:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:43:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:43:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:43:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:43:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:43:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:43:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:43:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:43:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:43:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:43:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:43:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:43:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:43:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:43:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:43:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:43:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:43:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:43:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:43:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:43:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:43:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:43:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:44:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:44:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:44:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:44:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:44:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:44:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:44:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:44:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:44:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:44:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:44:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:44:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:44:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:44:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:44:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:44:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:44:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:44:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:44:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:44:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:44:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:44:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:44:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:44:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:44:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:44:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:44:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:45:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:45:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:45:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:45:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:45:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:45:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:45:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:45:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:45:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:45:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:45:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:45:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:45:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:45:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:45:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:45:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:45:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:45:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:45:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:45:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:45:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:45:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:45:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:45:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:45:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:45:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:45:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:46:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:46:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:46:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:46:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:46:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:46:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:46:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:46:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:46:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:46:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:46:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:46:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:46:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:46:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:46:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:46:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:46:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:46:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:46:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:46:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:46:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:46:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:46:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:46:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:46:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:46:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:46:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:47:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:47:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:47:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:47:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:47:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:47:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:47:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:47:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:47:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:47:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:47:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:47:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:47:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:47:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:47:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:47:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:47:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:47:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:47:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:47:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:47:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:47:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:47:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:47:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:47:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:47:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:47:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:47:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:48:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:48:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:48:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:48:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:48:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:48:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:48:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:48:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:48:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:48:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:48:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:48:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:48:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:48:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:48:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:48:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:48:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:48:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:48:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:48:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:48:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:48:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:48:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:48:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:48:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:48:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:48:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:49:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:49:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:49:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:49:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:49:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:49:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:49:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:49:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:49:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:49:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:49:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:49:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:49:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:49:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:49:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:49:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:49:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:49:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:49:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:49:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:49:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:49:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:49:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:49:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:49:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:49:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:49:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:50:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:50:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:50:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:50:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:50:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:50:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:50:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:50:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:50:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:50:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:50:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:50:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:50:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:50:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:50:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:50:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:50:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:50:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:50:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:50:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:50:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:50:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:50:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:50:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:50:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:50:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:50:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:50:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:51:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:51:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:51:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:51:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:51:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:51:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:51:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:51:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:51:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:51:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:51:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:51:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:51:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:51:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:51:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:51:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:51:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:51:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:51:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:51:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:51:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:51:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:51:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:51:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:51:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:51:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:51:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:52:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:52:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:52:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:52:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:52:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:52:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:52:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:52:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:52:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:52:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:52:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:52:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:52:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:52:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:52:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:52:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:52:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:52:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:52:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:52:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:52:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:52:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:52:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:52:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:52:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:52:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:52:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:53:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:53:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:53:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:53:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:53:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:53:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:53:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:53:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:53:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:53:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:53:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:53:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:53:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:53:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:53:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:53:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:53:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:53:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:53:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:53:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:53:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:53:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:53:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:53:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:53:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:53:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:53:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:53:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:54:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:54:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:54:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:54:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:54:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:54:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:54:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:54:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:54:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:54:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:54:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:54:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:54:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:54:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:54:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:54:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:54:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:54:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:54:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:54:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:54:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:54:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:54:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:54:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:54:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:54:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:54:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:55:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:55:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:55:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:55:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:55:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:55:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:55:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:55:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:55:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:55:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:55:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:55:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:55:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:55:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:55:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:55:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:55:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:55:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:55:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:55:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:55:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:55:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:55:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:55:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:55:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:55:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:55:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:56:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:56:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:56:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:56:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:56:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:56:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:56:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:56:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:56:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:56:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:56:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:56:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:56:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:56:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:56:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:56:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:56:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:56:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:56:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:56:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:56:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:56:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:56:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:56:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:56:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:56:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:56:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:56:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:57:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:57:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:57:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:57:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:57:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:57:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:57:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:57:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:57:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:57:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:57:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:57:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:57:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:57:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:57:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:57:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:57:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:57:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:57:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:57:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:57:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:57:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:57:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:57:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:57:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:57:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:57:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:58:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:58:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:58:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:58:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:58:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:58:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:58:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:58:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:58:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:58:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:58:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:58:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:58:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:58:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:58:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:58:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:58:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:58:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:58:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:58:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:58:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:58:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:58:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:58:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:58:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:58:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:58:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:59:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:59:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:59:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:59:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:59:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:59:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:59:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:59:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:59:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:59:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:59:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:59:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:59:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:59:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:59:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:59:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:59:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:59:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:59:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 07:59:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:59:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:59:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:59:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:59:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 07:59:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 07:59:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 07:59:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 07:59:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:00:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:00:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:00:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:00:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:00:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:00:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:00:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:00:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:00:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:00:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:00:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:00:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:00:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:00:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:00:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:00:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:00:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:00:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:00:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:00:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:00:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:00:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:00:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:00:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:00:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:00:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:00:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:01:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:01:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:01:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:01:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:01:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:01:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:01:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:01:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:01:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:01:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:01:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:01:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:01:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:01:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:01:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:01:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:01:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:01:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:01:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:01:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:01:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:01:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:01:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:01:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:01:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:01:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:01:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:02:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:02:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:02:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:02:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:02:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:02:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:02:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:02:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:02:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:02:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:02:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:02:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:02:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:02:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:02:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:02:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:02:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:02:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:02:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:02:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:02:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:02:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:02:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:02:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:02:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:02:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:02:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:03:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:03:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:03:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:03:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:03:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:03:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:03:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:03:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:03:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:03:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:03:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:03:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:03:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:03:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:03:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:03:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:03:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:03:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:03:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:03:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:03:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:03:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:03:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:03:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:03:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:03:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:03:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:03:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:04:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:04:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:04:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:04:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:04:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:04:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:04:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:04:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:04:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:04:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:04:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:04:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:04:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:04:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:04:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:04:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:04:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:04:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:04:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:04:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:04:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:04:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:04:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:04:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:04:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:04:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:04:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:05:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:05:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:05:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:05:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:05:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:05:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:05:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:05:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:05:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:05:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:05:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:05:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:05:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:05:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:05:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:05:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:05:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:05:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:05:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:05:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:05:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:05:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:05:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:05:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:05:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:05:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:05:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:06:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:06:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:06:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:06:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:06:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:06:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:06:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:06:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:06:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:06:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:06:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:06:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:06:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:06:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:06:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:06:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:06:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:06:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:06:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:06:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:06:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:06:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:06:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:06:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:06:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:06:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:06:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:06:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:07:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:07:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:07:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:07:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:07:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:07:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:07:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:07:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:07:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:07:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:07:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:07:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:07:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:07:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:07:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:07:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:07:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:07:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:07:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:07:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:07:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:07:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:07:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:07:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:07:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:07:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:07:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:08:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:08:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:08:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:08:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:08:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:08:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:08:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:08:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:08:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:08:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:08:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:08:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:08:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:08:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:08:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:08:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:08:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:08:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:08:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:08:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:08:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:08:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:08:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:08:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:08:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:08:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:08:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:09:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:09:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:09:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:09:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:09:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:09:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:09:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:09:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:09:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:09:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:09:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:09:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:09:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:09:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:09:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:09:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:09:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:09:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:09:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:09:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:09:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:09:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:09:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:09:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:09:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:09:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:09:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:09:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:10:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:10:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:10:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:10:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:10:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:10:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:10:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:10:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:10:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:10:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:10:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:10:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:10:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:10:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:10:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:10:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:10:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:10:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:10:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:10:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:10:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:10:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:10:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:10:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:10:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:10:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:10:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:11:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:11:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:11:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:11:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:11:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:11:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:11:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:11:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:11:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:11:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:11:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:11:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:11:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:11:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:11:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:11:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:11:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:11:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:11:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:11:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:11:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:11:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:11:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:11:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:11:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:11:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:11:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:12:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:12:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:12:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:12:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:12:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:12:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:12:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:12:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:12:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:12:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:12:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:12:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:12:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:12:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:12:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:12:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:12:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:12:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:12:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:12:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:12:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:12:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:12:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:12:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:12:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:12:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:12:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:12:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:13:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:13:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:13:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:13:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:13:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:13:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:13:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:13:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:13:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:13:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:13:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:13:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:13:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:13:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:13:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:13:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:13:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:13:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:13:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:13:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:13:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:13:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:13:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:13:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:13:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:13:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:13:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:14:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:14:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:14:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:14:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:14:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:14:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:14:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:14:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:14:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:14:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:14:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:14:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:14:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:14:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:14:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:14:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:14:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:14:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:14:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:14:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:14:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:14:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:14:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:14:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:14:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:14:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:14:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:15:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:15:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:15:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:15:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:15:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:15:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:15:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:15:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:15:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:15:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:15:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:15:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:15:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:15:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:15:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:15:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:15:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:15:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:15:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:15:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:15:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:15:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:15:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:15:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:15:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:15:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:15:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:15:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:16:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:16:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:16:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:16:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:16:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:16:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:16:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:16:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:16:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:16:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:16:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:16:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:16:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:16:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:16:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:16:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:16:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:16:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:16:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:16:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:16:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:16:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:16:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:16:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:16:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:16:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:16:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:17:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:17:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:17:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:17:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:17:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:17:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:17:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:17:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:17:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:17:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:17:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:17:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:17:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:17:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:17:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:17:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:17:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:17:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:17:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:17:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:17:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:17:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:17:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:17:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:17:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:17:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:17:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:18:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:18:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:18:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:18:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:18:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:18:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:18:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:18:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:18:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:18:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:18:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:18:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:18:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:18:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:18:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:18:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:18:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:18:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:18:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:18:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:18:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:18:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:18:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:18:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:18:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:18:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:18:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:18:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:19:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:19:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:19:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:19:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:19:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:19:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:19:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:19:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:19:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:19:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:19:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:19:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:19:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:19:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:19:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:19:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:19:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:19:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:19:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:19:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:19:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:19:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:19:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:19:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:19:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:19:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:19:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:20:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:20:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:20:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:20:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:20:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:20:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:20:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:20:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:20:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:20:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:20:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:20:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:20:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:20:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:20:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:20:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:20:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:20:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:20:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:20:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:20:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:20:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:20:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:20:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:20:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:20:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:20:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:21:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:21:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:21:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:21:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:21:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:21:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:21:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:21:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:21:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:21:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:21:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:21:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:21:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:21:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:21:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:21:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:21:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:21:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:21:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:21:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:21:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:21:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:21:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:21:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:21:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:21:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:21:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:22:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:22:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:22:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:22:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:22:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:22:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:22:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:22:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:22:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:22:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:22:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:22:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:22:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:22:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:22:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:22:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:22:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:22:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:22:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:22:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:22:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:22:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:22:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:22:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:22:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:22:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:22:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:22:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:23:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:23:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:23:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:23:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:23:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:23:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:23:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:23:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:23:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:23:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:23:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:23:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:23:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:23:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:23:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:23:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:23:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:23:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:23:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:23:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:23:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:23:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:23:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:23:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:23:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:23:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:23:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:24:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:24:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:24:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:24:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:24:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:24:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:24:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:24:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:24:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:24:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:24:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:24:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:24:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:24:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:24:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:24:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:24:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:24:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:24:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:24:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:24:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:24:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:24:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:24:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:24:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:24:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:24:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:25:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:25:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:25:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:25:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:25:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:25:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:25:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:25:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:25:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:25:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:25:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:25:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:25:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:25:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:25:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:25:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:25:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:25:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:25:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:25:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:25:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:25:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:25:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:25:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:25:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:25:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:25:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:25:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:26:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:26:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:26:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:26:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:26:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:26:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:26:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:26:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:26:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:26:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:26:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:26:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:26:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:26:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:26:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:26:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:26:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:26:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:26:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:26:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:26:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:26:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:26:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:26:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:26:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:26:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:26:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:27:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:27:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:27:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:27:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:27:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:27:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:27:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:27:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:27:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:27:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:27:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:27:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:27:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:27:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:27:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:27:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:27:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:27:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:27:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:27:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:27:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:27:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:27:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:27:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:27:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:27:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:27:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:28:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:28:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:28:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:28:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:28:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:28:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:28:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:28:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:28:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:28:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:28:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:28:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:28:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:28:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:28:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:28:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:28:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:28:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:28:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:28:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:28:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:28:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:28:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:28:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:28:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:28:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:28:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:28:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:29:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:29:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:29:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:29:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:29:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:29:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:29:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:29:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:29:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:29:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:29:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:29:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:29:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:29:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:29:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:29:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:29:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:29:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:29:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:29:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:29:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:29:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:29:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:29:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:29:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:29:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:29:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:30:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:30:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:30:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:30:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:30:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:30:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:30:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:30:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:30:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:30:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:30:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:30:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:30:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:30:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:30:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:30:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:30:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:30:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:30:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:30:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:30:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:30:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:30:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:30:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:30:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:30:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:30:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:31:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:31:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:31:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:31:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:31:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:31:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:31:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:31:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:31:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:31:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:31:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:31:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:31:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:31:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:31:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:31:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:31:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:31:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:31:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:31:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:31:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:31:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:31:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:31:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:31:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:31:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:31:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:31:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:32:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:32:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:32:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:32:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:32:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:32:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:32:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:32:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:32:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:32:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:32:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:32:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:32:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:32:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:32:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:32:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:32:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:32:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:32:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:32:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:32:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:32:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:32:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:32:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:32:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:32:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:32:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:33:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:33:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:33:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:33:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:33:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:33:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:33:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:33:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:33:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:33:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:33:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:33:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:33:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:33:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:33:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:33:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:33:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:33:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:33:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:33:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:33:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:33:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:33:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:33:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:33:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:33:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:33:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:34:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:34:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:34:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:34:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:34:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:34:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:34:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:34:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:34:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:34:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:34:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:34:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:34:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:34:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:34:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:34:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:34:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:34:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:34:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:34:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:34:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:34:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:34:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:34:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:34:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:34:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:34:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:34:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:35:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:35:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:35:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:35:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:35:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:35:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:35:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:35:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:35:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:35:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:35:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:35:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:35:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:35:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:35:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:35:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:35:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:35:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:35:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:35:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:35:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:35:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:35:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:35:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:35:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:35:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:35:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:36:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:36:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:36:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:36:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:36:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:36:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:36:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:36:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:36:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:36:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:36:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:36:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:36:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:36:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:36:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:36:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:36:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:36:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:36:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:36:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:36:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:36:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:36:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:36:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:36:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:36:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:36:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:37:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:37:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:37:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:37:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:37:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:37:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:37:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:37:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:37:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:37:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:37:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:37:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:37:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:37:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:37:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:37:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:37:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:37:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:37:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:37:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:37:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:37:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:37:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:37:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:37:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:37:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:37:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:38:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:38:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:38:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:38:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:38:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:38:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:38:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:38:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:38:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:38:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:38:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:38:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:38:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:38:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:38:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:38:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:38:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:38:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:38:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:38:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:38:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:38:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:38:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:38:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:38:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:38:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:38:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:38:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:39:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:39:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:39:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:39:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:39:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:39:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:39:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:39:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:39:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:39:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:39:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:39:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:39:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:39:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:39:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:39:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:39:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:39:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:39:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:39:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:39:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:39:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:39:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:39:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:39:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:39:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:39:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:40:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:40:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:40:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:40:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:40:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:40:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:40:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:40:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:40:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:40:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:40:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:40:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:40:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:40:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:40:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:40:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:40:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:40:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:40:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:40:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:40:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:40:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:40:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:40:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:40:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:40:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:40:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:41:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:41:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:41:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:41:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:41:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:41:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:41:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:41:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:41:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:41:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:41:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:41:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:41:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:41:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:41:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:41:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:41:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:41:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:41:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:41:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:41:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:41:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:41:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:41:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:41:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:41:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:41:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:41:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:42:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:42:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:42:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:42:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:42:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:42:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:42:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:42:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:42:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:42:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:42:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:42:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:42:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:42:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:42:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:42:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:42:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:42:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:42:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:42:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:42:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:42:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:42:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:42:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:42:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:42:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:42:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:43:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:43:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:43:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:43:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:43:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:43:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:43:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:43:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:43:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:43:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:43:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:43:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:43:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:43:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:43:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:43:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:43:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:43:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:43:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:43:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:43:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:43:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:43:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:43:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:43:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:43:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:43:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:44:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:44:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:44:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:44:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:44:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:44:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:44:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:44:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:44:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:44:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:44:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:44:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:44:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:44:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:44:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:44:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:44:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:44:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:44:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:44:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:44:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:44:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:44:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:44:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:44:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:44:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:44:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:44:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:45:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:45:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:45:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:45:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:45:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:45:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:45:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:45:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:45:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:45:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:45:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:45:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:45:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:45:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:45:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:45:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:45:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:45:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:45:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:45:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:45:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:45:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:45:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:45:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:45:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:45:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:45:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:46:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:46:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:46:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:46:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:46:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:46:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:46:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:46:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:46:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:46:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:46:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:46:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:46:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:46:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:46:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:46:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:46:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:46:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:46:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:46:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:46:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:46:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:46:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:46:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:46:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:46:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:46:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:47:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:47:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:47:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:47:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:47:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:47:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:47:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:47:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:47:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:47:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:47:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:47:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:47:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:47:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:47:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:47:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:47:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:47:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:47:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:47:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:47:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:47:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:47:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:47:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:47:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:47:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:47:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:47:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:48:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:48:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:48:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:48:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:48:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:48:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:48:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:48:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:48:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:48:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:48:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:48:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:48:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:48:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:48:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:48:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:48:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:48:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:48:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:48:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:48:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:48:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:48:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:48:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:48:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:48:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:48:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:49:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:49:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:49:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:49:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:49:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:49:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:49:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:49:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:49:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:49:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:49:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:49:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:49:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:49:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:49:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:49:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:49:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:49:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:49:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:49:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:49:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:49:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:49:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:49:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:49:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:49:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:49:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:50:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:50:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:50:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:50:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:50:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:50:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:50:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:50:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:50:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:50:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:50:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:50:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:50:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:50:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:50:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:50:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:50:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:50:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:50:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:50:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:50:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:50:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:50:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:50:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:50:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:50:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:50:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:50:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:51:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:51:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:51:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:51:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:51:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:51:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:51:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:51:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:51:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:51:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:51:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:51:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:51:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:51:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:51:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:51:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:51:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:51:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:51:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:51:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:51:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:51:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:51:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:51:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:51:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:51:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:51:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:52:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:52:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:52:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:52:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:52:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:52:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:52:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:52:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:52:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:52:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:52:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:52:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:52:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:52:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:52:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:52:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:52:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:52:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:52:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:52:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:52:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:52:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:52:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:52:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:52:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:52:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:52:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:53:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:53:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:53:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:53:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:53:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:53:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:53:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:53:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:53:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:53:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:53:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:53:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:53:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:53:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:53:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:53:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:53:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:53:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:53:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:53:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:53:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:53:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:53:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:53:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:53:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:53:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:53:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:54:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:54:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:54:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:54:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:54:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:54:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:54:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:54:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:54:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:54:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:54:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:54:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:54:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:54:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:54:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:54:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:54:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:54:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:54:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:54:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:54:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:54:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:54:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:54:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:54:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:54:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:54:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:54:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:55:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:55:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:55:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:55:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:55:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:55:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:55:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:55:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:55:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:55:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:55:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:55:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:55:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:55:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:55:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:55:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:55:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:55:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:55:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:55:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:55:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:55:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:55:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:55:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:55:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:55:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:55:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:56:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:56:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:56:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:56:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:56:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:56:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:56:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:56:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:56:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:56:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:56:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:56:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:56:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:56:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:56:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:56:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:56:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:56:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:56:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:56:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:56:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:56:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:56:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:56:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:56:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:56:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:56:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:57:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:57:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:57:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:57:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:57:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:57:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:57:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:57:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:57:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:57:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:57:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:57:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:57:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:57:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:57:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:57:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:57:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:57:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:57:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:57:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:57:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:57:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:57:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:57:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:57:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:57:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:57:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:57:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:58:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:58:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:58:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:58:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:58:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:58:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:58:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:58:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:58:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:58:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:58:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:58:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:58:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:58:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:58:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:58:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:58:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:58:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:58:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:58:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:58:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:58:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:58:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:58:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:58:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:58:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:58:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:59:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:59:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:59:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:59:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:59:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:59:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:59:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:59:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:59:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:59:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:59:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:59:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:59:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:59:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:59:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:59:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:59:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:59:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:59:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:59:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:59:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:59:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 08:59:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 08:59:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 08:59:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 08:59:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 08:59:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:00:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:00:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:00:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:00:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:00:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:00:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:00:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:00:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:00:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:00:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:00:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:00:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:00:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:00:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:00:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:00:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:00:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:00:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:00:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:00:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:00:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:00:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:00:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:00:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:00:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:00:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:00:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:00:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:01:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:01:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:01:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:01:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:01:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:01:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:01:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:01:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:01:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:01:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:01:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:01:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:01:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:01:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:01:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:01:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:01:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:01:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:01:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:01:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:01:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:01:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:01:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:01:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:01:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:01:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:01:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:02:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:02:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:02:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:02:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:02:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:02:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:02:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:02:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:02:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:02:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:02:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:02:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:02:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:02:23 rdma06 systemd[1]: Starting dnf makecache...
Apr 16 09:02:24 rdma06 dnf[1097157]: Updating Subscription Management repositories.
Apr 16 09:02:24 rdma06 dnf[1097157]: Unable to read consumer identity
Apr 16 09:02:24 rdma06 dnf[1097157]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 16 09:02:24 rdma06 dnf[1097157]: Metadata cache refreshed recently.
Apr 16 09:02:24 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 16 09:02:24 rdma06 systemd[1]: Finished dnf makecache.
Apr 16 09:02:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:02:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:02:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:02:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:02:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:02:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:02:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:02:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:02:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:02:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:02:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:02:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:02:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:02:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:03:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:03:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:03:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:03:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:03:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:03:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:03:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:03:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:03:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:03:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:03:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:03:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:03:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:03:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:03:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:03:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:03:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:03:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:03:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:03:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:03:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:03:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:03:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:03:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:03:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:03:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:03:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:03:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:04:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:04:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:04:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:04:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:04:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:04:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:04:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:04:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:04:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:04:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:04:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:04:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:04:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:04:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:04:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:04:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:04:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:04:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:04:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:04:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:04:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:04:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:04:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:04:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:04:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:04:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:04:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:05:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:05:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:05:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:05:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:05:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:05:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:05:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:05:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:05:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:05:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:05:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:05:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:05:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:05:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:05:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:05:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:05:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:05:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:05:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:05:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:05:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:05:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:05:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:05:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:05:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:05:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:05:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:06:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:06:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:06:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:06:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:06:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:06:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:06:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:06:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:06:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:06:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:06:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:06:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:06:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:06:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:06:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:06:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:06:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:06:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:06:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:06:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:06:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:06:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:06:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:06:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:06:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:06:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:06:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:06:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:07:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:07:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:07:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:07:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:07:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:07:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:07:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:07:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:07:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:07:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:07:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:07:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:07:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:07:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:07:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:07:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:07:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:07:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:07:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:07:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:07:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:07:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:07:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:07:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:07:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:07:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:07:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:08:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:08:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:08:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:08:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:08:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:08:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:08:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:08:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:08:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:08:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:08:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:08:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:08:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:08:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:08:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:08:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:08:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:08:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:08:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:08:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:08:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:08:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:08:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:08:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:08:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:08:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:08:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:09:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:09:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:09:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:09:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:09:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:09:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:09:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:09:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:09:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:09:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:09:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:09:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:09:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:09:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:09:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:09:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:09:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:09:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:09:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:09:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:09:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:09:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:09:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:09:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:09:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:09:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:09:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:10:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:10:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:10:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:10:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:10:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:10:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:10:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:10:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:10:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:10:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:10:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:10:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:10:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:10:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:10:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:10:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:10:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:10:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:10:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:10:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:10:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:10:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:10:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:10:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:10:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:10:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:10:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:10:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:11:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:11:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:11:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:11:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:11:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:11:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:11:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:11:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:11:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:11:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:11:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:11:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:11:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:11:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:11:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:11:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:11:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:11:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:11:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:11:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:11:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:11:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:11:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:11:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:11:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:11:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:11:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:12:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:12:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:12:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:12:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:12:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:12:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:12:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:12:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:12:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:12:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:12:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:12:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:12:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:12:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:12:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:12:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:12:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:12:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:12:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:12:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:12:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:12:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:12:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:12:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:12:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:12:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:12:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:13:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:13:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:13:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:13:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:13:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:13:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:13:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:13:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:13:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:13:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:13:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:13:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:13:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:13:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:13:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:13:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:13:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:13:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:13:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:13:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:13:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:13:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:13:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:13:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:13:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:13:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:13:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:13:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:14:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:14:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:14:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:14:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:14:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:14:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:14:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:14:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:14:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:14:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:14:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:14:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:14:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:14:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:14:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:14:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:14:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:14:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:14:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:14:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:14:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:14:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:14:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:14:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:14:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:14:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:14:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:15:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:15:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:15:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:15:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:15:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:15:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:15:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:15:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:15:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:15:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:15:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:15:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:15:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:15:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:15:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:15:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:15:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:15:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:15:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:15:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:15:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:15:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:15:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:15:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:15:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:15:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:15:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:16:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:16:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:16:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:16:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:16:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:16:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:16:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:16:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:16:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:16:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:16:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:16:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:16:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:16:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:16:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:16:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:16:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:16:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:16:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:16:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:16:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:16:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:16:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:16:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:16:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:16:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:16:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:16:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:17:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:17:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:17:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:17:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:17:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:17:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:17:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:17:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:17:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:17:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:17:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:17:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:17:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:17:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:17:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:17:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:17:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:17:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:17:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:17:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:17:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:17:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:17:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:17:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:17:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:17:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:17:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:18:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:18:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:18:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:18:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:18:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:18:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:18:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:18:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:18:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:18:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:18:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:18:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:18:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:18:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:18:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:18:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:18:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:18:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:18:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:18:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:18:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:18:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:18:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:18:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:18:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:18:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:18:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:19:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:19:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:19:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:19:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:19:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:19:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:19:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:19:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:19:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:19:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:19:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:19:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:19:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:19:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:19:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:19:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:19:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:19:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:19:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:19:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:19:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:19:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:19:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:19:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:19:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:19:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:19:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:19:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:20:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:20:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:20:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:20:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:20:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:20:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:20:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:20:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:20:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:20:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:20:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:20:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:20:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:20:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:20:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:20:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:20:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:20:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:20:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:20:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:20:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:20:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:20:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:20:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:20:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:20:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:20:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:21:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:21:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:21:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:21:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:21:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:21:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:21:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:21:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:21:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:21:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:21:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:21:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:21:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:21:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:21:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:21:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:21:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:21:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:21:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:21:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:21:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:21:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:21:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:21:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:21:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:21:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:21:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:22:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:22:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:22:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:22:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:22:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:22:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:22:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:22:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:22:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:22:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:22:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:22:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:22:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:22:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:22:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:22:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:22:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:22:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:22:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:22:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:22:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:22:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:22:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:22:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:22:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:22:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:22:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:22:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:23:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:23:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:23:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:23:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:23:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:23:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:23:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:23:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:23:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:23:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:23:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:23:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:23:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:23:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:23:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:23:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:23:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:23:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:23:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:23:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:23:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:23:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:23:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:23:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:23:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:23:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:23:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:24:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:24:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:24:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:24:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:24:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:24:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:24:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:24:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:24:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:24:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:24:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:24:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:24:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:24:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:24:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:24:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:24:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:24:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:24:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:24:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:24:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:24:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:24:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:24:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:24:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:24:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:24:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:25:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:25:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:25:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:25:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:25:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:25:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:25:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:25:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:25:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:25:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:25:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:25:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:25:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:25:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:25:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:25:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:25:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:25:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:25:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:25:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:25:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:25:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:25:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:25:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:25:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:25:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:25:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:25:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:26:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:26:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:26:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:26:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:26:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:26:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:26:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:26:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:26:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:26:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:26:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:26:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:26:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:26:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:26:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:26:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:26:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:26:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:26:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:26:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:26:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:26:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:26:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:26:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:26:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:26:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:26:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:27:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:27:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:27:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:27:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:27:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:27:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:27:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:27:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:27:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:27:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:27:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:27:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:27:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:27:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:27:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:27:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:27:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:27:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:27:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:27:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:27:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:27:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:27:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:27:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:27:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:27:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:27:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:28:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:28:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:28:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:28:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:28:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:28:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:28:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:28:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:28:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:28:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:28:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:28:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:28:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:28:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:28:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:28:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:28:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:28:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:28:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:28:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:28:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:28:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:28:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:28:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:28:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:28:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:28:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:29:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:29:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:29:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:29:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:29:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:29:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:29:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:29:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:29:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:29:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:29:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:29:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:29:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:29:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:29:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:29:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:29:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:29:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:29:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:29:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:29:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:29:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:29:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:29:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:29:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:29:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:29:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:29:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:30:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:30:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:30:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:30:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:30:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:30:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:30:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:30:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:30:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:30:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:30:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:30:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:30:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:30:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:30:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:30:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:30:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:30:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:30:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:30:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:30:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:30:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:30:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:30:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:30:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:30:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:30:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:31:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:31:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:31:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:31:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:31:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:31:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:31:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:31:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:31:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:31:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:31:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:31:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:31:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:31:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:31:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:31:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:31:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:31:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:31:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:31:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:31:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:31:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:31:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:31:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:31:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:31:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:31:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:32:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:32:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:32:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:32:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:32:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:32:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:32:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:32:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:32:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:32:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:32:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:32:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:32:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:32:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:32:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:32:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:32:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:32:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:32:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:32:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:32:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:32:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:32:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:32:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:32:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:32:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:32:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:32:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:33:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:33:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:33:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:33:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:33:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:33:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:33:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:33:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:33:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:33:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:33:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:33:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:33:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:33:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:33:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:33:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:33:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:33:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:33:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:33:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:33:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:33:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:33:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:33:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:33:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:33:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:33:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:34:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:34:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:34:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:34:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:34:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:34:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:34:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:34:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:34:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:34:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:34:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:34:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:34:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:34:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:34:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:34:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:34:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:34:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:34:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:34:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:34:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:34:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:34:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:34:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:34:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:34:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:34:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:35:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:35:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:35:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:35:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:35:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:35:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:35:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:35:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:35:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:35:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:35:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:35:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:35:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:35:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:35:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:35:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:35:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:35:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:35:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:35:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:35:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:35:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:35:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:35:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:35:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:35:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:35:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:35:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:36:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:36:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:36:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:36:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:36:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:36:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:36:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:36:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:36:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:36:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:36:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:36:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:36:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:36:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:36:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:36:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:36:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:36:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:36:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:36:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:36:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:36:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:36:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:36:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:36:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:36:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:36:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:37:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:37:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:37:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:37:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:37:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:37:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:37:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:37:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:37:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:37:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:37:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:37:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:37:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:37:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:37:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:37:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:37:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:37:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:37:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:37:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:37:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:37:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:37:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:37:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:37:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:37:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:37:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:38:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:38:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:38:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:38:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:38:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:38:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:38:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:38:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:38:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:38:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:38:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:38:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:38:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:38:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:38:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:38:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:38:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:38:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:38:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:38:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:38:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:38:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:38:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:38:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:38:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:38:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:38:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:38:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:39:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:39:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:39:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:39:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:39:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:39:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:39:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:39:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:39:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:39:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:39:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:39:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:39:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:39:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:39:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:39:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:39:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:39:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:39:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:39:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:39:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:39:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:39:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:39:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:39:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:39:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:39:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:40:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:40:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:40:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:40:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:40:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:40:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:40:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:40:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:40:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:40:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:40:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:40:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:40:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:40:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:40:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:40:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:40:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:40:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:40:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:40:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:40:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:40:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:40:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:40:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:40:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:40:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:40:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:41:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:41:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:41:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:41:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:41:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:41:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:41:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:41:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:41:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:41:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:41:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:41:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:41:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:41:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:41:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:41:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:41:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:41:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:41:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:41:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:41:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:41:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:41:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:41:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:41:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:41:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:41:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:41:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:42:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:42:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:42:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:42:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:42:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:42:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:42:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:42:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:42:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:42:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:42:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:42:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:42:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:42:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:42:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:42:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:42:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:42:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:42:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:42:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:42:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:42:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:42:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:42:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:42:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:42:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:42:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:43:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:43:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:43:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:43:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:43:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:43:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:43:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:43:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:43:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:43:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:43:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:43:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:43:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:43:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:43:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:43:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:43:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:43:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:43:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:43:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:43:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:43:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:43:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:43:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:43:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:43:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:43:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:44:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:44:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:44:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:44:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:44:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:44:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:44:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:44:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:44:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:44:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:44:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:44:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:44:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:44:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:44:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:44:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:44:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:44:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:44:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:44:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:44:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:44:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:44:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:44:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:44:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:44:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:44:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:45:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:45:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:45:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:45:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:45:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:45:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:45:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:45:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:45:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:45:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:45:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:45:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:45:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:45:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:45:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:45:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:45:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:45:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:45:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:45:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:45:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:45:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:45:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:45:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:45:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:45:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:45:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:45:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:46:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:46:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:46:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:46:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:46:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:46:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:46:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:46:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:46:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:46:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:46:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:46:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:46:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:46:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:46:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:46:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:46:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:46:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:46:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:46:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:46:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:46:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:46:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:46:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:46:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:46:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:46:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:47:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:47:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:47:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:47:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:47:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:47:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:47:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:47:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:47:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:47:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:47:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:47:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:47:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:47:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:47:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:47:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:47:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:47:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:47:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:47:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:47:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:47:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:47:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:47:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:47:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:47:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:47:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:48:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:48:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:48:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:48:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:48:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:48:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:48:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:48:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:48:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:48:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:48:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:48:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:48:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:48:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:48:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:48:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:48:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:48:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:48:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:48:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:48:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:48:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:48:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:48:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:48:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:48:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:48:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:48:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:49:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:49:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:49:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:49:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:49:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:49:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:49:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:49:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:49:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:49:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:49:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:49:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:49:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:49:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:49:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:49:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:49:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:49:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:49:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:49:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:49:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:49:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:49:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:49:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:49:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:49:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:49:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:50:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:50:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:50:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:50:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:50:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:50:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:50:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:50:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:50:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:50:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:50:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:50:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:50:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:50:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:50:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:50:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:50:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:50:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:50:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:50:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:50:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:50:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:50:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:50:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:50:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:50:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:50:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:51:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:51:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:51:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:51:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:51:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:51:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:51:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:51:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:51:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:51:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:51:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:51:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:51:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:51:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:51:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:51:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:51:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:51:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:51:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:51:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:51:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:51:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:51:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:51:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:51:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:51:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:51:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:51:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:52:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:52:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:52:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:52:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:52:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:52:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:52:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:52:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:52:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:52:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:52:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:52:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:52:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:52:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:52:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:52:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:52:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:52:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:52:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:52:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:52:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:52:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:52:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:52:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:52:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:52:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:52:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:53:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:53:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:53:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:53:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:53:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:53:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:53:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:53:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:53:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:53:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:53:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:53:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:53:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:53:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:53:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:53:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:53:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:53:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:53:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:53:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:53:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:53:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:53:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:53:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:53:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:53:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:53:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:54:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:54:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:54:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:54:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:54:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:54:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:54:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:54:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:54:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:54:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:54:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:54:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:54:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:54:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:54:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:54:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:54:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:54:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:54:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:54:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:54:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:54:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:54:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:54:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:54:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:54:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:54:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:54:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:55:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:55:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:55:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:55:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:55:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:55:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:55:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:55:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:55:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:55:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:55:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:55:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:55:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:55:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:55:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:55:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:55:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:55:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:55:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:55:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:55:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:55:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:55:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:55:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:55:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:55:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:55:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:56:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:56:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:56:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:56:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:56:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:56:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:56:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:56:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:56:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:56:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:56:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:56:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:56:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:56:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:56:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:56:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:56:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:56:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:56:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:56:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:56:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:56:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:56:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:56:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:56:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:56:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:56:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:57:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:57:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:57:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:57:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:57:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:57:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:57:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:57:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:57:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:57:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:57:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:57:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:57:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:57:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:57:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:57:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:57:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:57:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:57:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:57:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:57:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:57:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:57:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:57:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:57:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:57:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:57:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:57:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:58:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:58:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:58:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:58:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:58:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:58:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:58:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:58:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:58:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:58:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:58:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:58:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:58:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:58:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:58:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:58:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:58:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:58:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:58:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:58:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:58:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:58:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:58:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:58:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:58:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:58:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:58:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:59:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:59:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:59:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:59:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:59:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:59:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:59:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:59:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:59:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:59:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:59:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:59:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:59:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:59:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:59:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:59:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:59:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:59:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:59:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:59:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:59:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:59:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 09:59:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 09:59:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 09:59:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 09:59:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 09:59:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:00:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:00:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:00:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:00:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:00:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:00:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:00:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:00:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:00:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:00:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:00:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:00:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:00:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:00:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:00:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:00:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:00:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:00:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:00:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:00:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:00:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:00:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:00:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:00:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:00:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:00:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:00:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:01:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:01:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:01:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:01:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:01:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:01:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:01:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:01:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:01:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:01:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:01:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:01:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:01:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:01:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:01:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:01:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:01:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:01:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:01:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:01:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:01:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:01:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:01:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:01:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:01:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:01:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:01:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:01:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:02:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:02:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:02:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:02:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:02:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:02:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:02:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:02:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:02:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:02:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:02:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:02:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:02:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:02:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:02:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:02:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:02:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:02:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:02:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:02:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:02:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:02:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:02:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:02:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:02:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:02:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:02:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:03:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:03:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:03:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:03:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:03:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:03:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:03:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:03:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:03:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:03:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:03:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:03:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:03:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:03:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:03:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:03:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:03:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:03:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:03:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:03:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:03:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:03:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:03:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:03:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:03:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:03:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:03:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:04:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:04:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:04:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:04:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:04:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:04:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:04:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:04:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:04:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:04:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:04:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:04:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:04:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:04:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:04:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:04:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:04:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:04:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:04:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:04:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:04:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:04:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:04:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:04:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:04:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:04:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:04:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:04:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:05:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:05:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:05:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:05:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:05:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:05:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:05:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:05:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:05:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:05:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:05:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:05:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:05:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:05:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:05:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:05:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:05:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:05:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:05:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:05:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:05:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:05:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:05:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:05:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:05:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:05:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:05:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:06:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:06:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:06:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:06:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:06:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:06:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:06:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:06:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:06:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:06:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:06:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:06:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:06:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:06:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:06:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:06:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:06:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:06:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:06:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:06:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:06:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:06:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:06:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:06:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:06:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:06:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:06:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:07:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:07:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:07:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:07:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:07:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:07:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:07:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:07:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:07:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:07:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:07:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:07:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:07:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:07:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:07:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:07:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:07:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:07:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:07:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:07:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:07:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:07:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:07:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:07:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:07:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:07:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:07:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:07:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:08:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:08:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:08:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:08:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:08:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:08:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:08:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:08:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:08:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:08:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:08:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:08:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:08:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:08:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:08:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:08:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:08:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:08:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:08:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:08:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:08:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:08:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:08:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:08:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:08:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:08:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:08:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:09:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:09:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:09:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:09:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:09:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:09:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:09:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:09:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:09:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:09:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:09:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:09:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:09:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:09:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:09:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:09:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:09:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:09:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:09:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:09:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:09:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:09:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:09:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:09:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:09:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:09:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:09:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:10:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:10:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:10:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:10:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:10:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:10:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:10:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:10:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:10:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:10:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:10:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:10:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:10:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:10:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:10:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:10:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:10:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:10:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:10:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:10:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:10:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:10:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:10:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:10:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:10:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:10:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:10:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:10:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:11:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:11:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:11:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:11:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:11:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:11:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:11:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:11:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:11:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:11:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:11:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:11:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:11:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:11:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:11:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:11:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:11:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:11:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:11:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:11:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:11:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:11:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:11:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:11:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:11:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:11:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:11:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:12:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:12:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:12:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:12:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:12:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:12:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:12:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:12:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:12:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:12:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:12:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:12:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:12:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:12:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:12:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:12:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:12:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:12:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:12:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:12:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:12:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:12:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:12:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:12:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:12:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:12:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:12:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:13:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:13:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:13:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:13:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:13:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:13:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:13:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:13:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:13:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:13:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:13:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:13:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:13:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:13:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:13:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:13:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:13:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:13:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:13:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:13:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:13:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:13:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:13:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:13:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:13:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:13:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:13:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:13:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:14:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:14:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:14:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:14:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:14:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:14:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:14:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:14:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:14:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:14:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:14:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:14:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:14:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:14:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:14:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:14:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:14:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:14:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:14:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:14:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:14:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:14:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:14:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:14:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:14:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:14:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:14:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:15:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:15:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:15:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:15:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:15:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:15:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:15:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:15:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:15:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:15:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:15:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:15:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:15:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:15:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:15:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:15:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:15:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:15:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:15:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:15:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:15:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:15:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:15:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:15:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:15:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:15:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:15:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:16:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:16:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:16:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:16:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:16:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:16:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:16:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:16:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:16:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:16:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:16:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:16:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:16:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:16:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:16:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:16:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:16:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:16:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:16:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:16:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:16:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:16:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:16:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:16:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:16:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:16:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:16:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:17:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:17:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:17:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:17:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:17:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:17:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:17:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:17:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:17:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:17:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:17:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:17:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:17:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:17:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:17:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:17:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:17:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:17:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:17:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:17:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:17:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:17:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:17:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:17:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:17:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:17:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:17:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:17:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:18:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:18:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:18:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:18:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:18:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:18:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:18:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:18:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:18:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:18:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:18:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:18:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:18:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:18:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:18:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:18:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:18:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:18:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:18:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:18:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:18:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:18:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:18:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:18:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:18:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:18:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:18:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:19:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:19:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:19:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:19:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:19:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:19:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:19:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:19:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:19:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:19:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:19:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:19:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:19:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:19:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:19:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:19:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:19:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:19:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:19:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:19:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:19:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:19:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:19:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:19:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:19:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:19:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:19:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:20:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:20:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:20:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:20:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:20:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:20:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:20:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:20:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:20:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:20:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:20:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:20:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:20:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:20:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:20:32 rdma06 NetworkManager[1025]: <info>  [1681654832.9129] dhcp4 (eno1): state changed new lease, address=10.73.131.45
Apr 16 10:20:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:20:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:20:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:20:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:20:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:20:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:20:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:20:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:20:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:20:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:20:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:20:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:20:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:20:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:21:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:21:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:21:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:21:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:21:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:21:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:21:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:21:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:21:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:21:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:21:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:21:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:21:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:21:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:21:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:21:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:21:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:21:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:21:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:21:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:21:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:21:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:21:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:21:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:21:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:21:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:21:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:22:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:22:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:22:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:22:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:22:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:22:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:22:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:22:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:22:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:22:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:22:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:22:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:22:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:22:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:22:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:22:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:22:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:22:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:22:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:22:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:22:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:22:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:22:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:22:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:22:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:22:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:22:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:23:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:23:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:23:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:23:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:23:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:23:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:23:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:23:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:23:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:23:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:23:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:23:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:23:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:23:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:23:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:23:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:23:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:23:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:23:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:23:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:23:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:23:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:23:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:23:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:23:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:23:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:23:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:23:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:24:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:24:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:24:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:24:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:24:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:24:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:24:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:24:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:24:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:24:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:24:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:24:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:24:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:24:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:24:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:24:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:24:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:24:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:24:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:24:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:24:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:24:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:24:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:24:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:24:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:24:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:24:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:25:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:25:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:25:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:25:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:25:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:25:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:25:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:25:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:25:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:25:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:25:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:25:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:25:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:25:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:25:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:25:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:25:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:25:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:25:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:25:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:25:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:25:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:25:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:25:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:25:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:25:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:25:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:26:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:26:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:26:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:26:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:26:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:26:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:26:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:26:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:26:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:26:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:26:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:26:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:26:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:26:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:26:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:26:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:26:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:26:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:26:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:26:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:26:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:26:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:26:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:26:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:26:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:26:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:26:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:26:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:27:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:27:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:27:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:27:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:27:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:27:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:27:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:27:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:27:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:27:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:27:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:27:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:27:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:27:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:27:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:27:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:27:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:27:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:27:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:27:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:27:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:27:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:27:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:27:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:27:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:27:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:27:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:28:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:28:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:28:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:28:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:28:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:28:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:28:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:28:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:28:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:28:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:28:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:28:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:28:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:28:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:28:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:28:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:28:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:28:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:28:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:28:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:28:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:28:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:28:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:28:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:28:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:28:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:28:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:29:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:29:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:29:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:29:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:29:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:29:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:29:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:29:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:29:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:29:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:29:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:29:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:29:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:29:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:29:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:29:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:29:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:29:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:29:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:29:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:29:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:29:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:29:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:29:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:29:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:29:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:29:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:29:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:30:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:30:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:30:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:30:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:30:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:30:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:30:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:30:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:30:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:30:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:30:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:30:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:30:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:30:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:30:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:30:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:30:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:30:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:30:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:30:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:30:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:30:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:30:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:30:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:30:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:30:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:30:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:31:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:31:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:31:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:31:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:31:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:31:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:31:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:31:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:31:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:31:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:31:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:31:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:31:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:31:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:31:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:31:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:31:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:31:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:31:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:31:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:31:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:31:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:31:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:31:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:31:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:31:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:31:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:32:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:32:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:32:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:32:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:32:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:32:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:32:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:32:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:32:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:32:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:32:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:32:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:32:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:32:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:32:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:32:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:32:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:32:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:32:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:32:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:32:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:32:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:32:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:32:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:32:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:32:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:32:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:32:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:33:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:33:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:33:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:33:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:33:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:33:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:33:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:33:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:33:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:33:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:33:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:33:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:33:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:33:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:33:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:33:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:33:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:33:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:33:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:33:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:33:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:33:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:33:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:33:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:33:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:33:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:33:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:34:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:34:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:34:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:34:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:34:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:34:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:34:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:34:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:34:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:34:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:34:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:34:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:34:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:34:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:34:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:34:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:34:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:34:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:34:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:34:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:34:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:34:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:34:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:34:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:34:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:34:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:34:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:35:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:35:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:35:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:35:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:35:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:35:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:35:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:35:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:35:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:35:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:35:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:35:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:35:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:35:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:35:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:35:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:35:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:35:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:35:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:35:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:35:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:35:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:35:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:35:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:35:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:35:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:35:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:36:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:36:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:36:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:36:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:36:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:36:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:36:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:36:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:36:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:36:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:36:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:36:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:36:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:36:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:36:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:36:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:36:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:36:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:36:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:36:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:36:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:36:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:36:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:36:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:36:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:36:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:36:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:36:53 rdma06 systemd[1]: Starting dnf makecache...
Apr 16 10:36:54 rdma06 dnf[1100982]: Updating Subscription Management repositories.
Apr 16 10:36:54 rdma06 dnf[1100982]: Unable to read consumer identity
Apr 16 10:36:54 rdma06 dnf[1100982]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 16 10:36:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:36:55 rdma06 dnf[1100982]: Copr repo for fmf owned by psss                 2.4 kB/s | 3.3 kB     00:01
Apr 16 10:36:55 rdma06 dnf[1100982]: beaker-AppStream                                 51 kB/s | 168  B     00:00
Apr 16 10:36:55 rdma06 dnf[1100982]: Errors during downloading metadata for repository 'beaker-AppStream':
Apr 16 10:36:55 rdma06 dnf[1100982]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 10:36:55 rdma06 dnf[1100982]: Error: Failed to download metadata for repo 'beaker-AppStream': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 10:36:55 rdma06 dnf[1100982]: beaker-AppStream-debuginfo                       62 kB/s | 168  B     00:00
Apr 16 10:36:55 rdma06 dnf[1100982]: Errors during downloading metadata for repository 'beaker-AppStream-debuginfo':
Apr 16 10:36:55 rdma06 dnf[1100982]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 10:36:55 rdma06 dnf[1100982]: Error: Failed to download metadata for repo 'beaker-AppStream-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 10:36:55 rdma06 dnf[1100982]: beaker-BaseOS                                    65 kB/s | 168  B     00:00
Apr 16 10:36:55 rdma06 dnf[1100982]: Errors during downloading metadata for repository 'beaker-BaseOS':
Apr 16 10:36:55 rdma06 dnf[1100982]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 10:36:55 rdma06 dnf[1100982]: Error: Failed to download metadata for repo 'beaker-BaseOS': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 10:36:55 rdma06 dnf[1100982]: beaker-BaseOS-debuginfo                          62 kB/s | 168  B     00:00
Apr 16 10:36:55 rdma06 dnf[1100982]: Errors during downloading metadata for repository 'beaker-BaseOS-debuginfo':
Apr 16 10:36:55 rdma06 dnf[1100982]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 10:36:55 rdma06 dnf[1100982]: Error: Failed to download metadata for repo 'beaker-BaseOS-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 10:36:55 rdma06 dnf[1100982]: beaker-CRB                                       62 kB/s | 168  B     00:00
Apr 16 10:36:55 rdma06 dnf[1100982]: Errors during downloading metadata for repository 'beaker-CRB':
Apr 16 10:36:55 rdma06 dnf[1100982]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 10:36:55 rdma06 dnf[1100982]: Error: Failed to download metadata for repo 'beaker-CRB': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 10:36:55 rdma06 dnf[1100982]: beaker-CRB-debuginfo                             66 kB/s | 168  B     00:00
Apr 16 10:36:55 rdma06 dnf[1100982]: Errors during downloading metadata for repository 'beaker-CRB-debuginfo':
Apr 16 10:36:55 rdma06 dnf[1100982]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 10:36:55 rdma06 dnf[1100982]: Error: Failed to download metadata for repo 'beaker-CRB-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 10:36:55 rdma06 dnf[1100982]: beaker-HighAvailability                          65 kB/s | 168  B     00:00
Apr 16 10:36:55 rdma06 dnf[1100982]: Errors during downloading metadata for repository 'beaker-HighAvailability':
Apr 16 10:36:55 rdma06 dnf[1100982]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 10:36:55 rdma06 dnf[1100982]: Error: Failed to download metadata for repo 'beaker-HighAvailability': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 10:36:55 rdma06 dnf[1100982]: beaker-HighAvailability-debuginfo                63 kB/s | 168  B     00:00
Apr 16 10:36:55 rdma06 dnf[1100982]: Errors during downloading metadata for repository 'beaker-HighAvailability-debuginfo':
Apr 16 10:36:55 rdma06 dnf[1100982]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 10:36:55 rdma06 dnf[1100982]: Error: Failed to download metadata for repo 'beaker-HighAvailability-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 10:36:55 rdma06 dnf[1100982]: beaker-NFV                                       55 kB/s | 168  B     00:00
Apr 16 10:36:55 rdma06 dnf[1100982]: Errors during downloading metadata for repository 'beaker-NFV':
Apr 16 10:36:55 rdma06 dnf[1100982]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 10:36:55 rdma06 dnf[1100982]: Error: Failed to download metadata for repo 'beaker-NFV': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 10:36:55 rdma06 dnf[1100982]: beaker-NFV-debuginfo                             61 kB/s | 168  B     00:00
Apr 16 10:36:55 rdma06 dnf[1100982]: Errors during downloading metadata for repository 'beaker-NFV-debuginfo':
Apr 16 10:36:55 rdma06 dnf[1100982]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 10:36:55 rdma06 dnf[1100982]: Error: Failed to download metadata for repo 'beaker-NFV-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 10:36:55 rdma06 dnf[1100982]: beaker-RT                                        61 kB/s | 168  B     00:00
Apr 16 10:36:55 rdma06 dnf[1100982]: Errors during downloading metadata for repository 'beaker-RT':
Apr 16 10:36:55 rdma06 dnf[1100982]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 10:36:55 rdma06 dnf[1100982]: Error: Failed to download metadata for repo 'beaker-RT': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 10:36:55 rdma06 dnf[1100982]: beaker-RT-debuginfo                              58 kB/s | 168  B     00:00
Apr 16 10:36:55 rdma06 dnf[1100982]: Errors during downloading metadata for repository 'beaker-RT-debuginfo':
Apr 16 10:36:55 rdma06 dnf[1100982]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 10:36:55 rdma06 dnf[1100982]: Error: Failed to download metadata for repo 'beaker-RT-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 10:36:55 rdma06 dnf[1100982]: beaker-ResilientStorage                          57 kB/s | 168  B     00:00
Apr 16 10:36:55 rdma06 dnf[1100982]: Errors during downloading metadata for repository 'beaker-ResilientStorage':
Apr 16 10:36:55 rdma06 dnf[1100982]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 10:36:55 rdma06 dnf[1100982]: Error: Failed to download metadata for repo 'beaker-ResilientStorage': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 10:36:55 rdma06 dnf[1100982]: beaker-ResilientStorage-debuginfo                60 kB/s | 168  B     00:00
Apr 16 10:36:55 rdma06 dnf[1100982]: Errors during downloading metadata for repository 'beaker-ResilientStorage-debuginfo':
Apr 16 10:36:55 rdma06 dnf[1100982]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 10:36:55 rdma06 dnf[1100982]: Error: Failed to download metadata for repo 'beaker-ResilientStorage-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 10:36:55 rdma06 dnf[1100982]: beaker-SAP                                       56 kB/s | 168  B     00:00
Apr 16 10:36:55 rdma06 dnf[1100982]: Errors during downloading metadata for repository 'beaker-SAP':
Apr 16 10:36:55 rdma06 dnf[1100982]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 10:36:55 rdma06 dnf[1100982]: Error: Failed to download metadata for repo 'beaker-SAP': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 10:36:55 rdma06 dnf[1100982]: beaker-SAP-debuginfo                             56 kB/s | 168  B     00:00
Apr 16 10:36:55 rdma06 dnf[1100982]: Errors during downloading metadata for repository 'beaker-SAP-debuginfo':
Apr 16 10:36:55 rdma06 dnf[1100982]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 10:36:55 rdma06 dnf[1100982]: Error: Failed to download metadata for repo 'beaker-SAP-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 10:36:55 rdma06 dnf[1100982]: beaker-SAPHANA                                   59 kB/s | 168  B     00:00
Apr 16 10:36:55 rdma06 dnf[1100982]: Errors during downloading metadata for repository 'beaker-SAPHANA':
Apr 16 10:36:55 rdma06 dnf[1100982]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 10:36:55 rdma06 dnf[1100982]: Error: Failed to download metadata for repo 'beaker-SAPHANA': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 10:36:55 rdma06 dnf[1100982]: beaker-SAPHANA-debuginfo                         59 kB/s | 168  B     00:00
Apr 16 10:36:55 rdma06 dnf[1100982]: Errors during downloading metadata for repository 'beaker-SAPHANA-debuginfo':
Apr 16 10:36:55 rdma06 dnf[1100982]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 10:36:55 rdma06 dnf[1100982]: Error: Failed to download metadata for repo 'beaker-SAPHANA-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 10:36:56 rdma06 dnf[1100982]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr 16 10:36:57 rdma06 dnf[1100982]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr 16 10:36:57 rdma06 dnf[1100982]: Ignoring repositories: beaker-AppStream, beaker-AppStream-debuginfo, beaker-BaseOS, beaker-BaseOS-debuginfo, beaker-CRB, beaker-CRB-debuginfo, beaker-HighAvailability, beaker-HighAvailability-debuginfo, beaker-NFV, beaker-NFV-debuginfo, beaker-RT, beaker-RT-debuginfo, beaker-ResilientStorage, beaker-ResilientStorage-debuginfo, beaker-SAP, beaker-SAP-debuginfo, beaker-SAPHANA, beaker-SAPHANA-debuginfo
Apr 16 10:36:57 rdma06 dnf[1100982]: Metadata cache created.
Apr 16 10:36:57 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 16 10:36:57 rdma06 systemd[1]: Finished dnf makecache.
Apr 16 10:37:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:37:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:37:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:37:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:37:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:37:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:37:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:37:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:37:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:37:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:37:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:37:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:37:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:37:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:37:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:37:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:37:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:37:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:37:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:37:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:37:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:37:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:37:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:37:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:37:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:37:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:37:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:38:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:38:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:38:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:38:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:38:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:38:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:38:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:38:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:38:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:38:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:38:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:38:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:38:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:38:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:38:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:38:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:38:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:38:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:38:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:38:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:38:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:38:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:38:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:38:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:38:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:38:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:38:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:39:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:39:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:39:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:39:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:39:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:39:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:39:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:39:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:39:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:39:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:39:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:39:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:39:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:39:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:39:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:39:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:39:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:39:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:39:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:39:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:39:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:39:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:39:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:39:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:39:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:39:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:39:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:39:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:40:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:40:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:40:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:40:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:40:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:40:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:40:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:40:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:40:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:40:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:40:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:40:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:40:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:40:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:40:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:40:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:40:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:40:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:40:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:40:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:40:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:40:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:40:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:40:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:40:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:40:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:40:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:41:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:41:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:41:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:41:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:41:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:41:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:41:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:41:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:41:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:41:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:41:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:41:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:41:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:41:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:41:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:41:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:41:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:41:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:41:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:41:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:41:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:41:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:41:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:41:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:41:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:41:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:41:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:42:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:42:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:42:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:42:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:42:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:42:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:42:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:42:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:42:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:42:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:42:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:42:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:42:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:42:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:42:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:42:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:42:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:42:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:42:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:42:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:42:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:42:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:42:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:42:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:42:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:42:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:42:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:42:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:43:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:43:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:43:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:43:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:43:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:43:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:43:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:43:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:43:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:43:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:43:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:43:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:43:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:43:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:43:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:43:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:43:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:43:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:43:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:43:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:43:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:43:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:43:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:43:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:43:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:43:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:43:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:44:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:44:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:44:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:44:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:44:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:44:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:44:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:44:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:44:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:44:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:44:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:44:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:44:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:44:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:44:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:44:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:44:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:44:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:44:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:44:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:44:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:44:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:44:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:44:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:44:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:44:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:44:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:45:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:45:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:45:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:45:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:45:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:45:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:45:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:45:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:45:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:45:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:45:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:45:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:45:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:45:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:45:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:45:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:45:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:45:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:45:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:45:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:45:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:45:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:45:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:45:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:45:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:45:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:45:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:45:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:46:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:46:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:46:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:46:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:46:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:46:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:46:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:46:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:46:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:46:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:46:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:46:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:46:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:46:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:46:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:46:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:46:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:46:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:46:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:46:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:46:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:46:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:46:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:46:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:46:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:46:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:46:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:47:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:47:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:47:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:47:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:47:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:47:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:47:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:47:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:47:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:47:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:47:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:47:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:47:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:47:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:47:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:47:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:47:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:47:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:47:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:47:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:47:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:47:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:47:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:47:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:47:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:47:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:47:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:48:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:48:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:48:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:48:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:48:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:48:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:48:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:48:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:48:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:48:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:48:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:48:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:48:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:48:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:48:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:48:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:48:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:48:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:48:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:48:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:48:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:48:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:48:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:48:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:48:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:48:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:48:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:48:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:49:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:49:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:49:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:49:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:49:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:49:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:49:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:49:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:49:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:49:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:49:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:49:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:49:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:49:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:49:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:49:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:49:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:49:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:49:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:49:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:49:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:49:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:49:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:49:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:49:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:49:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:49:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:50:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:50:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:50:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:50:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:50:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:50:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:50:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:50:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:50:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:50:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:50:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:50:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:50:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:50:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:50:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:50:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:50:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:50:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:50:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:50:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:50:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:50:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:50:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:50:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:50:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:50:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:50:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:51:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:51:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:51:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:51:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:51:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:51:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:51:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:51:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:51:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:51:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:51:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:51:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:51:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:51:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:51:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:51:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:51:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:51:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:51:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:51:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:51:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:51:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:51:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:51:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:51:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:51:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:51:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:52:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:52:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:52:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:52:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:52:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:52:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:52:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:52:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:52:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:52:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:52:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:52:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:52:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:52:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:52:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:52:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:52:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:52:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:52:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:52:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:52:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:52:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:52:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:52:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:52:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:52:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:52:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:52:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:53:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:53:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:53:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:53:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:53:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:53:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:53:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:53:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:53:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:53:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:53:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:53:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:53:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:53:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:53:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:53:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:53:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:53:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:53:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:53:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:53:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:53:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:53:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:53:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:53:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:53:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:53:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:54:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:54:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:54:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:54:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:54:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:54:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:54:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:54:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:54:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:54:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:54:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:54:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:54:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:54:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:54:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:54:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:54:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:54:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:54:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:54:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:54:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:54:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:54:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:54:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:54:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:54:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:54:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:55:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:55:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:55:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:55:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:55:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:55:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:55:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:55:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:55:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:55:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:55:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:55:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:55:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:55:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:55:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:55:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:55:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:55:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:55:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:55:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:55:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:55:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:55:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:55:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:55:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:55:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:55:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:55:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:56:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:56:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:56:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:56:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:56:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:56:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:56:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:56:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:56:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:56:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:56:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:56:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:56:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:56:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:56:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:56:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:56:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:56:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:56:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:56:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:56:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:56:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:56:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:56:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:56:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:56:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:56:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:57:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:57:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:57:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:57:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:57:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:57:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:57:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:57:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:57:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:57:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:57:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:57:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:57:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:57:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:57:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:57:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:57:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:57:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:57:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:57:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:57:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:57:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:57:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:57:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:57:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:57:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:57:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:58:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:58:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:58:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:58:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:58:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:58:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:58:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:58:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:58:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:58:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:58:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:58:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:58:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:58:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:58:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:58:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:58:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:58:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:58:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:58:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:58:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:58:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:58:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:58:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:58:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:58:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:58:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:58:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:59:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:59:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:59:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:59:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:59:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:59:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:59:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:59:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:59:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:59:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:59:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:59:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:59:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:59:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:59:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:59:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:59:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:59:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:59:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:59:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:59:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:59:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 10:59:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 10:59:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 10:59:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 10:59:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 10:59:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:00:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:00:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:00:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:00:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:00:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:00:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:00:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:00:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:00:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:00:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:00:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:00:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:00:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:00:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:00:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:00:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:00:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:00:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:00:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:00:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:00:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:00:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:00:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:00:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:00:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:00:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:00:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:01:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:01:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:01:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:01:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:01:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:01:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:01:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:01:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:01:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:01:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:01:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:01:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:01:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:01:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:01:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:01:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:01:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:01:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:01:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:01:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:01:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:01:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:01:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:01:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:01:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:01:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:01:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:01:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:02:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:02:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:02:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:02:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:02:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:02:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:02:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:02:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:02:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:02:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:02:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:02:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:02:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:02:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:02:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:02:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:02:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:02:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:02:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:02:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:02:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:02:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:02:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:02:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:02:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:02:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:02:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:03:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:03:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:03:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:03:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:03:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:03:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:03:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:03:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:03:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:03:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:03:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:03:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:03:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:03:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:03:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:03:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:03:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:03:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:03:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:03:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:03:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:03:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:03:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:03:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:03:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:03:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:03:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:04:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:04:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:04:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:04:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:04:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:04:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:04:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:04:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:04:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:04:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:04:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:04:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:04:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:04:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:04:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:04:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:04:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:04:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:04:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:04:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:04:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:04:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:04:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:04:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:04:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:04:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:04:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:04:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:05:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:05:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:05:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:05:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:05:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:05:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:05:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:05:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:05:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:05:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:05:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:05:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:05:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:05:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:05:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:05:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:05:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:05:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:05:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:05:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:05:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:05:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:05:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:05:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:05:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:05:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:05:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:06:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:06:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:06:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:06:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:06:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:06:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:06:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:06:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:06:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:06:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:06:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:06:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:06:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:06:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:06:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:06:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:06:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:06:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:06:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:06:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:06:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:06:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:06:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:06:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:06:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:06:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:06:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:07:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:07:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:07:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:07:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:07:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:07:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:07:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:07:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:07:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:07:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:07:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:07:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:07:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:07:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:07:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:07:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:07:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:07:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:07:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:07:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:07:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:07:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:07:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:07:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:07:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:07:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:07:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:08:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:08:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:08:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:08:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:08:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:08:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:08:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:08:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:08:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:08:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:08:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:08:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:08:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:08:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:08:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:08:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:08:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:08:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:08:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:08:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:08:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:08:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:08:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:08:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:08:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:08:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:08:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:08:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:09:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:09:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:09:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:09:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:09:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:09:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:09:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:09:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:09:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:09:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:09:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:09:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:09:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:09:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:09:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:09:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:09:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:09:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:09:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:09:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:09:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:09:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:09:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:09:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:09:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:09:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:09:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:10:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:10:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:10:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:10:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:10:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:10:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:10:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:10:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:10:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:10:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:10:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:10:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:10:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:10:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:10:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:10:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:10:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:10:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:10:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:10:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:10:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:10:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:10:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:10:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:10:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:10:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:10:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:11:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:11:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:11:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:11:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:11:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:11:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:11:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:11:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:11:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:11:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:11:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:11:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:11:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:11:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:11:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:11:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:11:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:11:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:11:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:11:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:11:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:11:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:11:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:11:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:11:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:11:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:11:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:11:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:12:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:12:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:12:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:12:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:12:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:12:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:12:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:12:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:12:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:12:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:12:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:12:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:12:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:12:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:12:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:12:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:12:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:12:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:12:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:12:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:12:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:12:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:12:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:12:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:12:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:12:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:12:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:13:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:13:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:13:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:13:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:13:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:13:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:13:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:13:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:13:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:13:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:13:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:13:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:13:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:13:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:13:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:13:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:13:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:13:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:13:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:13:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:13:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:13:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:13:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:13:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:13:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:13:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:13:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:14:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:14:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:14:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:14:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:14:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:14:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:14:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:14:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:14:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:14:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:14:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:14:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:14:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:14:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:14:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:14:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:14:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:14:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:14:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:14:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:14:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:14:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:14:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:14:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:14:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:14:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:14:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:14:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:15:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:15:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:15:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:15:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:15:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:15:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:15:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:15:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:15:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:15:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:15:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:15:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:15:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:15:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:15:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:15:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:15:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:15:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:15:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:15:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:15:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:15:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:15:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:15:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:15:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:15:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:15:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:16:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:16:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:16:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:16:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:16:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:16:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:16:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:16:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:16:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:16:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:16:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:16:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:16:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:16:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:16:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:16:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:16:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:16:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:16:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:16:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:16:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:16:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:16:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:16:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:16:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:16:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:16:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:17:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:17:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:17:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:17:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:17:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:17:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:17:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:17:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:17:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:17:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:17:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:17:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:17:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:17:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:17:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:17:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:17:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:17:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:17:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:17:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:17:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:17:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:17:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:17:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:17:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:17:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:17:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:17:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:18:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:18:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:18:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:18:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:18:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:18:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:18:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:18:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:18:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:18:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:18:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:18:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:18:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:18:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:18:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:18:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:18:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:18:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:18:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:18:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:18:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:18:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:18:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:18:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:18:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:18:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:18:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:19:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:19:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:19:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:19:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:19:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:19:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:19:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:19:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:19:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:19:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:19:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:19:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:19:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:19:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:19:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:19:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:19:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:19:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:19:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:19:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:19:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:19:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:19:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:19:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:19:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:19:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:19:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:20:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:20:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:20:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:20:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:20:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:20:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:20:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:20:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:20:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:20:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:20:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:20:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:20:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:20:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:20:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:20:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:20:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:20:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:20:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:20:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:20:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:20:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:20:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:20:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:20:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:20:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:20:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:20:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:21:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:21:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:21:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:21:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:21:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:21:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:21:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:21:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:21:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:21:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:21:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:21:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:21:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:21:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:21:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:21:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:21:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:21:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:21:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:21:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:21:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:21:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:21:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:21:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:21:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:21:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:21:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:22:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:22:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:22:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:22:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:22:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:22:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:22:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:22:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:22:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:22:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:22:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:22:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:22:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:22:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:22:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:22:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:22:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:22:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:22:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:22:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:22:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:22:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:22:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:22:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:22:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:22:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:22:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:23:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:23:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:23:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:23:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:23:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:23:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:23:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:23:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:23:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:23:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:23:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:23:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:23:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:23:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:23:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:23:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:23:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:23:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:23:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:23:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:23:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:23:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:23:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:23:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:23:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:23:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:23:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:24:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:24:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:24:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:24:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:24:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:24:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:24:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:24:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:24:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:24:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:24:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:24:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:24:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:24:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:24:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:24:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:24:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:24:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:24:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:24:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:24:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:24:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:24:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:24:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:24:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:24:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:24:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:24:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:25:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:25:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:25:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:25:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:25:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:25:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:25:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:25:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:25:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:25:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:25:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:25:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:25:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:25:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:25:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:25:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:25:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:25:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:25:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:25:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:25:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:25:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:25:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:25:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:25:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:25:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:25:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:26:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:26:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:26:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:26:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:26:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:26:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:26:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:26:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:26:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:26:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:26:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:26:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:26:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:26:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:26:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:26:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:26:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:26:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:26:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:26:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:26:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:26:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:26:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:26:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:26:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:26:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:26:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:27:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:27:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:27:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:27:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:27:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:27:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:27:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:27:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:27:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:27:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:27:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:27:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:27:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:27:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:27:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:27:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:27:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:27:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:27:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:27:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:27:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:27:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:27:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:27:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:27:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:27:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:27:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:27:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:28:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:28:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:28:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:28:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:28:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:28:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:28:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:28:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:28:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:28:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:28:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:28:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:28:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:28:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:28:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:28:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:28:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:28:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:28:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:28:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:28:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:28:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:28:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:28:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:28:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:28:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:28:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:29:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:29:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:29:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:29:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:29:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:29:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:29:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:29:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:29:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:29:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:29:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:29:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:29:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:29:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:29:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:29:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:29:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:29:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:29:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:29:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:29:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:29:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:29:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:29:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:29:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:29:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:29:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:30:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:30:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:30:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:30:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:30:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:30:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:30:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:30:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:30:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:30:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:30:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:30:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:30:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:30:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:30:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:30:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:30:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:30:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:30:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:30:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:30:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:30:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:30:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:30:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:30:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:30:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:30:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:30:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:31:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:31:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:31:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:31:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:31:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:31:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:31:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:31:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:31:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:31:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:31:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:31:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:31:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:31:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:31:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:31:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:31:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:31:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:31:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:31:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:31:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:31:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:31:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:31:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:31:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:31:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:31:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:32:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:32:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:32:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:32:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:32:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:32:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:32:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:32:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:32:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:32:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:32:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:32:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:32:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:32:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:32:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:32:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:32:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:32:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:32:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:32:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:32:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:32:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:32:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:32:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:32:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:32:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:32:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:33:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:33:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:33:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:33:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:33:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:33:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:33:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:33:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:33:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:33:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:33:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:33:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:33:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:33:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:33:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:33:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:33:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:33:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:33:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:33:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:33:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:33:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:33:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:33:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:33:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:33:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:33:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:33:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:34:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:34:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:34:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:34:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:34:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:34:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:34:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:34:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:34:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:34:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:34:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:34:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:34:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:34:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:34:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:34:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:34:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:34:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:34:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:34:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:34:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:34:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:34:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:34:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:34:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:34:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:34:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:35:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:35:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:35:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:35:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:35:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:35:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:35:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:35:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:35:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:35:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:35:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:35:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:35:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:35:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:35:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:35:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:35:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:35:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:35:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:35:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:35:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:35:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:35:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:35:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:35:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:35:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:35:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:36:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:36:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:36:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:36:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:36:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:36:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:36:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:36:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:36:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:36:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:36:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:36:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:36:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:36:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:36:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:36:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:36:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:36:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:36:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:36:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:36:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:36:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:36:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:36:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:36:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:36:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:36:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:36:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:37:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:37:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:37:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:37:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:37:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:37:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:37:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:37:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:37:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:37:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:37:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:37:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:37:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:37:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:37:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:37:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:37:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:37:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:37:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:37:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:37:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:37:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:37:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:37:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:37:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:37:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:37:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:38:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:38:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:38:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:38:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:38:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:38:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:38:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:38:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:38:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:38:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:38:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:38:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:38:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:38:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:38:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:38:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:38:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:38:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:38:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:38:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:38:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:38:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:38:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:38:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:38:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:38:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:38:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:39:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:39:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:39:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:39:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:39:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:39:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:39:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:39:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:39:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:39:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:39:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:39:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:39:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:39:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:39:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:39:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:39:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:39:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:39:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:39:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:39:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:39:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:39:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:39:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:39:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:39:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:39:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:39:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:40:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:40:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:40:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:40:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:40:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:40:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:40:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:40:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:40:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:40:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:40:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:40:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:40:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:40:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:40:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:40:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:40:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:40:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:40:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:40:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:40:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:40:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:40:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:40:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:40:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:40:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:40:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:41:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:41:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:41:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:41:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:41:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:41:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:41:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:41:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:41:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:41:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:41:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:41:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:41:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:41:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:41:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:41:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:41:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:41:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:41:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:41:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:41:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:41:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:41:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:41:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:41:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:41:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:41:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:42:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:42:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:42:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:42:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:42:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:42:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:42:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:42:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:42:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:42:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:42:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:42:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:42:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:42:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:42:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:42:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:42:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:42:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:42:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:42:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:42:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:42:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:42:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:42:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:42:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:42:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:42:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:43:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:43:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:43:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:43:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:43:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:43:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:43:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:43:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:43:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:43:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:43:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:43:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:43:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:43:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:43:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:43:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:43:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:43:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:43:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:43:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:43:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:43:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:43:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:43:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:43:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:43:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:43:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:43:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:44:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:44:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:44:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:44:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:44:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:44:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:44:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:44:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:44:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:44:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:44:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:44:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:44:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:44:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:44:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:44:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:44:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:44:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:44:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:44:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:44:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:44:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:44:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:44:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:44:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:44:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:44:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:45:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:45:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:45:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:45:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:45:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:45:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:45:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:45:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:45:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:45:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:45:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:45:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:45:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:45:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:45:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:45:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:45:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:45:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:45:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:45:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:45:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:45:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:45:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:45:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:45:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:45:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:45:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:46:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:46:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:46:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:46:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:46:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:46:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:46:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:46:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:46:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:46:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:46:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:46:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:46:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:46:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:46:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:46:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:46:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:46:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:46:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:46:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:46:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:46:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:46:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:46:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:46:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:46:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:46:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:46:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:47:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:47:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:47:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:47:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:47:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:47:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:47:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:47:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:47:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:47:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:47:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:47:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:47:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:47:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:47:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:47:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:47:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:47:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:47:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:47:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:47:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:47:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:47:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:47:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:47:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:47:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:47:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:48:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:48:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:48:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:48:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:48:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:48:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:48:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:48:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:48:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:48:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:48:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:48:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:48:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:48:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:48:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:48:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:48:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:48:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:48:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:48:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:48:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:48:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:48:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:48:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:48:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:48:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:48:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:49:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:49:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:49:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:49:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:49:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:49:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:49:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:49:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:49:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:49:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:49:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:49:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:49:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:49:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:49:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:49:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:49:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:49:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:49:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:49:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:49:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:49:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:49:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:49:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:49:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:49:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:49:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:49:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:50:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:50:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:50:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:50:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:50:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:50:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:50:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:50:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:50:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:50:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:50:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:50:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:50:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:50:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:50:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:50:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:50:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:50:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:50:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:50:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:50:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:50:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:50:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:50:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:50:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:50:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:50:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:51:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:51:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:51:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:51:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:51:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:51:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:51:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:51:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:51:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:51:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:51:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:51:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:51:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:51:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:51:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:51:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:51:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:51:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:51:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:51:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:51:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:51:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:51:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:51:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:51:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:51:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:51:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:52:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:52:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:52:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:52:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:52:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:52:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:52:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:52:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:52:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:52:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:52:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:52:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:52:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:52:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:52:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:52:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:52:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:52:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:52:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:52:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:52:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:52:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:52:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:52:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:52:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:52:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:52:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:52:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:53:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:53:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:53:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:53:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:53:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:53:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:53:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:53:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:53:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:53:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:53:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:53:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:53:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:53:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:53:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:53:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:53:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:53:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:53:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:53:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:53:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:53:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:53:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:53:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:53:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:53:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:53:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:54:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:54:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:54:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:54:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:54:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:54:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:54:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:54:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:54:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:54:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:54:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:54:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:54:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:54:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:54:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:54:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:54:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:54:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:54:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:54:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:54:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:54:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:54:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:54:47 rdma06 systemd[1]: Starting dnf makecache...
Apr 16 11:54:47 rdma06 dnf[1104149]: Updating Subscription Management repositories.
Apr 16 11:54:47 rdma06 dnf[1104149]: Unable to read consumer identity
Apr 16 11:54:47 rdma06 dnf[1104149]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 16 11:54:47 rdma06 dnf[1104149]: Metadata cache refreshed recently.
Apr 16 11:54:47 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 16 11:54:47 rdma06 systemd[1]: Finished dnf makecache.
Apr 16 11:54:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:54:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:54:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:54:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:55:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:55:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:55:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:55:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:55:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:55:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:55:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:55:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:55:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:55:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:55:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:55:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:55:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:55:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:55:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:55:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:55:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:55:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:55:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:55:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:55:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:55:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:55:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:55:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:55:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:55:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:55:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:55:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:56:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:56:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:56:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:56:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:56:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:56:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:56:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:56:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:56:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:56:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:56:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:56:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:56:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:56:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:56:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:56:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:56:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:56:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:56:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:56:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:56:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:56:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:56:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:56:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:56:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:56:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:56:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:57:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:57:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:57:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:57:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:57:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:57:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:57:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:57:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:57:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:57:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:57:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:57:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:57:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:57:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:57:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:57:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:57:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:57:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:57:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:57:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:57:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:57:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:57:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:57:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:57:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:57:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:57:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:58:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:58:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:58:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:58:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:58:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:58:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:58:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:58:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:58:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:58:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:58:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:58:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:58:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:58:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:58:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:58:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:58:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:58:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:58:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:58:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:58:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:58:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:58:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:58:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:58:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:58:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:58:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:59:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:59:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:59:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:59:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:59:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:59:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:59:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:59:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:59:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:59:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:59:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:59:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:59:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:59:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:59:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:59:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:59:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:59:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:59:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 11:59:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:59:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:59:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:59:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:59:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 11:59:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 11:59:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 11:59:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 11:59:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:00:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:00:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:00:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:00:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:00:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:00:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:00:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:00:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:00:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:00:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:00:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:00:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:00:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:00:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:00:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:00:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:00:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:00:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:00:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:00:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:00:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:00:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:00:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:00:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:00:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:00:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:00:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:01:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:01:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:01:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:01:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:01:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:01:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:01:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:01:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:01:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:01:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:01:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:01:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:01:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:01:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:01:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:01:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:01:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:01:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:01:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:01:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:01:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:01:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:01:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:01:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:01:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:01:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:01:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:02:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:02:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:02:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:02:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:02:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:02:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:02:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:02:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:02:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:02:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:02:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:02:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:02:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:02:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:02:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:02:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:02:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:02:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:02:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:02:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:02:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:02:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:02:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:02:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:02:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:02:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:02:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:02:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:03:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:03:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:03:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:03:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:03:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:03:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:03:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:03:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:03:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:03:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:03:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:03:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:03:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:03:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:03:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:03:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:03:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:03:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:03:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:03:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:03:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:03:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:03:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:03:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:03:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:03:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:03:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:04:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:04:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:04:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:04:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:04:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:04:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:04:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:04:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:04:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:04:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:04:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:04:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:04:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:04:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:04:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:04:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:04:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:04:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:04:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:04:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:04:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:04:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:04:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:04:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:04:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:04:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:04:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:05:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:05:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:05:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:05:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:05:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:05:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:05:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:05:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:05:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:05:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:05:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:05:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:05:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:05:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:05:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:05:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:05:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:05:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:05:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:05:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:05:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:05:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:05:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:05:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:05:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:05:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:05:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:05:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:06:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:06:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:06:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:06:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:06:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:06:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:06:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:06:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:06:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:06:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:06:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:06:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:06:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:06:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:06:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:06:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:06:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:06:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:06:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:06:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:06:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:06:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:06:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:06:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:06:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:06:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:06:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:07:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:07:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:07:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:07:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:07:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:07:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:07:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:07:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:07:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:07:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:07:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:07:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:07:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:07:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:07:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:07:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:07:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:07:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:07:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:07:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:07:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:07:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:07:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:07:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:07:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:07:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:07:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:08:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:08:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:08:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:08:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:08:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:08:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:08:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:08:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:08:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:08:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:08:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:08:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:08:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:08:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:08:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:08:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:08:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:08:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:08:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:08:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:08:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:08:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:08:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:08:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:08:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:08:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:08:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:08:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:09:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:09:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:09:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:09:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:09:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:09:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:09:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:09:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:09:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:09:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:09:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:09:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:09:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:09:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:09:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:09:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:09:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:09:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:09:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:09:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:09:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:09:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:09:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:09:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:09:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:09:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:09:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:10:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:10:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:10:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:10:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:10:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:10:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:10:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:10:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:10:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:10:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:10:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:10:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:10:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:10:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:10:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:10:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:10:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:10:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:10:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:10:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:10:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:10:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:10:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:10:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:10:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:10:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:10:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:11:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:11:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:11:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:11:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:11:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:11:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:11:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:11:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:11:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:11:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:11:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:11:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:11:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:11:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:11:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:11:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:11:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:11:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:11:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:11:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:11:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:11:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:11:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:11:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:11:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:11:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:11:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:11:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:12:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:12:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:12:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:12:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:12:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:12:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:12:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:12:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:12:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:12:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:12:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:12:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:12:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:12:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:12:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:12:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:12:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:12:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:12:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:12:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:12:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:12:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:12:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:12:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:12:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:12:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:12:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:13:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:13:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:13:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:13:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:13:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:13:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:13:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:13:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:13:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:13:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:13:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:13:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:13:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:13:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:13:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:13:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:13:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:13:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:13:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:13:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:13:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:13:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:13:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:13:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:13:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:13:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:13:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:14:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:14:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:14:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:14:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:14:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:14:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:14:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:14:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:14:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:14:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:14:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:14:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:14:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:14:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:14:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:14:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:14:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:14:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:14:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:14:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:14:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:14:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:14:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:14:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:14:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:14:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:14:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:15:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:15:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:15:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:15:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:15:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:15:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:15:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:15:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:15:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:15:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:15:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:15:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:15:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:15:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:15:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:15:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:15:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:15:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:15:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:15:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:15:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:15:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:15:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:15:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:15:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:15:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:15:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:15:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:16:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:16:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:16:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:16:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:16:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:16:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:16:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:16:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:16:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:16:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:16:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:16:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:16:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:16:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:16:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:16:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:16:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:16:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:16:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:16:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:16:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:16:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:16:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:16:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:16:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:16:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:16:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:17:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:17:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:17:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:17:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:17:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:17:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:17:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:17:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:17:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:17:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:17:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:17:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:17:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:17:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:17:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:17:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:17:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:17:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:17:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:17:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:17:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:17:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:17:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:17:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:17:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:17:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:17:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:18:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:18:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:18:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:18:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:18:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:18:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:18:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:18:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:18:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:18:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:18:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:18:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:18:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:18:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:18:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:18:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:18:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:18:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:18:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:18:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:18:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:18:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:18:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:18:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:18:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:18:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:18:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:18:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:19:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:19:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:19:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:19:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:19:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:19:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:19:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:19:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:19:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:19:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:19:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:19:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:19:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:19:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:19:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:19:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:19:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:19:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:19:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:19:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:19:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:19:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:19:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:19:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:19:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:19:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:19:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:20:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:20:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:20:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:20:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:20:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:20:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:20:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:20:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:20:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:20:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:20:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:20:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:20:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:20:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:20:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:20:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:20:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:20:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:20:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:20:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:20:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:20:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:20:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:20:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:20:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:20:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:20:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:21:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:21:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:21:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:21:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:21:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:21:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:21:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:21:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:21:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:21:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:21:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:21:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:21:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:21:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:21:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:21:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:21:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:21:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:21:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:21:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:21:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:21:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:21:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:21:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:21:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:21:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:21:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:21:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:22:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:22:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:22:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:22:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:22:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:22:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:22:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:22:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:22:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:22:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:22:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:22:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:22:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:22:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:22:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:22:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:22:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:22:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:22:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:22:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:22:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:22:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:22:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:22:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:22:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:22:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:22:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:23:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:23:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:23:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:23:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:23:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:23:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:23:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:23:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:23:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:23:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:23:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:23:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:23:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:23:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:23:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:23:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:23:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:23:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:23:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:23:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:23:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:23:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:23:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:23:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:23:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:23:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:23:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:24:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:24:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:24:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:24:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:24:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:24:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:24:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:24:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:24:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:24:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:24:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:24:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:24:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:24:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:24:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:24:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:24:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:24:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:24:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:24:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:24:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:24:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:24:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:24:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:24:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:24:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:24:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:24:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:25:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:25:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:25:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:25:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:25:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:25:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:25:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:25:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:25:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:25:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:25:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:25:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:25:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:25:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:25:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:25:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:25:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:25:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:25:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:25:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:25:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:25:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:25:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:25:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:25:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:25:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:25:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:26:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:26:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:26:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:26:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:26:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:26:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:26:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:26:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:26:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:26:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:26:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:26:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:26:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:26:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:26:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:26:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:26:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:26:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:26:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:26:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:26:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:26:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:26:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:26:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:26:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:26:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:26:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:27:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:27:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:27:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:27:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:27:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:27:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:27:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:27:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:27:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:27:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:27:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:27:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:27:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:27:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:27:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:27:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:27:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:27:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:27:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:27:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:27:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:27:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:27:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:27:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:27:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:27:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:27:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:27:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:28:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:28:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:28:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:28:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:28:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:28:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:28:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:28:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:28:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:28:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:28:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:28:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:28:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:28:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:28:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:28:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:28:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:28:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:28:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:28:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:28:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:28:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:28:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:28:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:28:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:28:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:28:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:29:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:29:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:29:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:29:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:29:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:29:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:29:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:29:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:29:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:29:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:29:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:29:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:29:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:29:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:29:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:29:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:29:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:29:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:29:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:29:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:29:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:29:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:29:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:29:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:29:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:29:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:29:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:30:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:30:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:30:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:30:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:30:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:30:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:30:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:30:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:30:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:30:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:30:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:30:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:30:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:30:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:30:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:30:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:30:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:30:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:30:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:30:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:30:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:30:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:30:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:30:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:30:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:30:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:30:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:31:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:31:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:31:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:31:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:31:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:31:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:31:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:31:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:31:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:31:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:31:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:31:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:31:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:31:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:31:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:31:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:31:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:31:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:31:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:31:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:31:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:31:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:31:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:31:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:31:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:31:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:31:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:31:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:32:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:32:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:32:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:32:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:32:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:32:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:32:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:32:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:32:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:32:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:32:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:32:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:32:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:32:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:32:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:32:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:32:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:32:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:32:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:32:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:32:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:32:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:32:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:32:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:32:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:32:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:32:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:33:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:33:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:33:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:33:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:33:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:33:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:33:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:33:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:33:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:33:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:33:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:33:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:33:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:33:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:33:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:33:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:33:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:33:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:33:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:33:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:33:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:33:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:33:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:33:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:33:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:33:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:33:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:34:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:34:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:34:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:34:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:34:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:34:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:34:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:34:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:34:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:34:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:34:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:34:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:34:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:34:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:34:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:34:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:34:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:34:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:34:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:34:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:34:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:34:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:34:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:34:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:34:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:34:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:34:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:34:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:35:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:35:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:35:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:35:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:35:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:35:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:35:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:35:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:35:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:35:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:35:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:35:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:35:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:35:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:35:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:35:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:35:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:35:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:35:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:35:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:35:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:35:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:35:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:35:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:35:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:35:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:35:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:36:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:36:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:36:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:36:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:36:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:36:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:36:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:36:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:36:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:36:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:36:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:36:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:36:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:36:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:36:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:36:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:36:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:36:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:36:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:36:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:36:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:36:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:36:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:36:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:36:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:36:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:36:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:37:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:37:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:37:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:37:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:37:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:37:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:37:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:37:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:37:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:37:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:37:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:37:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:37:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:37:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:37:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:37:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:37:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:37:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:37:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:37:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:37:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:37:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:37:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:37:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:37:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:37:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:37:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:37:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:38:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:38:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:38:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:38:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:38:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:38:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:38:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:38:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:38:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:38:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:38:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:38:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:38:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:38:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:38:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:38:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:38:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:38:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:38:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:38:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:38:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:38:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:38:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:38:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:38:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:38:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:38:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:39:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:39:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:39:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:39:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:39:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:39:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:39:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:39:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:39:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:39:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:39:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:39:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:39:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:39:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:39:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:39:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:39:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:39:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:39:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:39:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:39:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:39:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:39:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:39:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:39:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:39:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:39:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:40:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:40:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:40:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:40:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:40:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:40:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:40:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:40:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:40:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:40:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:40:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:40:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:40:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:40:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:40:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:40:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:40:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:40:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:40:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:40:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:40:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:40:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:40:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:40:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:40:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:40:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:40:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:40:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:41:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:41:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:41:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:41:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:41:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:41:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:41:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:41:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:41:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:41:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:41:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:41:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:41:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:41:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:41:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:41:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:41:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:41:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:41:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:41:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:41:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:41:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:41:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:41:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:41:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:41:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:41:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:42:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:42:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:42:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:42:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:42:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:42:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:42:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:42:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:42:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:42:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:42:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:42:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:42:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:42:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:42:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:42:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:42:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:42:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:42:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:42:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:42:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:42:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:42:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:42:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:42:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:42:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:42:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:43:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:43:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:43:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:43:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:43:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:43:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:43:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:43:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:43:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:43:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:43:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:43:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:43:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:43:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:43:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:43:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:43:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:43:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:43:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:43:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:43:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:43:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:43:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:43:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:43:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:43:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:43:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:43:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:44:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:44:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:44:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:44:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:44:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:44:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:44:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:44:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:44:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:44:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:44:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:44:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:44:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:44:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:44:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:44:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:44:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:44:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:44:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:44:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:44:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:44:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:44:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:44:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:44:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:44:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:44:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:45:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:45:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:45:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:45:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:45:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:45:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:45:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:45:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:45:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:45:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:45:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:45:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:45:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:45:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:45:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:45:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:45:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:45:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:45:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:45:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:45:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:45:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:45:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:45:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:45:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:45:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:45:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:46:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:46:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:46:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:46:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:46:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:46:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:46:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:46:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:46:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:46:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:46:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:46:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:46:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:46:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:46:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:46:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:46:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:46:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:46:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:46:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:46:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:46:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:46:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:46:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:46:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:46:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:46:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:46:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:47:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:47:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:47:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:47:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:47:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:47:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:47:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:47:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:47:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:47:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:47:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:47:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:47:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:47:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:47:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:47:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:47:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:47:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:47:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:47:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:47:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:47:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:47:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:47:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:47:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:47:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:47:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:48:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:48:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:48:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:48:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:48:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:48:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:48:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:48:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:48:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:48:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:48:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:48:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:48:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:48:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:48:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:48:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:48:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:48:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:48:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:48:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:48:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:48:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:48:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:48:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:48:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:48:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:48:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:49:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:49:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:49:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:49:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:49:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:49:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:49:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:49:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:49:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:49:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:49:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:49:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:49:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:49:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:49:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:49:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:49:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:49:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:49:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:49:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:49:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:49:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:49:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:49:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:49:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:49:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:49:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:50:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:50:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:50:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:50:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:50:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:50:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:50:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:50:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:50:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:50:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:50:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:50:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:50:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:50:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:50:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:50:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:50:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:50:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:50:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:50:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:50:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:50:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:50:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:50:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:50:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:50:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:50:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:50:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:51:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:51:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:51:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:51:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:51:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:51:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:51:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:51:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:51:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:51:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:51:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:51:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:51:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:51:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:51:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:51:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:51:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:51:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:51:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:51:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:51:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:51:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:51:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:51:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:51:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:51:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:51:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:52:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:52:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:52:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:52:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:52:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:52:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:52:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:52:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:52:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:52:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:52:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:52:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:52:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:52:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:52:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:52:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:52:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:52:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:52:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:52:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:52:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:52:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:52:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:52:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:52:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:52:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:52:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:53:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:53:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:53:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:53:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:53:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:53:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:53:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:53:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:53:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:53:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:53:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:53:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:53:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:53:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:53:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:53:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:53:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:53:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:53:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:53:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:53:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:53:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:53:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:53:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:53:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:53:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:53:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:53:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:54:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:54:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:54:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:54:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:54:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:54:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:54:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:54:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:54:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:54:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:54:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:54:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:54:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:54:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:54:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:54:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:54:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:54:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:54:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:54:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:54:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:54:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:54:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:54:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:54:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:54:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:54:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:55:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:55:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:55:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:55:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:55:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:55:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:55:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:55:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:55:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:55:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:55:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:55:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:55:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:55:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:55:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:55:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:55:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:55:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:55:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:55:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:55:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:55:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:55:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:55:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:55:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:55:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:55:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:56:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:56:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:56:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:56:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:56:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:56:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:56:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:56:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:56:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:56:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:56:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:56:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:56:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:56:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:56:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:56:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:56:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:56:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:56:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:56:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:56:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:56:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:56:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:56:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:56:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:56:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:56:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:56:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:57:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:57:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:57:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:57:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:57:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:57:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:57:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:57:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:57:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:57:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:57:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:57:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:57:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:57:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:57:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:57:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:57:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:57:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:57:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:57:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:57:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:57:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:57:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:57:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:57:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:57:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:57:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:58:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:58:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:58:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:58:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:58:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:58:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:58:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:58:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:58:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:58:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:58:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:58:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:58:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:58:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:58:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:58:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:58:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:58:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:58:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:58:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:58:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:58:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:58:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:58:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:58:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:58:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:58:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:59:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:59:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:59:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:59:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:59:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:59:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:59:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:59:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:59:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:59:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:59:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:59:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:59:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:59:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:59:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:59:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:59:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:59:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:59:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 12:59:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:59:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:59:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:59:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:59:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 12:59:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 12:59:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 12:59:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 12:59:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:00:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:00:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:00:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:00:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:00:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:00:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:00:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:00:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:00:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:00:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:00:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:00:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:00:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:00:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:00:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:00:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:00:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:00:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:00:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:00:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:00:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:00:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:00:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:00:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:00:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:00:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:00:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:01:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:01:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:01:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:01:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:01:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:01:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:01:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:01:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:01:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:01:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:01:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:01:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:01:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:01:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:01:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:01:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:01:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:01:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:01:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:01:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:01:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:01:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:01:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:01:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:01:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:01:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:01:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:02:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:02:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:02:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:02:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:02:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:02:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:02:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:02:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:02:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:02:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:02:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:02:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:02:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:02:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:02:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:02:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:02:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:02:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:02:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:02:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:02:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:02:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:02:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:02:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:02:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:02:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:02:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:02:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:03:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:03:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:03:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:03:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:03:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:03:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:03:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:03:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:03:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:03:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:03:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:03:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:03:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:03:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:03:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:03:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:03:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:03:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:03:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:03:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:03:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:03:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:03:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:03:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:03:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:03:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:03:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:04:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:04:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:04:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:04:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:04:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:04:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:04:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:04:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:04:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:04:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:04:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:04:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:04:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:04:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:04:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:04:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:04:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:04:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:04:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:04:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:04:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:04:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:04:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:04:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:04:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:04:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:04:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:05:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:05:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:05:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:05:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:05:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:05:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:05:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:05:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:05:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:05:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:05:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:05:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:05:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:05:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:05:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:05:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:05:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:05:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:05:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:05:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:05:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:05:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:05:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:05:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:05:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:05:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:05:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:06:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:06:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:06:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:06:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:06:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:06:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:06:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:06:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:06:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:06:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:06:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:06:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:06:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:06:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:06:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:06:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:06:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:06:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:06:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:06:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:06:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:06:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:06:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:06:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:06:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:06:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:06:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:06:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:07:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:07:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:07:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:07:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:07:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:07:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:07:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:07:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:07:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:07:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:07:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:07:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:07:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:07:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:07:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:07:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:07:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:07:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:07:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:07:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:07:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:07:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:07:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:07:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:07:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:07:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:07:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:08:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:08:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:08:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:08:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:08:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:08:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:08:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:08:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:08:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:08:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:08:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:08:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:08:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:08:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:08:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:08:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:08:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:08:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:08:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:08:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:08:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:08:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:08:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:08:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:08:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:08:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:08:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:09:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:09:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:09:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:09:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:09:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:09:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:09:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:09:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:09:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:09:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:09:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:09:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:09:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:09:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:09:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:09:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:09:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:09:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:09:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:09:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:09:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:09:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:09:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:09:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:09:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:09:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:09:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:09:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:10:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:10:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:10:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:10:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:10:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:10:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:10:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:10:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:10:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:10:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:10:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:10:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:10:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:10:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:10:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:10:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:10:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:10:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:10:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:10:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:10:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:10:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:10:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:10:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:10:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:10:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:10:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:11:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:11:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:11:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:11:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:11:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:11:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:11:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:11:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:11:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:11:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:11:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:11:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:11:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:11:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:11:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:11:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:11:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:11:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:11:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:11:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:11:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:11:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:11:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:11:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:11:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:11:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:11:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:12:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:12:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:12:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:12:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:12:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:12:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:12:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:12:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:12:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:12:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:12:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:12:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:12:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:12:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:12:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:12:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:12:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:12:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:12:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:12:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:12:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:12:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:12:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:12:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:12:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:12:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:12:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:12:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:13:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:13:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:13:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:13:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:13:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:13:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:13:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:13:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:13:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:13:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:13:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:13:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:13:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:13:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:13:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:13:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:13:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:13:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:13:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:13:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:13:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:13:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:13:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:13:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:13:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:13:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:13:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:14:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:14:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:14:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:14:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:14:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:14:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:14:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:14:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:14:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:14:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:14:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:14:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:14:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:14:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:14:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:14:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:14:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:14:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:14:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:14:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:14:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:14:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:14:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:14:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:14:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:14:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:14:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:15:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:15:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:15:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:15:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:15:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:15:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:15:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:15:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:15:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:15:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:15:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:15:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:15:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:15:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:15:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:15:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:15:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:15:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:15:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:15:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:15:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:15:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:15:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:15:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:15:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:15:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:15:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:15:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:16:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:16:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:16:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:16:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:16:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:16:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:16:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:16:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:16:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:16:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:16:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:16:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:16:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:16:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:16:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:16:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:16:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:16:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:16:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:16:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:16:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:16:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:16:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:16:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:16:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:16:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:16:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:17:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:17:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:17:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:17:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:17:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:17:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:17:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:17:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:17:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:17:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:17:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:17:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:17:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:17:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:17:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:17:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:17:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:17:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:17:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:17:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:17:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:17:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:17:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:17:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:17:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:17:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:17:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:18:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:18:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:18:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:18:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:18:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:18:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:18:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:18:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:18:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:18:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:18:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:18:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:18:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:18:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:18:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:18:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:18:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:18:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:18:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:18:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:18:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:18:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:18:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:18:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:18:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:18:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:18:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:18:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:19:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:19:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:19:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:19:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:19:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:19:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:19:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:19:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:19:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:19:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:19:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:19:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:19:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:19:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:19:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:19:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:19:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:19:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:19:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:19:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:19:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:19:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:19:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:19:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:19:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:19:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:19:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:20:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:20:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:20:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:20:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:20:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:20:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:20:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:20:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:20:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:20:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:20:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:20:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:20:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:20:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:20:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:20:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:20:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:20:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:20:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:20:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:20:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:20:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:20:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:20:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:20:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:20:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:20:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:21:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:21:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:21:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:21:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:21:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:21:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:21:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:21:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:21:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:21:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:21:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:21:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:21:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:21:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:21:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:21:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:21:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:21:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:21:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:21:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:21:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:21:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:21:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:21:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:21:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:21:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:21:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:22:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:22:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:22:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:22:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:22:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:22:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:22:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:22:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:22:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:22:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:22:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:22:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:22:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:22:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:22:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:22:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:22:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:22:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:22:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:22:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:22:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:22:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:22:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:22:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:22:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:22:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:22:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:22:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:23:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:23:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:23:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:23:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:23:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:23:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:23:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:23:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:23:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:23:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:23:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:23:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:23:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:23:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:23:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:23:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:23:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:23:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:23:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:23:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:23:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:23:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:23:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:23:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:23:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:23:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:23:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:24:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:24:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:24:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:24:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:24:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:24:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:24:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:24:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:24:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:24:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:24:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:24:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:24:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:24:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:24:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:24:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:24:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:24:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:24:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:24:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:24:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:24:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:24:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:24:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:24:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:24:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:24:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:25:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:25:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:25:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:25:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:25:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:25:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:25:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:25:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:25:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:25:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:25:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:25:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:25:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:25:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:25:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:25:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:25:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:25:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:25:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:25:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:25:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:25:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:25:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:25:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:25:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:25:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:25:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:25:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:26:00 rdma06 systemd[1]: Starting dnf makecache...
Apr 16 13:26:01 rdma06 dnf[1107858]: Updating Subscription Management repositories.
Apr 16 13:26:01 rdma06 dnf[1107858]: Unable to read consumer identity
Apr 16 13:26:01 rdma06 dnf[1107858]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 16 13:26:01 rdma06 dnf[1107858]: Metadata cache refreshed recently.
Apr 16 13:26:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 16 13:26:01 rdma06 systemd[1]: Finished dnf makecache.
Apr 16 13:26:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:26:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:26:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:26:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:26:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:26:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:26:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:26:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:26:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:26:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:26:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:26:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:26:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:26:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:26:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:26:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:26:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:26:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:26:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:26:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:26:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:26:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:26:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:26:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:26:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:26:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:26:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:27:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:27:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:27:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:27:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:27:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:27:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:27:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:27:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:27:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:27:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:27:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:27:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:27:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:27:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:27:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:27:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:27:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:27:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:27:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:27:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:27:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:27:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:27:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:27:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:27:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:27:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:27:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:28:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:28:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:28:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:28:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:28:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:28:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:28:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:28:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:28:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:28:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:28:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:28:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:28:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:28:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:28:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:28:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:28:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:28:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:28:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:28:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:28:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:28:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:28:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:28:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:28:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:28:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:28:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:28:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:29:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:29:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:29:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:29:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:29:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:29:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:29:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:29:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:29:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:29:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:29:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:29:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:29:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:29:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:29:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:29:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:29:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:29:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:29:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:29:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:29:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:29:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:29:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:29:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:29:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:29:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:29:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:30:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:30:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:30:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:30:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:30:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:30:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:30:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:30:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:30:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:30:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:30:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:30:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:30:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:30:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:30:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:30:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:30:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:30:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:30:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:30:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:30:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:30:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:30:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:30:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:30:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:30:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:30:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:31:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:31:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:31:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:31:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:31:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:31:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:31:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:31:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:31:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:31:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:31:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:31:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:31:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:31:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:31:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:31:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:31:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:31:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:31:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:31:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:31:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:31:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:31:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:31:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:31:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:31:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:31:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:31:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:32:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:32:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:32:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:32:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:32:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:32:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:32:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:32:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:32:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:32:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:32:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:32:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:32:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:32:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:32:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:32:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:32:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:32:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:32:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:32:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:32:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:32:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:32:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:32:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:32:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:32:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:32:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:33:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:33:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:33:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:33:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:33:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:33:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:33:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:33:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:33:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:33:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:33:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:33:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:33:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:33:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:33:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:33:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:33:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:33:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:33:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:33:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:33:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:33:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:33:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:33:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:33:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:33:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:33:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:34:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:34:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:34:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:34:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:34:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:34:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:34:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:34:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:34:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:34:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:34:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:34:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:34:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:34:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:34:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:34:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:34:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:34:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:34:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:34:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:34:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:34:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:34:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:34:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:34:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:34:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:34:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:34:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:35:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:35:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:35:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:35:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:35:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:35:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:35:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:35:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:35:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:35:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:35:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:35:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:35:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:35:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:35:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:35:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:35:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:35:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:35:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:35:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:35:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:35:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:35:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:35:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:35:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:35:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:35:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:36:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:36:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:36:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:36:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:36:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:36:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:36:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:36:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:36:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:36:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:36:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:36:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:36:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:36:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:36:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:36:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:36:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:36:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:36:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:36:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:36:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:36:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:36:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:36:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:36:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:36:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:36:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:37:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:37:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:37:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:37:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:37:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:37:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:37:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:37:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:37:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:37:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:37:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:37:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:37:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:37:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:37:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:37:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:37:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:37:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:37:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:37:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:37:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:37:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:37:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:37:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:37:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:37:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:37:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:38:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:38:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:38:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:38:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:38:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:38:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:38:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:38:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:38:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:38:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:38:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:38:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:38:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:38:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:38:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:38:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:38:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:38:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:38:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:38:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:38:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:38:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:38:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:38:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:38:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:38:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:38:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:38:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:39:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:39:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:39:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:39:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:39:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:39:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:39:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:39:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:39:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:39:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:39:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:39:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:39:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:39:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:39:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:39:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:39:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:39:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:39:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:39:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:39:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:39:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:39:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:39:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:39:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:39:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:39:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:40:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:40:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:40:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:40:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:40:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:40:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:40:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:40:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:40:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:40:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:40:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:40:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:40:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:40:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:40:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:40:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:40:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:40:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:40:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:40:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:40:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:40:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:40:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:40:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:40:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:40:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:40:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:41:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:41:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:41:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:41:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:41:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:41:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:41:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:41:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:41:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:41:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:41:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:41:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:41:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:41:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:41:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:41:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:41:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:41:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:41:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:41:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:41:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:41:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:41:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:41:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:41:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:41:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:41:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:41:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:42:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:42:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:42:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:42:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:42:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:42:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:42:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:42:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:42:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:42:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:42:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:42:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:42:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:42:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:42:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:42:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:42:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:42:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:42:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:42:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:42:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:42:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:42:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:42:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:42:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:42:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:42:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:43:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:43:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:43:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:43:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:43:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:43:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:43:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:43:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:43:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:43:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:43:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:43:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:43:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:43:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:43:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:43:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:43:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:43:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:43:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:43:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:43:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:43:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:43:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:43:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:43:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:43:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:43:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:44:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:44:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:44:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:44:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:44:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:44:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:44:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:44:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:44:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:44:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:44:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:44:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:44:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:44:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:44:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:44:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:44:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:44:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:44:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:44:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:44:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:44:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:44:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:44:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:44:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:44:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:44:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:44:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:45:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:45:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:45:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:45:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:45:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:45:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:45:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:45:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:45:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:45:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:45:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:45:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:45:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:45:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:45:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:45:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:45:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:45:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:45:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:45:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:45:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:45:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:45:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:45:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:45:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:45:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:45:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:46:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:46:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:46:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:46:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:46:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:46:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:46:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:46:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:46:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:46:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:46:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:46:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:46:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:46:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:46:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:46:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:46:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:46:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:46:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:46:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:46:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:46:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:46:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:46:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:46:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:46:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:46:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:47:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:47:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:47:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:47:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:47:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:47:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:47:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:47:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:47:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:47:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:47:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:47:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:47:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:47:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:47:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:47:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:47:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:47:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:47:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:47:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:47:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:47:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:47:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:47:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:47:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:47:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:47:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:47:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:48:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:48:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:48:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:48:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:48:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:48:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:48:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:48:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:48:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:48:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:48:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:48:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:48:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:48:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:48:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:48:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:48:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:48:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:48:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:48:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:48:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:48:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:48:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:48:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:48:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:48:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:48:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:49:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:49:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:49:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:49:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:49:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:49:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:49:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:49:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:49:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:49:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:49:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:49:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:49:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:49:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:49:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:49:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:49:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:49:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:49:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:49:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:49:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:49:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:49:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:49:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:49:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:49:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:49:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:50:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:50:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:50:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:50:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:50:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:50:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:50:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:50:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:50:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:50:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:50:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:50:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:50:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:50:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:50:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:50:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:50:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:50:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:50:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:50:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:50:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:50:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:50:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:50:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:50:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:50:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:50:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:50:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:51:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:51:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:51:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:51:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:51:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:51:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:51:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:51:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:51:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:51:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:51:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:51:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:51:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:51:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:51:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:51:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:51:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:51:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:51:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:51:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:51:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:51:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:51:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:51:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:51:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:51:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:51:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:52:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:52:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:52:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:52:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:52:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:52:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:52:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:52:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:52:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:52:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:52:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:52:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:52:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:52:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:52:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:52:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:52:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:52:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:52:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:52:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:52:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:52:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:52:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:52:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:52:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:52:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:52:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:53:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:53:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:53:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:53:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:53:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:53:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:53:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:53:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:53:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:53:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:53:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:53:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:53:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:53:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:53:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:53:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:53:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:53:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:53:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:53:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:53:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:53:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:53:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:53:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:53:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:53:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:53:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:53:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:54:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:54:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:54:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:54:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:54:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:54:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:54:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:54:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:54:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:54:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:54:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:54:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:54:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:54:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:54:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:54:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:54:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:54:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:54:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:54:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:54:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:54:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:54:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:54:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:54:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:54:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:54:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:55:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:55:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:55:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:55:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:55:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:55:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:55:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:55:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:55:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:55:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:55:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:55:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:55:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:55:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:55:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:55:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:55:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:55:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:55:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:55:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:55:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:55:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:55:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:55:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:55:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:55:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:55:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:56:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:56:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:56:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:56:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:56:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:56:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:56:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:56:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:56:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:56:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:56:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:56:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:56:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:56:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:56:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:56:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:56:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:56:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:56:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:56:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:56:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:56:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:56:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:56:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:56:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:56:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:56:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:57:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:57:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:57:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:57:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:57:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:57:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:57:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:57:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:57:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:57:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:57:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:57:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:57:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:57:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:57:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:57:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:57:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:57:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:57:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:57:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:57:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:57:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:57:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:57:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:57:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:57:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:57:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:57:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:58:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:58:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:58:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:58:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:58:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:58:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:58:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:58:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:58:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:58:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:58:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:58:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:58:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:58:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:58:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:58:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:58:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:58:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:58:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:58:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:58:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:58:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:58:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:58:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:58:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:58:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:58:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:59:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:59:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:59:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:59:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:59:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:59:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:59:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:59:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:59:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:59:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:59:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:59:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:59:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:59:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:59:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:59:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:59:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:59:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:59:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:59:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:59:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:59:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 13:59:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 13:59:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 13:59:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 13:59:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 13:59:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:00:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:00:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:00:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:00:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:00:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:00:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:00:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:00:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:00:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:00:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:00:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:00:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:00:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:00:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:00:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:00:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:00:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:00:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:00:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:00:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:00:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:00:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:00:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:00:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:00:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:00:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:00:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:00:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:01:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:01:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:01:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:01:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:01:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:01:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:01:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:01:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:01:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:01:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:01:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:01:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:01:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:01:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:01:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:01:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:01:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:01:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:01:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:01:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:01:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:01:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:01:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:01:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:01:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:01:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:01:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:02:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:02:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:02:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:02:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:02:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:02:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:02:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:02:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:02:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:02:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:02:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:02:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:02:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:02:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:02:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:02:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:02:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:02:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:02:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:02:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:02:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:02:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:02:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:02:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:02:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:02:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:02:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:03:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:03:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:03:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:03:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:03:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:03:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:03:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:03:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:03:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:03:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:03:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:03:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:03:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:03:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:03:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:03:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:03:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:03:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:03:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:03:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:03:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:03:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:03:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:03:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:03:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:03:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:03:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:03:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:04:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:04:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:04:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:04:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:04:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:04:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:04:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:04:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:04:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:04:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:04:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:04:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:04:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:04:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:04:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:04:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:04:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:04:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:04:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:04:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:04:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:04:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:04:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:04:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:04:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:04:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:04:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:05:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:05:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:05:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:05:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:05:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:05:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:05:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:05:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:05:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:05:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:05:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:05:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:05:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:05:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:05:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:05:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:05:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:05:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:05:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:05:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:05:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:05:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:05:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:05:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:05:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:05:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:05:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:06:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:06:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:06:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:06:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:06:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:06:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:06:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:06:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:06:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:06:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:06:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:06:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:06:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:06:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:06:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:06:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:06:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:06:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:06:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:06:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:06:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:06:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:06:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:06:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:06:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:06:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:06:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:06:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:07:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:07:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:07:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:07:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:07:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:07:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:07:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:07:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:07:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:07:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:07:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:07:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:07:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:07:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:07:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:07:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:07:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:07:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:07:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:07:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:07:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:07:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:07:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:07:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:07:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:07:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:07:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:08:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:08:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:08:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:08:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:08:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:08:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:08:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:08:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:08:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:08:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:08:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:08:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:08:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:08:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:08:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:08:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:08:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:08:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:08:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:08:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:08:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:08:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:08:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:08:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:08:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:08:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:08:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:09:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:09:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:09:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:09:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:09:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:09:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:09:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:09:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:09:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:09:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:09:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:09:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:09:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:09:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:09:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:09:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:09:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:09:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:09:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:09:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:09:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:09:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:09:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:09:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:09:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:09:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:09:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:09:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:10:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:10:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:10:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:10:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:10:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:10:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:10:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:10:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:10:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:10:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:10:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:10:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:10:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:10:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:10:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:10:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:10:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:10:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:10:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:10:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:10:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:10:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:10:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:10:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:10:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:10:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:10:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:11:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:11:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:11:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:11:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:11:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:11:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:11:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:11:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:11:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:11:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:11:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:11:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:11:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:11:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:11:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:11:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:11:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:11:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:11:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:11:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:11:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:11:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:11:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:11:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:11:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:11:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:11:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:12:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:12:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:12:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:12:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:12:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:12:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:12:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:12:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:12:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:12:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:12:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:12:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:12:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:12:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:12:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:12:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:12:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:12:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:12:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:12:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:12:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:12:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:12:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:12:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:12:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:12:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:12:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:13:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:13:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:13:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:13:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:13:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:13:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:13:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:13:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:13:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:13:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:13:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:13:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:13:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:13:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:13:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:13:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:13:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:13:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:13:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:13:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:13:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:13:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:13:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:13:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:13:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:13:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:13:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:13:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:14:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:14:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:14:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:14:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:14:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:14:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:14:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:14:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:14:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:14:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:14:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:14:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:14:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:14:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:14:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:14:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:14:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:14:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:14:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:14:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:14:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:14:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:14:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:14:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:14:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:14:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:14:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:15:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:15:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:15:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:15:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:15:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:15:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:15:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:15:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:15:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:15:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:15:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:15:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:15:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:15:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:15:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:15:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:15:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:15:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:15:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:15:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:15:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:15:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:15:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:15:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:15:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:15:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:15:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:16:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:16:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:16:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:16:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:16:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:16:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:16:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:16:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:16:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:16:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:16:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:16:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:16:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:16:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:16:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:16:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:16:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:16:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:16:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:16:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:16:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:16:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:16:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:16:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:16:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:16:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:16:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:16:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:17:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:17:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:17:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:17:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:17:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:17:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:17:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:17:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:17:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:17:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:17:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:17:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:17:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:17:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:17:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:17:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:17:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:17:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:17:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:17:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:17:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:17:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:17:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:17:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:17:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:17:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:17:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:18:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:18:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:18:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:18:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:18:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:18:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:18:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:18:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:18:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:18:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:18:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:18:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:18:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:18:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:18:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:18:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:18:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:18:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:18:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:18:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:18:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:18:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:18:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:18:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:18:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:18:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:18:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:19:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:19:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:19:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:19:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:19:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:19:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:19:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:19:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:19:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:19:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:19:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:19:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:19:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:19:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:19:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:19:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:19:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:19:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:19:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:19:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:19:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:19:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:19:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:19:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:19:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:19:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:19:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:19:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:20:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:20:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:20:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:20:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:20:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:20:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:20:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:20:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:20:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:20:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:20:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:20:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:20:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:20:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:20:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:20:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:20:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:20:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:20:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:20:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:20:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:20:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:20:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:20:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:20:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:20:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:20:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:21:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:21:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:21:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:21:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:21:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:21:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:21:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:21:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:21:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:21:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:21:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:21:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:21:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:21:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:21:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:21:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:21:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:21:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:21:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:21:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:21:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:21:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:21:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:21:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:21:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:21:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:21:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:22:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:22:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:22:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:22:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:22:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:22:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:22:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:22:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:22:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:22:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:22:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:22:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:22:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:22:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:22:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:22:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:22:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:22:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:22:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:22:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:22:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:22:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:22:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:22:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:22:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:22:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:22:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:22:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:23:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:23:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:23:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:23:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:23:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:23:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:23:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:23:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:23:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:23:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:23:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:23:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:23:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:23:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:23:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:23:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:23:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:23:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:23:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:23:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:23:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:23:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:23:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:23:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:23:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:23:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:23:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:24:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:24:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:24:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:24:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:24:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:24:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:24:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:24:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:24:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:24:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:24:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:24:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:24:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:24:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:24:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:24:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:24:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:24:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:24:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:24:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:24:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:24:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:24:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:24:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:24:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:24:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:24:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:25:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:25:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:25:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:25:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:25:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:25:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:25:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:25:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:25:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:25:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:25:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:25:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:25:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:25:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:25:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:25:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:25:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:25:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:25:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:25:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:25:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:25:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:25:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:25:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:25:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:25:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:25:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:25:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:26:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:26:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:26:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:26:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:26:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:26:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:26:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:26:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:26:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:26:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:26:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:26:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:26:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:26:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:26:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:26:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:26:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:26:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:26:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:26:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:26:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:26:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:26:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:26:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:26:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:26:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:26:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:27:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:27:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:27:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:27:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:27:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:27:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:27:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:27:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:27:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:27:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:27:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:27:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:27:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:27:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:27:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:27:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:27:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:27:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:27:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:27:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:27:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:27:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:27:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:27:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:27:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:27:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:27:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:28:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:28:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:28:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:28:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:28:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:28:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:28:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:28:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:28:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:28:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:28:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:28:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:28:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:28:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:28:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:28:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:28:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:28:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:28:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:28:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:28:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:28:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:28:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:28:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:28:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:28:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:28:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:29:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:29:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:29:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:29:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:29:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:29:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:29:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:29:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:29:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:29:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:29:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:29:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:29:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:29:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:29:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:29:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:29:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:29:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:29:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:29:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:29:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:29:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:29:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:29:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:29:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:29:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:29:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:29:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:30:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:30:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:30:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:30:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:30:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:30:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:30:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:30:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:30:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:30:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:30:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:30:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:30:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:30:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:30:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:30:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:30:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:30:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:30:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:30:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:30:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:30:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:30:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:30:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:30:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:30:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:30:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:31:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:31:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:31:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:31:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:31:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:31:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:31:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:31:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:31:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:31:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:31:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:31:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:31:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:31:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:31:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:31:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:31:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:31:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:31:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:31:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:31:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:31:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:31:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:31:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:31:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:31:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:31:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:32:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:32:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:32:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:32:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:32:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:32:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:32:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:32:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:32:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:32:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:32:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:32:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:32:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:32:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:32:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:32:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:32:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:32:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:32:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:32:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:32:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:32:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:32:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:32:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:32:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:32:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:32:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:32:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:33:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:33:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:33:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:33:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:33:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:33:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:33:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:33:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:33:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:33:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:33:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:33:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:33:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:33:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:33:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:33:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:33:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:33:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:33:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:33:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:33:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:33:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:33:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:33:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:33:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:33:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:33:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:34:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:34:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:34:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:34:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:34:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:34:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:34:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:34:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:34:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:34:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:34:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:34:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:34:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:34:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:34:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:34:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:34:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:34:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:34:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:34:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:34:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:34:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:34:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:34:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:34:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:34:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:34:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:35:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:35:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:35:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:35:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:35:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:35:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:35:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:35:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:35:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:35:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:35:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:35:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:35:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:35:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:35:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:35:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:35:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:35:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:35:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:35:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:35:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:35:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:35:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:35:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:35:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:35:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:35:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:35:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:36:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:36:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:36:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:36:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:36:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:36:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:36:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:36:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:36:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:36:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:36:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:36:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:36:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:36:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:36:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:36:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:36:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:36:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:36:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:36:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:36:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:36:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:36:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:36:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:36:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:36:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:36:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:37:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:37:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:37:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:37:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:37:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:37:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:37:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:37:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:37:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:37:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:37:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:37:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:37:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:37:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:37:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:37:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:37:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:37:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:37:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:37:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:37:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:37:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:37:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:37:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:37:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:37:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:37:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:38:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:38:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:38:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:38:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:38:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:38:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:38:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:38:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:38:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:38:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:38:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:38:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:38:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:38:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:38:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:38:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:38:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:38:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:38:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:38:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:38:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:38:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:38:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:38:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:38:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:38:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:38:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:38:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:39:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:39:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:39:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:39:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:39:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:39:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:39:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:39:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:39:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:39:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:39:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:39:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:39:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:39:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:39:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:39:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:39:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:39:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:39:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:39:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:39:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:39:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:39:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:39:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:39:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:39:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:39:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:40:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:40:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:40:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:40:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:40:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:40:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:40:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:40:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:40:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:40:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:40:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:40:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:40:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:40:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:40:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:40:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:40:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:40:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:40:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:40:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:40:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:40:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:40:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:40:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:40:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:40:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:40:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:41:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:41:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:41:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:41:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:41:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:41:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:41:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:41:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:41:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:41:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:41:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:41:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:41:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:41:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:41:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:41:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:41:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:41:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:41:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:41:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:41:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:41:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:41:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:41:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:41:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:41:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:41:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:41:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:42:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:42:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:42:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:42:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:42:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:42:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:42:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:42:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:42:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:42:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:42:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:42:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:42:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:42:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:42:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:42:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:42:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:42:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:42:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:42:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:42:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:42:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:42:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:42:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:42:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:42:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:42:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:43:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:43:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:43:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:43:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:43:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:43:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:43:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:43:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:43:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:43:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:43:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:43:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:43:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:43:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:43:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:43:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:43:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:43:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:43:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:43:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:43:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:43:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:43:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:43:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:43:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:43:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:43:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:44:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:44:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:44:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:44:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:44:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:44:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:44:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:44:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:44:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:44:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:44:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:44:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:44:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:44:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:44:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:44:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:44:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:44:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:44:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:44:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:44:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:44:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:44:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:44:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:44:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:44:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:44:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:45:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:45:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:45:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:45:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:45:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:45:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:45:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:45:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:45:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:45:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:45:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:45:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:45:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:45:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:45:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:45:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:45:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:45:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:45:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:45:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:45:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:45:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:45:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:45:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:45:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:45:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:45:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:45:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:46:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:46:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:46:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:46:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:46:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:46:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:46:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:46:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:46:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:46:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:46:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:46:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:46:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:46:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:46:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:46:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:46:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:46:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:46:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:46:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:46:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:46:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:46:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:46:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:46:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:46:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:46:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:47:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:47:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:47:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:47:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:47:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:47:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:47:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:47:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:47:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:47:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:47:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:47:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:47:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:47:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:47:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:47:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:47:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:47:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:47:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:47:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:47:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:47:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:47:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:47:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:47:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:47:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:47:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:48:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:48:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:48:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:48:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:48:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:48:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:48:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:48:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:48:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:48:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:48:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:48:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:48:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:48:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:48:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:48:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:48:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:48:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:48:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:48:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:48:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:48:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:48:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:48:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:48:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:48:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:48:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:48:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:49:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:49:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:49:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:49:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:49:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:49:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:49:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:49:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:49:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:49:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:49:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:49:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:49:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:49:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:49:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:49:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:49:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:49:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:49:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:49:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:49:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:49:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:49:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:49:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:49:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:49:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:49:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:50:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:50:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:50:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:50:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:50:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:50:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:50:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:50:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:50:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:50:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:50:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:50:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:50:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:50:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:50:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:50:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:50:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:50:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:50:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:50:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:50:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:50:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:50:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:50:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:50:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:50:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:50:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:51:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:51:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:51:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:51:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:51:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:51:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:51:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:51:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:51:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:51:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:51:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:51:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:51:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:51:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:51:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:51:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:51:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:51:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:51:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:51:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:51:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:51:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:51:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:51:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:51:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:51:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:51:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:51:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:52:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:52:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:52:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:52:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:52:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:52:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:52:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:52:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:52:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:52:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:52:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:52:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:52:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:52:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:52:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:52:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:52:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:52:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:52:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:52:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:52:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:52:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:52:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:52:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:52:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:52:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:52:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:53:00 rdma06 systemd[1]: Starting dnf makecache...
Apr 16 14:53:01 rdma06 dnf[1111382]: Updating Subscription Management repositories.
Apr 16 14:53:01 rdma06 dnf[1111382]: Unable to read consumer identity
Apr 16 14:53:01 rdma06 dnf[1111382]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 16 14:53:02 rdma06 dnf[1111382]: Copr repo for fmf owned by psss                 2.3 kB/s | 3.3 kB     00:01
Apr 16 14:53:02 rdma06 dnf[1111382]: beaker-AppStream                                 41 kB/s | 168  B     00:00
Apr 16 14:53:02 rdma06 dnf[1111382]: Errors during downloading metadata for repository 'beaker-AppStream':
Apr 16 14:53:02 rdma06 dnf[1111382]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 14:53:02 rdma06 dnf[1111382]: Error: Failed to download metadata for repo 'beaker-AppStream': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 14:53:02 rdma06 dnf[1111382]: beaker-AppStream-debuginfo                       51 kB/s | 168  B     00:00
Apr 16 14:53:02 rdma06 dnf[1111382]: Errors during downloading metadata for repository 'beaker-AppStream-debuginfo':
Apr 16 14:53:02 rdma06 dnf[1111382]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 14:53:02 rdma06 dnf[1111382]: Error: Failed to download metadata for repo 'beaker-AppStream-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 14:53:02 rdma06 dnf[1111382]: beaker-BaseOS                                    54 kB/s | 168  B     00:00
Apr 16 14:53:02 rdma06 dnf[1111382]: Errors during downloading metadata for repository 'beaker-BaseOS':
Apr 16 14:53:02 rdma06 dnf[1111382]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 14:53:02 rdma06 dnf[1111382]: Error: Failed to download metadata for repo 'beaker-BaseOS': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 14:53:02 rdma06 dnf[1111382]: beaker-BaseOS-debuginfo                          59 kB/s | 168  B     00:00
Apr 16 14:53:02 rdma06 dnf[1111382]: Errors during downloading metadata for repository 'beaker-BaseOS-debuginfo':
Apr 16 14:53:02 rdma06 dnf[1111382]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 14:53:02 rdma06 dnf[1111382]: Error: Failed to download metadata for repo 'beaker-BaseOS-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 14:53:02 rdma06 dnf[1111382]: beaker-CRB                                       58 kB/s | 168  B     00:00
Apr 16 14:53:02 rdma06 dnf[1111382]: Errors during downloading metadata for repository 'beaker-CRB':
Apr 16 14:53:02 rdma06 dnf[1111382]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 14:53:02 rdma06 dnf[1111382]: Error: Failed to download metadata for repo 'beaker-CRB': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 14:53:02 rdma06 dnf[1111382]: beaker-CRB-debuginfo                             58 kB/s | 168  B     00:00
Apr 16 14:53:02 rdma06 dnf[1111382]: Errors during downloading metadata for repository 'beaker-CRB-debuginfo':
Apr 16 14:53:02 rdma06 dnf[1111382]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 14:53:02 rdma06 dnf[1111382]: Error: Failed to download metadata for repo 'beaker-CRB-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 14:53:02 rdma06 dnf[1111382]: beaker-HighAvailability                          56 kB/s | 168  B     00:00
Apr 16 14:53:02 rdma06 dnf[1111382]: Errors during downloading metadata for repository 'beaker-HighAvailability':
Apr 16 14:53:02 rdma06 dnf[1111382]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 14:53:02 rdma06 dnf[1111382]: Error: Failed to download metadata for repo 'beaker-HighAvailability': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 14:53:02 rdma06 dnf[1111382]: beaker-HighAvailability-debuginfo                58 kB/s | 168  B     00:00
Apr 16 14:53:02 rdma06 dnf[1111382]: Errors during downloading metadata for repository 'beaker-HighAvailability-debuginfo':
Apr 16 14:53:02 rdma06 dnf[1111382]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 14:53:02 rdma06 dnf[1111382]: Error: Failed to download metadata for repo 'beaker-HighAvailability-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 14:53:02 rdma06 dnf[1111382]: beaker-NFV                                       60 kB/s | 168  B     00:00
Apr 16 14:53:02 rdma06 dnf[1111382]: Errors during downloading metadata for repository 'beaker-NFV':
Apr 16 14:53:02 rdma06 dnf[1111382]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 14:53:02 rdma06 dnf[1111382]: Error: Failed to download metadata for repo 'beaker-NFV': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 14:53:02 rdma06 dnf[1111382]: beaker-NFV-debuginfo                             54 kB/s | 168  B     00:00
Apr 16 14:53:02 rdma06 dnf[1111382]: Errors during downloading metadata for repository 'beaker-NFV-debuginfo':
Apr 16 14:53:02 rdma06 dnf[1111382]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 14:53:02 rdma06 dnf[1111382]: Error: Failed to download metadata for repo 'beaker-NFV-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 14:53:02 rdma06 dnf[1111382]: beaker-RT                                        55 kB/s | 168  B     00:00
Apr 16 14:53:02 rdma06 dnf[1111382]: Errors during downloading metadata for repository 'beaker-RT':
Apr 16 14:53:02 rdma06 dnf[1111382]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 14:53:02 rdma06 dnf[1111382]: Error: Failed to download metadata for repo 'beaker-RT': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 14:53:02 rdma06 dnf[1111382]: beaker-RT-debuginfo                              57 kB/s | 168  B     00:00
Apr 16 14:53:02 rdma06 dnf[1111382]: Errors during downloading metadata for repository 'beaker-RT-debuginfo':
Apr 16 14:53:02 rdma06 dnf[1111382]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 14:53:02 rdma06 dnf[1111382]: Error: Failed to download metadata for repo 'beaker-RT-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 14:53:02 rdma06 dnf[1111382]: beaker-ResilientStorage                          54 kB/s | 168  B     00:00
Apr 16 14:53:02 rdma06 dnf[1111382]: Errors during downloading metadata for repository 'beaker-ResilientStorage':
Apr 16 14:53:02 rdma06 dnf[1111382]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 14:53:02 rdma06 dnf[1111382]: Error: Failed to download metadata for repo 'beaker-ResilientStorage': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 14:53:02 rdma06 dnf[1111382]: beaker-ResilientStorage-debuginfo                44 kB/s | 168  B     00:00
Apr 16 14:53:02 rdma06 dnf[1111382]: Errors during downloading metadata for repository 'beaker-ResilientStorage-debuginfo':
Apr 16 14:53:02 rdma06 dnf[1111382]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 14:53:02 rdma06 dnf[1111382]: Error: Failed to download metadata for repo 'beaker-ResilientStorage-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 14:53:02 rdma06 dnf[1111382]: beaker-SAP                                       65 kB/s | 168  B     00:00
Apr 16 14:53:02 rdma06 dnf[1111382]: Errors during downloading metadata for repository 'beaker-SAP':
Apr 16 14:53:02 rdma06 dnf[1111382]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 14:53:02 rdma06 dnf[1111382]: Error: Failed to download metadata for repo 'beaker-SAP': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 14:53:02 rdma06 dnf[1111382]: beaker-SAP-debuginfo                             64 kB/s | 168  B     00:00
Apr 16 14:53:02 rdma06 dnf[1111382]: Errors during downloading metadata for repository 'beaker-SAP-debuginfo':
Apr 16 14:53:02 rdma06 dnf[1111382]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 14:53:02 rdma06 dnf[1111382]: Error: Failed to download metadata for repo 'beaker-SAP-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 14:53:02 rdma06 dnf[1111382]: beaker-SAPHANA                                   60 kB/s | 168  B     00:00
Apr 16 14:53:02 rdma06 dnf[1111382]: Errors during downloading metadata for repository 'beaker-SAPHANA':
Apr 16 14:53:02 rdma06 dnf[1111382]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 14:53:02 rdma06 dnf[1111382]: Error: Failed to download metadata for repo 'beaker-SAPHANA': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 14:53:02 rdma06 dnf[1111382]: beaker-SAPHANA-debuginfo                         54 kB/s | 168  B     00:00
Apr 16 14:53:02 rdma06 dnf[1111382]: Errors during downloading metadata for repository 'beaker-SAPHANA-debuginfo':
Apr 16 14:53:02 rdma06 dnf[1111382]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 14:53:02 rdma06 dnf[1111382]: Error: Failed to download metadata for repo 'beaker-SAPHANA-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 14:53:03 rdma06 dnf[1111382]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr 16 14:53:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:53:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:53:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:53:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:53:04 rdma06 dnf[1111382]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr 16 14:53:04 rdma06 dnf[1111382]: Ignoring repositories: beaker-AppStream, beaker-AppStream-debuginfo, beaker-BaseOS, beaker-BaseOS-debuginfo, beaker-CRB, beaker-CRB-debuginfo, beaker-HighAvailability, beaker-HighAvailability-debuginfo, beaker-NFV, beaker-NFV-debuginfo, beaker-RT, beaker-RT-debuginfo, beaker-ResilientStorage, beaker-ResilientStorage-debuginfo, beaker-SAP, beaker-SAP-debuginfo, beaker-SAPHANA, beaker-SAPHANA-debuginfo
Apr 16 14:53:04 rdma06 dnf[1111382]: Metadata cache created.
Apr 16 14:53:04 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 16 14:53:04 rdma06 systemd[1]: Finished dnf makecache.
Apr 16 14:53:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:53:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:53:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:53:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:53:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:53:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:53:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:53:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:53:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:53:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:53:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:53:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:53:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:53:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:53:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:53:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:53:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:53:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:53:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:53:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:53:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:53:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:53:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:54:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:54:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:54:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:54:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:54:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:54:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:54:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:54:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:54:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:54:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:54:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:54:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:54:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:54:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:54:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:54:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:54:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:54:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:54:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:54:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:54:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:54:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:54:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:54:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:54:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:54:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:54:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:54:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:55:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:55:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:55:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:55:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:55:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:55:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:55:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:55:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:55:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:55:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:55:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:55:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:55:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:55:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:55:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:55:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:55:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:55:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:55:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:55:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:55:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:55:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:55:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:55:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:55:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:55:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:55:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:56:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:56:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:56:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:56:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:56:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:56:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:56:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:56:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:56:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:56:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:56:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:56:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:56:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:56:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:56:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:56:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:56:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:56:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:56:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:56:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:56:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:56:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:56:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:56:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:56:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:56:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:56:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:57:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:57:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:57:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:57:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:57:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:57:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:57:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:57:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:57:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:57:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:57:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:57:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:57:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:57:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:57:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:57:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:57:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:57:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:57:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:57:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:57:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:57:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:57:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:57:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:57:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:57:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:57:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:57:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:58:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:58:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:58:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:58:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:58:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:58:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:58:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:58:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:58:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:58:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:58:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:58:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:58:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:58:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:58:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:58:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:58:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:58:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:58:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:58:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:58:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:58:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:58:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:58:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:58:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:58:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:58:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:59:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:59:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:59:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:59:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:59:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:59:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:59:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:59:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:59:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:59:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:59:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:59:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:59:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:59:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:59:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:59:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:59:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:59:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:59:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:59:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:59:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:59:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 14:59:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 14:59:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 14:59:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 14:59:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 14:59:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:00:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:00:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:00:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:00:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:00:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:00:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:00:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:00:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:00:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:00:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:00:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:00:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:00:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:00:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:00:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:00:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:00:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:00:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:00:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:00:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:00:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:00:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:00:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:00:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:00:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:00:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:00:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:00:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:01:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:01:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:01:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:01:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:01:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:01:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:01:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:01:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:01:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:01:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:01:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:01:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:01:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:01:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:01:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:01:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:01:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:01:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:01:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:01:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:01:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:01:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:01:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:01:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:01:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:01:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:01:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:02:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:02:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:02:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:02:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:02:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:02:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:02:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:02:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:02:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:02:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:02:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:02:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:02:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:02:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:02:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:02:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:02:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:02:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:02:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:02:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:02:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:02:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:02:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:02:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:02:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:02:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:02:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:03:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:03:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:03:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:03:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:03:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:03:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:03:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:03:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:03:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:03:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:03:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:03:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:03:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:03:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:03:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:03:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:03:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:03:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:03:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:03:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:03:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:03:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:03:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:03:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:03:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:03:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:03:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:04:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:04:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:04:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:04:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:04:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:04:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:04:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:04:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:04:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:04:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:04:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:04:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:04:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:04:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:04:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:04:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:04:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:04:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:04:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:04:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:04:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:04:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:04:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:04:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:04:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:04:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:04:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:04:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:05:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:05:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:05:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:05:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:05:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:05:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:05:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:05:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:05:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:05:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:05:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:05:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:05:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:05:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:05:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:05:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:05:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:05:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:05:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:05:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:05:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:05:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:05:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:05:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:05:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:05:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:05:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:06:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:06:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:06:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:06:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:06:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:06:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:06:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:06:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:06:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:06:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:06:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:06:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:06:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:06:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:06:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:06:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:06:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:06:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:06:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:06:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:06:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:06:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:06:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:06:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:06:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:06:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:06:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:07:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:07:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:07:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:07:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:07:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:07:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:07:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:07:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:07:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:07:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:07:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:07:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:07:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:07:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:07:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:07:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:07:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:07:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:07:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:07:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:07:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:07:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:07:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:07:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:07:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:07:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:07:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:07:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:08:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:08:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:08:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:08:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:08:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:08:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:08:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:08:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:08:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:08:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:08:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:08:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:08:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:08:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:08:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:08:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:08:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:08:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:08:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:08:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:08:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:08:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:08:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:08:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:08:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:08:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:08:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:09:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:09:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:09:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:09:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:09:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:09:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:09:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:09:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:09:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:09:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:09:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:09:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:09:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:09:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:09:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:09:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:09:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:09:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:09:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:09:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:09:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:09:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:09:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:09:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:09:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:09:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:09:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:10:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:10:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:10:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:10:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:10:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:10:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:10:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:10:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:10:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:10:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:10:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:10:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:10:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:10:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:10:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:10:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:10:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:10:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:10:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:10:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:10:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:10:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:10:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:10:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:10:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:10:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:10:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:10:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:11:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:11:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:11:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:11:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:11:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:11:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:11:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:11:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:11:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:11:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:11:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:11:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:11:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:11:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:11:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:11:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:11:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:11:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:11:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:11:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:11:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:11:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:11:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:11:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:11:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:11:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:11:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:12:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:12:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:12:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:12:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:12:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:12:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:12:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:12:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:12:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:12:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:12:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:12:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:12:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:12:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:12:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:12:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:12:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:12:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:12:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:12:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:12:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:12:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:12:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:12:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:12:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:12:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:12:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:13:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:13:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:13:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:13:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:13:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:13:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:13:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:13:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:13:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:13:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:13:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:13:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:13:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:13:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:13:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:13:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:13:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:13:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:13:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:13:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:13:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:13:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:13:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:13:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:13:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:13:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:13:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:13:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:14:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:14:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:14:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:14:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:14:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:14:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:14:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:14:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:14:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:14:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:14:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:14:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:14:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:14:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:14:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:14:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:14:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:14:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:14:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:14:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:14:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:14:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:14:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:14:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:14:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:14:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:14:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:15:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:15:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:15:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:15:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:15:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:15:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:15:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:15:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:15:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:15:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:15:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:15:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:15:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:15:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:15:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:15:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:15:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:15:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:15:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:15:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:15:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:15:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:15:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:15:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:15:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:15:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:15:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:16:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:16:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:16:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:16:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:16:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:16:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:16:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:16:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:16:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:16:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:16:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:16:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:16:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:16:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:16:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:16:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:16:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:16:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:16:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:16:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:16:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:16:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:16:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:16:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:16:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:16:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:16:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:16:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:17:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:17:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:17:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:17:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:17:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:17:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:17:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:17:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:17:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:17:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:17:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:17:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:17:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:17:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:17:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:17:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:17:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:17:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:17:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:17:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:17:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:17:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:17:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:17:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:17:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:17:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:17:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:18:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:18:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:18:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:18:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:18:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:18:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:18:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:18:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:18:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:18:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:18:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:18:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:18:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:18:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:18:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:18:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:18:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:18:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:18:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:18:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:18:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:18:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:18:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:18:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:18:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:18:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:18:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:19:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:19:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:19:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:19:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:19:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:19:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:19:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:19:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:19:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:19:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:19:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:19:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:19:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:19:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:19:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:19:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:19:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:19:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:19:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:19:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:19:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:19:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:19:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:19:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:19:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:19:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:19:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:20:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:20:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:20:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:20:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:20:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:20:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:20:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:20:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:20:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:20:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:20:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:20:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:20:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:20:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:20:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:20:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:20:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:20:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:20:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:20:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:20:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:20:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:20:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:20:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:20:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:20:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:20:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:20:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:21:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:21:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:21:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:21:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:21:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:21:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:21:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:21:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:21:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:21:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:21:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:21:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:21:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:21:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:21:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:21:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:21:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:21:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:21:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:21:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:21:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:21:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:21:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:21:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:21:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:21:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:21:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:22:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:22:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:22:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:22:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:22:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:22:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:22:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:22:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:22:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:22:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:22:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:22:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:22:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:22:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:22:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:22:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:22:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:22:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:22:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:22:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:22:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:22:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:22:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:22:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:22:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:22:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:22:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:23:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:23:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:23:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:23:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:23:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:23:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:23:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:23:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:23:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:23:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:23:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:23:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:23:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:23:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:23:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:23:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:23:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:23:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:23:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:23:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:23:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:23:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:23:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:23:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:23:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:23:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:23:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:23:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:24:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:24:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:24:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:24:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:24:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:24:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:24:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:24:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:24:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:24:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:24:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:24:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:24:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:24:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:24:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:24:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:24:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:24:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:24:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:24:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:24:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:24:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:24:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:24:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:24:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:24:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:24:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:25:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:25:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:25:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:25:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:25:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:25:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:25:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:25:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:25:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:25:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:25:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:25:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:25:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:25:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:25:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:25:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:25:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:25:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:25:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:25:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:25:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:25:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:25:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:25:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:25:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:25:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:25:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:26:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:26:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:26:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:26:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:26:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:26:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:26:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:26:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:26:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:26:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:26:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:26:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:26:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:26:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:26:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:26:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:26:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:26:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:26:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:26:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:26:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:26:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:26:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:26:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:26:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:26:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:26:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:26:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:27:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:27:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:27:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:27:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:27:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:27:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:27:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:27:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:27:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:27:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:27:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:27:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:27:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:27:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:27:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:27:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:27:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:27:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:27:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:27:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:27:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:27:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:27:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:27:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:27:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:27:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:27:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:28:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:28:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:28:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:28:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:28:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:28:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:28:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:28:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:28:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:28:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:28:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:28:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:28:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:28:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:28:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:28:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:28:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:28:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:28:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:28:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:28:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:28:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:28:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:28:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:28:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:28:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:28:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:29:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:29:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:29:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:29:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:29:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:29:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:29:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:29:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:29:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:29:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:29:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:29:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:29:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:29:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:29:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:29:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:29:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:29:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:29:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:29:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:29:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:29:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:29:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:29:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:29:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:29:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:29:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:29:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:30:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:30:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:30:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:30:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:30:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:30:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:30:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:30:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:30:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:30:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:30:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:30:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:30:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:30:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:30:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:30:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:30:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:30:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:30:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:30:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:30:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:30:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:30:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:30:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:30:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:30:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:30:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:31:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:31:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:31:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:31:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:31:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:31:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:31:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:31:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:31:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:31:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:31:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:31:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:31:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:31:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:31:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:31:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:31:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:31:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:31:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:31:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:31:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:31:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:31:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:31:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:31:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:31:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:31:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:32:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:32:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:32:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:32:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:32:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:32:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:32:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:32:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:32:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:32:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:32:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:32:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:32:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:32:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:32:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:32:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:32:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:32:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:32:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:32:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:32:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:32:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:32:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:32:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:32:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:32:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:32:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:32:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:33:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:33:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:33:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:33:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:33:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:33:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:33:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:33:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:33:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:33:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:33:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:33:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:33:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:33:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:33:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:33:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:33:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:33:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:33:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:33:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:33:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:33:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:33:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:33:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:33:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:33:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:33:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:34:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:34:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:34:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:34:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:34:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:34:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:34:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:34:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:34:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:34:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:34:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:34:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:34:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:34:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:34:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:34:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:34:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:34:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:34:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:34:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:34:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:34:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:34:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:34:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:34:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:34:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:34:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:35:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:35:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:35:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:35:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:35:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:35:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:35:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:35:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:35:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:35:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:35:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:35:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:35:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:35:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:35:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:35:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:35:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:35:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:35:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:35:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:35:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:35:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:35:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:35:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:35:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:35:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:35:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:36:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:36:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:36:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:36:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:36:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:36:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:36:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:36:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:36:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:36:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:36:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:36:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:36:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:36:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:36:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:36:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:36:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:36:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:36:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:36:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:36:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:36:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:36:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:36:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:36:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:36:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:36:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:36:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:37:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:37:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:37:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:37:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:37:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:37:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:37:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:37:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:37:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:37:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:37:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:37:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:37:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:37:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:37:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:37:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:37:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:37:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:37:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:37:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:37:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:37:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:37:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:37:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:37:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:37:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:37:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:38:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:38:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:38:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:38:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:38:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:38:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:38:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:38:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:38:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:38:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:38:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:38:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:38:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:38:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:38:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:38:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:38:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:38:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:38:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:38:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:38:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:38:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:38:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:38:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:38:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:38:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:38:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:39:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:39:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:39:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:39:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:39:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:39:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:39:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:39:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:39:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:39:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:39:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:39:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:39:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:39:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:39:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:39:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:39:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:39:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:39:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:39:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:39:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:39:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:39:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:39:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:39:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:39:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:39:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:39:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:40:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:40:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:40:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:40:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:40:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:40:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:40:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:40:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:40:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:40:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:40:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:40:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:40:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:40:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:40:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:40:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:40:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:40:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:40:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:40:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:40:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:40:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:40:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:40:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:40:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:40:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:40:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:41:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:41:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:41:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:41:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:41:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:41:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:41:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:41:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:41:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:41:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:41:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:41:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:41:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:41:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:41:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:41:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:41:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:41:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:41:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:41:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:41:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:41:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:41:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:41:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:41:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:41:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:41:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:42:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:42:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:42:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:42:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:42:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:42:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:42:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:42:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:42:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:42:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:42:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:42:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:42:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:42:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:42:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:42:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:42:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:42:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:42:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:42:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:42:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:42:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:42:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:42:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:42:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:42:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:42:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:42:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:43:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:43:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:43:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:43:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:43:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:43:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:43:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:43:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:43:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:43:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:43:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:43:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:43:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:43:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:43:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:43:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:43:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:43:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:43:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:43:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:43:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:43:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:43:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:43:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:43:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:43:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:43:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:44:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:44:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:44:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:44:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:44:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:44:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:44:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:44:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:44:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:44:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:44:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:44:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:44:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:44:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:44:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:44:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:44:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:44:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:44:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:44:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:44:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:44:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:44:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:44:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:44:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:44:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:44:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:45:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:45:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:45:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:45:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:45:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:45:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:45:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:45:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:45:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:45:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:45:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:45:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:45:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:45:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:45:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:45:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:45:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:45:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:45:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:45:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:45:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:45:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:45:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:45:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:45:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:45:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:45:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:45:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:46:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:46:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:46:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:46:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:46:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:46:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:46:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:46:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:46:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:46:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:46:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:46:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:46:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:46:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:46:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:46:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:46:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:46:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:46:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:46:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:46:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:46:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:46:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:46:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:46:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:46:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:46:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:47:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:47:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:47:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:47:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:47:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:47:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:47:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:47:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:47:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:47:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:47:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:47:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:47:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:47:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:47:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:47:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:47:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:47:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:47:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:47:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:47:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:47:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:47:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:47:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:47:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:47:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:47:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:48:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:48:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:48:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:48:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:48:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:48:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:48:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:48:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:48:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:48:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:48:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:48:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:48:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:48:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:48:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:48:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:48:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:48:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:48:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:48:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:48:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:48:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:48:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:48:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:48:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:48:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:48:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:48:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:49:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:49:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:49:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:49:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:49:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:49:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:49:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:49:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:49:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:49:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:49:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:49:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:49:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:49:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:49:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:49:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:49:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:49:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:49:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:49:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:49:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:49:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:49:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:49:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:49:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:49:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:49:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:50:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:50:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:50:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:50:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:50:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:50:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:50:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:50:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:50:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:50:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:50:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:50:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:50:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:50:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:50:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:50:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:50:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:50:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:50:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:50:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:50:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:50:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:50:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:50:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:50:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:50:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:50:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:51:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:51:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:51:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:51:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:51:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:51:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:51:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:51:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:51:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:51:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:51:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:51:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:51:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:51:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:51:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:51:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:51:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:51:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:51:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:51:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:51:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:51:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:51:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:51:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:51:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:51:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:51:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:52:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:52:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:52:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:52:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:52:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:52:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:52:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:52:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:52:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:52:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:52:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:52:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:52:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:52:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:52:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:52:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:52:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:52:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:52:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:52:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:52:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:52:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:52:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:52:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:52:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:52:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:52:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:52:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:53:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:53:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:53:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:53:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:53:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:53:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:53:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:53:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:53:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:53:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:53:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:53:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:53:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:53:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:53:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:53:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:53:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:53:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:53:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:53:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:53:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:53:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:53:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:53:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:53:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:53:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:53:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:54:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:54:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:54:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:54:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:54:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:54:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:54:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:54:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:54:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:54:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:54:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:54:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:54:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:54:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:54:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:54:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:54:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:54:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:54:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:54:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:54:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:54:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:54:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:54:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:54:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:54:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:54:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:55:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:55:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:55:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:55:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:55:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:55:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:55:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:55:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:55:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:55:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:55:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:55:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:55:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:55:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:55:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:55:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:55:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:55:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:55:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:55:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:55:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:55:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:55:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:55:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:55:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:55:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:55:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:55:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:56:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:56:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:56:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:56:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:56:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:56:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:56:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:56:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:56:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:56:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:56:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:56:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:56:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:56:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:56:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:56:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:56:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:56:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:56:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:56:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:56:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:56:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:56:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:56:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:56:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:56:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:56:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:57:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:57:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:57:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:57:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:57:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:57:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:57:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:57:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:57:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:57:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:57:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:57:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:57:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:57:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:57:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:57:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:57:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:57:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:57:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:57:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:57:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:57:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:57:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:57:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:57:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:57:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:57:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:58:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:58:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:58:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:58:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:58:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:58:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:58:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:58:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:58:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:58:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:58:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:58:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:58:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:58:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:58:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:58:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:58:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:58:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:58:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:58:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:58:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:58:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:58:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:58:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:58:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:58:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:58:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:58:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:59:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:59:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:59:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:59:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:59:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:59:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:59:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:59:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:59:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:59:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:59:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:59:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:59:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:59:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:59:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:59:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:59:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:59:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:59:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:59:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:59:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:59:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 15:59:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 15:59:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 15:59:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 15:59:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 15:59:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:00:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:00:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:00:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:00:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:00:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:00:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:00:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:00:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:00:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:00:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:00:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:00:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:00:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:00:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:00:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:00:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:00:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:00:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:00:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:00:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:00:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:00:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:00:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:00:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:00:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:00:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:00:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:01:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:01:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:01:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:01:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:01:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:01:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:01:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:01:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:01:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:01:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:01:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:01:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:01:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:01:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:01:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:01:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:01:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:01:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:01:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:01:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:01:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:01:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:01:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:01:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:01:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:01:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:01:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:01:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:02:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:02:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:02:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:02:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:02:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:02:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:02:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:02:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:02:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:02:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:02:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:02:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:02:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:02:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:02:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:02:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:02:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:02:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:02:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:02:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:02:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:02:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:02:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:02:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:02:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:02:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:02:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:03:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:03:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:03:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:03:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:03:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:03:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:03:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:03:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:03:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:03:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:03:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:03:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:03:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:03:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:03:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:03:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:03:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:03:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:03:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:03:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:03:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:03:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:03:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:03:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:03:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:03:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:03:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:04:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:04:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:04:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:04:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:04:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:04:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:04:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:04:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:04:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:04:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:04:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:04:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:04:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:04:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:04:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:04:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:04:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:04:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:04:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:04:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:04:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:04:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:04:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:04:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:04:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:04:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:04:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:04:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:05:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:05:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:05:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:05:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:05:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:05:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:05:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:05:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:05:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:05:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:05:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:05:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:05:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:05:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:05:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:05:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:05:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:05:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:05:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:05:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:05:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:05:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:05:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:05:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:05:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:05:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:05:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:06:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:06:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:06:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:06:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:06:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:06:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:06:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:06:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:06:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:06:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:06:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:06:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:06:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:06:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:06:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:06:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:06:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:06:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:06:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:06:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:06:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:06:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:06:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:06:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:06:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:06:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:06:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:07:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:07:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:07:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:07:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:07:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:07:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:07:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:07:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:07:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:07:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:07:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:07:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:07:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:07:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:07:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:07:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:07:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:07:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:07:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:07:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:07:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:07:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:07:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:07:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:07:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:07:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:07:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:07:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:08:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:08:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:08:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:08:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:08:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:08:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:08:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:08:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:08:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:08:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:08:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:08:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:08:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:08:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:08:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:08:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:08:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:08:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:08:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:08:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:08:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:08:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:08:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:08:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:08:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:08:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:08:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:09:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:09:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:09:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:09:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:09:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:09:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:09:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:09:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:09:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:09:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:09:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:09:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:09:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:09:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:09:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:09:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:09:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:09:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:09:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:09:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:09:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:09:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:09:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:09:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:09:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:09:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:09:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:10:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:10:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:10:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:10:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:10:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:10:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:10:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:10:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:10:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:10:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:10:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:10:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:10:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:10:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:10:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:10:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:10:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:10:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:10:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:10:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:10:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:10:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:10:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:10:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:10:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:10:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:10:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:11:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:11:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:11:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:11:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:11:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:11:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:11:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:11:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:11:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:11:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:11:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:11:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:11:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:11:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:11:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:11:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:11:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:11:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:11:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:11:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:11:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:11:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:11:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:11:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:11:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:11:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:11:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:11:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:12:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:12:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:12:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:12:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:12:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:12:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:12:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:12:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:12:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:12:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:12:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:12:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:12:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:12:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:12:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:12:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:12:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:12:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:12:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:12:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:12:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:12:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:12:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:12:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:12:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:12:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:12:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:13:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:13:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:13:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:13:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:13:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:13:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:13:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:13:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:13:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:13:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:13:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:13:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:13:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:13:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:13:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:13:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:13:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:13:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:13:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:13:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:13:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:13:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:13:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:13:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:13:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:13:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:13:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:14:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:14:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:14:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:14:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:14:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:14:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:14:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:14:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:14:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:14:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:14:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:14:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:14:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:14:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:14:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:14:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:14:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:14:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:14:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:14:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:14:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:14:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:14:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:14:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:14:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:14:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:14:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:14:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:15:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:15:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:15:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:15:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:15:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:15:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:15:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:15:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:15:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:15:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:15:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:15:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:15:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:15:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:15:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:15:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:15:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:15:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:15:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:15:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:15:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:15:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:15:44 rdma06 systemd[1]: Starting dnf makecache...
Apr 16 16:15:44 rdma06 dnf[1114762]: Updating Subscription Management repositories.
Apr 16 16:15:44 rdma06 dnf[1114762]: Unable to read consumer identity
Apr 16 16:15:44 rdma06 dnf[1114762]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 16 16:15:44 rdma06 dnf[1114762]: Metadata cache refreshed recently.
Apr 16 16:15:44 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 16 16:15:44 rdma06 systemd[1]: Finished dnf makecache.
Apr 16 16:15:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:15:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:15:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:15:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:15:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:16:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:16:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:16:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:16:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:16:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:16:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:16:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:16:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:16:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:16:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:16:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:16:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:16:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:16:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:16:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:16:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:16:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:16:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:16:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:16:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:16:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:16:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:16:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:16:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:16:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:16:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:16:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:17:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:17:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:17:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:17:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:17:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:17:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:17:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:17:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:17:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:17:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:17:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:17:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:17:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:17:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:17:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:17:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:17:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:17:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:17:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:17:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:17:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:17:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:17:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:17:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:17:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:17:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:17:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:17:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:18:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:18:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:18:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:18:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:18:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:18:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:18:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:18:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:18:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:18:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:18:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:18:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:18:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:18:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:18:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:18:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:18:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:18:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:18:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:18:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:18:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:18:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:18:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:18:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:18:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:18:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:18:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:19:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:19:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:19:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:19:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:19:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:19:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:19:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:19:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:19:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:19:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:19:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:19:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:19:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:19:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:19:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:19:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:19:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:19:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:19:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:19:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:19:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:19:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:19:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:19:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:19:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:19:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:19:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:20:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:20:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:20:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:20:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:20:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:20:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:20:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:20:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:20:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:20:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:20:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:20:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:20:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:20:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:20:32 rdma06 NetworkManager[1025]: <info>  [1681676432.8409] dhcp4 (eno1): state changed new lease, address=10.73.131.45
Apr 16 16:20:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:20:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:20:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:20:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:20:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:20:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:20:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:20:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:20:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:20:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:20:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:20:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:20:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:20:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:21:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:21:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:21:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:21:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:21:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:21:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:21:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:21:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:21:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:21:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:21:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:21:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:21:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:21:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:21:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:21:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:21:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:21:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:21:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:21:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:21:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:21:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:21:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:21:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:21:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:21:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:21:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:22:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:22:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:22:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:22:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:22:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:22:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:22:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:22:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:22:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:22:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:22:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:22:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:22:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:22:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:22:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:22:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:22:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:22:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:22:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:22:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:22:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:22:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:22:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:22:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:22:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:22:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:22:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:23:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:23:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:23:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:23:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:23:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:23:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:23:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:23:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:23:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:23:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:23:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:23:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:23:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:23:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:23:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:23:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:23:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:23:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:23:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:23:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:23:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:23:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:23:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:23:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:23:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:23:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:23:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:23:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:24:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:24:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:24:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:24:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:24:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:24:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:24:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:24:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:24:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:24:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:24:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:24:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:24:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:24:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:24:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:24:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:24:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:24:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:24:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:24:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:24:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:24:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:24:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:24:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:24:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:24:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:24:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:25:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:25:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:25:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:25:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:25:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:25:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:25:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:25:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:25:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:25:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:25:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:25:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:25:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:25:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:25:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:25:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:25:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:25:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:25:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:25:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:25:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:25:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:25:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:25:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:25:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:25:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:25:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:26:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:26:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:26:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:26:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:26:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:26:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:26:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:26:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:26:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:26:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:26:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:26:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:26:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:26:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:26:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:26:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:26:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:26:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:26:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:26:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:26:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:26:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:26:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:26:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:26:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:26:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:26:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:27:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:27:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:27:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:27:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:27:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:27:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:27:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:27:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:27:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:27:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:27:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:27:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:27:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:27:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:27:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:27:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:27:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:27:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:27:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:27:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:27:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:27:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:27:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:27:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:27:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:27:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:27:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:27:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:28:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:28:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:28:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:28:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:28:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:28:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:28:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:28:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:28:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:28:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:28:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:28:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:28:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:28:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:28:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:28:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:28:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:28:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:28:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:28:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:28:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:28:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:28:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:28:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:28:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:28:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:28:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:29:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:29:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:29:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:29:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:29:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:29:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:29:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:29:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:29:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:29:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:29:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:29:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:29:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:29:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:29:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:29:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:29:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:29:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:29:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:29:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:29:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:29:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:29:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:29:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:29:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:29:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:29:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:30:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:30:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:30:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:30:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:30:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:30:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:30:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:30:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:30:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:30:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:30:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:30:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:30:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:30:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:30:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:30:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:30:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:30:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:30:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:30:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:30:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:30:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:30:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:30:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:30:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:30:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:30:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:30:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:31:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:31:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:31:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:31:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:31:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:31:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:31:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:31:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:31:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:31:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:31:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:31:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:31:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:31:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:31:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:31:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:31:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:31:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:31:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:31:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:31:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:31:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:31:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:31:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:31:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:31:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:31:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:32:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:32:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:32:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:32:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:32:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:32:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:32:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:32:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:32:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:32:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:32:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:32:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:32:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:32:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:32:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:32:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:32:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:32:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:32:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:32:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:32:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:32:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:32:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:32:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:32:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:32:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:32:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:33:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:33:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:33:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:33:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:33:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:33:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:33:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:33:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:33:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:33:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:33:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:33:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:33:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:33:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:33:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:33:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:33:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:33:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:33:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:33:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:33:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:33:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:33:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:33:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:33:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:33:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:33:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:33:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:34:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:34:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:34:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:34:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:34:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:34:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:34:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:34:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:34:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:34:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:34:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:34:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:34:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:34:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:34:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:34:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:34:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:34:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:34:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:34:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:34:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:34:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:34:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:34:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:34:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:34:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:34:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:35:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:35:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:35:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:35:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:35:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:35:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:35:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:35:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:35:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:35:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:35:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:35:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:35:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:35:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:35:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:35:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:35:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:35:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:35:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:35:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:35:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:35:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:35:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:35:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:35:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:35:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:35:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:36:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:36:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:36:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:36:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:36:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:36:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:36:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:36:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:36:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:36:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:36:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:36:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:36:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:36:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:36:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:36:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:36:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:36:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:36:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:36:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:36:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:36:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:36:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:36:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:36:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:36:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:36:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:36:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:37:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:37:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:37:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:37:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:37:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:37:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:37:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:37:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:37:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:37:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:37:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:37:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:37:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:37:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:37:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:37:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:37:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:37:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:37:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:37:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:37:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:37:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:37:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:37:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:37:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:37:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:37:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:38:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:38:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:38:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:38:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:38:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:38:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:38:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:38:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:38:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:38:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:38:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:38:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:38:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:38:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:38:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:38:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:38:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:38:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:38:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:38:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:38:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:38:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:38:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:38:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:38:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:38:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:38:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:39:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:39:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:39:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:39:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:39:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:39:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:39:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:39:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:39:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:39:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:39:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:39:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:39:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:39:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:39:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:39:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:39:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:39:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:39:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:39:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:39:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:39:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:39:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:39:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:39:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:39:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:39:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:39:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:40:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:40:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:40:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:40:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:40:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:40:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:40:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:40:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:40:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:40:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:40:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:40:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:40:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:40:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:40:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:40:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:40:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:40:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:40:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:40:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:40:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:40:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:40:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:40:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:40:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:40:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:40:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:41:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:41:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:41:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:41:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:41:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:41:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:41:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:41:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:41:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:41:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:41:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:41:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:41:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:41:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:41:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:41:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:41:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:41:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:41:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:41:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:41:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:41:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:41:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:41:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:41:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:41:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:41:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:42:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:42:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:42:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:42:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:42:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:42:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:42:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:42:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:42:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:42:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:42:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:42:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:42:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:42:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:42:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:42:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:42:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:42:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:42:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:42:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:42:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:42:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:42:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:42:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:42:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:42:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:42:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:43:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:43:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:43:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:43:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:43:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:43:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:43:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:43:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:43:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:43:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:43:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:43:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:43:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:43:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:43:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:43:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:43:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:43:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:43:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:43:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:43:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:43:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:43:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:43:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:43:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:43:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:43:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:43:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:44:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:44:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:44:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:44:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:44:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:44:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:44:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:44:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:44:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:44:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:44:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:44:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:44:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:44:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:44:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:44:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:44:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:44:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:44:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:44:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:44:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:44:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:44:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:44:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:44:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:44:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:44:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:45:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:45:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:45:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:45:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:45:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:45:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:45:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:45:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:45:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:45:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:45:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:45:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:45:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:45:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:45:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:45:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:45:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:45:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:45:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:45:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:45:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:45:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:45:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:45:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:45:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:45:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:45:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:46:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:46:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:46:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:46:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:46:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:46:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:46:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:46:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:46:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:46:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:46:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:46:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:46:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:46:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:46:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:46:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:46:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:46:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:46:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:46:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:46:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:46:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:46:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:46:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:46:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:46:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:46:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:46:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:47:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:47:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:47:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:47:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:47:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:47:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:47:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:47:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:47:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:47:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:47:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:47:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:47:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:47:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:47:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:47:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:47:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:47:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:47:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:47:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:47:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:47:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:47:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:47:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:47:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:47:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:47:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:48:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:48:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:48:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:48:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:48:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:48:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:48:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:48:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:48:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:48:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:48:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:48:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:48:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:48:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:48:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:48:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:48:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:48:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:48:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:48:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:48:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:48:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:48:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:48:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:48:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:48:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:48:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:49:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:49:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:49:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:49:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:49:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:49:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:49:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:49:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:49:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:49:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:49:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:49:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:49:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:49:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:49:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:49:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:49:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:49:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:49:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:49:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:49:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:49:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:49:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:49:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:49:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:49:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:49:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:49:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:50:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:50:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:50:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:50:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:50:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:50:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:50:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:50:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:50:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:50:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:50:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:50:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:50:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:50:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:50:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:50:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:50:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:50:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:50:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:50:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:50:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:50:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:50:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:50:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:50:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:50:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:50:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:51:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:51:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:51:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:51:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:51:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:51:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:51:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:51:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:51:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:51:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:51:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:51:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:51:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:51:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:51:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:51:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:51:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:51:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:51:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:51:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:51:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:51:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:51:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:51:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:51:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:51:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:51:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:52:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:52:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:52:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:52:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:52:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:52:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:52:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:52:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:52:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:52:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:52:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:52:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:52:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:52:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:52:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:52:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:52:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:52:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:52:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:52:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:52:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:52:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:52:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:52:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:52:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:52:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:52:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:52:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:53:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:53:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:53:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:53:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:53:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:53:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:53:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:53:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:53:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:53:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:53:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:53:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:53:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:53:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:53:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:53:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:53:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:53:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:53:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:53:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:53:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:53:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:53:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:53:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:53:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:53:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:53:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:54:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:54:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:54:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:54:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:54:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:54:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:54:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:54:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:54:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:54:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:54:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:54:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:54:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:54:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:54:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:54:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:54:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:54:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:54:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:54:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:54:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:54:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:54:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:54:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:54:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:54:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:54:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:55:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:55:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:55:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:55:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:55:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:55:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:55:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:55:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:55:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:55:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:55:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:55:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:55:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:55:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:55:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:55:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:55:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:55:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:55:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:55:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:55:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:55:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:55:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:55:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:55:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:55:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:55:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:55:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:56:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:56:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:56:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:56:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:56:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:56:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:56:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:56:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:56:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:56:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:56:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:56:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:56:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:56:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:56:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:56:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:56:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:56:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:56:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:56:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:56:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:56:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:56:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:56:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:56:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:56:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:56:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:57:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:57:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:57:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:57:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:57:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:57:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:57:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:57:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:57:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:57:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:57:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:57:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:57:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:57:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:57:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:57:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:57:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:57:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:57:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:57:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:57:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:57:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:57:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:57:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:57:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:57:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:57:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:58:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:58:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:58:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:58:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:58:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:58:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:58:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:58:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:58:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:58:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:58:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:58:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:58:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:58:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:58:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:58:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:58:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:58:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:58:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:58:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:58:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:58:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:58:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:58:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:58:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:58:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:58:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:59:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:59:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:59:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:59:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:59:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:59:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:59:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:59:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:59:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:59:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:59:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:59:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:59:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:59:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:59:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:59:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:59:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:59:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:59:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 16:59:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:59:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:59:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:59:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:59:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 16:59:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 16:59:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 16:59:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 16:59:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:00:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:00:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:00:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:00:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:00:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:00:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:00:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:00:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:00:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:00:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:00:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:00:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:00:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:00:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:00:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:00:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:00:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:00:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:00:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:00:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:00:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:00:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:00:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:00:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:00:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:00:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:00:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:01:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:01:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:01:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:01:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:01:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:01:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:01:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:01:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:01:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:01:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:01:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:01:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:01:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:01:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:01:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:01:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:01:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:01:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:01:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:01:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:01:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:01:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:01:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:01:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:01:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:01:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:01:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:02:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:02:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:02:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:02:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:02:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:02:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:02:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:02:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:02:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:02:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:02:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:02:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:02:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:02:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:02:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:02:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:02:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:02:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:02:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:02:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:02:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:02:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:02:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:02:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:02:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:02:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:02:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:02:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:03:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:03:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:03:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:03:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:03:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:03:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:03:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:03:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:03:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:03:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:03:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:03:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:03:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:03:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:03:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:03:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:03:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:03:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:03:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:03:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:03:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:03:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:03:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:03:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:03:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:03:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:03:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:04:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:04:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:04:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:04:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:04:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:04:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:04:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:04:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:04:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:04:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:04:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:04:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:04:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:04:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:04:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:04:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:04:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:04:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:04:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:04:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:04:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:04:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:04:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:04:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:04:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:04:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:04:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:05:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:05:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:05:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:05:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:05:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:05:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:05:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:05:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:05:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:05:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:05:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:05:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:05:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:05:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:05:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:05:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:05:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:05:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:05:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:05:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:05:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:05:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:05:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:05:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:05:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:05:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:05:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:05:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:06:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:06:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:06:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:06:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:06:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:06:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:06:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:06:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:06:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:06:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:06:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:06:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:06:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:06:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:06:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:06:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:06:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:06:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:06:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:06:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:06:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:06:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:06:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:06:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:06:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:06:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:06:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:07:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:07:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:07:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:07:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:07:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:07:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:07:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:07:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:07:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:07:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:07:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:07:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:07:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:07:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:07:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:07:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:07:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:07:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:07:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:07:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:07:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:07:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:07:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:07:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:07:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:07:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:07:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:08:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:08:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:08:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:08:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:08:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:08:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:08:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:08:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:08:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:08:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:08:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:08:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:08:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:08:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:08:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:08:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:08:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:08:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:08:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:08:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:08:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:08:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:08:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:08:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:08:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:08:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:08:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:08:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:09:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:09:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:09:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:09:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:09:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:09:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:09:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:09:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:09:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:09:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:09:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:09:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:09:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:09:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:09:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:09:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:09:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:09:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:09:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:09:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:09:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:09:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:09:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:09:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:09:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:09:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:09:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:10:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:10:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:10:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:10:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:10:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:10:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:10:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:10:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:10:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:10:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:10:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:10:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:10:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:10:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:10:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:10:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:10:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:10:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:10:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:10:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:10:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:10:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:10:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:10:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:10:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:10:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:10:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:11:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:11:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:11:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:11:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:11:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:11:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:11:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:11:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:11:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:11:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:11:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:11:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:11:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:11:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:11:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:11:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:11:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:11:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:11:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:11:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:11:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:11:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:11:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:11:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:11:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:11:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:11:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:11:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:12:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:12:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:12:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:12:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:12:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:12:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:12:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:12:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:12:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:12:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:12:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:12:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:12:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:12:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:12:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:12:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:12:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:12:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:12:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:12:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:12:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:12:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:12:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:12:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:12:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:12:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:12:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:13:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:13:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:13:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:13:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:13:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:13:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:13:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:13:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:13:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:13:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:13:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:13:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:13:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:13:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:13:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:13:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:13:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:13:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:13:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:13:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:13:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:13:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:13:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:13:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:13:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:13:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:13:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:14:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:14:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:14:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:14:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:14:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:14:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:14:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:14:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:14:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:14:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:14:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:14:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:14:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:14:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:14:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:14:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:14:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:14:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:14:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:14:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:14:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:14:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:14:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:14:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:14:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:14:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:14:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:14:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:15:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:15:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:15:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:15:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:15:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:15:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:15:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:15:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:15:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:15:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:15:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:15:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:15:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:15:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:15:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:15:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:15:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:15:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:15:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:15:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:15:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:15:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:15:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:15:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:15:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:15:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:15:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:16:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:16:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:16:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:16:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:16:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:16:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:16:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:16:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:16:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:16:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:16:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:16:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:16:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:16:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:16:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:16:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:16:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:16:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:16:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:16:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:16:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:16:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:16:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:16:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:16:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:16:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:16:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:17:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:17:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:17:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:17:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:17:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:17:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:17:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:17:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:17:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:17:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:17:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:17:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:17:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:17:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:17:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:17:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:17:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:17:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:17:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:17:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:17:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:17:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:17:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:17:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:17:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:17:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:17:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:18:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:18:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:18:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:18:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:18:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:18:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:18:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:18:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:18:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:18:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:18:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:18:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:18:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:18:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:18:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:18:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:18:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:18:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:18:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:18:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:18:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:18:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:18:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:18:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:18:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:18:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:18:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:18:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:19:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:19:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:19:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:19:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:19:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:19:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:19:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:19:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:19:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:19:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:19:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:19:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:19:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:19:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:19:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:19:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:19:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:19:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:19:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:19:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:19:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:19:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:19:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:19:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:19:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:19:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:19:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:20:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:20:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:20:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:20:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:20:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:20:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:20:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:20:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:20:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:20:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:20:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:20:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:20:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:20:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:20:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:20:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:20:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:20:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:20:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:20:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:20:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:20:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:20:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:20:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:20:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:20:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:20:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:21:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:21:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:21:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:21:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:21:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:21:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:21:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:21:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:21:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:21:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:21:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:21:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:21:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:21:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:21:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:21:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:21:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:21:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:21:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:21:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:21:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:21:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:21:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:21:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:21:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:21:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:21:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:21:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:22:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:22:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:22:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:22:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:22:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:22:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:22:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:22:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:22:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:22:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:22:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:22:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:22:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:22:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:22:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:22:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:22:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:22:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:22:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:22:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:22:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:22:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:22:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:22:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:22:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:22:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:22:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:23:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:23:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:23:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:23:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:23:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:23:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:23:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:23:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:23:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:23:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:23:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:23:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:23:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:23:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:23:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:23:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:23:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:23:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:23:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:23:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:23:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:23:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:23:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:23:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:23:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:23:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:23:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:24:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:24:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:24:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:24:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:24:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:24:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:24:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:24:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:24:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:24:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:24:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:24:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:24:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:24:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:24:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:24:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:24:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:24:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:24:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:24:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:24:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:24:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:24:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:24:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:24:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:24:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:24:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:24:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:25:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:25:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:25:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:25:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:25:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:25:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:25:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:25:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:25:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:25:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:25:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:25:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:25:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:25:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:25:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:25:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:25:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:25:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:25:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:25:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:25:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:25:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:25:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:25:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:25:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:25:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:25:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:26:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:26:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:26:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:26:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:26:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:26:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:26:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:26:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:26:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:26:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:26:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:26:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:26:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:26:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:26:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:26:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:26:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:26:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:26:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:26:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:26:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:26:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:26:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:26:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:26:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:26:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:26:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:27:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:27:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:27:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:27:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:27:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:27:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:27:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:27:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:27:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:27:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:27:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:27:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:27:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:27:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:27:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:27:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:27:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:27:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:27:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:27:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:27:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:27:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:27:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:27:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:27:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:27:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:27:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:27:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:28:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:28:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:28:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:28:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:28:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:28:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:28:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:28:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:28:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:28:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:28:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:28:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:28:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:28:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:28:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:28:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:28:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:28:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:28:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:28:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:28:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:28:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:28:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:28:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:28:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:28:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:28:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:29:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:29:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:29:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:29:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:29:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:29:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:29:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:29:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:29:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:29:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:29:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:29:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:29:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:29:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:29:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:29:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:29:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:29:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:29:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:29:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:29:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:29:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:29:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:29:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:29:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:29:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:29:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:30:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:30:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:30:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:30:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:30:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:30:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:30:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:30:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:30:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:30:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:30:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:30:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:30:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:30:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:30:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:30:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:30:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:30:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:30:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:30:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:30:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:30:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:30:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:30:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:30:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:30:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:30:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:30:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:31:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:31:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:31:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:31:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:31:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:31:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:31:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:31:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:31:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:31:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:31:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:31:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:31:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:31:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:31:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:31:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:31:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:31:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:31:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:31:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:31:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:31:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:31:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:31:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:31:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:31:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:31:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:32:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:32:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:32:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:32:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:32:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:32:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:32:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:32:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:32:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:32:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:32:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:32:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:32:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:32:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:32:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:32:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:32:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:32:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:32:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:32:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:32:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:32:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:32:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:32:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:32:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:32:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:32:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:33:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:33:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:33:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:33:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:33:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:33:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:33:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:33:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:33:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:33:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:33:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:33:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:33:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:33:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:33:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:33:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:33:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:33:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:33:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:33:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:33:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:33:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:33:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:33:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:33:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:33:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:33:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:34:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:34:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:34:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:34:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:34:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:34:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:34:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:34:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:34:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:34:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:34:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:34:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:34:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:34:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:34:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:34:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:34:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:34:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:34:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:34:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:34:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:34:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:34:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:34:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:34:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:34:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:34:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:34:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:35:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:35:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:35:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:35:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:35:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:35:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:35:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:35:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:35:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:35:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:35:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:35:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:35:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:35:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:35:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:35:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:35:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:35:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:35:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:35:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:35:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:35:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:35:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:35:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:35:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:35:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:35:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:36:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:36:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:36:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:36:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:36:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:36:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:36:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:36:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:36:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:36:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:36:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:36:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:36:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:36:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:36:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:36:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:36:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:36:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:36:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:36:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:36:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:36:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:36:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:36:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:36:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:36:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:36:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:37:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:37:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:37:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:37:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:37:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:37:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:37:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:37:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:37:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:37:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:37:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:37:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:37:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:37:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:37:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:37:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:37:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:37:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:37:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:37:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:37:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:37:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:37:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:37:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:37:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:37:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:37:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:37:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:38:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:38:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:38:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:38:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:38:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:38:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:38:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:38:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:38:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:38:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:38:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:38:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:38:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:38:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:38:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:38:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:38:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:38:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:38:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:38:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:38:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:38:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:38:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:38:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:38:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:38:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:38:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:39:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:39:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:39:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:39:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:39:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:39:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:39:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:39:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:39:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:39:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:39:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:39:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:39:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:39:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:39:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:39:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:39:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:39:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:39:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:39:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:39:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:39:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:39:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:39:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:39:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:39:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:39:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:40:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:40:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:40:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:40:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:40:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:40:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:40:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:40:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:40:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:40:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:40:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:40:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:40:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:40:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:40:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:40:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:40:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:40:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:40:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:40:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:40:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:40:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:40:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:40:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:40:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:40:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:40:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:40:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:41:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:41:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:41:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:41:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:41:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:41:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:41:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:41:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:41:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:41:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:41:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:41:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:41:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:41:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:41:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:41:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:41:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:41:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:41:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:41:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:41:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:41:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:41:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:41:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:41:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:41:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:41:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:42:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:42:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:42:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:42:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:42:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:42:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:42:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:42:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:42:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:42:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:42:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:42:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:42:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:42:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:42:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:42:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:42:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:42:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:42:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:42:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:42:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:42:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:42:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:42:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:42:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:42:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:42:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:43:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:43:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:43:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:43:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:43:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:43:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:43:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:43:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:43:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:43:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:43:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:43:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:43:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:43:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:43:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:43:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:43:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:43:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:43:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:43:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:43:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:43:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:43:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:43:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:43:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:43:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:43:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:43:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:44:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:44:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:44:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:44:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:44:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:44:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:44:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:44:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:44:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:44:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:44:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:44:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:44:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:44:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:44:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:44:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:44:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:44:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:44:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:44:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:44:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:44:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:44:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:44:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:44:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:44:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:44:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:45:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:45:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:45:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:45:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:45:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:45:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:45:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:45:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:45:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:45:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:45:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:45:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:45:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:45:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:45:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:45:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:45:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:45:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:45:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:45:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:45:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:45:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:45:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:45:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:45:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:45:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:45:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:46:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:46:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:46:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:46:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:46:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:46:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:46:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:46:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:46:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:46:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:46:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:46:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:46:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:46:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:46:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:46:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:46:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:46:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:46:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:46:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:46:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:46:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:46:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:46:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:46:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:46:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:46:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:46:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:47:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:47:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:47:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:47:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:47:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:47:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:47:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:47:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:47:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:47:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:47:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:47:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:47:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:47:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:47:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:47:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:47:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:47:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:47:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:47:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:47:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:47:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:47:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:47:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:47:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:47:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:47:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:48:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:48:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:48:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:48:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:48:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:48:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:48:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:48:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:48:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:48:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:48:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:48:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:48:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:48:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:48:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:48:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:48:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:48:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:48:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:48:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:48:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:48:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:48:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:48:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:48:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:48:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:48:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:49:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:49:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:49:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:49:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:49:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:49:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:49:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:49:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:49:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:49:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:49:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:49:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:49:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:49:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:49:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:49:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:49:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:49:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:49:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:49:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:49:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:49:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:49:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:49:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:49:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:49:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:49:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:50:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:50:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:50:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:50:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:50:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:50:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:50:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:50:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:50:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:50:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:50:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:50:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:50:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:50:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:50:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:50:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:50:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:50:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:50:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:50:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:50:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:50:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:50:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:50:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:50:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:50:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:50:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:50:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:51:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:51:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:51:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:51:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:51:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:51:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:51:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:51:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:51:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:51:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:51:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:51:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:51:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:51:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:51:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:51:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:51:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:51:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:51:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:51:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:51:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:51:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:51:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:51:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:51:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:51:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:51:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:52:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:52:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:52:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:52:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:52:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:52:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:52:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:52:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:52:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:52:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:52:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:52:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:52:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:52:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:52:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:52:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:52:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:52:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:52:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:52:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:52:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:52:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:52:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:52:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:52:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:52:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:52:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:53:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:53:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:53:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:53:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:53:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:53:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:53:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:53:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:53:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:53:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:53:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:53:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:53:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:53:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:53:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:53:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:53:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:53:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:53:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:53:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:53:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:53:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:53:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:53:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:53:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:53:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:53:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:53:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:54:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:54:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:54:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:54:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:54:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:54:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:54:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:54:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:54:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:54:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:54:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:54:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:54:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:54:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:54:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:54:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:54:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:54:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:54:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:54:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:54:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:54:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:54:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:54:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:54:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:54:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:54:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:55:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:55:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:55:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:55:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:55:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:55:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:55:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:55:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:55:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:55:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:55:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:55:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:55:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:55:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:55:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:55:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:55:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:55:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:55:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:55:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:55:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:55:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:55:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:55:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:55:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:55:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:55:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:56:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:56:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:56:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:56:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:56:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:56:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:56:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:56:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:56:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:56:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:56:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:56:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:56:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:56:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:56:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:56:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:56:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:56:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:56:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:56:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:56:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:56:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:56:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:56:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:56:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:56:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:56:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:56:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:57:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:57:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:57:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:57:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:57:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:57:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:57:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:57:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:57:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:57:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:57:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:57:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:57:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:57:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:57:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:57:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:57:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:57:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:57:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:57:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:57:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:57:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:57:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:57:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:57:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:57:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:57:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:58:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:58:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:58:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:58:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:58:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:58:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:58:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:58:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:58:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:58:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:58:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:58:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:58:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:58:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:58:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:58:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:58:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:58:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:58:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:58:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:58:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:58:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:58:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:58:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:58:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:58:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:58:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:59:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:59:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:59:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:59:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:59:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:59:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:59:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:59:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:59:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:59:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:59:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:59:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:59:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:59:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:59:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:59:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:59:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:59:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:59:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 17:59:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:59:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:59:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:59:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:59:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 17:59:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 17:59:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 17:59:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 17:59:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:00:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:00:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:00:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:00:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:00:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:00:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:00:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:00:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:00:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:00:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:00:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:00:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:00:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:00:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:00:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:00:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:00:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:00:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:00:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:00:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:00:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:00:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:00:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:00:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:00:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:00:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:00:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:01:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:01:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:01:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:01:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:01:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:01:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:01:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:01:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:01:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:01:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:01:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:01:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:01:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:01:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:01:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:01:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:01:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:01:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:01:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:01:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:01:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:01:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:01:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:01:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:01:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:01:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:01:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:02:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:02:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:02:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:02:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:02:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:02:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:02:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:02:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:02:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:02:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:02:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:02:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:02:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:02:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:02:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:02:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:02:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:02:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:02:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:02:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:02:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:02:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:02:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:02:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:02:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:02:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:02:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:02:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:03:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:03:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:03:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:03:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:03:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:03:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:03:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:03:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:03:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:03:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:03:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:03:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:03:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:03:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:03:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:03:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:03:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:03:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:03:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:03:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:03:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:03:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:03:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:03:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:03:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:03:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:03:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:04:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:04:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:04:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:04:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:04:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:04:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:04:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:04:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:04:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:04:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:04:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:04:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:04:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:04:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:04:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:04:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:04:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:04:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:04:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:04:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:04:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:04:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:04:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:04:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:04:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:04:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:04:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:05:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:05:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:05:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:05:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:05:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:05:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:05:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:05:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:05:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:05:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:05:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:05:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:05:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:05:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:05:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:05:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:05:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:05:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:05:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:05:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:05:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:05:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:05:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:05:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:05:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:05:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:05:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:06:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:06:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:06:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:06:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:06:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:06:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:06:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:06:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:06:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:06:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:06:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:06:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:06:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:06:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:06:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:06:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:06:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:06:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:06:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:06:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:06:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:06:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:06:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:06:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:06:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:06:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:06:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:06:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:07:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:07:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:07:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:07:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:07:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:07:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:07:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:07:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:07:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:07:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:07:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:07:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:07:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:07:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:07:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:07:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:07:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:07:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:07:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:07:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:07:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:07:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:07:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:07:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:07:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:07:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:07:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:08:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:08:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:08:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:08:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:08:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:08:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:08:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:08:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:08:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:08:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:08:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:08:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:08:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:08:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:08:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:08:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:08:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:08:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:08:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:08:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:08:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:08:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:08:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:08:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:08:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:08:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:08:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:09:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:09:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:09:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:09:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:09:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:09:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:09:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:09:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:09:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:09:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:09:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:09:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:09:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:09:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:09:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:09:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:09:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:09:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:09:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:09:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:09:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:09:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:09:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:09:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:09:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:09:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:09:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:09:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:10:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:10:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:10:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:10:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:10:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:10:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:10:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:10:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:10:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:10:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:10:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:10:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:10:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:10:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:10:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:10:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:10:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:10:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:10:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:10:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:10:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:10:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:10:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:10:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:10:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:10:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:10:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:11:00 rdma06 systemd[1]: Starting dnf makecache...
Apr 16 18:11:01 rdma06 dnf[1119435]: Updating Subscription Management repositories.
Apr 16 18:11:01 rdma06 dnf[1119435]: Unable to read consumer identity
Apr 16 18:11:01 rdma06 dnf[1119435]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 16 18:11:02 rdma06 dnf[1119435]: Copr repo for fmf owned by psss                 2.4 kB/s | 3.3 kB     00:01
Apr 16 18:11:02 rdma06 dnf[1119435]: beaker-AppStream                                 35 kB/s | 168  B     00:00
Apr 16 18:11:02 rdma06 dnf[1119435]: Errors during downloading metadata for repository 'beaker-AppStream':
Apr 16 18:11:02 rdma06 dnf[1119435]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 18:11:02 rdma06 dnf[1119435]: Error: Failed to download metadata for repo 'beaker-AppStream': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 18:11:02 rdma06 dnf[1119435]: beaker-AppStream-debuginfo                       61 kB/s | 168  B     00:00
Apr 16 18:11:02 rdma06 dnf[1119435]: Errors during downloading metadata for repository 'beaker-AppStream-debuginfo':
Apr 16 18:11:02 rdma06 dnf[1119435]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 18:11:02 rdma06 dnf[1119435]: Error: Failed to download metadata for repo 'beaker-AppStream-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 18:11:02 rdma06 dnf[1119435]: beaker-BaseOS                                    62 kB/s | 168  B     00:00
Apr 16 18:11:02 rdma06 dnf[1119435]: Errors during downloading metadata for repository 'beaker-BaseOS':
Apr 16 18:11:02 rdma06 dnf[1119435]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 18:11:02 rdma06 dnf[1119435]: Error: Failed to download metadata for repo 'beaker-BaseOS': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 18:11:02 rdma06 dnf[1119435]: beaker-BaseOS-debuginfo                          66 kB/s | 168  B     00:00
Apr 16 18:11:02 rdma06 dnf[1119435]: Errors during downloading metadata for repository 'beaker-BaseOS-debuginfo':
Apr 16 18:11:02 rdma06 dnf[1119435]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 18:11:02 rdma06 dnf[1119435]: Error: Failed to download metadata for repo 'beaker-BaseOS-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 18:11:02 rdma06 dnf[1119435]: beaker-CRB                                       61 kB/s | 168  B     00:00
Apr 16 18:11:02 rdma06 dnf[1119435]: Errors during downloading metadata for repository 'beaker-CRB':
Apr 16 18:11:02 rdma06 dnf[1119435]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 18:11:02 rdma06 dnf[1119435]: Error: Failed to download metadata for repo 'beaker-CRB': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 18:11:02 rdma06 dnf[1119435]: beaker-CRB-debuginfo                             58 kB/s | 168  B     00:00
Apr 16 18:11:02 rdma06 dnf[1119435]: Errors during downloading metadata for repository 'beaker-CRB-debuginfo':
Apr 16 18:11:02 rdma06 dnf[1119435]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 18:11:02 rdma06 dnf[1119435]: Error: Failed to download metadata for repo 'beaker-CRB-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 18:11:02 rdma06 dnf[1119435]: beaker-HighAvailability                          64 kB/s | 168  B     00:00
Apr 16 18:11:02 rdma06 dnf[1119435]: Errors during downloading metadata for repository 'beaker-HighAvailability':
Apr 16 18:11:02 rdma06 dnf[1119435]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 18:11:02 rdma06 dnf[1119435]: Error: Failed to download metadata for repo 'beaker-HighAvailability': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 18:11:02 rdma06 dnf[1119435]: beaker-HighAvailability-debuginfo                66 kB/s | 168  B     00:00
Apr 16 18:11:02 rdma06 dnf[1119435]: Errors during downloading metadata for repository 'beaker-HighAvailability-debuginfo':
Apr 16 18:11:02 rdma06 dnf[1119435]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 18:11:02 rdma06 dnf[1119435]: Error: Failed to download metadata for repo 'beaker-HighAvailability-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 18:11:02 rdma06 dnf[1119435]: beaker-NFV                                       56 kB/s | 168  B     00:00
Apr 16 18:11:02 rdma06 dnf[1119435]: Errors during downloading metadata for repository 'beaker-NFV':
Apr 16 18:11:02 rdma06 dnf[1119435]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 18:11:02 rdma06 dnf[1119435]: Error: Failed to download metadata for repo 'beaker-NFV': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 18:11:02 rdma06 dnf[1119435]: beaker-NFV-debuginfo                             60 kB/s | 168  B     00:00
Apr 16 18:11:02 rdma06 dnf[1119435]: Errors during downloading metadata for repository 'beaker-NFV-debuginfo':
Apr 16 18:11:02 rdma06 dnf[1119435]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 18:11:02 rdma06 dnf[1119435]: Error: Failed to download metadata for repo 'beaker-NFV-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 18:11:02 rdma06 dnf[1119435]: beaker-RT                                        61 kB/s | 168  B     00:00
Apr 16 18:11:02 rdma06 dnf[1119435]: Errors during downloading metadata for repository 'beaker-RT':
Apr 16 18:11:02 rdma06 dnf[1119435]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 18:11:02 rdma06 dnf[1119435]: Error: Failed to download metadata for repo 'beaker-RT': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 18:11:02 rdma06 dnf[1119435]: beaker-RT-debuginfo                              63 kB/s | 168  B     00:00
Apr 16 18:11:02 rdma06 dnf[1119435]: Errors during downloading metadata for repository 'beaker-RT-debuginfo':
Apr 16 18:11:02 rdma06 dnf[1119435]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 18:11:02 rdma06 dnf[1119435]: Error: Failed to download metadata for repo 'beaker-RT-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 18:11:02 rdma06 dnf[1119435]: beaker-ResilientStorage                          66 kB/s | 168  B     00:00
Apr 16 18:11:02 rdma06 dnf[1119435]: Errors during downloading metadata for repository 'beaker-ResilientStorage':
Apr 16 18:11:02 rdma06 dnf[1119435]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 18:11:02 rdma06 dnf[1119435]: Error: Failed to download metadata for repo 'beaker-ResilientStorage': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 18:11:02 rdma06 dnf[1119435]: beaker-ResilientStorage-debuginfo                65 kB/s | 168  B     00:00
Apr 16 18:11:02 rdma06 dnf[1119435]: Errors during downloading metadata for repository 'beaker-ResilientStorage-debuginfo':
Apr 16 18:11:02 rdma06 dnf[1119435]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 18:11:02 rdma06 dnf[1119435]: Error: Failed to download metadata for repo 'beaker-ResilientStorage-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 18:11:02 rdma06 dnf[1119435]: beaker-SAP                                       65 kB/s | 168  B     00:00
Apr 16 18:11:02 rdma06 dnf[1119435]: Errors during downloading metadata for repository 'beaker-SAP':
Apr 16 18:11:02 rdma06 dnf[1119435]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 18:11:02 rdma06 dnf[1119435]: Error: Failed to download metadata for repo 'beaker-SAP': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 18:11:02 rdma06 dnf[1119435]: beaker-SAP-debuginfo                             64 kB/s | 168  B     00:00
Apr 16 18:11:02 rdma06 dnf[1119435]: Errors during downloading metadata for repository 'beaker-SAP-debuginfo':
Apr 16 18:11:02 rdma06 dnf[1119435]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 18:11:02 rdma06 dnf[1119435]: Error: Failed to download metadata for repo 'beaker-SAP-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 18:11:02 rdma06 dnf[1119435]: beaker-SAPHANA                                   64 kB/s | 168  B     00:00
Apr 16 18:11:02 rdma06 dnf[1119435]: Errors during downloading metadata for repository 'beaker-SAPHANA':
Apr 16 18:11:02 rdma06 dnf[1119435]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 18:11:02 rdma06 dnf[1119435]: Error: Failed to download metadata for repo 'beaker-SAPHANA': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 18:11:02 rdma06 dnf[1119435]: beaker-SAPHANA-debuginfo                         63 kB/s | 168  B     00:00
Apr 16 18:11:02 rdma06 dnf[1119435]: Errors during downloading metadata for repository 'beaker-SAPHANA-debuginfo':
Apr 16 18:11:02 rdma06 dnf[1119435]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 18:11:02 rdma06 dnf[1119435]: Error: Failed to download metadata for repo 'beaker-SAPHANA-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 18:11:03 rdma06 dnf[1119435]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr 16 18:11:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:11:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:11:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:11:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:11:04 rdma06 dnf[1119435]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr 16 18:11:04 rdma06 dnf[1119435]: Ignoring repositories: beaker-AppStream, beaker-AppStream-debuginfo, beaker-BaseOS, beaker-BaseOS-debuginfo, beaker-CRB, beaker-CRB-debuginfo, beaker-HighAvailability, beaker-HighAvailability-debuginfo, beaker-NFV, beaker-NFV-debuginfo, beaker-RT, beaker-RT-debuginfo, beaker-ResilientStorage, beaker-ResilientStorage-debuginfo, beaker-SAP, beaker-SAP-debuginfo, beaker-SAPHANA, beaker-SAPHANA-debuginfo
Apr 16 18:11:04 rdma06 dnf[1119435]: Metadata cache created.
Apr 16 18:11:04 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 16 18:11:04 rdma06 systemd[1]: Finished dnf makecache.
Apr 16 18:11:04 rdma06 systemd[1]: dnf-makecache.service: Consumed 1.003s CPU time.
Apr 16 18:11:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:11:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:11:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:11:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:11:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:11:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:11:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:11:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:11:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:11:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:11:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:11:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:11:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:11:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:11:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:11:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:11:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:11:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:11:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:11:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:11:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:11:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:11:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:12:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:12:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:12:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:12:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:12:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:12:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:12:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:12:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:12:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:12:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:12:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:12:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:12:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:12:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:12:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:12:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:12:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:12:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:12:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:12:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:12:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:12:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:12:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:12:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:12:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:12:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:12:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:12:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:13:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:13:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:13:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:13:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:13:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:13:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:13:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:13:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:13:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:13:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:13:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:13:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:13:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:13:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:13:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:13:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:13:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:13:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:13:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:13:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:13:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:13:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:13:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:13:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:13:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:13:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:13:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:14:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:14:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:14:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:14:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:14:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:14:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:14:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:14:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:14:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:14:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:14:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:14:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:14:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:14:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:14:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:14:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:14:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:14:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:14:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:14:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:14:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:14:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:14:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:14:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:14:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:14:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:14:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:15:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:15:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:15:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:15:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:15:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:15:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:15:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:15:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:15:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:15:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:15:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:15:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:15:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:15:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:15:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:15:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:15:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:15:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:15:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:15:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:15:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:15:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:15:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:15:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:15:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:15:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:15:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:15:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:16:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:16:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:16:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:16:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:16:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:16:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:16:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:16:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:16:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:16:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:16:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:16:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:16:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:16:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:16:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:16:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:16:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:16:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:16:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:16:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:16:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:16:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:16:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:16:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:16:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:16:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:16:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:17:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:17:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:17:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:17:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:17:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:17:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:17:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:17:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:17:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:17:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:17:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:17:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:17:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:17:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:17:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:17:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:17:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:17:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:17:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:17:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:17:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:17:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:17:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:17:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:17:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:17:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:17:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:18:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:18:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:18:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:18:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:18:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:18:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:18:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:18:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:18:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:18:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:18:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:18:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:18:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:18:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:18:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:18:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:18:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:18:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:18:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:18:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:18:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:18:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:18:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:18:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:18:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:18:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:18:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:18:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:19:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:19:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:19:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:19:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:19:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:19:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:19:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:19:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:19:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:19:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:19:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:19:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:19:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:19:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:19:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:19:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:19:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:19:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:19:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:19:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:19:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:19:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:19:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:19:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:19:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:19:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:19:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:20:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:20:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:20:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:20:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:20:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:20:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:20:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:20:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:20:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:20:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:20:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:20:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:20:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:20:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:20:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:20:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:20:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:20:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:20:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:20:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:20:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:20:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:20:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:20:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:20:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:20:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:20:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:21:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:21:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:21:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:21:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:21:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:21:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:21:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:21:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:21:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:21:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:21:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:21:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:21:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:21:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:21:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:21:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:21:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:21:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:21:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:21:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:21:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:21:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:21:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:21:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:21:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:21:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:21:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:21:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:22:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:22:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:22:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:22:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:22:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:22:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:22:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:22:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:22:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:22:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:22:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:22:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:22:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:22:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:22:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:22:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:22:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:22:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:22:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:22:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:22:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:22:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:22:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:22:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:22:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:22:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:22:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:23:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:23:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:23:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:23:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:23:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:23:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:23:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:23:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:23:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:23:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:23:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:23:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:23:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:23:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:23:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:23:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:23:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:23:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:23:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:23:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:23:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:23:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:23:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:23:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:23:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:23:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:23:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:24:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:24:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:24:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:24:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:24:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:24:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:24:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:24:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:24:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:24:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:24:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:24:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:24:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:24:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:24:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:24:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:24:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:24:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:24:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:24:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:24:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:24:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:24:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:24:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:24:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:24:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:24:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:25:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:25:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:25:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:25:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:25:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:25:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:25:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:25:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:25:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:25:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:25:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:25:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:25:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:25:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:25:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:25:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:25:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:25:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:25:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:25:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:25:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:25:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:25:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:25:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:25:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:25:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:25:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:25:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:26:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:26:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:26:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:26:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:26:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:26:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:26:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:26:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:26:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:26:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:26:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:26:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:26:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:26:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:26:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:26:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:26:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:26:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:26:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:26:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:26:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:26:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:26:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:26:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:26:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:26:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:26:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:27:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:27:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:27:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:27:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:27:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:27:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:27:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:27:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:27:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:27:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:27:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:27:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:27:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:27:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:27:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:27:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:27:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:27:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:27:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:27:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:27:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:27:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:27:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:27:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:27:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:27:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:27:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:28:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:28:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:28:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:28:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:28:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:28:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:28:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:28:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:28:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:28:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:28:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:28:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:28:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:28:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:28:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:28:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:28:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:28:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:28:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:28:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:28:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:28:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:28:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:28:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:28:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:28:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:28:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:28:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:29:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:29:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:29:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:29:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:29:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:29:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:29:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:29:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:29:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:29:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:29:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:29:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:29:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:29:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:29:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:29:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:29:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:29:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:29:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:29:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:29:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:29:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:29:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:29:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:29:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:29:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:29:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:30:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:30:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:30:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:30:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:30:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:30:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:30:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:30:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:30:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:30:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:30:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:30:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:30:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:30:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:30:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:30:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:30:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:30:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:30:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:30:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:30:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:30:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:30:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:30:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:30:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:30:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:30:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:31:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:31:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:31:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:31:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:31:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:31:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:31:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:31:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:31:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:31:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:31:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:31:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:31:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:31:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:31:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:31:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:31:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:31:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:31:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:31:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:31:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:31:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:31:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:31:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:31:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:31:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:31:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:31:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:32:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:32:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:32:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:32:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:32:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:32:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:32:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:32:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:32:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:32:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:32:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:32:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:32:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:32:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:32:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:32:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:32:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:32:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:32:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:32:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:32:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:32:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:32:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:32:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:32:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:32:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:32:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:33:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:33:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:33:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:33:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:33:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:33:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:33:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:33:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:33:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:33:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:33:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:33:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:33:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:33:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:33:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:33:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:33:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:33:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:33:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:33:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:33:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:33:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:33:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:33:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:33:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:33:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:33:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:34:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:34:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:34:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:34:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:34:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:34:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:34:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:34:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:34:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:34:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:34:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:34:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:34:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:34:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:34:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:34:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:34:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:34:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:34:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:34:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:34:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:34:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:34:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:34:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:34:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:34:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:34:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:34:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:35:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:35:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:35:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:35:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:35:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:35:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:35:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:35:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:35:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:35:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:35:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:35:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:35:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:35:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:35:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:35:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:35:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:35:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:35:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:35:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:35:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:35:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:35:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:35:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:35:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:35:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:35:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:36:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:36:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:36:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:36:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:36:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:36:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:36:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:36:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:36:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:36:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:36:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:36:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:36:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:36:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:36:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:36:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:36:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:36:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:36:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:36:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:36:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:36:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:36:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:36:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:36:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:36:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:36:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:37:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:37:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:37:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:37:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:37:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:37:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:37:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:37:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:37:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:37:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:37:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:37:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:37:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:37:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:37:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:37:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:37:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:37:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:37:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:37:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:37:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:37:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:37:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:37:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:37:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:37:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:37:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:37:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:38:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:38:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:38:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:38:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:38:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:38:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:38:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:38:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:38:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:38:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:38:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:38:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:38:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:38:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:38:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:38:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:38:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:38:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:38:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:38:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:38:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:38:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:38:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:38:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:38:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:38:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:38:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:39:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:39:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:39:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:39:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:39:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:39:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:39:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:39:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:39:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:39:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:39:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:39:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:39:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:39:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:39:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:39:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:39:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:39:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:39:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:39:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:39:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:39:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:39:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:39:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:39:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:39:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:39:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:40:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:40:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:40:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:40:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:40:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:40:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:40:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:40:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:40:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:40:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:40:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:40:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:40:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:40:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:40:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:40:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:40:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:40:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:40:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:40:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:40:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:40:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:40:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:40:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:40:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:40:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:40:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:41:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:41:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:41:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:41:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:41:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:41:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:41:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:41:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:41:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:41:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:41:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:41:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:41:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:41:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:41:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:41:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:41:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:41:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:41:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:41:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:41:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:41:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:41:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:41:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:41:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:41:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:41:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:41:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:42:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:42:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:42:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:42:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:42:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:42:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:42:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:42:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:42:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:42:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:42:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:42:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:42:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:42:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:42:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:42:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:42:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:42:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:42:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:42:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:42:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:42:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:42:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:42:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:42:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:42:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:42:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:43:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:43:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:43:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:43:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:43:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:43:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:43:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:43:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:43:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:43:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:43:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:43:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:43:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:43:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:43:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:43:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:43:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:43:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:43:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:43:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:43:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:43:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:43:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:43:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:43:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:43:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:43:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:44:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:44:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:44:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:44:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:44:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:44:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:44:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:44:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:44:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:44:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:44:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:44:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:44:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:44:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:44:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:44:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:44:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:44:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:44:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:44:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:44:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:44:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:44:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:44:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:44:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:44:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:44:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:44:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:45:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:45:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:45:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:45:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:45:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:45:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:45:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:45:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:45:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:45:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:45:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:45:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:45:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:45:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:45:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:45:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:45:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:45:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:45:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:45:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:45:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:45:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:45:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:45:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:45:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:45:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:45:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:46:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:46:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:46:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:46:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:46:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:46:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:46:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:46:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:46:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:46:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:46:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:46:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:46:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:46:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:46:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:46:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:46:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:46:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:46:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:46:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:46:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:46:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:46:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:46:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:46:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:46:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:46:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:47:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:47:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:47:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:47:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:47:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:47:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:47:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:47:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:47:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:47:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:47:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:47:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:47:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:47:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:47:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:47:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:47:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:47:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:47:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:47:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:47:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:47:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:47:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:47:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:47:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:47:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:47:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:47:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:48:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:48:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:48:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:48:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:48:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:48:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:48:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:48:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:48:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:48:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:48:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:48:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:48:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:48:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:48:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:48:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:48:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:48:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:48:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:48:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:48:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:48:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:48:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:48:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:48:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:48:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:48:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:49:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:49:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:49:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:49:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:49:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:49:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:49:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:49:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:49:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:49:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:49:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:49:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:49:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:49:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:49:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:49:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:49:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:49:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:49:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:49:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:49:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:49:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:49:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:49:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:49:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:49:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:49:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:50:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:50:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:50:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:50:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:50:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:50:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:50:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:50:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:50:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:50:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:50:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:50:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:50:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:50:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:50:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:50:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:50:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:50:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:50:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:50:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:50:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:50:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:50:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:50:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:50:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:50:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:50:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:50:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:51:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:51:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:51:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:51:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:51:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:51:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:51:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:51:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:51:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:51:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:51:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:51:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:51:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:51:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:51:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:51:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:51:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:51:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:51:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:51:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:51:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:51:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:51:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:51:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:51:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:51:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:51:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:52:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:52:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:52:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:52:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:52:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:52:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:52:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:52:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:52:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:52:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:52:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:52:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:52:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:52:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:52:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:52:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:52:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:52:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:52:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:52:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:52:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:52:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:52:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:52:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:52:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:52:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:52:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:53:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:53:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:53:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:53:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:53:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:53:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:53:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:53:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:53:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:53:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:53:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:53:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:53:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:53:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:53:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:53:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:53:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:53:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:53:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:53:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:53:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:53:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:53:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:53:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:53:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:53:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:53:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:53:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:54:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:54:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:54:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:54:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:54:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:54:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:54:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:54:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:54:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:54:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:54:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:54:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:54:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:54:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:54:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:54:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:54:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:54:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:54:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:54:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:54:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:54:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:54:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:54:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:54:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:54:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:54:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:55:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:55:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:55:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:55:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:55:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:55:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:55:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:55:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:55:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:55:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:55:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:55:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:55:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:55:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:55:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:55:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:55:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:55:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:55:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:55:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:55:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:55:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:55:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:55:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:55:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:55:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:55:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:56:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:56:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:56:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:56:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:56:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:56:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:56:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:56:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:56:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:56:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:56:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:56:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:56:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:56:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:56:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:56:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:56:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:56:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:56:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:56:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:56:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:56:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:56:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:56:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:56:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:56:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:56:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:57:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:57:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:57:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:57:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:57:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:57:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:57:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:57:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:57:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:57:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:57:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:57:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:57:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:57:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:57:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:57:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:57:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:57:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:57:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:57:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:57:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:57:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:57:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:57:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:57:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:57:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:57:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:57:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:58:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:58:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:58:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:58:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:58:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:58:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:58:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:58:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:58:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:58:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:58:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:58:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:58:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:58:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:58:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:58:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:58:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:58:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:58:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:58:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:58:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:58:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:58:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:58:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:58:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:58:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:58:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:59:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:59:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:59:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:59:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:59:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:59:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:59:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:59:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:59:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:59:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:59:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:59:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:59:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:59:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:59:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:59:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:59:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:59:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:59:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 18:59:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:59:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:59:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:59:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 18:59:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 18:59:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 18:59:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 18:59:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:00:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:00:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:00:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:00:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:00:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:00:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:00:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:00:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:00:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:00:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:00:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:00:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:00:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:00:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:00:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:00:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:00:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:00:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:00:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:00:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:00:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:00:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:00:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:00:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:00:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:00:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:00:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:00:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:01:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:01:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:01:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:01:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:01:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:01:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:01:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:01:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:01:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:01:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:01:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:01:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:01:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:01:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:01:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:01:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:01:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:01:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:01:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:01:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:01:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:01:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:01:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:01:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:01:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:01:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:01:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:02:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:02:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:02:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:02:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:02:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:02:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:02:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:02:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:02:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:02:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:02:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:02:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:02:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:02:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:02:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:02:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:02:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:02:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:02:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:02:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:02:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:02:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:02:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:02:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:02:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:02:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:02:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:03:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:03:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:03:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:03:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:03:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:03:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:03:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:03:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:03:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:03:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:03:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:03:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:03:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:03:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:03:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:03:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:03:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:03:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:03:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:03:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:03:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:03:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:03:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:03:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:03:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:03:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:03:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:03:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:04:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:04:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:04:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:04:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:04:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:04:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:04:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:04:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:04:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:04:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:04:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:04:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:04:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:04:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:04:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:04:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:04:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:04:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:04:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:04:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:04:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:04:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:04:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:04:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:04:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:04:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:04:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:05:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:05:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:05:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:05:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:05:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:05:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:05:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:05:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:05:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:05:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:05:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:05:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:05:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:05:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:05:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:05:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:05:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:05:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:05:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:05:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:05:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:05:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:05:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:05:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:05:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:05:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:05:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:06:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:06:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:06:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:06:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:06:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:06:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:06:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:06:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:06:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:06:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:06:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:06:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:06:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:06:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:06:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:06:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:06:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:06:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:06:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:06:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:06:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:06:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:06:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:06:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:06:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:06:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:06:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:06:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:07:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:07:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:07:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:07:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:07:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:07:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:07:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:07:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:07:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:07:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:07:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:07:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:07:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:07:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:07:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:07:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:07:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:07:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:07:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:07:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:07:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:07:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:07:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:07:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:07:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:07:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:07:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:08:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:08:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:08:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:08:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:08:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:08:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:08:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:08:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:08:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:08:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:08:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:08:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:08:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:08:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:08:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:08:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:08:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:08:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:08:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:08:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:08:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:08:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:08:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:08:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:08:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:08:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:08:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:09:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:09:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:09:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:09:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:09:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:09:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:09:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:09:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:09:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:09:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:09:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:09:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:09:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:09:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:09:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:09:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:09:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:09:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:09:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:09:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:09:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:09:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:09:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:09:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:09:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:09:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:09:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:09:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:10:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:10:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:10:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:10:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:10:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:10:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:10:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:10:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:10:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:10:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:10:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:10:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:10:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:10:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:10:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:10:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:10:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:10:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:10:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:10:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:10:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:10:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:10:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:10:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:10:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:10:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:10:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:11:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:11:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:11:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:11:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:11:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:11:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:11:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:11:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:11:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:11:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:11:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:11:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:11:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:11:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:11:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:11:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:11:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:11:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:11:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:11:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:11:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:11:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:11:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:11:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:11:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:11:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:11:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:12:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:12:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:12:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:12:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:12:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:12:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:12:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:12:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:12:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:12:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:12:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:12:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:12:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:12:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:12:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:12:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:12:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:12:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:12:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:12:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:12:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:12:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:12:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:12:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:12:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:12:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:12:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:13:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:13:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:13:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:13:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:13:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:13:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:13:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:13:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:13:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:13:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:13:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:13:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:13:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:13:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:13:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:13:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:13:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:13:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:13:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:13:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:13:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:13:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:13:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:13:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:13:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:13:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:13:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:13:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:14:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:14:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:14:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:14:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:14:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:14:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:14:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:14:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:14:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:14:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:14:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:14:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:14:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:14:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:14:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:14:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:14:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:14:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:14:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:14:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:14:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:14:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:14:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:14:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:14:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:14:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:14:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:15:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:15:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:15:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:15:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:15:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:15:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:15:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:15:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:15:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:15:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:15:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:15:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:15:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:15:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:15:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:15:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:15:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:15:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:15:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:15:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:15:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:15:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:15:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:15:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:15:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:15:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:15:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:16:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:16:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:16:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:16:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:16:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:16:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:16:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:16:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:16:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:16:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:16:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:16:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:16:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:16:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:16:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:16:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:16:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:16:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:16:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:16:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:16:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:16:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:16:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:16:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:16:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:16:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:16:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:16:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:17:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:17:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:17:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:17:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:17:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:17:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:17:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:17:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:17:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:17:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:17:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:17:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:17:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:17:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:17:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:17:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:17:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:17:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:17:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:17:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:17:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:17:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:17:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:17:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:17:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:17:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:17:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:18:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:18:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:18:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:18:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:18:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:18:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:18:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:18:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:18:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:18:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:18:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:18:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:18:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:18:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:18:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:18:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:18:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:18:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:18:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:18:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:18:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:18:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:18:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:18:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:18:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:18:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:18:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:19:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:19:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:19:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:19:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:19:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:19:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:19:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:19:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:19:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:19:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:19:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:19:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:19:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:19:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:19:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:19:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:19:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:19:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:19:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:19:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:19:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:19:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:19:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:19:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:19:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:19:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:19:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:19:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:20:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:20:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:20:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:20:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:20:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:20:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:20:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:20:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:20:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:20:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:20:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:20:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:20:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:20:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:20:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:20:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:20:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:20:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:20:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:20:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:20:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:20:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:20:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:20:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:20:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:20:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:20:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:21:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:21:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:21:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:21:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:21:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:21:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:21:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:21:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:21:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:21:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:21:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:21:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:21:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:21:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:21:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:21:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:21:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:21:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:21:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:21:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:21:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:21:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:21:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:21:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:21:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:21:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:21:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:22:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:22:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:22:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:22:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:22:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:22:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:22:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:22:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:22:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:22:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:22:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:22:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:22:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:22:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:22:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:22:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:22:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:22:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:22:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:22:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:22:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:22:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:22:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:22:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:22:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:22:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:22:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:22:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:23:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:23:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:23:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:23:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:23:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:23:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:23:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:23:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:23:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:23:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:23:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:23:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:23:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:23:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:23:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:23:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:23:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:23:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:23:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:23:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:23:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:23:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:23:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:23:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:23:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:23:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:23:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:24:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:24:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:24:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:24:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:24:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:24:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:24:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:24:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:24:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:24:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:24:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:24:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:24:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:24:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:24:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:24:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:24:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:24:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:24:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:24:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:24:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:24:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:24:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:24:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:24:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:24:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:24:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:25:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:25:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:25:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:25:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:25:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:25:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:25:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:25:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:25:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:25:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:25:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:25:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:25:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:25:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:25:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:25:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:25:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:25:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:25:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:25:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:25:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:25:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:25:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:25:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:25:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:25:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:25:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:25:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:26:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:26:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:26:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:26:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:26:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:26:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:26:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:26:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:26:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:26:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:26:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:26:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:26:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:26:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:26:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:26:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:26:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:26:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:26:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:26:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:26:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:26:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:26:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:26:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:26:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:26:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:26:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:27:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:27:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:27:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:27:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:27:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:27:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:27:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:27:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:27:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:27:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:27:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:27:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:27:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:27:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:27:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:27:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:27:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:27:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:27:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:27:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:27:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:27:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:27:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:27:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:27:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:27:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:27:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:28:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:28:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:28:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:28:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:28:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:28:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:28:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:28:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:28:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:28:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:28:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:28:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:28:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:28:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:28:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:28:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:28:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:28:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:28:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:28:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:28:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:28:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:28:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:28:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:28:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:28:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:28:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:28:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:29:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:29:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:29:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:29:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:29:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:29:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:29:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:29:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:29:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:29:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:29:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:29:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:29:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:29:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:29:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:29:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:29:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:29:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:29:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:29:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:29:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:29:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:29:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:29:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:29:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:29:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:29:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:30:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:30:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:30:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:30:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:30:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:30:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:30:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:30:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:30:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:30:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:30:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:30:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:30:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:30:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:30:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:30:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:30:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:30:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:30:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:30:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:30:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:30:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:30:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:30:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:30:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:30:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:30:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:31:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:31:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:31:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:31:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:31:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:31:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:31:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:31:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:31:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:31:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:31:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:31:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:31:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:31:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:31:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:31:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:31:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:31:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:31:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:31:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:31:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:31:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:31:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:31:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:31:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:31:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:31:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:32:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:32:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:32:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:32:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:32:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:32:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:32:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:32:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:32:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:32:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:32:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:32:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:32:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:32:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:32:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:32:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:32:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:32:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:32:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:32:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:32:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:32:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:32:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:32:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:32:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:32:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:32:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:32:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:33:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:33:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:33:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:33:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:33:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:33:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:33:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:33:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:33:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:33:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:33:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:33:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:33:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:33:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:33:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:33:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:33:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:33:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:33:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:33:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:33:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:33:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:33:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:33:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:33:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:33:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:33:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:34:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:34:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:34:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:34:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:34:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:34:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:34:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:34:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:34:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:34:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:34:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:34:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:34:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:34:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:34:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:34:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:34:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:34:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:34:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:34:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:34:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:34:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:34:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:34:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:34:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:34:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:34:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:35:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:35:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:35:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:35:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:35:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:35:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:35:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:35:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:35:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:35:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:35:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:35:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:35:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:35:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:35:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:35:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:35:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:35:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:35:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:35:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:35:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:35:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:35:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:35:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:35:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:35:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:35:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:35:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:36:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:36:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:36:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:36:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:36:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:36:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:36:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:36:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:36:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:36:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:36:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:36:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:36:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:36:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:36:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:36:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:36:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:36:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:36:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:36:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:36:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:36:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:36:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:36:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:36:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:36:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:36:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:37:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:37:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:37:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:37:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:37:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:37:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:37:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:37:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:37:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:37:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:37:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:37:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:37:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:37:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:37:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:37:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:37:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:37:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:37:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:37:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:37:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:37:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:37:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:37:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:37:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:37:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:37:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:38:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:38:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:38:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:38:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:38:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:38:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:38:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:38:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:38:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:38:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:38:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:38:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:38:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:38:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:38:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:38:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:38:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:38:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:38:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:38:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:38:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:38:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:38:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:38:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:38:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:38:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:38:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:38:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:39:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:39:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:39:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:39:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:39:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:39:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:39:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:39:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:39:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:39:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:39:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:39:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:39:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:39:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:39:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:39:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:39:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:39:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:39:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:39:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:39:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:39:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:39:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:39:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:39:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:39:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:39:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:40:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:40:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:40:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:40:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:40:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:40:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:40:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:40:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:40:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:40:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:40:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:40:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:40:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:40:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:40:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:40:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:40:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:40:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:40:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:40:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:40:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:40:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:40:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:40:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:40:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:40:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:40:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:41:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:41:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:41:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:41:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:41:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:41:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:41:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:41:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:41:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:41:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:41:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:41:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:41:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:41:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:41:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:41:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:41:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:41:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:41:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:41:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:41:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:41:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:41:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:41:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:41:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:41:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:41:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:41:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:42:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:42:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:42:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:42:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:42:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:42:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:42:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:42:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:42:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:42:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:42:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:42:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:42:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:42:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:42:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:42:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:42:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:42:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:42:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:42:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:42:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:42:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:42:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:42:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:42:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:42:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:42:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:43:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:43:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:43:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:43:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:43:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:43:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:43:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:43:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:43:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:43:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:43:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:43:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:43:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:43:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:43:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:43:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:43:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:43:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:43:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:43:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:43:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:43:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:43:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:43:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:43:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:43:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:43:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:44:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:44:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:44:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:44:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:44:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:44:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:44:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:44:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:44:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:44:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:44:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:44:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:44:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:44:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:44:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:44:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:44:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:44:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:44:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:44:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:44:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:44:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:44:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:44:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:44:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:44:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:44:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:44:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:45:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:45:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:45:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:45:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:45:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:45:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:45:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:45:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:45:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:45:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:45:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:45:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:45:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:45:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:45:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:45:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:45:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:45:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:45:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:45:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:45:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:45:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:45:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:45:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:45:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:45:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:45:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:46:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:46:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:46:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:46:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:46:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:46:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:46:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:46:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:46:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:46:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:46:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:46:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:46:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:46:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:46:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:46:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:46:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:46:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:46:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:46:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:46:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:46:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:46:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:46:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:46:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:46:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:46:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:47:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:47:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:47:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:47:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:47:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:47:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:47:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:47:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:47:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:47:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:47:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:47:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:47:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:47:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:47:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:47:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:47:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:47:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:47:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:47:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:47:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:47:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:47:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:47:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:47:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:47:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:47:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:48:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:48:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:48:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:48:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:48:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:48:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:48:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:48:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:48:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:48:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:48:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:48:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:48:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:48:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:48:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:48:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:48:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:48:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:48:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:48:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:48:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:48:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:48:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:48:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:48:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:48:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:48:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:48:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:49:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:49:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:49:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:49:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:49:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:49:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:49:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:49:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:49:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:49:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:49:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:49:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:49:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:49:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:49:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:49:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:49:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:49:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:49:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:49:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:49:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:49:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:49:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:49:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:49:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:49:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:49:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:50:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:50:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:50:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:50:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:50:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:50:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:50:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:50:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:50:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:50:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:50:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:50:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:50:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:50:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:50:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:50:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:50:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:50:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:50:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:50:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:50:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:50:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:50:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:50:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:50:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:50:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:50:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:51:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:51:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:51:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:51:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:51:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:51:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:51:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:51:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:51:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:51:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:51:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:51:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:51:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:51:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:51:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:51:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:51:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:51:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:51:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:51:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:51:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:51:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:51:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:51:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:51:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:51:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:51:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:51:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:52:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:52:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:52:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:52:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:52:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:52:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:52:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:52:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:52:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:52:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:52:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:52:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:52:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:52:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:52:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:52:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:52:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:52:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:52:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:52:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:52:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:52:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:52:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:52:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:52:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:52:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:52:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:53:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:53:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:53:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:53:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:53:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:53:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:53:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:53:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:53:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:53:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:53:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:53:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:53:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:53:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:53:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:53:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:53:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:53:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:53:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:53:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:53:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:53:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:53:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:53:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:53:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:53:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:53:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:54:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:54:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:54:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:54:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:54:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:54:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:54:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:54:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:54:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:54:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:54:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:54:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:54:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:54:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:54:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:54:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:54:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:54:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:54:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:54:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:54:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:54:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:54:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:54:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:54:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:54:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:54:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:54:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:55:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:55:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:55:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:55:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:55:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:55:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:55:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:55:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:55:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:55:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:55:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:55:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:55:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:55:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:55:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:55:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:55:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:55:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:55:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:55:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:55:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:55:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:55:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:55:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:55:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:55:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:55:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:56:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:56:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:56:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:56:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:56:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:56:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:56:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:56:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:56:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:56:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:56:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:56:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:56:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:56:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:56:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:56:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:56:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:56:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:56:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:56:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:56:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:56:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:56:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:56:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:56:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:56:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:56:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:57:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:57:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:57:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:57:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:57:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:57:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:57:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:57:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:57:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:57:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:57:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:57:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:57:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:57:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:57:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:57:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:57:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:57:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:57:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:57:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:57:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:57:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:57:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:57:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:57:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:57:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:57:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:57:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:58:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:58:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:58:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:58:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:58:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:58:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:58:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:58:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:58:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:58:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:58:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:58:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:58:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:58:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:58:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:58:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:58:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:58:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:58:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:58:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:58:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:58:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:58:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:58:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:58:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:58:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:58:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:59:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:59:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:59:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:59:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:59:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:59:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:59:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:59:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:59:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:59:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:59:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:59:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:59:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:59:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:59:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:59:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:59:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:59:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:59:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:59:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:59:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:59:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 19:59:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 19:59:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 19:59:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 19:59:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 19:59:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:00:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:00:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:00:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:00:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:00:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:00:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:00:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:00:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:00:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:00:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:00:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:00:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:00:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:00:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:00:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:00:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:00:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:00:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:00:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:00:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:00:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:00:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:00:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:00:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:00:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:00:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:00:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:00:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:01:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:01:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:01:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:01:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:01:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:01:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:01:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:01:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:01:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:01:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:01:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:01:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:01:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:01:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:01:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:01:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:01:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:01:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:01:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:01:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:01:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:01:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:01:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:01:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:01:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:01:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:01:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:02:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:02:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:02:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:02:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:02:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:02:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:02:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:02:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:02:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:02:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:02:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:02:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:02:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:02:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:02:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:02:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:02:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:02:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:02:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:02:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:02:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:02:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:02:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:02:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:02:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:02:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:02:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:03:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:03:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:03:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:03:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:03:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:03:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:03:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:03:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:03:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:03:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:03:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:03:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:03:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:03:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:03:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:03:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:03:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:03:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:03:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:03:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:03:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:03:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:03:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:03:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:03:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:03:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:03:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:04:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:04:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:04:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:04:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:04:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:04:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:04:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:04:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:04:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:04:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:04:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:04:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:04:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:04:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:04:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:04:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:04:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:04:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:04:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:04:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:04:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:04:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:04:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:04:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:04:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:04:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:04:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:04:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:05:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:05:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:05:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:05:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:05:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:05:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:05:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:05:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:05:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:05:14 rdma06 systemd[1]: Starting dnf makecache...
Apr 16 20:05:14 rdma06 dnf[1124078]: Updating Subscription Management repositories.
Apr 16 20:05:14 rdma06 dnf[1124078]: Unable to read consumer identity
Apr 16 20:05:14 rdma06 dnf[1124078]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 16 20:05:14 rdma06 dnf[1124078]: Metadata cache refreshed recently.
Apr 16 20:05:14 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 16 20:05:14 rdma06 systemd[1]: Finished dnf makecache.
Apr 16 20:05:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:05:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:05:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:05:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:05:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:05:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:05:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:05:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:05:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:05:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:05:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:05:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:05:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:05:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:05:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:05:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:05:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:05:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:06:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:06:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:06:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:06:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:06:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:06:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:06:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:06:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:06:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:06:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:06:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:06:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:06:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:06:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:06:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:06:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:06:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:06:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:06:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:06:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:06:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:06:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:06:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:06:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:06:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:06:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:06:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:07:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:07:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:07:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:07:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:07:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:07:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:07:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:07:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:07:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:07:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:07:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:07:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:07:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:07:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:07:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:07:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:07:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:07:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:07:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:07:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:07:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:07:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:07:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:07:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:07:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:07:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:07:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:07:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:08:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:08:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:08:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:08:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:08:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:08:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:08:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:08:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:08:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:08:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:08:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:08:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:08:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:08:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:08:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:08:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:08:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:08:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:08:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:08:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:08:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:08:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:08:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:08:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:08:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:08:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:08:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:09:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:09:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:09:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:09:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:09:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:09:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:09:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:09:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:09:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:09:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:09:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:09:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:09:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:09:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:09:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:09:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:09:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:09:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:09:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:09:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:09:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:09:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:09:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:09:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:09:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:09:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:09:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:10:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:10:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:10:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:10:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:10:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:10:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:10:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:10:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:10:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:10:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:10:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:10:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:10:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:10:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:10:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:10:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:10:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:10:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:10:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:10:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:10:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:10:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:10:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:10:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:10:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:10:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:10:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:10:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:11:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:11:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:11:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:11:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:11:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:11:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:11:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:11:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:11:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:11:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:11:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:11:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:11:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:11:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:11:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:11:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:11:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:11:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:11:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:11:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:11:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:11:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:11:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:11:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:11:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:11:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:11:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:12:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:12:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:12:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:12:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:12:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:12:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:12:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:12:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:12:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:12:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:12:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:12:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:12:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:12:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:12:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:12:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:12:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:12:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:12:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:12:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:12:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:12:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:12:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:12:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:12:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:12:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:12:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:13:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:13:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:13:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:13:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:13:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:13:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:13:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:13:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:13:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:13:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:13:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:13:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:13:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:13:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:13:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:13:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:13:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:13:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:13:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:13:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:13:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:13:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:13:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:13:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:13:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:13:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:13:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:13:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:14:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:14:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:14:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:14:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:14:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:14:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:14:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:14:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:14:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:14:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:14:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:14:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:14:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:14:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:14:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:14:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:14:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:14:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:14:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:14:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:14:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:14:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:14:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:14:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:14:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:14:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:14:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:15:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:15:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:15:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:15:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:15:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:15:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:15:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:15:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:15:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:15:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:15:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:15:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:15:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:15:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:15:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:15:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:15:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:15:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:15:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:15:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:15:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:15:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:15:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:15:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:15:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:15:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:15:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:16:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:16:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:16:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:16:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:16:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:16:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:16:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:16:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:16:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:16:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:16:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:16:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:16:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:16:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:16:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:16:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:16:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:16:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:16:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:16:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:16:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:16:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:16:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:16:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:16:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:16:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:16:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:16:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:17:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:17:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:17:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:17:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:17:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:17:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:17:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:17:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:17:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:17:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:17:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:17:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:17:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:17:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:17:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:17:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:17:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:17:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:17:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:17:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:17:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:17:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:17:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:17:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:17:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:17:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:17:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:18:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:18:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:18:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:18:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:18:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:18:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:18:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:18:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:18:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:18:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:18:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:18:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:18:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:18:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:18:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:18:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:18:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:18:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:18:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:18:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:18:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:18:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:18:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:18:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:18:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:18:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:18:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:19:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:19:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:19:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:19:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:19:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:19:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:19:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:19:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:19:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:19:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:19:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:19:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:19:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:19:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:19:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:19:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:19:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:19:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:19:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:19:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:19:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:19:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:19:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:19:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:19:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:19:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:19:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:20:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:20:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:20:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:20:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:20:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:20:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:20:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:20:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:20:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:20:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:20:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:20:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:20:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:20:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:20:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:20:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:20:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:20:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:20:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:20:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:20:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:20:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:20:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:20:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:20:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:20:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:20:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:20:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:21:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:21:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:21:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:21:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:21:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:21:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:21:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:21:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:21:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:21:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:21:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:21:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:21:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:21:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:21:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:21:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:21:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:21:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:21:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:21:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:21:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:21:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:21:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:21:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:21:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:21:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:21:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:22:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:22:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:22:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:22:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:22:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:22:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:22:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:22:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:22:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:22:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:22:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:22:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:22:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:22:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:22:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:22:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:22:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:22:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:22:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:22:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:22:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:22:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:22:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:22:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:22:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:22:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:22:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:23:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:23:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:23:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:23:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:23:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:23:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:23:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:23:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:23:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:23:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:23:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:23:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:23:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:23:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:23:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:23:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:23:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:23:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:23:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:23:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:23:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:23:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:23:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:23:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:23:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:23:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:23:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:23:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:24:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:24:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:24:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:24:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:24:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:24:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:24:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:24:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:24:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:24:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:24:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:24:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:24:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:24:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:24:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:24:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:24:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:24:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:24:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:24:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:24:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:24:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:24:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:24:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:24:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:24:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:24:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:25:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:25:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:25:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:25:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:25:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:25:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:25:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:25:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:25:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:25:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:25:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:25:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:25:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:25:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:25:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:25:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:25:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:25:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:25:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:25:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:25:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:25:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:25:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:25:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:25:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:25:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:25:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:26:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:26:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:26:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:26:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:26:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:26:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:26:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:26:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:26:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:26:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:26:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:26:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:26:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:26:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:26:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:26:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:26:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:26:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:26:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:26:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:26:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:26:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:26:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:26:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:26:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:26:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:26:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:26:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:27:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:27:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:27:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:27:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:27:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:27:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:27:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:27:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:27:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:27:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:27:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:27:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:27:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:27:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:27:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:27:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:27:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:27:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:27:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:27:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:27:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:27:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:27:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:27:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:27:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:27:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:27:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:28:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:28:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:28:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:28:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:28:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:28:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:28:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:28:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:28:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:28:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:28:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:28:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:28:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:28:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:28:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:28:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:28:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:28:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:28:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:28:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:28:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:28:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:28:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:28:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:28:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:28:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:28:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:29:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:29:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:29:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:29:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:29:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:29:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:29:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:29:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:29:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:29:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:29:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:29:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:29:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:29:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:29:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:29:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:29:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:29:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:29:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:29:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:29:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:29:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:29:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:29:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:29:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:29:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:29:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:29:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:30:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:30:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:30:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:30:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:30:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:30:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:30:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:30:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:30:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:30:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:30:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:30:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:30:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:30:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:30:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:30:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:30:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:30:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:30:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:30:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:30:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:30:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:30:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:30:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:30:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:30:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:30:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:31:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:31:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:31:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:31:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:31:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:31:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:31:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:31:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:31:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:31:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:31:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:31:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:31:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:31:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:31:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:31:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:31:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:31:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:31:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:31:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:31:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:31:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:31:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:31:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:31:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:31:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:31:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:32:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:32:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:32:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:32:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:32:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:32:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:32:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:32:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:32:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:32:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:32:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:32:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:32:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:32:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:32:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:32:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:32:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:32:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:32:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:32:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:32:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:32:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:32:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:32:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:32:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:32:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:32:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:32:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:33:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:33:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:33:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:33:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:33:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:33:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:33:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:33:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:33:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:33:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:33:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:33:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:33:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:33:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:33:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:33:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:33:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:33:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:33:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:33:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:33:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:33:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:33:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:33:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:33:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:33:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:33:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:34:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:34:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:34:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:34:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:34:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:34:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:34:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:34:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:34:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:34:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:34:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:34:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:34:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:34:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:34:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:34:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:34:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:34:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:34:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:34:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:34:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:34:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:34:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:34:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:34:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:34:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:34:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:35:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:35:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:35:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:35:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:35:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:35:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:35:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:35:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:35:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:35:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:35:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:35:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:35:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:35:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:35:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:35:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:35:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:35:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:35:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:35:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:35:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:35:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:35:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:35:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:35:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:35:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:35:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:35:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:36:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:36:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:36:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:36:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:36:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:36:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:36:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:36:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:36:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:36:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:36:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:36:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:36:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:36:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:36:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:36:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:36:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:36:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:36:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:36:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:36:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:36:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:36:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:36:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:36:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:36:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:36:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:37:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:37:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:37:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:37:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:37:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:37:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:37:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:37:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:37:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:37:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:37:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:37:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:37:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:37:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:37:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:37:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:37:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:37:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:37:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:37:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:37:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:37:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:37:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:37:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:37:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:37:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:37:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:38:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:38:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:38:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:38:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:38:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:38:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:38:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:38:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:38:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:38:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:38:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:38:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:38:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:38:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:38:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:38:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:38:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:38:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:38:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:38:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:38:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:38:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:38:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:38:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:38:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:38:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:38:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:39:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:39:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:39:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:39:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:39:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:39:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:39:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:39:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:39:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:39:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:39:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:39:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:39:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:39:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:39:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:39:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:39:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:39:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:39:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:39:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:39:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:39:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:39:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:39:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:39:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:39:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:39:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:39:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:40:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:40:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:40:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:40:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:40:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:40:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:40:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:40:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:40:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:40:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:40:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:40:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:40:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:40:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:40:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:40:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:40:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:40:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:40:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:40:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:40:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:40:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:40:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:40:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:40:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:40:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:40:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:41:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:41:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:41:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:41:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:41:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:41:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:41:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:41:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:41:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:41:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:41:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:41:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:41:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:41:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:41:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:41:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:41:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:41:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:41:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:41:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:41:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:41:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:41:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:41:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:41:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:41:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:41:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:42:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:42:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:42:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:42:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:42:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:42:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:42:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:42:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:42:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:42:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:42:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:42:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:42:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:42:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:42:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:42:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:42:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:42:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:42:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:42:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:42:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:42:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:42:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:42:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:42:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:42:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:42:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:42:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:43:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:43:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:43:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:43:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:43:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:43:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:43:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:43:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:43:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:43:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:43:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:43:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:43:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:43:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:43:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:43:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:43:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:43:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:43:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:43:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:43:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:43:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:43:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:43:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:43:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:43:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:43:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:44:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:44:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:44:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:44:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:44:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:44:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:44:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:44:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:44:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:44:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:44:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:44:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:44:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:44:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:44:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:44:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:44:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:44:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:44:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:44:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:44:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:44:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:44:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:44:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:44:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:44:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:44:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:45:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:45:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:45:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:45:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:45:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:45:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:45:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:45:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:45:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:45:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:45:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:45:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:45:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:45:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:45:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:45:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:45:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:45:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:45:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:45:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:45:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:45:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:45:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:45:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:45:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:45:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:45:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:45:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:46:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:46:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:46:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:46:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:46:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:46:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:46:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:46:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:46:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:46:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:46:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:46:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:46:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:46:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:46:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:46:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:46:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:46:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:46:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:46:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:46:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:46:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:46:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:46:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:46:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:46:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:46:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:47:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:47:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:47:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:47:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:47:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:47:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:47:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:47:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:47:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:47:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:47:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:47:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:47:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:47:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:47:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:47:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:47:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:47:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:47:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:47:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:47:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:47:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:47:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:47:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:47:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:47:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:47:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:48:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:48:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:48:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:48:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:48:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:48:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:48:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:48:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:48:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:48:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:48:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:48:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:48:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:48:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:48:33 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:48:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:48:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:48:33 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:48:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:48:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:48:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:48:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:48:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:48:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:48:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:48:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:48:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:48:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:49:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:49:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:49:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:49:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:49:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:49:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:49:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:49:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:49:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:49:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:49:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:49:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:49:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:49:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:49:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:49:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:49:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:49:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:49:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:49:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:49:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:49:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:49:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:49:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:49:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:49:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:49:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:50:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:50:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:50:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:50:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:50:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:50:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:50:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:50:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:50:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:50:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:50:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:50:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:50:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:50:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:50:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:50:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:50:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:50:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:50:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:50:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:50:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:50:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:50:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:50:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:50:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:50:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:50:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:51:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:51:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:51:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:51:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:51:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:51:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:51:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:51:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:51:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:51:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:51:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:51:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:51:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:51:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:51:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:51:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:51:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:51:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:51:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:51:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:51:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:51:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:51:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:51:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:51:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:51:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:51:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:51:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:52:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:52:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:52:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:52:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:52:13 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:52:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:52:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:52:13 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:52:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:52:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:52:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:52:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:52:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:52:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:52:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:52:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:52:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:52:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:52:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:52:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:52:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:52:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:52:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:52:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:52:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:52:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:52:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:53:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:53:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:53:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:53:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:53:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:53:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:53:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:53:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:53:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:53:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:53:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:53:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:53:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:53:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:53:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:53:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:53:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:53:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:53:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:53:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:53:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:53:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:53:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:53:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:53:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:53:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:53:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:54:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:54:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:54:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:54:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:54:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:54:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:54:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:54:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:54:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:54:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:54:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:54:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:54:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:54:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:54:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:54:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:54:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:54:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:54:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:54:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:54:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:54:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:54:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:54:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:54:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:54:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:54:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:55:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:55:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:55:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:55:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:55:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:55:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:55:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:55:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:55:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:55:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:55:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:55:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:55:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:55:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:55:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:55:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:55:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:55:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:55:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:55:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:55:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:55:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:55:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:55:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:55:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:55:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:55:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:55:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:56:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:56:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:56:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:56:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:56:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:56:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:56:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:56:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:56:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:56:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:56:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:56:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:56:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:56:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:56:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:56:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:56:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:56:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:56:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:56:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:56:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:56:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:56:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:56:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:56:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:56:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:56:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:57:03 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:57:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:57:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:57:03 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:57:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:57:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:57:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:57:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:57:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:57:23 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:57:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:57:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:57:23 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:57:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:57:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:57:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:57:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:57:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:57:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:57:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:57:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:57:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:57:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:57:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:57:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:57:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:57:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:58:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:58:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:58:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:58:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:58:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:58:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:58:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:58:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:58:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:58:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:58:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:58:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:58:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:58:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:58:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:58:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:58:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:58:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:58:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:58:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:58:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:58:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:58:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:58:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:58:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:58:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:58:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:58:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:59:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:59:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:59:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:59:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:59:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:59:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:59:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:59:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:59:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:59:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:59:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:59:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:59:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:59:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:59:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:59:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:59:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:59:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:59:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:59:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:59:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:59:43 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 20:59:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 20:59:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 20:59:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 20:59:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 20:59:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:00:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:00:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:00:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:00:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:00:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:00:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:00:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:00:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:00:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:00:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:00:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:00:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:00:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:00:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:00:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:00:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:00:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:00:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:00:43 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:00:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:00:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:00:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:00:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:00:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:00:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:00:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:00:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:01:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:01:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:01:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:01:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:01:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:01:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:01:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:01:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:01:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:01:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:01:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:01:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:01:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:01:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:01:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:01:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:01:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:01:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:01:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:01:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:01:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:01:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:01:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:01:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:01:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:01:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:01:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:01:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:02:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:02:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:02:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:02:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:02:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:02:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:02:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:02:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:02:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:02:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:02:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:02:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:02:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:02:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:02:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:02:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:02:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:02:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:02:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:02:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:02:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:02:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:02:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:02:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:02:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:02:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:02:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:03:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:03:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:03:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:03:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:03:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:03:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:03:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:03:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:03:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:03:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:03:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:03:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:03:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:03:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:03:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:03:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:03:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:03:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:03:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:03:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:03:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:03:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:03:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:03:53 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:03:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:03:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:03:53 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:04:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:04:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:04:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:04:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:04:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:04:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:04:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:04:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:04:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:04:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:04:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:04:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:04:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:04:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:04:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:04:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:04:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:04:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:04:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:04:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:04:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:04:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:04:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:04:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:04:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:04:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:04:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:04:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:05:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:05:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:05:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:05:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:05:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:05:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:05:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:05:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:05:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:05:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:05:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:05:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:05:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:05:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:05:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:05:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:05:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:05:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:05:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:05:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:05:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:05:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:05:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:05:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:05:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:05:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:05:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:06:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:06:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:06:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:06:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:06:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:06:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:06:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:06:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:06:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:06:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:06:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:06:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:06:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:06:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:06:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:06:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:06:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:06:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:06:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:06:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:06:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:06:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:06:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:06:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:06:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:06:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:06:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:07:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:07:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:07:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:07:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:07:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:07:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:07:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:07:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:07:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:07:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:07:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:07:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:07:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:07:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:07:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:07:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:07:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:07:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:07:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:07:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:07:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:07:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:07:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:07:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:07:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:07:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:07:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:07:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:08:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:08:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:08:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:08:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:08:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:08:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:08:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:08:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:08:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:08:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:08:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:08:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:08:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:08:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:08:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:08:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:08:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:08:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:08:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:08:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:08:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:08:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:08:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:08:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:08:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:08:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:08:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:09:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:09:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:09:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:09:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:09:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:09:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:09:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:09:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:09:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:09:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:09:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:09:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:09:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:09:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:09:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:09:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:09:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:09:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:09:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:09:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:09:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:09:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:09:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:09:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:09:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:09:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:09:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:10:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:10:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:10:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:10:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:10:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:10:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:10:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:10:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:10:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:10:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:10:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:10:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:10:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:10:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:10:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:10:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:10:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:10:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:10:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:10:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:10:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:10:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:10:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:10:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:10:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:10:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:10:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:11:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:11:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:11:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:11:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:11:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:11:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:11:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:11:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:11:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:11:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:11:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:11:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:11:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:11:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:11:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:11:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:11:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:11:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:11:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:11:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:11:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:11:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:11:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:11:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:11:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:11:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:11:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:11:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:12:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:12:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:12:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:12:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:12:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:12:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:12:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:12:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:12:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:12:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:12:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:12:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:12:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:12:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:12:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:12:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:12:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:12:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:12:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:12:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:12:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:12:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:12:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:12:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:12:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:12:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:12:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:13:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:13:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:13:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:13:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:13:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:13:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:13:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:13:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:13:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:13:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:13:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:13:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:13:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:13:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:13:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:13:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:13:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:13:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:13:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:13:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:13:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:13:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:13:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:13:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:13:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:13:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:13:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:14:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:14:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:14:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:14:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:14:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:14:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:14:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:14:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:14:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:14:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:14:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:14:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:14:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:14:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:14:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:14:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:14:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:14:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:14:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:14:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:14:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:14:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:14:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:14:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:14:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:14:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:14:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:14:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:15:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:15:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:15:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:15:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:15:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:15:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:15:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:15:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:15:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:15:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:15:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:15:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:15:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:15:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:15:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:15:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:15:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:15:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:15:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:15:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:15:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:15:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:15:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:15:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:15:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:15:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:15:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:16:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:16:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:16:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:16:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:16:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:16:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:16:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:16:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:16:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:16:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:16:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:16:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:16:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:16:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:16:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:16:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:16:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:16:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:16:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:16:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:16:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:16:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:16:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:16:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:16:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:16:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:16:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:17:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:17:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:17:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:17:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:17:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:17:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:17:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:17:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:17:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:17:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:17:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:17:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:17:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:17:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:17:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:17:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:17:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:17:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:17:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:17:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:17:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:17:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:17:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:17:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:17:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:17:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:17:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:17:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:18:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:18:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:18:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:18:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:18:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:18:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:18:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:18:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:18:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:18:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:18:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:18:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:18:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:18:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:18:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:18:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:18:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:18:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:18:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:18:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:18:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:18:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:18:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:18:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:18:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:18:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:18:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:19:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:19:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:19:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:19:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:19:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:19:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:19:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:19:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:19:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:19:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:19:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:19:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:19:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:19:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:19:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:19:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:19:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:19:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:19:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:19:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:19:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:19:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:19:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:19:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:19:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:19:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:19:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:20:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:20:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:20:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:20:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:20:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:20:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:20:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:20:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:20:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:20:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:20:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:20:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:20:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:20:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:20:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:20:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:20:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:20:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:20:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:20:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:20:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:20:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:20:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:20:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:20:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:20:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:20:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:20:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:21:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:21:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:21:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:21:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:21:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:21:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:21:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:21:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:21:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:21:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:21:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:21:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:21:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:21:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:21:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:21:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:21:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:21:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:21:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:21:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:21:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:21:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:21:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:21:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:21:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:21:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:21:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:22:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:22:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:22:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:22:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:22:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:22:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:22:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:22:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:22:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:22:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:22:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:22:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:22:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:22:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:22:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:22:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:22:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:22:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:22:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:22:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:22:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:22:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:22:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:22:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:22:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:22:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:22:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:23:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:23:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:23:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:23:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:23:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:23:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:23:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:23:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:23:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:23:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:23:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:23:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:23:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:23:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:23:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:23:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:23:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:23:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:23:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:23:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:23:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:23:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:23:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:23:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:23:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:23:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:23:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:23:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:24:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:24:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:24:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:24:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:24:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:24:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:24:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:24:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:24:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:24:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:24:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:24:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:24:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:24:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:24:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:24:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:24:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:24:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:24:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:24:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:24:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:24:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:24:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:24:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:24:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:24:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:24:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:25:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:25:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:25:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:25:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:25:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:25:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:25:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:25:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:25:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:25:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:25:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:25:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:25:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:25:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:25:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:25:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:25:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:25:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:25:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:25:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:25:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:25:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:25:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:25:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:25:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:25:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:25:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:26:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:26:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:26:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:26:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:26:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:26:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:26:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:26:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:26:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:26:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:26:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:26:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:26:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:26:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:26:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:26:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:26:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:26:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:26:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:26:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:26:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:26:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:26:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:26:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:26:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:26:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:26:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:27:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:27:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:27:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:27:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:27:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:27:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:27:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:27:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:27:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:27:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:27:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:27:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:27:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:27:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:27:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:27:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:27:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:27:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:27:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:27:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:27:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:27:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:27:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:27:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:27:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:27:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:27:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:27:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:28:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:28:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:28:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:28:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:28:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:28:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:28:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:28:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:28:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:28:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:28:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:28:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:28:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:28:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:28:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:28:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:28:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:28:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:28:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:28:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:28:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:28:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:28:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:28:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:28:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:28:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:28:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:29:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:29:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:29:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:29:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:29:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:29:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:29:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:29:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:29:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:29:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:29:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:29:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:29:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:29:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:29:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:29:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:29:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:29:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:29:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:29:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:29:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:29:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:29:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:29:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:29:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:29:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:29:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:30:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:30:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:30:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:30:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:30:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:30:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:30:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:30:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:30:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:30:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:30:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:30:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:30:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:30:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:30:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:30:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:30:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:30:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:30:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:30:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:30:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:30:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:30:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:30:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:30:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:30:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:30:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:30:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:31:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:31:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:31:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:31:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:31:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:31:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:31:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:31:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:31:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:31:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:31:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:31:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:31:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:31:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:31:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:31:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:31:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:31:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:31:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:31:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:31:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:31:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:31:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:31:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:31:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:31:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:31:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:32:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:32:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:32:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:32:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:32:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:32:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:32:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:32:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:32:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:32:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:32:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:32:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:32:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:32:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:32:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:32:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:32:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:32:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:32:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:32:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:32:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:32:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:32:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:32:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:32:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:32:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:32:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:33:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:33:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:33:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:33:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:33:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:33:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:33:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:33:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:33:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:33:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:33:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:33:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:33:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:33:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:33:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:33:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:33:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:33:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:33:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:33:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:33:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:33:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:33:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:33:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:33:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:33:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:33:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:33:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:34:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:34:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:34:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:34:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:34:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:34:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:34:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:34:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:34:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:34:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:34:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:34:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:34:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:34:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:34:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:34:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:34:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:34:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:34:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:34:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:34:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:34:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:34:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:34:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:34:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:34:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:34:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:35:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:35:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:35:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:35:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:35:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:35:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:35:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:35:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:35:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:35:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:35:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:35:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:35:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:35:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:35:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:35:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:35:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:35:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:35:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:35:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:35:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:35:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:35:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:35:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:35:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:35:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:35:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:36:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:36:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:36:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:36:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:36:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:36:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:36:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:36:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:36:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:36:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:36:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:36:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:36:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:36:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:36:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:36:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:36:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:36:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:36:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:36:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:36:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:36:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:36:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:36:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:36:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:36:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:36:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:36:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:37:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:37:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:37:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:37:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:37:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:37:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:37:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:37:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:37:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:37:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:37:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:37:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:37:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:37:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:37:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:37:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:37:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:37:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:37:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:37:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:37:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:37:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:37:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:37:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:37:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:37:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:37:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:38:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:38:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:38:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:38:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:38:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:38:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:38:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:38:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:38:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:38:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:38:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:38:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:38:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:38:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:38:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:38:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:38:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:38:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:38:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:38:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:38:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:38:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:38:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:38:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:38:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:38:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:38:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:39:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:39:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:39:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:39:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:39:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:39:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:39:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:39:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:39:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:39:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:39:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:39:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:39:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:39:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:39:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:39:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:39:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:39:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:39:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:39:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:39:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:39:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:39:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:39:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:39:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:39:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:39:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:39:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:40:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:40:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:40:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:40:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:40:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:40:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:40:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:40:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:40:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:40:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:40:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:40:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:40:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:40:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:40:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:40:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:40:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:40:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:40:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:40:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:40:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:40:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:40:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:40:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:40:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:40:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:40:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:41:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:41:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:41:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:41:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:41:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:41:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:41:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:41:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:41:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:41:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:41:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:41:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:41:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:41:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:41:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:41:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:41:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:41:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:41:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:41:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:41:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:41:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:41:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:41:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:41:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:41:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:41:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:42:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:42:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:42:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:42:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:42:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:42:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:42:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:42:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:42:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:42:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:42:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:42:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:42:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:42:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:42:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:42:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:42:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:42:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:42:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:42:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:42:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:42:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:42:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:42:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:42:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:42:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:42:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:43:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:43:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:43:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:43:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:43:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:43:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:43:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:43:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:43:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:43:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:43:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:43:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:43:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:43:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:43:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:43:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:43:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:43:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:43:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:43:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:43:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:43:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:43:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:43:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:43:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:43:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:43:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:43:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:44:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:44:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:44:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:44:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:44:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:44:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:44:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:44:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:44:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:44:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:44:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:44:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:44:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:44:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:44:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:44:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:44:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:44:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:44:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:44:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:44:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:44:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:44:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:44:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:44:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:44:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:44:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:45:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:45:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:45:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:45:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:45:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:45:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:45:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:45:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:45:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:45:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:45:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:45:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:45:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:45:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:45:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:45:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:45:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:45:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:45:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:45:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:45:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:45:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:45:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:45:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:45:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:45:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:45:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:46:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:46:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:46:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:46:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:46:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:46:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:46:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:46:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:46:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:46:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:46:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:46:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:46:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:46:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:46:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:46:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:46:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:46:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:46:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:46:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:46:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:46:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:46:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:46:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:46:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:46:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:46:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:46:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:47:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:47:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:47:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:47:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:47:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:47:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:47:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:47:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:47:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:47:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:47:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:47:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:47:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:47:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:47:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:47:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:47:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:47:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:47:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:47:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:47:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:47:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:47:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:47:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:47:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:47:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:47:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:48:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:48:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:48:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:48:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:48:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:48:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:48:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:48:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:48:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:48:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:48:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:48:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:48:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:48:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:48:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:48:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:48:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:48:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:48:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:48:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:48:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:48:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:48:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:48:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:48:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:48:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:48:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:49:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:49:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:49:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:49:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:49:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:49:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:49:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:49:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:49:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:49:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:49:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:49:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:49:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:49:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:49:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:49:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:49:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:49:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:49:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:49:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:49:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:49:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:49:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:49:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:49:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:49:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:49:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:49:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:50:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:50:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:50:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:50:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:50:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:50:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:50:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:50:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:50:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:50:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:50:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:50:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:50:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:50:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:50:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:50:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:50:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:50:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:50:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:50:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:50:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:50:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:50:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:50:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:50:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:50:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:50:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:51:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:51:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:51:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:51:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:51:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:51:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:51:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:51:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:51:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:51:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:51:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:51:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:51:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:51:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:51:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:51:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:51:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:51:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:51:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:51:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:51:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:51:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:51:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:51:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:51:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:51:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:51:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:52:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:52:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:52:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:52:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:52:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:52:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:52:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:52:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:52:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:52:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:52:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:52:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:52:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:52:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:52:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:52:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:52:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:52:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:52:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:52:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:52:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:52:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:52:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:52:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:52:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:52:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:52:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:52:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:53:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:53:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:53:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:53:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:53:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:53:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:53:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:53:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:53:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:53:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:53:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:53:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:53:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:53:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:53:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:53:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:53:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:53:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:53:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:53:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:53:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:53:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:53:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:53:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:53:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:53:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:53:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:54:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:54:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:54:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:54:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:54:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:54:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:54:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:54:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:54:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:54:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:54:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:54:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:54:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:54:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:54:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:54:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:54:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:54:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:54:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:54:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:54:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:54:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:54:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:54:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:54:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:54:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:54:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:55:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:55:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:55:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:55:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:55:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:55:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:55:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:55:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:55:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:55:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:55:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:55:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:55:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:55:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:55:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:55:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:55:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:55:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:55:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:55:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:55:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:55:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:55:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:55:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:55:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:55:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:55:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:55:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:56:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:56:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:56:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:56:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:56:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:56:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:56:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:56:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:56:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:56:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:56:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:56:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:56:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:56:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:56:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:56:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:56:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:56:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:56:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:56:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:56:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:56:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:56:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:56:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:56:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:56:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:56:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:57:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:57:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:57:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:57:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:57:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:57:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:57:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:57:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:57:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:57:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:57:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:57:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:57:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:57:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:57:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:57:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:57:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:57:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:57:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:57:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:57:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:57:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:57:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:57:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:57:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:57:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:57:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:58:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:58:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:58:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:58:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:58:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:58:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:58:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:58:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:58:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:58:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:58:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:58:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:58:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:58:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:58:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:58:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:58:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:58:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:58:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:58:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:58:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:58:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:58:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:58:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:58:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:58:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:58:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:58:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:59:00 rdma06 systemd[1]: Starting dnf makecache...
Apr 16 21:59:01 rdma06 dnf[1128680]: Updating Subscription Management repositories.
Apr 16 21:59:01 rdma06 dnf[1128680]: Unable to read consumer identity
Apr 16 21:59:01 rdma06 dnf[1128680]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 16 21:59:02 rdma06 dnf[1128680]: Copr repo for fmf owned by psss                 2.5 kB/s | 3.3 kB     00:01
Apr 16 21:59:02 rdma06 dnf[1128680]: beaker-AppStream                                7.0 kB/s | 168  B     00:00
Apr 16 21:59:02 rdma06 dnf[1128680]: Errors during downloading metadata for repository 'beaker-AppStream':
Apr 16 21:59:02 rdma06 dnf[1128680]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 21:59:02 rdma06 dnf[1128680]: Error: Failed to download metadata for repo 'beaker-AppStream': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 21:59:02 rdma06 dnf[1128680]: beaker-AppStream-debuginfo                       13 kB/s | 168  B     00:00
Apr 16 21:59:02 rdma06 dnf[1128680]: Errors during downloading metadata for repository 'beaker-AppStream-debuginfo':
Apr 16 21:59:02 rdma06 dnf[1128680]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 21:59:02 rdma06 dnf[1128680]: Error: Failed to download metadata for repo 'beaker-AppStream-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 21:59:02 rdma06 dnf[1128680]: beaker-BaseOS                                    57 kB/s | 168  B     00:00
Apr 16 21:59:02 rdma06 dnf[1128680]: Errors during downloading metadata for repository 'beaker-BaseOS':
Apr 16 21:59:02 rdma06 dnf[1128680]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 21:59:02 rdma06 dnf[1128680]: Error: Failed to download metadata for repo 'beaker-BaseOS': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 21:59:02 rdma06 dnf[1128680]: beaker-BaseOS-debuginfo                          56 kB/s | 168  B     00:00
Apr 16 21:59:02 rdma06 dnf[1128680]: Errors during downloading metadata for repository 'beaker-BaseOS-debuginfo':
Apr 16 21:59:02 rdma06 dnf[1128680]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 21:59:02 rdma06 dnf[1128680]: Error: Failed to download metadata for repo 'beaker-BaseOS-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 21:59:02 rdma06 dnf[1128680]: beaker-CRB                                       17 kB/s | 168  B     00:00
Apr 16 21:59:02 rdma06 dnf[1128680]: Errors during downloading metadata for repository 'beaker-CRB':
Apr 16 21:59:02 rdma06 dnf[1128680]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 21:59:02 rdma06 dnf[1128680]: Error: Failed to download metadata for repo 'beaker-CRB': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 21:59:02 rdma06 dnf[1128680]: beaker-CRB-debuginfo                            9.5 kB/s | 168  B     00:00
Apr 16 21:59:02 rdma06 dnf[1128680]: Errors during downloading metadata for repository 'beaker-CRB-debuginfo':
Apr 16 21:59:02 rdma06 dnf[1128680]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 21:59:02 rdma06 dnf[1128680]: Error: Failed to download metadata for repo 'beaker-CRB-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 21:59:02 rdma06 dnf[1128680]: beaker-HighAvailability                          13 kB/s | 168  B     00:00
Apr 16 21:59:02 rdma06 dnf[1128680]: Errors during downloading metadata for repository 'beaker-HighAvailability':
Apr 16 21:59:02 rdma06 dnf[1128680]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 21:59:02 rdma06 dnf[1128680]: Error: Failed to download metadata for repo 'beaker-HighAvailability': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 21:59:02 rdma06 dnf[1128680]: beaker-HighAvailability-debuginfo               8.7 kB/s | 168  B     00:00
Apr 16 21:59:02 rdma06 dnf[1128680]: Errors during downloading metadata for repository 'beaker-HighAvailability-debuginfo':
Apr 16 21:59:02 rdma06 dnf[1128680]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 21:59:02 rdma06 dnf[1128680]: Error: Failed to download metadata for repo 'beaker-HighAvailability-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 21:59:02 rdma06 dnf[1128680]: beaker-NFV                                      3.8 kB/s | 168  B     00:00
Apr 16 21:59:02 rdma06 dnf[1128680]: Errors during downloading metadata for repository 'beaker-NFV':
Apr 16 21:59:02 rdma06 dnf[1128680]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 21:59:02 rdma06 dnf[1128680]: Error: Failed to download metadata for repo 'beaker-NFV': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 21:59:02 rdma06 dnf[1128680]: beaker-NFV-debuginfo                            9.9 kB/s | 168  B     00:00
Apr 16 21:59:02 rdma06 dnf[1128680]: Errors during downloading metadata for repository 'beaker-NFV-debuginfo':
Apr 16 21:59:02 rdma06 dnf[1128680]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 21:59:02 rdma06 dnf[1128680]: Error: Failed to download metadata for repo 'beaker-NFV-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 21:59:02 rdma06 dnf[1128680]: beaker-RT                                        56 kB/s | 168  B     00:00
Apr 16 21:59:02 rdma06 dnf[1128680]: Errors during downloading metadata for repository 'beaker-RT':
Apr 16 21:59:02 rdma06 dnf[1128680]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 21:59:02 rdma06 dnf[1128680]: Error: Failed to download metadata for repo 'beaker-RT': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 21:59:02 rdma06 dnf[1128680]: beaker-RT-debuginfo                              58 kB/s | 168  B     00:00
Apr 16 21:59:02 rdma06 dnf[1128680]: Errors during downloading metadata for repository 'beaker-RT-debuginfo':
Apr 16 21:59:02 rdma06 dnf[1128680]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 21:59:02 rdma06 dnf[1128680]: Error: Failed to download metadata for repo 'beaker-RT-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 21:59:02 rdma06 dnf[1128680]: beaker-ResilientStorage                          59 kB/s | 168  B     00:00
Apr 16 21:59:02 rdma06 dnf[1128680]: Errors during downloading metadata for repository 'beaker-ResilientStorage':
Apr 16 21:59:02 rdma06 dnf[1128680]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 21:59:02 rdma06 dnf[1128680]: Error: Failed to download metadata for repo 'beaker-ResilientStorage': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 21:59:02 rdma06 dnf[1128680]: beaker-ResilientStorage-debuginfo                57 kB/s | 168  B     00:00
Apr 16 21:59:02 rdma06 dnf[1128680]: Errors during downloading metadata for repository 'beaker-ResilientStorage-debuginfo':
Apr 16 21:59:02 rdma06 dnf[1128680]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 21:59:02 rdma06 dnf[1128680]: Error: Failed to download metadata for repo 'beaker-ResilientStorage-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 21:59:02 rdma06 dnf[1128680]: beaker-SAP                                       26 kB/s | 168  B     00:00
Apr 16 21:59:02 rdma06 dnf[1128680]: Errors during downloading metadata for repository 'beaker-SAP':
Apr 16 21:59:02 rdma06 dnf[1128680]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 21:59:02 rdma06 dnf[1128680]: Error: Failed to download metadata for repo 'beaker-SAP': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 21:59:02 rdma06 dnf[1128680]: beaker-SAP-debuginfo                             17 kB/s | 168  B     00:00
Apr 16 21:59:02 rdma06 dnf[1128680]: Errors during downloading metadata for repository 'beaker-SAP-debuginfo':
Apr 16 21:59:02 rdma06 dnf[1128680]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 21:59:02 rdma06 dnf[1128680]: Error: Failed to download metadata for repo 'beaker-SAP-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 21:59:02 rdma06 dnf[1128680]: beaker-SAPHANA                                   28 kB/s | 168  B     00:00
Apr 16 21:59:02 rdma06 dnf[1128680]: Errors during downloading metadata for repository 'beaker-SAPHANA':
Apr 16 21:59:02 rdma06 dnf[1128680]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 21:59:02 rdma06 dnf[1128680]: Error: Failed to download metadata for repo 'beaker-SAPHANA': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 21:59:02 rdma06 dnf[1128680]: beaker-SAPHANA-debuginfo                         24 kB/s | 168  B     00:00
Apr 16 21:59:02 rdma06 dnf[1128680]: Errors during downloading metadata for repository 'beaker-SAPHANA-debuginfo':
Apr 16 21:59:02 rdma06 dnf[1128680]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 21:59:02 rdma06 dnf[1128680]: Error: Failed to download metadata for repo 'beaker-SAPHANA-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 16 21:59:03 rdma06 dnf[1128680]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr 16 21:59:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:59:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:59:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:59:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:59:04 rdma06 dnf[1128680]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr 16 21:59:04 rdma06 dnf[1128680]: Ignoring repositories: beaker-AppStream, beaker-AppStream-debuginfo, beaker-BaseOS, beaker-BaseOS-debuginfo, beaker-CRB, beaker-CRB-debuginfo, beaker-HighAvailability, beaker-HighAvailability-debuginfo, beaker-NFV, beaker-NFV-debuginfo, beaker-RT, beaker-RT-debuginfo, beaker-ResilientStorage, beaker-ResilientStorage-debuginfo, beaker-SAP, beaker-SAP-debuginfo, beaker-SAPHANA, beaker-SAPHANA-debuginfo
Apr 16 21:59:04 rdma06 dnf[1128680]: Metadata cache created.
Apr 16 21:59:04 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 16 21:59:04 rdma06 systemd[1]: Finished dnf makecache.
Apr 16 21:59:04 rdma06 systemd[1]: dnf-makecache.service: Consumed 1.012s CPU time.
Apr 16 21:59:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:59:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:59:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:59:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:59:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:59:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:59:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:59:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:59:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:59:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:59:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:59:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:59:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:59:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 21:59:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:59:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:59:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:59:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:59:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 21:59:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 21:59:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 21:59:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 21:59:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:00:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:00:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:00:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:00:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:00:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:00:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:00:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:00:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:00:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:00:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:00:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:00:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:00:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:00:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:00:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:00:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:00:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:00:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:00:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:00:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:00:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:00:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:00:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:00:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:00:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:00:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:00:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:01:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:01:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:01:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:01:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:01:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:01:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:01:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:01:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:01:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:01:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:01:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:01:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:01:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:01:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:01:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:01:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:01:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:01:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:01:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:01:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:01:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:01:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:01:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:01:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:01:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:01:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:01:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:02:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:02:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:02:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:02:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:02:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:02:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:02:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:02:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:02:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:02:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:02:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:02:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:02:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:02:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:02:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:02:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:02:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:02:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:02:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:02:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:02:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:02:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:02:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:02:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:02:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:02:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:02:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:02:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:03:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:03:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:03:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:03:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:03:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:03:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:03:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:03:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:03:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:03:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:03:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:03:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:03:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:03:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:03:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:03:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:03:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:03:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:03:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:03:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:03:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:03:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:03:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:03:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:03:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:03:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:03:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:04:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:04:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:04:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:04:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:04:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:04:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:04:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:04:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:04:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:04:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:04:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:04:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:04:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:04:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:04:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:04:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:04:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:04:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:04:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:04:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:04:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:04:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:04:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:04:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:04:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:04:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:04:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:05:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:05:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:05:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:05:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:05:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:05:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:05:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:05:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:05:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:05:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:05:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:05:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:05:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:05:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:05:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:05:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:05:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:05:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:05:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:05:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:05:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:05:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:05:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:05:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:05:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:05:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:05:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:05:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:06:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:06:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:06:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:06:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:06:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:06:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:06:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:06:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:06:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:06:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:06:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:06:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:06:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:06:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:06:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:06:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:06:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:06:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:06:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:06:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:06:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:06:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:06:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:06:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:06:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:06:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:06:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:07:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:07:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:07:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:07:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:07:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:07:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:07:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:07:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:07:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:07:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:07:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:07:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:07:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:07:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:07:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:07:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:07:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:07:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:07:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:07:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:07:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:07:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:07:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:07:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:07:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:07:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:07:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:08:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:08:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:08:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:08:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:08:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:08:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:08:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:08:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:08:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:08:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:08:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:08:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:08:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:08:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:08:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:08:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:08:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:08:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:08:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:08:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:08:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:08:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:08:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:08:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:08:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:08:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:08:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:08:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:09:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:09:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:09:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:09:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:09:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:09:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:09:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:09:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:09:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:09:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:09:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:09:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:09:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:09:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:09:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:09:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:09:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:09:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:09:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:09:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:09:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:09:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:09:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:09:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:09:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:09:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:09:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:10:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:10:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:10:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:10:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:10:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:10:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:10:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:10:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:10:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:10:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:10:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:10:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:10:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:10:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:10:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:10:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:10:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:10:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:10:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:10:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:10:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:10:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:10:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:10:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:10:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:10:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:10:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:11:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:11:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:11:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:11:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:11:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:11:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:11:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:11:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:11:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:11:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:11:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:11:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:11:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:11:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:11:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:11:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:11:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:11:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:11:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:11:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:11:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:11:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:11:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:11:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:11:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:11:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:11:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:11:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:12:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:12:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:12:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:12:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:12:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:12:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:12:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:12:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:12:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:12:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:12:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:12:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:12:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:12:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:12:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:12:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:12:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:12:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:12:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:12:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:12:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:12:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:12:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:12:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:12:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:12:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:12:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:13:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:13:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:13:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:13:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:13:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:13:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:13:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:13:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:13:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:13:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:13:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:13:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:13:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:13:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:13:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:13:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:13:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:13:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:13:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:13:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:13:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:13:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:13:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:13:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:13:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:13:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:13:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:14:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:14:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:14:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:14:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:14:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:14:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:14:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:14:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:14:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:14:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:14:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:14:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:14:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:14:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:14:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:14:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:14:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:14:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:14:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:14:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:14:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:14:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:14:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:14:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:14:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:14:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:14:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:14:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:15:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:15:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:15:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:15:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:15:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:15:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:15:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:15:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:15:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:15:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:15:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:15:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:15:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:15:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:15:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:15:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:15:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:15:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:15:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:15:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:15:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:15:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:15:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:15:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:15:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:15:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:15:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:16:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:16:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:16:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:16:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:16:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:16:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:16:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:16:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:16:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:16:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:16:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:16:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:16:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:16:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:16:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:16:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:16:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:16:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:16:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:16:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:16:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:16:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:16:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:16:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:16:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:16:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:16:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:17:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:17:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:17:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:17:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:17:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:17:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:17:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:17:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:17:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:17:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:17:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:17:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:17:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:17:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:17:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:17:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:17:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:17:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:17:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:17:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:17:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:17:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:17:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:17:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:17:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:17:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:17:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:18:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:18:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:18:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:18:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:18:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:18:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:18:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:18:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:18:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:18:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:18:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:18:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:18:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:18:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:18:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:18:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:18:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:18:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:18:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:18:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:18:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:18:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:18:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:18:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:18:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:18:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:18:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:18:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:19:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:19:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:19:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:19:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:19:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:19:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:19:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:19:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:19:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:19:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:19:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:19:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:19:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:19:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:19:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:19:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:19:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:19:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:19:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:19:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:19:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:19:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:19:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:19:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:19:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:19:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:19:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:20:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:20:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:20:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:20:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:20:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:20:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:20:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:20:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:20:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:20:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:20:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:20:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:20:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:20:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:20:32 rdma06 NetworkManager[1025]: <info>  [1681698032.8327] dhcp4 (eno1): state changed new lease, address=10.73.131.45
Apr 16 22:20:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:20:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:20:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:20:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:20:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:20:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:20:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:20:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:20:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:20:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:20:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:20:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:20:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:21:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:21:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:21:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:21:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:21:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:21:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:21:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:21:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:21:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:21:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:21:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:21:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:21:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:21:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:21:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:21:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:21:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:21:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:21:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:21:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:21:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:21:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:21:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:21:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:21:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:21:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:21:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:21:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:22:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:22:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:22:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:22:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:22:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:22:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:22:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:22:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:22:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:22:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:22:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:22:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:22:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:22:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:22:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:22:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:22:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:22:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:22:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:22:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:22:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:22:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:22:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:22:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:22:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:22:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:22:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:23:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:23:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:23:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:23:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:23:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:23:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:23:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:23:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:23:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:23:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:23:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:23:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:23:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:23:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:23:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:23:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:23:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:23:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:23:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:23:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:23:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:23:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:23:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:23:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:23:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:23:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:23:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:24:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:24:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:24:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:24:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:24:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:24:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:24:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:24:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:24:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:24:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:24:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:24:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:24:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:24:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:24:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:24:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:24:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:24:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:24:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:24:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:24:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:24:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:24:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:24:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:24:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:24:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:24:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:24:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:25:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:25:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:25:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:25:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:25:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:25:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:25:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:25:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:25:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:25:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:25:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:25:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:25:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:25:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:25:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:25:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:25:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:25:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:25:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:25:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:25:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:25:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:25:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:25:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:25:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:25:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:25:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:26:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:26:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:26:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:26:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:26:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:26:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:26:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:26:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:26:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:26:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:26:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:26:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:26:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:26:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:26:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:26:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:26:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:26:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:26:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:26:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:26:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:26:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:26:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:26:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:26:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:26:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:26:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:27:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:27:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:27:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:27:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:27:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:27:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:27:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:27:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:27:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:27:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:27:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:27:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:27:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:27:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:27:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:27:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:27:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:27:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:27:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:27:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:27:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:27:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:27:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:27:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:27:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:27:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:27:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:27:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:28:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:28:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:28:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:28:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:28:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:28:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:28:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:28:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:28:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:28:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:28:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:28:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:28:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:28:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:28:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:28:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:28:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:28:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:28:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:28:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:28:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:28:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:28:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:28:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:28:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:28:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:28:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:29:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:29:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:29:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:29:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:29:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:29:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:29:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:29:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:29:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:29:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:29:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:29:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:29:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:29:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:29:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:29:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:29:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:29:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:29:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:29:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:29:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:29:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:29:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:29:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:29:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:29:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:29:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:30:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:30:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:30:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:30:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:30:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:30:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:30:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:30:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:30:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:30:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:30:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:30:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:30:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:30:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:30:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:30:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:30:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:30:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:30:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:30:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:30:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:30:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:30:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:30:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:30:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:30:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:30:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:30:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:31:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:31:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:31:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:31:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:31:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:31:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:31:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:31:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:31:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:31:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:31:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:31:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:31:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:31:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:31:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:31:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:31:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:31:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:31:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:31:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:31:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:31:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:31:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:31:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:31:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:31:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:31:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:32:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:32:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:32:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:32:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:32:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:32:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:32:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:32:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:32:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:32:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:32:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:32:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:32:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:32:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:32:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:32:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:32:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:32:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:32:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:32:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:32:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:32:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:32:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:32:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:32:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:32:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:32:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:33:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:33:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:33:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:33:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:33:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:33:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:33:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:33:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:33:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:33:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:33:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:33:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:33:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:33:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:33:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:33:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:33:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:33:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:33:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:33:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:33:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:33:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:33:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:33:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:33:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:33:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:33:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:34:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:34:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:34:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:34:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:34:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:34:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:34:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:34:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:34:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:34:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:34:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:34:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:34:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:34:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:34:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:34:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:34:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:34:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:34:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:34:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:34:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:34:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:34:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:34:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:34:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:34:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:34:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:34:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:35:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:35:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:35:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:35:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:35:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:35:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:35:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:35:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:35:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:35:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:35:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:35:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:35:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:35:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:35:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:35:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:35:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:35:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:35:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:35:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:35:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:35:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:35:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:35:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:35:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:35:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:35:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:36:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:36:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:36:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:36:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:36:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:36:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:36:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:36:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:36:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:36:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:36:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:36:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:36:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:36:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:36:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:36:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:36:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:36:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:36:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:36:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:36:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:36:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:36:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:36:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:36:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:36:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:36:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:37:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:37:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:37:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:37:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:37:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:37:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:37:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:37:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:37:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:37:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:37:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:37:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:37:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:37:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:37:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:37:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:37:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:37:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:37:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:37:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:37:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:37:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:37:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:37:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:37:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:37:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:37:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:37:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:38:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:38:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:38:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:38:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:38:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:38:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:38:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:38:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:38:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:38:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:38:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:38:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:38:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:38:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:38:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:38:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:38:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:38:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:38:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:38:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:38:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:38:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:38:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:38:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:38:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:38:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:38:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:39:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:39:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:39:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:39:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:39:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:39:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:39:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:39:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:39:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:39:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:39:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:39:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:39:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:39:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:39:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:39:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:39:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:39:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:39:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:39:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:39:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:39:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:39:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:39:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:39:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:39:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:39:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:40:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:40:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:40:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:40:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:40:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:40:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:40:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:40:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:40:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:40:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:40:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:40:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:40:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:40:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:40:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:40:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:40:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:40:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:40:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:40:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:40:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:40:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:40:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:40:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:40:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:40:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:40:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:40:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:41:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:41:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:41:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:41:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:41:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:41:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:41:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:41:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:41:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:41:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:41:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:41:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:41:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:41:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:41:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:41:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:41:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:41:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:41:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:41:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:41:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:41:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:41:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:41:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:41:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:41:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:41:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:42:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:42:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:42:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:42:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:42:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:42:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:42:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:42:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:42:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:42:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:42:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:42:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:42:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:42:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:42:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:42:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:42:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:42:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:42:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:42:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:42:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:42:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:42:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:42:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:42:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:42:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:42:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:43:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:43:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:43:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:43:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:43:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:43:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:43:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:43:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:43:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:43:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:43:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:43:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:43:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:43:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:43:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:43:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:43:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:43:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:43:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:43:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:43:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:43:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:43:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:43:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:43:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:43:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:43:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:43:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:44:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:44:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:44:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:44:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:44:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:44:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:44:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:44:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:44:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:44:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:44:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:44:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:44:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:44:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:44:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:44:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:44:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:44:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:44:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:44:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:44:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:44:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:44:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:44:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:44:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:44:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:44:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:45:00 rdma06 systemd[5685]: Starting Cleanup of User's Temporary Files and Directories...
Apr 16 22:45:00 rdma06 systemd[5685]: Finished Cleanup of User's Temporary Files and Directories.
Apr 16 22:45:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:45:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:45:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:45:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:45:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:45:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:45:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:45:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:45:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:45:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:45:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:45:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:45:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:45:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:45:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:45:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:45:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:45:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:45:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:45:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:45:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:45:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:45:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:45:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:45:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:45:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:45:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:46:00 rdma06 systemd[1]: Starting Cleanup of Temporary Directories...
Apr 16 22:46:00 rdma06 systemd[1]: systemd-tmpfiles-clean.service: Deactivated successfully.
Apr 16 22:46:00 rdma06 systemd[1]: Finished Cleanup of Temporary Directories.
Apr 16 22:46:00 rdma06 systemd[1]: run-credentials-systemd\x2dtmpfiles\x2dclean.service.mount: Deactivated successfully.
Apr 16 22:46:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:46:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:46:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:46:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:46:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:46:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:46:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:46:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:46:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:46:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:46:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:46:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:46:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:46:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:46:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:46:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:46:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:46:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:46:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:46:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:46:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:46:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:46:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:46:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:46:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:46:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:46:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:46:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:47:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:47:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:47:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:47:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:47:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:47:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:47:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:47:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:47:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:47:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:47:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:47:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:47:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:47:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:47:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:47:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:47:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:47:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:47:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:47:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:47:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:47:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:47:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:47:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:47:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:47:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:47:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:48:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:48:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:48:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:48:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:48:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:48:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:48:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:48:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:48:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:48:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:48:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:48:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:48:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:48:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:48:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:48:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:48:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:48:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:48:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:48:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:48:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:48:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:48:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:48:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:48:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:48:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:48:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:49:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:49:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:49:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:49:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:49:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:49:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:49:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:49:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:49:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:49:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:49:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:49:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:49:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:49:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:49:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:49:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:49:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:49:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:49:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:49:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:49:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:49:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:49:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:49:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:49:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:49:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:49:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:50:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:50:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:50:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:50:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:50:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:50:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:50:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:50:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:50:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:50:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:50:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:50:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:50:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:50:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:50:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:50:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:50:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:50:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:50:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:50:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:50:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:50:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:50:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:50:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:50:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:50:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:50:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:50:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:51:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:51:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:51:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:51:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:51:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:51:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:51:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:51:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:51:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:51:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:51:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:51:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:51:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:51:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:51:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:51:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:51:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:51:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:51:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:51:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:51:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:51:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:51:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:51:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:51:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:51:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:51:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:52:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:52:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:52:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:52:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:52:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:52:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:52:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:52:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:52:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:52:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:52:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:52:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:52:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:52:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:52:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:52:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:52:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:52:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:52:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:52:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:52:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:52:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:52:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:52:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:52:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:52:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:52:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:53:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:53:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:53:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:53:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:53:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:53:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:53:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:53:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:53:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:53:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:53:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:53:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:53:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:53:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:53:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:53:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:53:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:53:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:53:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:53:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:53:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:53:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:53:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:53:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:53:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:53:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:53:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:53:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:54:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:54:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:54:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:54:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:54:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:54:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:54:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:54:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:54:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:54:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:54:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:54:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:54:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:54:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:54:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:54:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:54:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:54:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:54:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:54:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:54:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:54:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:54:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:54:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:54:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:54:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:54:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:55:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:55:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:55:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:55:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:55:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:55:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:55:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:55:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:55:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:55:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:55:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:55:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:55:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:55:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:55:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:55:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:55:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:55:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:55:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:55:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:55:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:55:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:55:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:55:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:55:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:55:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:55:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:56:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:56:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:56:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:56:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:56:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:56:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:56:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:56:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:56:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:56:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:56:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:56:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:56:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:56:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:56:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:56:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:56:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:56:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:56:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:56:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:56:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:56:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:56:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:56:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:56:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:56:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:56:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:56:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:57:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:57:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:57:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:57:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:57:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:57:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:57:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:57:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:57:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:57:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:57:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:57:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:57:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:57:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:57:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:57:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:57:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:57:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:57:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:57:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:57:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:57:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:57:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:57:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:57:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:57:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:57:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:58:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:58:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:58:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:58:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:58:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:58:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:58:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:58:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:58:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:58:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:58:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:58:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:58:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:58:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:58:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:58:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:58:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:58:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:58:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:58:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:58:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:58:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:58:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:58:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:58:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:58:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:58:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:59:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:59:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:59:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:59:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:59:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:59:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:59:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:59:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:59:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:59:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:59:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:59:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:59:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:59:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:59:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:59:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:59:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:59:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:59:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 22:59:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:59:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:59:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:59:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:59:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 22:59:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 22:59:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 22:59:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 22:59:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:00:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:00:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:00:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:00:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:00:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:00:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:00:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:00:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:00:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:00:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:00:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:00:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:00:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:00:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:00:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:00:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:00:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:00:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:00:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:00:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:00:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:00:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:00:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:00:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:00:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:00:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:00:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:01:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:01:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:01:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:01:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:01:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:01:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:01:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:01:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:01:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:01:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:01:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:01:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:01:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:01:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:01:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:01:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:01:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:01:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:01:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:01:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:01:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:01:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:01:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:01:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:01:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:01:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:01:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:02:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:02:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:02:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:02:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:02:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:02:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:02:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:02:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:02:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:02:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:02:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:02:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:02:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:02:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:02:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:02:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:02:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:02:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:02:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:02:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:02:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:02:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:02:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:02:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:02:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:02:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:02:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:02:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:03:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:03:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:03:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:03:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:03:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:03:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:03:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:03:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:03:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:03:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:03:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:03:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:03:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:03:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:03:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:03:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:03:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:03:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:03:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:03:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:03:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:03:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:03:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:03:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:03:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:03:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:03:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:04:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:04:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:04:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:04:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:04:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:04:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:04:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:04:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:04:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:04:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:04:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:04:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:04:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:04:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:04:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:04:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:04:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:04:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:04:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:04:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:04:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:04:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:04:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:04:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:04:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:04:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:04:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:05:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:05:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:05:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:05:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:05:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:05:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:05:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:05:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:05:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:05:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:05:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:05:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:05:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:05:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:05:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:05:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:05:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:05:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:05:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:05:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:05:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:05:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:05:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:05:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:05:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:05:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:05:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:05:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:06:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:06:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:06:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:06:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:06:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:06:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:06:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:06:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:06:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:06:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:06:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:06:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:06:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:06:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:06:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:06:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:06:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:06:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:06:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:06:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:06:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:06:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:06:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:06:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:06:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:06:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:06:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:07:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:07:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:07:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:07:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:07:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:07:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:07:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:07:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:07:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:07:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:07:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:07:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:07:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:07:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:07:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:07:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:07:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:07:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:07:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:07:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:07:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:07:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:07:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:07:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:07:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:07:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:07:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:08:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:08:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:08:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:08:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:08:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:08:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:08:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:08:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:08:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:08:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:08:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:08:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:08:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:08:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:08:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:08:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:08:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:08:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:08:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:08:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:08:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:08:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:08:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:08:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:08:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:08:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:08:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:09:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:09:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:09:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:09:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:09:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:09:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:09:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:09:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:09:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:09:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:09:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:09:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:09:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:09:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:09:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:09:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:09:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:09:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:09:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:09:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:09:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:09:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:09:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:09:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:09:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:09:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:09:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:09:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:10:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:10:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:10:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:10:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:10:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:10:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:10:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:10:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:10:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:10:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:10:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:10:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:10:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:10:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:10:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:10:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:10:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:10:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:10:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:10:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:10:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:10:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:10:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:10:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:10:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:10:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:10:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:11:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:11:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:11:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:11:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:11:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:11:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:11:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:11:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:11:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:11:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:11:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:11:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:11:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:11:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:11:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:11:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:11:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:11:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:11:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:11:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:11:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:11:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:11:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:11:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:11:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:11:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:11:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:12:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:12:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:12:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:12:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:12:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:12:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:12:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:12:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:12:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:12:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:12:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:12:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:12:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:12:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:12:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:12:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:12:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:12:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:12:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:12:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:12:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:12:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:12:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:12:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:12:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:12:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:12:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:12:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:13:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:13:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:13:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:13:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:13:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:13:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:13:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:13:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:13:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:13:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:13:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:13:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:13:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:13:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:13:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:13:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:13:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:13:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:13:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:13:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:13:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:13:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:13:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:13:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:13:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:13:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:13:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:14:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:14:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:14:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:14:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:14:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:14:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:14:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:14:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:14:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:14:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:14:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:14:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:14:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:14:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:14:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:14:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:14:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:14:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:14:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:14:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:14:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:14:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:14:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:14:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:14:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:14:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:14:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:15:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:15:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:15:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:15:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:15:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:15:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:15:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:15:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:15:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:15:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:15:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:15:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:15:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:15:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:15:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:15:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:15:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:15:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:15:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:15:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:15:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:15:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:15:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:15:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:15:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:15:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:15:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:15:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:16:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:16:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:16:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:16:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:16:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:16:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:16:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:16:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:16:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:16:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:16:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:16:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:16:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:16:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:16:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:16:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:16:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:16:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:16:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:16:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:16:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:16:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:16:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:16:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:16:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:16:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:16:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:17:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:17:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:17:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:17:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:17:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:17:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:17:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:17:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:17:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:17:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:17:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:17:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:17:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:17:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:17:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:17:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:17:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:17:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:17:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:17:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:17:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:17:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:17:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:17:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:17:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:17:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:17:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:18:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:18:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:18:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:18:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:18:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:18:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:18:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:18:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:18:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:18:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:18:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:18:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:18:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:18:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:18:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:18:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:18:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:18:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:18:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:18:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:18:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:18:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:18:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:18:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:18:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:18:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:18:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:18:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:19:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:19:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:19:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:19:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:19:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:19:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:19:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:19:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:19:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:19:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:19:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:19:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:19:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:19:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:19:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:19:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:19:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:19:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:19:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:19:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:19:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:19:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:19:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:19:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:19:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:19:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:19:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:20:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:20:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:20:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:20:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:20:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:20:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:20:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:20:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:20:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:20:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:20:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:20:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:20:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:20:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:20:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:20:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:20:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:20:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:20:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:20:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:20:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:20:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:20:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:20:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:20:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:20:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:20:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:21:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:21:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:21:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:21:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:21:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:21:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:21:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:21:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:21:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:21:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:21:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:21:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:21:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:21:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:21:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:21:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:21:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:21:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:21:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:21:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:21:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:21:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:21:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:21:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:21:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:21:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:21:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:21:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:22:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:22:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:22:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:22:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:22:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:22:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:22:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:22:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:22:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:22:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:22:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:22:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:22:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:22:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:22:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:22:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:22:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:22:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:22:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:22:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:22:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:22:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:22:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:22:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:22:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:22:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:22:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:23:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:23:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:23:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:23:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:23:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:23:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:23:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:23:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:23:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:23:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:23:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:23:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:23:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:23:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:23:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:23:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:23:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:23:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:23:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:23:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:23:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:23:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:23:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:23:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:23:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:23:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:23:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:24:00 rdma06 systemd[1]: Starting dnf makecache...
Apr 16 23:24:01 rdma06 dnf[1132159]: Updating Subscription Management repositories.
Apr 16 23:24:01 rdma06 dnf[1132159]: Unable to read consumer identity
Apr 16 23:24:01 rdma06 dnf[1132159]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 16 23:24:01 rdma06 dnf[1132159]: Metadata cache refreshed recently.
Apr 16 23:24:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 16 23:24:01 rdma06 systemd[1]: Finished dnf makecache.
Apr 16 23:24:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:24:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:24:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:24:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:24:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:24:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:24:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:24:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:24:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:24:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:24:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:24:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:24:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:24:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:24:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:24:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:24:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:24:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:24:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:24:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:24:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:24:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:24:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:24:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:24:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:24:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:24:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:25:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:25:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:25:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:25:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:25:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:25:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:25:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:25:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:25:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:25:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:25:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:25:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:25:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:25:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:25:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:25:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:25:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:25:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:25:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:25:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:25:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:25:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:25:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:25:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:25:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:25:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:25:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:25:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:26:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:26:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:26:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:26:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:26:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:26:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:26:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:26:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:26:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:26:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:26:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:26:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:26:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:26:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:26:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:26:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:26:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:26:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:26:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:26:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:26:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:26:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:26:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:26:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:26:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:26:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:26:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:27:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:27:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:27:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:27:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:27:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:27:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:27:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:27:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:27:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:27:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:27:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:27:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:27:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:27:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:27:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:27:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:27:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:27:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:27:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:27:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:27:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:27:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:27:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:27:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:27:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:27:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:27:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:28:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:28:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:28:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:28:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:28:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:28:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:28:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:28:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:28:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:28:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:28:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:28:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:28:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:28:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:28:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:28:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:28:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:28:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:28:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:28:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:28:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:28:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:28:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:28:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:28:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:28:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:28:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:28:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:29:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:29:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:29:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:29:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:29:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:29:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:29:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:29:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:29:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:29:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:29:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:29:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:29:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:29:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:29:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:29:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:29:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:29:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:29:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:29:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:29:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:29:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:29:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:29:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:29:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:29:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:29:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:30:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:30:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:30:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:30:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:30:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:30:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:30:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:30:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:30:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:30:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:30:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:30:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:30:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:30:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:30:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:30:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:30:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:30:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:30:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:30:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:30:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:30:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:30:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:30:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:30:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:30:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:30:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:31:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:31:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:31:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:31:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:31:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:31:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:31:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:31:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:31:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:31:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:31:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:31:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:31:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:31:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:31:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:31:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:31:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:31:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:31:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:31:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:31:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:31:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:31:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:31:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:31:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:31:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:31:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:31:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:32:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:32:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:32:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:32:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:32:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:32:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:32:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:32:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:32:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:32:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:32:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:32:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:32:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:32:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:32:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:32:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:32:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:32:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:32:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:32:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:32:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:32:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:32:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:32:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:32:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:32:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:32:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:33:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:33:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:33:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:33:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:33:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:33:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:33:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:33:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:33:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:33:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:33:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:33:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:33:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:33:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:33:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:33:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:33:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:33:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:33:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:33:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:33:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:33:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:33:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:33:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:33:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:33:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:33:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:34:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:34:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:34:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:34:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:34:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:34:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:34:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:34:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:34:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:34:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:34:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:34:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:34:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:34:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:34:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:34:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:34:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:34:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:34:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:34:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:34:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:34:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:34:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:34:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:34:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:34:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:34:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:34:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:35:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:35:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:35:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:35:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:35:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:35:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:35:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:35:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:35:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:35:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:35:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:35:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:35:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:35:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:35:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:35:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:35:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:35:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:35:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:35:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:35:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:35:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:35:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:35:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:35:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:35:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:35:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:36:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:36:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:36:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:36:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:36:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:36:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:36:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:36:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:36:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:36:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:36:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:36:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:36:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:36:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:36:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:36:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:36:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:36:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:36:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:36:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:36:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:36:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:36:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:36:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:36:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:36:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:36:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:37:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:37:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:37:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:37:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:37:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:37:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:37:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:37:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:37:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:37:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:37:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:37:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:37:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:37:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:37:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:37:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:37:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:37:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:37:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:37:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:37:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:37:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:37:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:37:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:37:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:37:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:37:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:37:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:38:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:38:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:38:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:38:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:38:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:38:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:38:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:38:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:38:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:38:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:38:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:38:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:38:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:38:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:38:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:38:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:38:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:38:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:38:38 rdma06 sshd[1132750]: main: sshd: ssh-rsa algorithm is disabled
Apr 16 23:38:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:38:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:38:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:38:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:38:44 rdma06 systemd-logind[1064]: New session 14 of user root.
Apr 16 23:38:44 rdma06 systemd[1]: Started Session 14 of User root.
Apr 16 23:38:45 rdma06 systemd[1]: Starting Hostname Service...
Apr 16 23:38:45 rdma06 systemd[1]: Started Hostname Service.
Apr 16 23:38:53 rdma06 journal[2657]: Adjustment to local watchdog ignored since 'no_localwatchdog' metadata is set
Apr 16 23:38:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:38:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:38:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:38:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:38:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:39:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:39:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:39:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:39:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:39:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:39:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:39:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:39:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:39:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:39:15 rdma06 systemd[1]: systemd-hostnamed.service: Deactivated successfully.
Apr 16 23:39:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:39:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:39:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:39:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:39:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:39:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:39:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:39:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:39:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:39:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:39:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:39:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:39:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:39:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:39:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:39:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:39:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:39:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:40:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:40:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:40:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:40:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:40:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:40:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:40:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:40:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:40:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:40:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:40:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:40:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:40:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:40:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:40:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:40:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:40:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:40:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:40:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:40:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:40:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:40:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:40:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:40:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:40:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:40:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:40:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:41:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:41:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:41:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:41:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:41:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:41:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:41:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:41:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:41:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:41:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:41:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:41:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:41:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:41:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:41:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:41:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:41:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:41:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:41:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:41:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:41:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:41:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:41:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:41:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:41:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:41:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:41:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:41:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:42:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:42:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:42:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:42:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:42:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:42:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:42:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:42:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:42:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:42:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:42:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:42:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:42:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:42:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:42:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:42:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:42:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:42:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:42:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:42:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:42:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:42:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:42:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:42:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:42:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:42:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:42:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:43:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:43:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:43:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:43:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:43:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:43:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:43:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:43:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:43:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:43:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:43:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:43:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:43:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:43:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:43:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:43:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:43:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:43:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:43:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:43:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:43:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:43:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:43:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:43:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:43:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:43:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:43:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:44:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:44:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:44:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:44:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:44:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:44:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:44:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:44:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:44:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:44:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:44:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:44:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:44:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:44:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:44:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:44:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:44:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:44:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:44:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:44:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:44:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:44:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:44:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:44:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:44:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:44:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:44:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:44:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:45:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:45:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:45:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:45:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:45:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:45:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:45:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:45:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:45:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:45:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:45:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:45:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:45:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:45:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:45:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:45:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:45:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:45:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:45:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:45:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:45:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:45:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:45:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:45:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:45:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:45:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:45:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:46:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:46:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:46:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:46:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:46:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:46:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:46:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:46:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:46:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:46:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:46:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:46:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:46:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:46:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:46:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:46:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:46:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:46:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:46:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:46:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:46:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:46:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:46:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:46:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:46:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:46:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:46:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:47:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:47:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:47:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:47:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:47:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:47:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:47:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:47:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:47:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:47:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:47:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:47:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:47:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:47:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:47:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:47:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:47:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:47:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:47:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:47:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:47:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:47:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:47:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:47:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:47:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:47:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:47:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:47:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:48:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:48:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:48:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:48:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:48:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:48:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:48:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:48:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:48:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:48:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:48:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:48:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:48:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:48:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:48:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:48:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:48:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:48:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:48:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:48:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:48:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:48:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:48:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:48:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:48:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:48:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:48:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:49:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:49:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:49:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:49:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:49:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:49:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:49:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:49:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:49:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:49:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:49:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:49:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:49:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:49:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:49:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:49:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:49:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:49:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:49:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:49:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:49:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:49:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:49:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:49:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:49:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:49:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:49:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:50:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:50:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:50:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:50:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:50:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:50:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:50:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:50:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:50:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:50:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:50:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:50:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:50:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:50:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:50:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:50:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:50:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:50:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:50:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:50:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:50:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:50:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:50:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:50:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:50:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:50:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:50:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:50:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:51:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:51:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:51:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:51:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:51:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:51:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:51:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:51:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:51:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:51:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:51:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:51:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:51:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:51:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:51:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:51:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:51:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:51:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:51:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:51:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:51:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:51:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:51:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:51:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:51:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:51:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:51:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:52:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:52:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:52:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:52:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:52:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:52:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:52:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:52:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:52:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:52:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:52:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:52:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:52:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:52:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:52:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:52:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:52:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:52:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:52:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:52:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:52:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:52:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:52:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:52:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:52:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:52:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:52:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:53:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:53:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:53:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:53:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:53:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:53:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:53:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:53:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:53:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:53:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:53:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:53:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:53:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:53:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:53:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:53:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:53:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:53:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:53:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:53:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:53:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:53:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:53:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:53:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:53:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:53:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:53:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:53:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:54:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:54:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:54:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:54:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:54:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:54:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:54:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:54:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:54:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:54:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:54:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:54:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:54:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:54:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:54:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:54:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:54:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:54:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:54:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:54:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:54:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:54:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:54:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:54:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:54:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:54:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:54:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:55:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:55:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:55:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:55:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:55:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:55:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:55:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:55:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:55:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:55:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:55:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:55:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:55:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:55:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:55:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:55:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:55:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:55:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:55:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:55:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:55:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:55:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:55:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:55:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:55:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:55:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:55:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:56:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:56:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:56:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:56:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:56:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:56:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:56:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:56:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:56:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:56:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:56:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:56:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:56:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:56:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:56:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:56:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:56:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:56:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:56:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:56:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:56:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:56:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:56:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:56:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:56:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:56:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:56:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:57:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:57:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:57:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:57:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:57:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:57:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:57:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:57:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:57:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:57:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:57:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:57:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:57:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:57:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:57:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:57:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:57:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:57:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:57:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:57:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:57:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:57:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:57:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:57:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:57:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:57:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:57:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:57:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:58:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:58:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:58:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:58:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:58:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:58:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:58:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:58:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:58:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:58:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:58:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:58:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:58:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:58:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:58:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:58:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:58:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:58:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:58:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:58:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:58:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:58:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:58:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:58:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:58:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:58:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:58:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:59:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:59:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:59:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:59:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:59:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:59:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:59:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:59:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:59:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:59:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:59:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:59:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:59:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:59:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:59:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:59:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:59:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:59:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:59:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 16 23:59:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:59:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:59:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:59:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 16 23:59:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 16 23:59:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 16 23:59:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 16 23:59:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:00:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:00:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:00:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:00:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:00:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:00:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:00:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:00:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:00:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:00:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:00:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:00:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:00:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:00:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:00:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:00:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:00:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:00:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:00:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:00:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:00:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:00:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:00:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:00:44 rdma06 systemd[1]: Starting Rotate log files...
Apr 17 00:00:44 rdma06 systemd[1]: logrotate.service: Deactivated successfully.
Apr 17 00:00:44 rdma06 systemd[1]: Finished Rotate log files.
Apr 17 00:00:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:00:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:00:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:00:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:00:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:01:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:01:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:01:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:01:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:01:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:01:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:01:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:01:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:01:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:01:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:01:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:01:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:01:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:01:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:01:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:01:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:01:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:01:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:01:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:01:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:01:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:01:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:01:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:01:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:01:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:01:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:01:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:02:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:02:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:02:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:02:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:02:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:02:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:02:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:02:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:02:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:02:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:02:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:02:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:02:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:02:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:02:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:02:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:02:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:02:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:02:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:02:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:02:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:02:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:02:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:02:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:02:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:02:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:02:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:03:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:03:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:03:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:03:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:03:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:03:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:03:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:03:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:03:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:03:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:03:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:03:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:03:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:03:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:03:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:03:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:03:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:03:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:03:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:03:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:03:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:03:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:03:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:03:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:03:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:03:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:03:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:03:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:04:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:04:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:04:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:04:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:04:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:04:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:04:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:04:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:04:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:04:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:04:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:04:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:04:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:04:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:04:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:04:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:04:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:04:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:04:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:04:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:04:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:04:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:04:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:04:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:04:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:04:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:04:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:05:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:05:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:05:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:05:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:05:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:05:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:05:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:05:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:05:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:05:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:05:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:05:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:05:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:05:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:05:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:05:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:05:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:05:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:05:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:05:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:05:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:05:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:05:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:05:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:05:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:05:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:05:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:06:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:06:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:06:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:06:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:06:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:06:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:06:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:06:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:06:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:06:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:06:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:06:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:06:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:06:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:06:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:06:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:06:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:06:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:06:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:06:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:06:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:06:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:06:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:06:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:06:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:06:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:06:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:06:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:07:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:07:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:07:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:07:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:07:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:07:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:07:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:07:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:07:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:07:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:07:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:07:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:07:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:07:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:07:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:07:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:07:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:07:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:07:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:07:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:07:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:07:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:07:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:07:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:07:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:07:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:07:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:08:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:08:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:08:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:08:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:08:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:08:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:08:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:08:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:08:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:08:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:08:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:08:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:08:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:08:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:08:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:08:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:08:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:08:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:08:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:08:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:08:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:08:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:08:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:08:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:08:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:08:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:08:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:09:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:09:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:09:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:09:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:09:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:09:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:09:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:09:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:09:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:09:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:09:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:09:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:09:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:09:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:09:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:09:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:09:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:09:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:09:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:09:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:09:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:09:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:09:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:09:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:09:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:09:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:09:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:09:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:10:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:10:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:10:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:10:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:10:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:10:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:10:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:10:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:10:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:10:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:10:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:10:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:10:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:10:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:10:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:10:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:10:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:10:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:10:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:10:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:10:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:10:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:10:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:10:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:10:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:10:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:10:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:11:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:11:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:11:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:11:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:11:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:11:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:11:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:11:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:11:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:11:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:11:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:11:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:11:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:11:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:11:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:11:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:11:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:11:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:11:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:11:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:11:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:11:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:11:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:11:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:11:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:11:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:11:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:12:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:12:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:12:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:12:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:12:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:12:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:12:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:12:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:12:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:12:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:12:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:12:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:12:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:12:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:12:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:12:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:12:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:12:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:12:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:12:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:12:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:12:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:12:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:12:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:12:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:12:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:12:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:12:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:13:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:13:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:13:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:13:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:13:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:13:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:13:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:13:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:13:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:13:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:13:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:13:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:13:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:13:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:13:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:13:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:13:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:13:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:13:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:13:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:13:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:13:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:13:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:13:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:13:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:13:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:13:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:14:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:14:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:14:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:14:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:14:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:14:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:14:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:14:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:14:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:14:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:14:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:14:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:14:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:14:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:14:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:14:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:14:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:14:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:14:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:14:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:14:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:14:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:14:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:14:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:14:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:14:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:14:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:15:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:15:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:15:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:15:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:15:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:15:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:15:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:15:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:15:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:15:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:15:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:15:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:15:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:15:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:15:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:15:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:15:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:15:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:15:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:15:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:15:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:15:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:15:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:15:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:15:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:15:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:15:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:16:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:16:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:16:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:16:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:16:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:16:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:16:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:16:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:16:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:16:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:16:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:16:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:16:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:16:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:16:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:16:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:16:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:16:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:16:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:16:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:16:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:16:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:16:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:16:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:16:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:16:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:16:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:16:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:17:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:17:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:17:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:17:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:17:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:17:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:17:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:17:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:17:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:17:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:17:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:17:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:17:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:17:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:17:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:17:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:17:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:17:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:17:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:17:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:17:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:17:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:17:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:17:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:17:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:17:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:17:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:18:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:18:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:18:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:18:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:18:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:18:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:18:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:18:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:18:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:18:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:18:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:18:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:18:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:18:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:18:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:18:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:18:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:18:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:18:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:18:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:18:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:18:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:18:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:18:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:18:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:18:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:18:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:19:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:19:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:19:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:19:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:19:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:19:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:19:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:19:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:19:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:19:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:19:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:19:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:19:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:19:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:19:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:19:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:19:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:19:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:19:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:19:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:19:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:19:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:19:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:19:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:19:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:19:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:19:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:19:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:20:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:20:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:20:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:20:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:20:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:20:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:20:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:20:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:20:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:20:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:20:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:20:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:20:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:20:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:20:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:20:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:20:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:20:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:20:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:20:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:20:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:20:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:20:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:20:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:20:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:20:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:20:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:21:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:21:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:21:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:21:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:21:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:21:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:21:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:21:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:21:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:21:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:21:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:21:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:21:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:21:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:21:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:21:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:21:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:21:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:21:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:21:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:21:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:21:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:21:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:21:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:21:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:21:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:21:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:22:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:22:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:22:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:22:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:22:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:22:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:22:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:22:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:22:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:22:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:22:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:22:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:22:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:22:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:22:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:22:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:22:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:22:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:22:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:22:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:22:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:22:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:22:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:22:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:22:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:22:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:22:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:22:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:23:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:23:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:23:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:23:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:23:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:23:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:23:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:23:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:23:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:23:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:23:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:23:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:23:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:23:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:23:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:23:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:23:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:23:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:23:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:23:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:23:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:23:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:23:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:23:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:23:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:23:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:23:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:24:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:24:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:24:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:24:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:24:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:24:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:24:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:24:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:24:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:24:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:24:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:24:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:24:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:24:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:24:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:24:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:24:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:24:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:24:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:24:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:24:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:24:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:24:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:24:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:24:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:24:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:24:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:25:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:25:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:25:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:25:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:25:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:25:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:25:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:25:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:25:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:25:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:25:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:25:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:25:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:25:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:25:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:25:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:25:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:25:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:25:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:25:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:25:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:25:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:25:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:25:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:25:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:25:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:25:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:25:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:26:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:26:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:26:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:26:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:26:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:26:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:26:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:26:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:26:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:26:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:26:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:26:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:26:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:26:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:26:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:26:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:26:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:26:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:26:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:26:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:26:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:26:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:26:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:26:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:26:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:26:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:26:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:27:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:27:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:27:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:27:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:27:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:27:11 rdma06 systemd[1]: session-14.scope: Deactivated successfully.
Apr 17 00:27:11 rdma06 systemd[1]: session-14.scope: Consumed 10.246s CPU time.
Apr 17 00:27:11 rdma06 systemd-logind[1064]: Session 14 logged out. Waiting for processes to exit.
Apr 17 00:27:11 rdma06 systemd-logind[1064]: Removed session 14.
Apr 17 00:27:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:27:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:27:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:27:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:27:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:27:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:27:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:27:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:27:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:27:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:27:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:27:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:27:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:27:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:27:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:27:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:27:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:27:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:27:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:27:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:27:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:27:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:28:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:28:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:28:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:28:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:28:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:28:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:28:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:28:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:28:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:28:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:28:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:28:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:28:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:28:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:28:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:28:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:28:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:28:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:28:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:28:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:28:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:28:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:28:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:28:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:28:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:28:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:28:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:28:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:29:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:29:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:29:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:29:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:29:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:29:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:29:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:29:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:29:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:29:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:29:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:29:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:29:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:29:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:29:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:29:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:29:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:29:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:29:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:29:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:29:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:29:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:29:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:29:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:29:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:29:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:29:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:30:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:30:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:30:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:30:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:30:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:30:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:30:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:30:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:30:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:30:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:30:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:30:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:30:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:30:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:30:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:30:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:30:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:30:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:30:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:30:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:30:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:30:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:30:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:30:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:30:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:30:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:30:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:31:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:31:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:31:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:31:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:31:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:31:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:31:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:31:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:31:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:31:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:31:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:31:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:31:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:31:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:31:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:31:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:31:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:31:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:31:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:31:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:31:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:31:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:31:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:31:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:31:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:31:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:31:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:32:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:32:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:32:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:32:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:32:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:32:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:32:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:32:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:32:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:32:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:32:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:32:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:32:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:32:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:32:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:32:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:32:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:32:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:32:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:32:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:32:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:32:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:32:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:32:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:32:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:32:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:32:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:32:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:33:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:33:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:33:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:33:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:33:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:33:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:33:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:33:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:33:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:33:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:33:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:33:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:33:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:33:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:33:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:33:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:33:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:33:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:33:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:33:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:33:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:33:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:33:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:33:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:33:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:33:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:33:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:34:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:34:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:34:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:34:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:34:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:34:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:34:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:34:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:34:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:34:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:34:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:34:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:34:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:34:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:34:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:34:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:34:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:34:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:34:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:34:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:34:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:34:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:34:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:34:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:34:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:34:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:34:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:35:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:35:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:35:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:35:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:35:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:35:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:35:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:35:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:35:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:35:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:35:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:35:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:35:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:35:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:35:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:35:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:35:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:35:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:35:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:35:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:35:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:35:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:35:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:35:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:35:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:35:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:35:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:35:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:36:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:36:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:36:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:36:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:36:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:36:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:36:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:36:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:36:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:36:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:36:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:36:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:36:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:36:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:36:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:36:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:36:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:36:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:36:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:36:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:36:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:36:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:36:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:36:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:36:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:36:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:36:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:37:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:37:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:37:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:37:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:37:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:37:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:37:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:37:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:37:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:37:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:37:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:37:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:37:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:37:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:37:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:37:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:37:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:37:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:37:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:37:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:37:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:37:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:37:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:37:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:37:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:37:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:37:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:38:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:38:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:38:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:38:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:38:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:38:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:38:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:38:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:38:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:38:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:38:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:38:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:38:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:38:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:38:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:38:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:38:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:38:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:38:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:38:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:38:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:38:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:38:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:38:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:38:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:38:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:38:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:38:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:39:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:39:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:39:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:39:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:39:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:39:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:39:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:39:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:39:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:39:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:39:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:39:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:39:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:39:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:39:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:39:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:39:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:39:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:39:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:39:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:39:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:39:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:39:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:39:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:39:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:39:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:39:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:40:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:40:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:40:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:40:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:40:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:40:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:40:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:40:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:40:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:40:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:40:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:40:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:40:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:40:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:40:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:40:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:40:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:40:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:40:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:40:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:40:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:40:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:40:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:40:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:40:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:40:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:40:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:41:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:41:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:41:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:41:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:41:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:41:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:41:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:41:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:41:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:41:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:41:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:41:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:41:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:41:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:41:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:41:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:41:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:41:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:41:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:41:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:41:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:41:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:41:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:41:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:41:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:41:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:41:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:41:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:42:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:42:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:42:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:42:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:42:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:42:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:42:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:42:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:42:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:42:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:42:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:42:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:42:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:42:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:42:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:42:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:42:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:42:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:42:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:42:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:42:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:42:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:42:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:42:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:42:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:42:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:42:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:43:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:43:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:43:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:43:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:43:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:43:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:43:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:43:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:43:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:43:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:43:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:43:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:43:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:43:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:43:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:43:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:43:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:43:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:43:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:43:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:43:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:43:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:43:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:43:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:43:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:43:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:43:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:44:00 rdma06 systemd[1]: Starting dnf makecache...
Apr 17 00:44:01 rdma06 dnf[1135583]: Updating Subscription Management repositories.
Apr 17 00:44:01 rdma06 dnf[1135583]: Unable to read consumer identity
Apr 17 00:44:01 rdma06 dnf[1135583]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 17 00:44:01 rdma06 dnf[1135583]: Metadata cache refreshed recently.
Apr 17 00:44:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 17 00:44:01 rdma06 systemd[1]: Finished dnf makecache.
Apr 17 00:44:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:44:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:44:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:44:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:44:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:44:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:44:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:44:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:44:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:44:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:44:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:44:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:44:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:44:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:44:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:44:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:44:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:44:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:44:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:44:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:44:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:44:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:44:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:44:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:44:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:44:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:44:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:44:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:45:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:45:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:45:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:45:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:45:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:45:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:45:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:45:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:45:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:45:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:45:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:45:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:45:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:45:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:45:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:45:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:45:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:45:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:45:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:45:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:45:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:45:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:45:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:45:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:45:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:45:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:45:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:46:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:46:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:46:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:46:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:46:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:46:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:46:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:46:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:46:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:46:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:46:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:46:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:46:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:46:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:46:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:46:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:46:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:46:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:46:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:46:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:46:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:46:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:46:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:46:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:46:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:46:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:46:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:47:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:47:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:47:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:47:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:47:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:47:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:47:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:47:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:47:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:47:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:47:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:47:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:47:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:47:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:47:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:47:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:47:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:47:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:47:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:47:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:47:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:47:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:47:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:47:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:47:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:47:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:47:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:48:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:48:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:48:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:48:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:48:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:48:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:48:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:48:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:48:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:48:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:48:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:48:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:48:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:48:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:48:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:48:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:48:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:48:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:48:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:48:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:48:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:48:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:48:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:48:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:48:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:48:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:48:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:48:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:49:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:49:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:49:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:49:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:49:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:49:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:49:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:49:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:49:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:49:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:49:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:49:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:49:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:49:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:49:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:49:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:49:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:49:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:49:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:49:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:49:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:49:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:49:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:49:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:49:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:49:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:49:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:50:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:50:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:50:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:50:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:50:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:50:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:50:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:50:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:50:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:50:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:50:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:50:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:50:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:50:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:50:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:50:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:50:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:50:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:50:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:50:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:50:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:50:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:50:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:50:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:50:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:50:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:50:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:51:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:51:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:51:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:51:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:51:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:51:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:51:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:51:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:51:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:51:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:51:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:51:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:51:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:51:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:51:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:51:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:51:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:51:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:51:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:51:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:51:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:51:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:51:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:51:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:51:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:51:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:51:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:51:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:52:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:52:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:52:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:52:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:52:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:52:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:52:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:52:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:52:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:52:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:52:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:52:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:52:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:52:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:52:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:52:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:52:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:52:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:52:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:52:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:52:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:52:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:52:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:52:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:52:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:52:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:52:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:53:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:53:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:53:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:53:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:53:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:53:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:53:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:53:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:53:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:53:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:53:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:53:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:53:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:53:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:53:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:53:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:53:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:53:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:53:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:53:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:53:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:53:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:53:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:53:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:53:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:53:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:53:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:54:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:54:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:54:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:54:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:54:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:54:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:54:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:54:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:54:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:54:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:54:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:54:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:54:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:54:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:54:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:54:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:54:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:54:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:54:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:54:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:54:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:54:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:54:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:54:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:54:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:54:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:54:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:54:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:55:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:55:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:55:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:55:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:55:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:55:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:55:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:55:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:55:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:55:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:55:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:55:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:55:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:55:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:55:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:55:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:55:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:55:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:55:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:55:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:55:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:55:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:55:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:55:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:55:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:55:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:55:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:56:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:56:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:56:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:56:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:56:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:56:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:56:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:56:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:56:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:56:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:56:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:56:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:56:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:56:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:56:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:56:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:56:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:56:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:56:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:56:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:56:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:56:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:56:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:56:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:56:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:56:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:56:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:57:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:57:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:57:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:57:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:57:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:57:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:57:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:57:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:57:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:57:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:57:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:57:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:57:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:57:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:57:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:57:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:57:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:57:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:57:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:57:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:57:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:57:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:57:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:57:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:57:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:57:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:57:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:57:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:58:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:58:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:58:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:58:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:58:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:58:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:58:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:58:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:58:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:58:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:58:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:58:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:58:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:58:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:58:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:58:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:58:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:58:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:58:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:58:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:58:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:58:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:58:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:58:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:58:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:58:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:58:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:59:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:59:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:59:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:59:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:59:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:59:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:59:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:59:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:59:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:59:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:59:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:59:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:59:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:59:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:59:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:59:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:59:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:59:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:59:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:59:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:59:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:59:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 00:59:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 00:59:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 00:59:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 00:59:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 00:59:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:00:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:00:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:00:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:00:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:00:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:00:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:00:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:00:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:00:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:00:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:00:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:00:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:00:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:00:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:00:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:00:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:00:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:00:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:00:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:00:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:00:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:00:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:00:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:00:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:00:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:00:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:00:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:00:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:01:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:01:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:01:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:01:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:01:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:01:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:01:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:01:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:01:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:01:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:01:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:01:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:01:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:01:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:01:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:01:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:01:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:01:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:01:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:01:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:01:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:01:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:01:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:01:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:01:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:01:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:01:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:02:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:02:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:02:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:02:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:02:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:02:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:02:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:02:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:02:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:02:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:02:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:02:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:02:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:02:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:02:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:02:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:02:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:02:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:02:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:02:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:02:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:02:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:02:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:02:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:02:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:02:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:02:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:03:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:03:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:03:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:03:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:03:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:03:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:03:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:03:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:03:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:03:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:03:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:03:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:03:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:03:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:03:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:03:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:03:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:03:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:03:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:03:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:03:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:03:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:03:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:03:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:03:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:03:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:03:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:04:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:04:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:04:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:04:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:04:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:04:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:04:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:04:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:04:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:04:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:04:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:04:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:04:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:04:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:04:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:04:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:04:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:04:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:04:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:04:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:04:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:04:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:04:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:04:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:04:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:04:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:04:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:04:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:05:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:05:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:05:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:05:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:05:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:05:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:05:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:05:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:05:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:05:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:05:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:05:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:05:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:05:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:05:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:05:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:05:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:05:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:05:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:05:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:05:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:05:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:05:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:05:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:05:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:05:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:05:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:06:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:06:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:06:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:06:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:06:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:06:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:06:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:06:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:06:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:06:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:06:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:06:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:06:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:06:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:06:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:06:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:06:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:06:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:06:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:06:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:06:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:06:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:06:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:06:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:06:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:06:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:06:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:07:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:07:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:07:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:07:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:07:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:07:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:07:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:07:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:07:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:07:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:07:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:07:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:07:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:07:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:07:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:07:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:07:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:07:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:07:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:07:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:07:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:07:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:07:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:07:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:07:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:07:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:07:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:07:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:08:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:08:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:08:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:08:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:08:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:08:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:08:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:08:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:08:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:08:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:08:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:08:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:08:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:08:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:08:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:08:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:08:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:08:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:08:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:08:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:08:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:08:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:08:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:08:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:08:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:08:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:08:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:09:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:09:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:09:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:09:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:09:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:09:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:09:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:09:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:09:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:09:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:09:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:09:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:09:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:09:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:09:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:09:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:09:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:09:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:09:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:09:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:09:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:09:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:09:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:09:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:09:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:09:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:09:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:10:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:10:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:10:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:10:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:10:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:10:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:10:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:10:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:10:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:10:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:10:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:10:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:10:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:10:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:10:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:10:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:10:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:10:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:10:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:10:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:10:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:10:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:10:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:10:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:10:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:10:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:10:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:10:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:11:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:11:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:11:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:11:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:11:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:11:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:11:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:11:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:11:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:11:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:11:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:11:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:11:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:11:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:11:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:11:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:11:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:11:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:11:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:11:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:11:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:11:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:11:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:11:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:11:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:11:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:11:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:12:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:12:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:12:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:12:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:12:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:12:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:12:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:12:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:12:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:12:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:12:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:12:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:12:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:12:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:12:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:12:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:12:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:12:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:12:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:12:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:12:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:12:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:12:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:12:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:12:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:12:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:12:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:13:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:13:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:13:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:13:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:13:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:13:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:13:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:13:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:13:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:13:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:13:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:13:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:13:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:13:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:13:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:13:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:13:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:13:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:13:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:13:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:13:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:13:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:13:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:13:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:13:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:13:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:13:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:13:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:14:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:14:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:14:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:14:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:14:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:14:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:14:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:14:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:14:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:14:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:14:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:14:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:14:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:14:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:14:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:14:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:14:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:14:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:14:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:14:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:14:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:14:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:14:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:14:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:14:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:14:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:14:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:15:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:15:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:15:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:15:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:15:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:15:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:15:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:15:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:15:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:15:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:15:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:15:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:15:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:15:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:15:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:15:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:15:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:15:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:15:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:15:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:15:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:15:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:15:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:15:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:15:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:15:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:15:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:16:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:16:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:16:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:16:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:16:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:16:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:16:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:16:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:16:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:16:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:16:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:16:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:16:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:16:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:16:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:16:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:16:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:16:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:16:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:16:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:16:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:16:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:16:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:16:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:16:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:16:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:16:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:16:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:17:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:17:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:17:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:17:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:17:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:17:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:17:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:17:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:17:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:17:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:17:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:17:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:17:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:17:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:17:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:17:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:17:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:17:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:17:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:17:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:17:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:17:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:17:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:17:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:17:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:17:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:17:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:18:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:18:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:18:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:18:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:18:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:18:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:18:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:18:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:18:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:18:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:18:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:18:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:18:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:18:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:18:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:18:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:18:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:18:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:18:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:18:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:18:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:18:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:18:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:18:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:18:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:18:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:18:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:19:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:19:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:19:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:19:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:19:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:19:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:19:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:19:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:19:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:19:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:19:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:19:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:19:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:19:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:19:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:19:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:19:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:19:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:19:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:19:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:19:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:19:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:19:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:19:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:19:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:19:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:19:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:19:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:20:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:20:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:20:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:20:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:20:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:20:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:20:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:20:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:20:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:20:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:20:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:20:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:20:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:20:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:20:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:20:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:20:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:20:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:20:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:20:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:20:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:20:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:20:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:20:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:20:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:20:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:20:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:21:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:21:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:21:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:21:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:21:05 rdma06 sshd[1137094]: main: sshd: ssh-rsa algorithm is disabled
Apr 17 01:21:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:21:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:21:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:21:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:21:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:21:19 rdma06 systemd-logind[1064]: New session 15 of user root.
Apr 17 01:21:19 rdma06 systemd[1]: Started Session 15 of User root.
Apr 17 01:21:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:21:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:21:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:21:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:21:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:21:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:21:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:21:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:21:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:21:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:21:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:21:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:21:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:21:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:21:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:21:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:21:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:21:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:21:58 rdma06 systemd[1]: session-15.scope: Deactivated successfully.
Apr 17 01:21:58 rdma06 systemd-logind[1064]: Session 15 logged out. Waiting for processes to exit.
Apr 17 01:21:58 rdma06 systemd-logind[1064]: Removed session 15.
Apr 17 01:22:02 rdma06 sshd[1137139]: main: sshd: ssh-rsa algorithm is disabled
Apr 17 01:22:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:22:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:22:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:22:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:22:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:22:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:22:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:22:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:22:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:22:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:22:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:22:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:22:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:22:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:22:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:22:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:22:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:22:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:22:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:22:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:22:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:22:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:22:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:22:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:22:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:22:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:22:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:23:00 rdma06 sshd[1137181]: main: sshd: ssh-rsa algorithm is disabled
Apr 17 01:23:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:23:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:23:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:23:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:23:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:23:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:23:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:23:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:23:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:23:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:23:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:23:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:23:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:23:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:23:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:23:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:23:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:23:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:23:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:23:43 rdma06 systemd-logind[1064]: New session 16 of user root.
Apr 17 01:23:43 rdma06 systemd[1]: Started Session 16 of User root.
Apr 17 01:23:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:23:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:23:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:23:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:23:45 rdma06 systemd[1]: Starting Hostname Service...
Apr 17 01:23:45 rdma06 systemd[1]: Started Hostname Service.
Apr 17 01:23:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:23:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:23:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:23:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:23:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:24:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:24:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:24:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:24:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:24:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:24:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:24:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:24:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:24:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:24:15 rdma06 systemd[1]: systemd-hostnamed.service: Deactivated successfully.
Apr 17 01:24:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:24:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:24:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:24:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:24:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:24:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:24:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:24:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:24:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:24:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:24:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:24:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:24:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:24:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:24:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:24:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:24:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:24:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:25:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:25:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:25:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:25:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:25:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:25:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:25:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:25:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:25:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:25:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:25:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:25:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:25:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:25:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:25:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:25:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:25:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:25:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:25:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:25:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:25:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:25:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:25:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:25:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:25:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:25:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:25:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:26:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:26:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:26:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:26:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:26:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:26:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:26:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:26:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:26:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:26:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:26:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:26:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:26:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:26:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:26:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:26:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:26:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:26:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:26:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:26:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:26:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:26:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:26:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:26:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:26:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:26:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:26:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:26:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:27:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:27:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:27:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:27:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:27:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:27:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:27:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:27:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:27:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:27:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:27:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:27:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:27:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:27:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:27:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:27:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:27:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:27:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:27:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:27:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:27:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:27:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:27:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:27:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:27:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:27:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:27:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:28:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:28:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:28:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:28:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:28:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:28:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:28:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:28:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:28:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:28:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:28:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:28:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:28:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:28:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:28:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:28:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:28:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:28:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:28:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:28:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:28:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:28:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:28:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:28:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:28:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:28:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:28:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:29:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:29:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:29:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:29:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:29:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:29:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:29:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:29:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:29:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:29:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:29:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:29:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:29:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:29:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:29:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:29:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:29:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:29:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:29:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:29:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:29:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:29:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:29:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:29:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:29:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:29:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:29:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:29:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:30:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:30:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:30:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:30:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:30:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:30:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:30:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:30:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:30:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:30:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:30:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:30:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:30:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:30:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:30:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:30:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:30:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:30:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:30:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:30:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:30:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:30:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:30:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:30:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:30:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:30:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:30:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:31:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:31:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:31:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:31:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:31:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:31:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:31:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:31:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:31:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:31:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:31:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:31:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:31:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:31:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:31:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:31:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:31:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:31:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:31:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:31:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:31:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:31:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:31:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:31:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:31:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:31:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:31:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:32:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:32:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:32:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:32:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:32:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:32:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:32:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:32:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:32:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:32:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:32:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:32:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:32:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:32:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:32:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:32:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:32:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:32:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:32:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:32:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:32:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:32:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:32:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:32:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:32:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:32:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:32:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:32:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:33:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:33:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:33:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:33:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:33:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:33:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:33:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:33:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:33:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:33:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:33:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:33:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:33:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:33:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:33:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:33:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:33:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:33:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:33:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:33:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:33:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:33:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:33:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:33:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:33:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:33:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:33:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:34:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:34:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:34:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:34:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:34:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:34:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:34:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:34:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:34:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:34:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:34:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:34:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:34:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:34:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:34:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:34:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:34:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:34:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:34:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:34:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:34:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:34:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:34:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:34:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:34:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:34:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:34:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:35:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:35:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:35:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:35:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:35:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:35:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:35:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:35:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:35:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:35:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:35:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:35:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:35:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:35:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:35:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:35:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:35:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:35:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:35:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:35:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:35:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:35:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:35:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:35:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:35:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:35:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:35:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:35:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:36:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:36:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:36:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:36:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:36:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:36:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:36:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:36:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:36:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:36:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:36:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:36:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:36:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:36:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:36:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:36:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:36:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:36:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:36:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:36:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:36:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:36:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:36:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:36:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:36:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:36:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:36:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:37:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:37:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:37:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:37:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:37:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:37:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:37:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:37:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:37:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:37:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:37:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:37:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:37:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:37:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:37:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:37:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:37:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:37:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:37:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:37:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:37:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:37:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:37:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:37:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:37:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:37:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:37:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:38:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:38:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:38:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:38:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:38:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:38:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:38:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:38:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:38:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:38:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:38:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:38:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:38:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:38:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:38:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:38:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:38:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:38:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:38:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:38:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:38:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:38:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:38:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:38:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:38:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:38:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:38:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:39:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:39:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:39:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:39:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:39:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:39:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:39:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:39:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:39:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:39:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:39:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:39:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:39:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:39:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:39:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:39:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:39:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:39:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:39:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:39:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:39:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:39:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:39:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:39:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:39:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:39:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:39:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:39:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:40:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:40:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:40:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:40:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:40:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:40:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:40:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:40:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:40:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:40:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:40:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:40:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:40:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:40:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:40:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:40:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:40:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:40:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:40:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:40:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:40:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:40:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:40:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:40:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:40:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:40:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:40:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:41:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:41:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:41:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:41:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:41:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:41:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:41:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:41:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:41:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:41:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:41:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:41:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:41:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:41:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:41:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:41:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:41:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:41:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:41:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:41:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:41:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:41:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:41:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:41:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:41:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:41:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:41:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:42:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:42:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:42:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:42:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:42:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:42:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:42:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:42:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:42:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:42:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:42:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:42:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:42:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:42:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:42:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:42:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:42:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:42:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:42:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:42:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:42:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:42:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:42:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:42:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:42:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:42:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:42:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:42:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:43:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:43:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:43:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:43:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:43:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:43:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:43:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:43:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:43:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:43:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:43:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:43:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:43:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:43:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:43:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:43:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:43:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:43:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:43:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:43:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:43:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:43:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:43:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:43:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:43:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:43:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:43:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:44:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:44:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:44:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:44:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:44:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:44:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:44:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:44:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:44:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:44:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:44:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:44:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:44:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:44:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:44:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:44:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:44:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:44:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:44:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:44:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:44:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:44:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:44:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:44:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:44:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:44:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:44:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:45:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:45:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:45:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:45:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:45:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:45:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:45:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:45:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:45:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:45:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:45:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:45:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:45:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:45:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:45:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:45:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:45:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:45:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:45:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:45:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:45:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:45:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:45:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:45:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:45:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:45:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:45:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:45:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:46:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:46:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:46:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:46:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:46:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:46:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:46:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:46:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:46:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:46:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:46:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:46:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:46:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:46:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:46:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:46:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:46:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:46:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:46:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:46:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:46:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:46:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:46:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:46:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:46:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:46:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:46:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:47:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:47:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:47:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:47:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:47:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:47:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:47:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:47:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:47:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:47:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:47:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:47:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:47:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:47:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:47:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:47:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:47:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:47:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:47:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:47:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:47:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:47:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:47:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:47:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:47:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:47:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:47:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:48:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:48:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:48:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:48:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:48:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:48:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:48:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:48:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:48:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:48:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:48:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:48:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:48:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:48:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:48:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:48:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:48:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:48:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:48:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:48:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:48:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:48:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:48:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:48:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:48:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:48:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:48:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:48:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:49:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:49:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:49:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:49:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:49:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:49:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:49:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:49:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:49:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:49:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:49:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:49:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:49:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:49:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:49:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:49:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:49:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:49:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:49:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:49:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:49:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:49:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:49:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:49:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:49:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:49:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:49:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:50:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:50:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:50:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:50:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:50:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:50:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:50:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:50:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:50:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:50:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:50:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:50:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:50:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:50:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:50:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:50:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:50:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:50:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:50:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:50:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:50:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:50:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:50:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:50:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:50:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:50:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:50:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:51:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:51:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:51:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:51:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:51:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:51:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:51:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:51:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:51:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:51:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:51:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:51:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:51:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:51:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:51:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:51:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:51:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:51:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:51:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:51:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:51:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:51:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:51:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:51:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:51:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:51:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:51:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:51:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:52:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:52:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:52:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:52:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:52:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:52:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:52:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:52:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:52:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:52:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:52:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:52:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:52:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:52:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:52:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:52:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:52:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:52:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:52:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:52:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:52:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:52:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:52:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:52:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:52:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:52:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:52:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:53:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:53:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:53:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:53:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:53:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:53:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:53:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:53:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:53:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:53:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:53:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:53:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:53:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:53:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:53:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:53:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:53:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:53:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:53:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:53:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:53:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:53:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:53:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:53:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:53:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:53:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:53:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:54:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:54:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:54:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:54:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:54:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:54:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:54:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:54:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:54:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:54:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:54:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:54:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:54:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:54:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:54:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:54:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:54:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:54:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:54:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:54:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:54:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:54:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:54:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:54:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:54:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:54:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:54:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:55:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:55:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:55:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:55:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:55:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:55:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:55:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:55:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:55:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:55:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:55:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:55:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:55:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:55:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:55:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:55:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:55:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:55:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:55:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:55:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:55:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:55:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:55:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:55:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:55:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:55:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:55:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:55:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:56:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:56:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:56:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:56:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:56:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:56:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:56:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:56:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:56:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:56:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:56:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:56:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:56:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:56:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:56:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:56:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:56:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:56:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:56:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:56:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:56:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:56:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:56:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:56:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:56:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:56:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:56:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:57:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:57:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:57:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:57:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:57:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:57:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:57:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:57:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:57:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:57:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:57:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:57:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:57:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:57:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:57:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:57:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:57:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:57:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:57:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:57:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:57:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:57:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:57:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:57:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:57:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:57:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:57:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:58:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:58:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:58:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:58:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:58:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:58:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:58:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:58:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:58:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:58:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:58:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:58:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:58:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:58:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:58:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:58:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:58:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:58:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:58:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:58:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:58:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:58:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:58:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:58:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:58:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:58:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:58:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:58:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:59:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:59:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:59:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:59:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:59:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:59:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:59:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:59:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:59:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:59:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:59:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:59:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:59:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:59:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:59:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:59:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:59:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:59:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:59:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:59:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:59:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:59:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 01:59:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 01:59:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 01:59:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 01:59:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 01:59:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:00:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:00:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:00:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:00:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:00:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:00:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:00:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:00:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:00:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:00:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:00:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:00:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:00:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:00:26 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:00:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:00:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:00:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:00:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:00:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:00:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:00:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:00:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:00:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:00:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:00:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:00:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:00:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:01:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:01:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:01:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:01:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:01:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:01:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:01:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:01:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:01:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:01:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:01:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:01:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:01:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:01:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:01:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:01:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:01:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:01:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:01:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:01:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:01:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:01:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:01:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:01:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:01:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:01:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:01:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:01:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:02:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:02:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:02:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:02:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:02:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:02:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:02:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:02:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:02:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:02:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:02:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:02:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:02:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:02:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:02:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:02:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:02:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:02:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:02:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:02:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:02:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:02:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:02:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:02:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:02:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:02:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:02:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:03:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:03:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:03:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:03:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:03:09 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:03:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:03:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:03:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:03:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:03:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:03:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:03:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:03:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:03:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:03:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:03:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:03:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:03:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:03:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:03:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:03:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:03:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:03:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:03:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:03:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:03:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:03:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:04:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:04:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:04:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:04:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:04:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:04:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:04:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:04:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:04:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:04:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:04:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:04:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:04:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:04:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:04:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:04:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:04:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:04:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:04:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:04:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:04:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:04:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:04:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:04:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:04:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:04:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:04:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:04:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:05:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:05:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:05:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:05:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:05:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:05:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:05:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:05:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:05:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:05:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:05:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:05:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:05:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:05:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:05:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:05:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:05:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:05:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:05:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:05:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:05:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:05:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:05:52 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:05:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:05:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:05:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:05:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:06:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:06:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:06:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:06:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:06:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:06:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:06:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:06:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:06:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:06:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:06:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:06:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:06:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:06:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:06:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:06:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:06:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:06:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:06:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:06:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:06:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:06:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:06:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:06:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:06:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:06:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:06:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:07:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:07:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:07:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:07:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:07:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:07:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:07:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:07:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:07:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:07:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:07:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:07:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:07:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:07:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:07:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:07:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:07:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:07:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:07:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:07:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:07:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:07:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:07:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:07:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:07:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:07:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:07:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:07:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:08:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:08:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:08:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:08:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:08:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:08:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:08:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:08:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:08:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:08:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:08:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:08:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:08:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:08:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:08:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:08:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:08:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:08:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:08:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:08:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:08:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:08:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:08:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:08:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:08:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:08:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:08:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:09:00 rdma06 systemd[1]: Starting dnf makecache...
Apr 17 02:09:01 rdma06 dnf[1139098]: Updating Subscription Management repositories.
Apr 17 02:09:01 rdma06 dnf[1139098]: Unable to read consumer identity
Apr 17 02:09:01 rdma06 dnf[1139098]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 17 02:09:02 rdma06 dnf[1139098]: Copr repo for fmf owned by psss                 2.4 kB/s | 3.3 kB     00:01
Apr 17 02:09:02 rdma06 dnf[1139098]: beaker-AppStream                                 51 kB/s | 168  B     00:00
Apr 17 02:09:02 rdma06 dnf[1139098]: Errors during downloading metadata for repository 'beaker-AppStream':
Apr 17 02:09:02 rdma06 dnf[1139098]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 02:09:02 rdma06 dnf[1139098]: Error: Failed to download metadata for repo 'beaker-AppStream': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 02:09:02 rdma06 dnf[1139098]: beaker-AppStream-debuginfo                       61 kB/s | 168  B     00:00
Apr 17 02:09:02 rdma06 dnf[1139098]: Errors during downloading metadata for repository 'beaker-AppStream-debuginfo':
Apr 17 02:09:02 rdma06 dnf[1139098]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 02:09:02 rdma06 dnf[1139098]: Error: Failed to download metadata for repo 'beaker-AppStream-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 02:09:02 rdma06 dnf[1139098]: beaker-BaseOS                                    38 kB/s | 168  B     00:00
Apr 17 02:09:02 rdma06 dnf[1139098]: Errors during downloading metadata for repository 'beaker-BaseOS':
Apr 17 02:09:02 rdma06 dnf[1139098]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 02:09:02 rdma06 dnf[1139098]: Error: Failed to download metadata for repo 'beaker-BaseOS': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 02:09:02 rdma06 dnf[1139098]: beaker-BaseOS-debuginfo                          38 kB/s | 168  B     00:00
Apr 17 02:09:02 rdma06 dnf[1139098]: Errors during downloading metadata for repository 'beaker-BaseOS-debuginfo':
Apr 17 02:09:02 rdma06 dnf[1139098]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 02:09:02 rdma06 dnf[1139098]: Error: Failed to download metadata for repo 'beaker-BaseOS-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 02:09:02 rdma06 dnf[1139098]: beaker-CRB                                       22 kB/s | 168  B     00:00
Apr 17 02:09:02 rdma06 dnf[1139098]: Errors during downloading metadata for repository 'beaker-CRB':
Apr 17 02:09:02 rdma06 dnf[1139098]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 02:09:02 rdma06 dnf[1139098]: Error: Failed to download metadata for repo 'beaker-CRB': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 02:09:02 rdma06 dnf[1139098]: beaker-CRB-debuginfo                             66 kB/s | 168  B     00:00
Apr 17 02:09:02 rdma06 dnf[1139098]: Errors during downloading metadata for repository 'beaker-CRB-debuginfo':
Apr 17 02:09:02 rdma06 dnf[1139098]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 02:09:02 rdma06 dnf[1139098]: Error: Failed to download metadata for repo 'beaker-CRB-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 02:09:02 rdma06 dnf[1139098]: beaker-HighAvailability                          64 kB/s | 168  B     00:00
Apr 17 02:09:02 rdma06 dnf[1139098]: Errors during downloading metadata for repository 'beaker-HighAvailability':
Apr 17 02:09:02 rdma06 dnf[1139098]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 02:09:02 rdma06 dnf[1139098]: Error: Failed to download metadata for repo 'beaker-HighAvailability': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 02:09:02 rdma06 dnf[1139098]: beaker-HighAvailability-debuginfo                62 kB/s | 168  B     00:00
Apr 17 02:09:02 rdma06 dnf[1139098]: Errors during downloading metadata for repository 'beaker-HighAvailability-debuginfo':
Apr 17 02:09:02 rdma06 dnf[1139098]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 02:09:02 rdma06 dnf[1139098]: Error: Failed to download metadata for repo 'beaker-HighAvailability-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 02:09:02 rdma06 dnf[1139098]: beaker-NFV                                       66 kB/s | 168  B     00:00
Apr 17 02:09:02 rdma06 dnf[1139098]: Errors during downloading metadata for repository 'beaker-NFV':
Apr 17 02:09:02 rdma06 dnf[1139098]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 02:09:02 rdma06 dnf[1139098]: Error: Failed to download metadata for repo 'beaker-NFV': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 02:09:02 rdma06 dnf[1139098]: beaker-NFV-debuginfo                             61 kB/s | 168  B     00:00
Apr 17 02:09:02 rdma06 dnf[1139098]: Errors during downloading metadata for repository 'beaker-NFV-debuginfo':
Apr 17 02:09:02 rdma06 dnf[1139098]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 02:09:02 rdma06 dnf[1139098]: Error: Failed to download metadata for repo 'beaker-NFV-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 02:09:02 rdma06 dnf[1139098]: beaker-RT                                        61 kB/s | 168  B     00:00
Apr 17 02:09:02 rdma06 dnf[1139098]: Errors during downloading metadata for repository 'beaker-RT':
Apr 17 02:09:02 rdma06 dnf[1139098]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 02:09:02 rdma06 dnf[1139098]: Error: Failed to download metadata for repo 'beaker-RT': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 02:09:02 rdma06 dnf[1139098]: beaker-RT-debuginfo                              19 kB/s | 168  B     00:00
Apr 17 02:09:02 rdma06 dnf[1139098]: Errors during downloading metadata for repository 'beaker-RT-debuginfo':
Apr 17 02:09:02 rdma06 dnf[1139098]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 02:09:02 rdma06 dnf[1139098]: Error: Failed to download metadata for repo 'beaker-RT-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 02:09:02 rdma06 dnf[1139098]: beaker-ResilientStorage                          61 kB/s | 168  B     00:00
Apr 17 02:09:02 rdma06 dnf[1139098]: Errors during downloading metadata for repository 'beaker-ResilientStorage':
Apr 17 02:09:02 rdma06 dnf[1139098]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 02:09:02 rdma06 dnf[1139098]: Error: Failed to download metadata for repo 'beaker-ResilientStorage': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 02:09:02 rdma06 dnf[1139098]: beaker-ResilientStorage-debuginfo                65 kB/s | 168  B     00:00
Apr 17 02:09:02 rdma06 dnf[1139098]: Errors during downloading metadata for repository 'beaker-ResilientStorage-debuginfo':
Apr 17 02:09:02 rdma06 dnf[1139098]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 02:09:02 rdma06 dnf[1139098]: Error: Failed to download metadata for repo 'beaker-ResilientStorage-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 02:09:02 rdma06 dnf[1139098]: beaker-SAP                                       45 kB/s | 168  B     00:00
Apr 17 02:09:02 rdma06 dnf[1139098]: Errors during downloading metadata for repository 'beaker-SAP':
Apr 17 02:09:02 rdma06 dnf[1139098]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 02:09:02 rdma06 dnf[1139098]: Error: Failed to download metadata for repo 'beaker-SAP': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 02:09:02 rdma06 dnf[1139098]: beaker-SAP-debuginfo                             68 kB/s | 168  B     00:00
Apr 17 02:09:02 rdma06 dnf[1139098]: Errors during downloading metadata for repository 'beaker-SAP-debuginfo':
Apr 17 02:09:02 rdma06 dnf[1139098]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 02:09:02 rdma06 dnf[1139098]: Error: Failed to download metadata for repo 'beaker-SAP-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 02:09:02 rdma06 dnf[1139098]: beaker-SAPHANA                                   70 kB/s | 168  B     00:00
Apr 17 02:09:02 rdma06 dnf[1139098]: Errors during downloading metadata for repository 'beaker-SAPHANA':
Apr 17 02:09:02 rdma06 dnf[1139098]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 02:09:02 rdma06 dnf[1139098]: Error: Failed to download metadata for repo 'beaker-SAPHANA': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 02:09:02 rdma06 dnf[1139098]: beaker-SAPHANA-debuginfo                         66 kB/s | 168  B     00:00
Apr 17 02:09:02 rdma06 dnf[1139098]: Errors during downloading metadata for repository 'beaker-SAPHANA-debuginfo':
Apr 17 02:09:02 rdma06 dnf[1139098]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 02:09:02 rdma06 dnf[1139098]: Error: Failed to download metadata for repo 'beaker-SAPHANA-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 02:09:03 rdma06 dnf[1139098]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr 17 02:09:04 rdma06 dnf[1139098]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr 17 02:09:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:09:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:09:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:09:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:09:04 rdma06 dnf[1139098]: Ignoring repositories: beaker-AppStream, beaker-AppStream-debuginfo, beaker-BaseOS, beaker-BaseOS-debuginfo, beaker-CRB, beaker-CRB-debuginfo, beaker-HighAvailability, beaker-HighAvailability-debuginfo, beaker-NFV, beaker-NFV-debuginfo, beaker-RT, beaker-RT-debuginfo, beaker-ResilientStorage, beaker-ResilientStorage-debuginfo, beaker-SAP, beaker-SAP-debuginfo, beaker-SAPHANA, beaker-SAPHANA-debuginfo
Apr 17 02:09:04 rdma06 dnf[1139098]: Metadata cache created.
Apr 17 02:09:04 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 17 02:09:04 rdma06 systemd[1]: Finished dnf makecache.
Apr 17 02:09:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:09:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:09:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:09:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:09:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:09:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:09:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:09:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:09:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:09:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:09:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:09:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:09:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:09:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:09:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:09:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:09:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:09:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:09:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:09:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:09:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:09:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:09:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:10:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:10:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:10:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:10:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:10:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:10:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:10:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:10:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:10:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:10:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:10:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:10:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:10:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:10:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:10:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:10:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:10:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:10:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:10:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:10:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:10:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:10:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:10:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:10:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:10:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:10:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:10:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:11:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:11:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:11:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:11:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:11:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:11:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:11:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:11:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:11:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:11:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:11:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:11:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:11:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:11:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:11:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:11:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:11:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:11:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:11:36 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:11:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:11:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:11:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:11:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:11:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:11:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:11:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:11:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:11:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:12:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:12:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:12:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:12:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:12:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:12:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:12:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:12:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:12:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:12:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:12:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:12:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:12:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:12:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:12:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:12:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:12:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:12:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:12:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:12:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:12:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:12:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:12:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:12:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:12:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:12:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:12:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:13:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:13:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:13:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:13:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:13:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:13:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:13:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:13:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:13:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:13:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:13:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:13:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:13:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:13:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:13:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:13:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:13:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:13:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:13:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:13:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:13:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:13:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:13:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:13:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:13:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:13:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:13:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:14:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:14:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:14:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:14:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:14:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:14:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:14:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:14:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:14:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:14:19 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:14:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:14:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:14:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:14:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:14:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:14:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:14:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:14:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:14:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:14:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:14:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:14:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:14:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:14:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:14:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:14:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:14:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:14:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:15:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:15:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:15:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:15:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:15:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:15:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:15:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:15:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:15:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:15:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:15:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:15:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:15:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:15:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:15:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:15:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:15:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:15:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:15:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:15:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:15:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:15:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:15:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:15:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:15:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:15:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:15:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:16:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:16:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:16:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:16:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:16:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:16:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:16:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:16:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:16:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:16:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:16:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:16:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:16:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:16:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:16:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:16:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:16:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:16:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:16:44 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:16:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:16:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:16:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:16:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:16:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:16:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:16:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:16:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:17:02 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:17:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:17:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:17:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:17:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:17:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:17:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:17:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:17:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:17:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:17:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:17:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:17:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:17:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:17:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:17:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:17:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:17:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:17:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:17:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:17:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:17:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:17:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:17:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:17:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:17:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:17:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:17:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:18:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:18:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:18:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:18:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:18:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:18:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:18:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:18:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:18:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:18:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:18:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:18:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:18:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:18:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:18:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:18:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:18:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:18:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:18:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:18:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:18:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:18:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:18:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:18:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:18:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:18:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:18:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:19:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:19:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:19:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:19:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:19:08 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:19:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:19:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:19:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:19:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:19:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:19:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:19:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:19:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:19:27 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:19:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:19:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:19:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:19:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:19:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:19:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:19:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:19:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:19:45 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:19:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:19:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:19:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:19:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:20:03 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:20:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:20:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:20:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:20:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:20:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:20:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:20:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:20:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:20:21 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:20:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:20:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:20:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:20:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:20:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:20:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:20:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:20:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:20:39 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:20:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:20:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:20:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:20:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:20:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:20:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:20:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:20:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:20:57 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:21:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:21:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:21:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:21:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:21:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:21:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:21:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:21:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:21:15 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:21:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:21:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:21:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:21:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:21:33 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:21:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:21:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:21:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:21:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:21:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:21:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:21:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:21:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:21:51 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:21:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:21:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:21:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:21:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:22:03 rdma06 systemd[1]: session-16.scope: Deactivated successfully.
Apr 17 02:22:03 rdma06 systemd[1]: session-16.scope: Consumed 1.166s CPU time.
Apr 17 02:22:03 rdma06 systemd-logind[1064]: Session 16 logged out. Waiting for processes to exit.
Apr 17 02:22:03 rdma06 systemd-logind[1064]: Removed session 16.
Apr 17 02:22:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:22:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:22:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:22:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:22:10 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:22:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:22:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:22:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:22:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:22:14 rdma06 sshd[1139668]: main: sshd: ssh-rsa algorithm is disabled
Apr 17 02:22:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:22:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:22:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:22:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:22:28 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:22:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:22:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:22:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:22:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:22:44 rdma06 systemd-logind[1064]: New session 17 of user root.
Apr 17 02:22:44 rdma06 systemd[1]: Started Session 17 of User root.
Apr 17 02:22:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:22:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:22:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:22:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:22:44 rdma06 systemd[1]: Starting Hostname Service...
Apr 17 02:22:44 rdma06 systemd[1]: Started Hostname Service.
Apr 17 02:22:46 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:22:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:22:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:22:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:22:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:23:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:23:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:23:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:23:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:23:04 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:23:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:23:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:23:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:23:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:23:14 rdma06 systemd[1]: systemd-hostnamed.service: Deactivated successfully.
Apr 17 02:23:22 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:23:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:23:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:23:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:23:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:23:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:23:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:23:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:23:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:23:40 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:23:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:23:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:23:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:23:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:23:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:23:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:23:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:23:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:23:58 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:24:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:24:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:24:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:24:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:24:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:24:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:24:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:24:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:24:16 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:24:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:24:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:24:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:24:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:24:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:24:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:24:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:24:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:24:34 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:24:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:24:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:24:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:24:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:24:53 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:24:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:24:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:24:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:24:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:25:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:25:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:25:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:25:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:25:11 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:25:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:25:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:25:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:25:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:25:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:25:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:25:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:25:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:25:29 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:25:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:25:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:25:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:25:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:25:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:25:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:25:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:25:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:25:47 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:25:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:25:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:25:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:25:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:26:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:26:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:26:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:26:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:26:05 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:26:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:26:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:26:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:26:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:26:23 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:26:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:26:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:26:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:26:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:26:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:26:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:26:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:26:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:26:41 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:26:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:26:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:26:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:26:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:26:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:26:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:26:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:26:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:26:59 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:27:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:27:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:27:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:27:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:27:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:27:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:27:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:27:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:27:17 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:27:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:27:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:27:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:27:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:27:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:27:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:27:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:27:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:27:35 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:27:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:27:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:27:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:27:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:27:54 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:27:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:27:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:27:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:27:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:28:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:28:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:28:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:28:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:28:12 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:28:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:28:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:28:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:28:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:28:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:28:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:28:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:28:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:28:30 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:28:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:28:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:28:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:28:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:28:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:28:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:28:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:28:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:28:48 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:28:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:28:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:28:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:28:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:29:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:29:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:29:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:29:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:29:06 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:29:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:29:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:29:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:29:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:29:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:29:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:29:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:29:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:29:24 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:29:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:29:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:29:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:29:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:29:42 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:29:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:29:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:29:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:29:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:29:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:29:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:29:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:29:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:30:00 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:30:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:30:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:30:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:30:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:30:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:30:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:30:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:30:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:30:18 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:30:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:30:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:30:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:30:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:30:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:30:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:30:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:30:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:30:37 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:30:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:30:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:30:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:30:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:30:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:30:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:30:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:30:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:30:55 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:31:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:31:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:31:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:31:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:31:13 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:31:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:31:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:31:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:31:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:31:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:31:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:31:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:31:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:31:31 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:31:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:31:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:31:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:31:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:31:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:31:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:31:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:31:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:31:49 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:31:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:31:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:31:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:31:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:32:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:32:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:32:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:32:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:32:07 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:32:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:32:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:32:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:32:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:32:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:32:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:32:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:32:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:32:25 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:32:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:32:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:32:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:32:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:32:43 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:32:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:32:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:32:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:32:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:32:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:32:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:32:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:32:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:33:01 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:33:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:33:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:33:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:33:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:33:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:33:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:33:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:33:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:33:20 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:33:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:33:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:33:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:33:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:33:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:33:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:33:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:33:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:33:38 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:33:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:33:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:33:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:33:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:33:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:33:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:33:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:33:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:33:56 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:34:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:34:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:34:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:34:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:34:14 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:34:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:34:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:34:14 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:34:14 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:34:24 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:34:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:34:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:34:24 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:34:32 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:34:34 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:34:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:34:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:34:34 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:34:44 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:34:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:34:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:34:44 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:34:50 rdma06 fm0_sm[380791]: PROGR[pm]: PM: pm_main: Can't get port GUID from SA, will try later
Apr 17 02:34:54 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:34:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:34:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:34:54 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:35:04 rdma06 fm0_sm[380791]: WARN [PmAsyncRcv]: PM: PmPrintFailPort: Unable to Get(PortStatus) rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1
Apr 17 02:35:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmSweepAllPortCounters: Unable to get 1 Ports on 1 Nodes
Apr 17 02:35:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: Integrity of 1240 Exceeded Threshold of 100. rdma06 hfi1_0 Guid 0x001175010174479a LID 0x1 Port 1.  Neighbor: rdma05 hfi1_0 Guid 0x0011750101744776 LID 0x2 Port 1. Details: LQI=0
Apr 17 02:35:04 rdma06 fm0_sm[380791]: WARN [PmEngine]: PM: PmFinalizeAllPortStats: Unable to get 1 Port on 1 Node
Apr 17 02:35:05 rdma06 systemd[1]: Stopping OPA Fabric Manager...
Apr 17 02:35:05 rdma06 opafmd[380779]: Stopped instance 0 of SM.
Apr 17 02:35:05 rdma06 fm0_sm[380791]: PROGR[main]: SM: sm_control_shutdown: FM exiting.
Apr 17 02:35:05 rdma06 systemd[1]: opafm.service: Deactivated successfully.
Apr 17 02:35:05 rdma06 systemd[1]: Stopped OPA Fabric Manager.
Apr 17 02:35:05 rdma06 systemd[1]: opafm.service: Consumed 52min 23.515s CPU time.
Apr 17 02:35:05 rdma06 systemd[1]: Starting OPA Fabric Manager...
Apr 17 02:35:05 rdma06 systemd[1]: Started OPA Fabric Manager.
Apr 17 02:35:05 rdma06 opafmd[1140263]: WARN [main]: APP: XmlParsePrintWarning: Parse warning at line 2293 in tag 'Fm': VirtualFabric configuration:: Multicast group matching XML parse error - Implicit MC Groups did not match any enabled VF.
Apr 17 02:35:05 rdma06 opafmd[1140263]: Parse warning at line 2293 in tag 'Fm': VirtualFabric configuration:: Multicast group matching XML parse error - Implicit MC Groups did not match any enabled VF.
Apr 17 02:35:05 rdma06 opafmd[1140263]: Started instance 0 of SM.
Apr 17 02:35:05 rdma06 fm0_sm[1140265]: [main]: Subnet Manager starting up. LogLevel: 2 LogMode: 0
Apr 17 02:35:05 rdma06 fm0_sm[1140265]: [main]: SM: Version: 10.11.2.0.3
Apr 17 02:35:05 rdma06 fm0_sm[1140265]: [main]: SM: Enabling CoreDumps to /var/crash/opafm up to 10G bytes
Apr 17 02:35:05 rdma06 fm0_sm[1140265]: WARN [main]: SM: RenderVirtualFabrics: VirtualFabric configuration:: Multicast group matching XML parse error - Implicit MC Groups did not match any enabled VF.
Apr 17 02:35:05 rdma06 fm0_sm[1140265]: [main]: SM: Using dynamic packet lifetime values 14, 15, 15, 16, 16, 16, 16, 17, 17
Apr 17 02:35:05 rdma06 fm0_sm[1140265]: [main]: SM: GidPrefix=0xfe80000000000000, Key=0x0000000000000000, MKey=0x0000000000000000 : protect_level=1 : lease=0 seconds, dbsync interval=900 seconds
Apr 17 02:35:05 rdma06 fm0_sm[1140265]: [main]: SM: Size Limits: EndNodePorts=9216 Nodes=11904 Ports=76416 Links=55296
Apr 17 02:35:05 rdma06 fm0_sm[1140265]: [main]: SM: Memory: Pool=262144K SA Resp=38208K
Apr 17 02:35:05 rdma06 fm0_sm[1140265]: PROGR[main]: SM: sm_main: Routing Algorithm in use: shortestpath
Apr 17 02:35:05 rdma06 fm0_sm[1140265]: PROGR[main]: SM: sm_routing_func_assign_sls: 2 QOSGroups require 2 SLs and 2 SCs for operation
Apr 17 02:35:05 rdma06 fm0_sm[1140265]: PROGR[main]: SM: sm_update_bw: QOSGroup Bandwidth Allocations : Total allocated bandwidth is 100%
Apr 17 02:35:05 rdma06 fm0_sm[1140265]: PROGR[main]: SM: sm_update_bw: QoS(LowPriority) shared by 1 VFs; Assigned 100% Bandwidth on BaseSL 0
Apr 17 02:35:05 rdma06 fm0_sm[1140265]: PROGR[main]: SM: sm_update_bw: QoS(Tenant_QOS) shared by 0 VFs; Assigned 0% Bandwidth on BaseSL 1
Apr 17 02:35:05 rdma06 fm0_sm[1140265]: PROGR[main]: SM: [VF:Admin] : Base SL:0 Resp SL:0 Requires Resp SL:0 Multicast SL:0 QOS:1 HP:0 PKey:0x7fff
Apr 17 02:35:07 rdma06 fm0_sm[1140265]: [main]: SM: Using: HFI 1 Port 1 PortGuid 0x001175010174479a
Apr 17 02:35:09 rdma06 fm0_sm[1140265]: PROGR[topology]: SM: sm_transition: SM STATE TRANSITION from NOTACTIVE to DISCOVERING
Apr 17 02:35:09 rdma06 fm0_sm[1140265]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Initial sweep.
Apr 17 02:35:09 rdma06 fm0_sm[1140265]: WARN [topology]: SM: _query_local_port: Waiting for port (portnum=1) to reach state INIT (portstate=Down)...
Apr 17 02:35:10 rdma06 fm0_sm[1140265]: PROGR[sm_dbsync]: IF3: sm_wait_ready: Service waiting for SM to go into MASTER/STANDBY state
Apr 17 02:35:10 rdma06 fm0_sm[1140265]: PROGR[pm]: PM: sm_wait_ready: Service waiting for SM to go into MASTER/STANDBY state
Apr 17 02:36:38 rdma06 systemd[1]: Stopping OPA Fabric Manager...
Apr 17 02:36:38 rdma06 opafmd[1140263]: Stopped instance 0 of SM.
Apr 17 02:36:38 rdma06 fm0_sm[1140265]: PROGR[main]: SM: sm_control_shutdown: FM exiting.
Apr 17 02:36:39 rdma06 systemd[1]: opafm.service: Deactivated successfully.
Apr 17 02:36:39 rdma06 systemd[1]: Stopped OPA Fabric Manager.
Apr 17 02:36:39 rdma06 systemd[1]: Starting OPA Fabric Manager...
Apr 17 02:36:39 rdma06 systemd[1]: Started OPA Fabric Manager.
Apr 17 02:36:39 rdma06 opafmd[1140339]: WARN [main]: APP: XmlParsePrintWarning: Parse warning at line 2293 in tag 'Fm': VirtualFabric configuration:: Multicast group matching XML parse error - Implicit MC Groups did not match any enabled VF.
Apr 17 02:36:39 rdma06 opafmd[1140339]: Parse warning at line 2293 in tag 'Fm': VirtualFabric configuration:: Multicast group matching XML parse error - Implicit MC Groups did not match any enabled VF.
Apr 17 02:36:39 rdma06 opafmd[1140339]: Started instance 0 of SM.
Apr 17 02:36:39 rdma06 fm0_sm[1140340]: [main]: Subnet Manager starting up. LogLevel: 2 LogMode: 0
Apr 17 02:36:39 rdma06 fm0_sm[1140340]: [main]: SM: Version: 10.11.2.0.3
Apr 17 02:36:39 rdma06 fm0_sm[1140340]: [main]: SM: Enabling CoreDumps to /var/crash/opafm up to 10G bytes
Apr 17 02:36:39 rdma06 fm0_sm[1140340]: WARN [main]: SM: RenderVirtualFabrics: VirtualFabric configuration:: Multicast group matching XML parse error - Implicit MC Groups did not match any enabled VF.
Apr 17 02:36:39 rdma06 fm0_sm[1140340]: [main]: SM: Using dynamic packet lifetime values 14, 15, 15, 16, 16, 16, 16, 17, 17
Apr 17 02:36:39 rdma06 fm0_sm[1140340]: [main]: SM: GidPrefix=0xfe80000000000000, Key=0x0000000000000000, MKey=0x0000000000000000 : protect_level=1 : lease=0 seconds, dbsync interval=900 seconds
Apr 17 02:36:39 rdma06 fm0_sm[1140340]: [main]: SM: Size Limits: EndNodePorts=9216 Nodes=11904 Ports=76416 Links=55296
Apr 17 02:36:39 rdma06 fm0_sm[1140340]: [main]: SM: Memory: Pool=262144K SA Resp=38208K
Apr 17 02:36:39 rdma06 fm0_sm[1140340]: PROGR[main]: SM: sm_main: Routing Algorithm in use: shortestpath
Apr 17 02:36:39 rdma06 fm0_sm[1140340]: PROGR[main]: SM: sm_routing_func_assign_sls: 2 QOSGroups require 2 SLs and 2 SCs for operation
Apr 17 02:36:39 rdma06 fm0_sm[1140340]: PROGR[main]: SM: sm_update_bw: QOSGroup Bandwidth Allocations : Total allocated bandwidth is 100%
Apr 17 02:36:39 rdma06 fm0_sm[1140340]: PROGR[main]: SM: sm_update_bw: QoS(LowPriority) shared by 1 VFs; Assigned 100% Bandwidth on BaseSL 0
Apr 17 02:36:39 rdma06 fm0_sm[1140340]: PROGR[main]: SM: sm_update_bw: QoS(Tenant_QOS) shared by 0 VFs; Assigned 0% Bandwidth on BaseSL 1
Apr 17 02:36:39 rdma06 fm0_sm[1140340]: PROGR[main]: SM: [VF:Admin] : Base SL:0 Resp SL:0 Requires Resp SL:0 Multicast SL:0 QOS:1 HP:0 PKey:0x7fff
Apr 17 02:36:40 rdma06 fm0_sm[1140340]: [main]: SM: Using: HFI 1 Port 1 PortGuid 0x001175010174479a
Apr 17 02:36:43 rdma06 fm0_sm[1140340]: PROGR[topology]: SM: sm_transition: SM STATE TRANSITION from NOTACTIVE to DISCOVERING
Apr 17 02:36:43 rdma06 fm0_sm[1140340]: PROGR[topology]: SM: topology_main: TT: DISCOVERY CYCLE START - REASON: Initial sweep.
Apr 17 02:36:43 rdma06 fm0_sm[1140340]: WARN [topology]: SM: _query_local_port: Waiting for port (portnum=1) to reach state INIT (portstate=Down)...
Apr 17 02:36:43 rdma06 fm0_sm[1140340]: PROGR[sm_dbsync]: IF3: sm_wait_ready: Service waiting for SM to go into MASTER/STANDBY state
Apr 17 02:36:44 rdma06 fm0_sm[1140340]: PROGR[pm]: PM: sm_wait_ready: Service waiting for SM to go into MASTER/STANDBY state
Apr 17 04:03:00 rdma06 systemd[1]: Starting dnf makecache...
Apr 17 04:03:01 rdma06 dnf[1142564]: Updating Subscription Management repositories.
Apr 17 04:03:01 rdma06 dnf[1142564]: Unable to read consumer identity
Apr 17 04:03:01 rdma06 dnf[1142564]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 17 04:03:01 rdma06 dnf[1142564]: Metadata cache refreshed recently.
Apr 17 04:03:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 17 04:03:01 rdma06 systemd[1]: Finished dnf makecache.
Apr 17 04:20:32 rdma06 NetworkManager[1025]: <info>  [1681719632.8825] dhcp4 (eno1): state changed new lease, address=10.73.131.45
Apr 17 04:23:34 rdma06 opafmd[1140340]:         Seconds since last cleared: 6413
Apr 17 04:23:34 rdma06 opafmd[1140340]:                  Time last cleared: Mon Apr 17 02:36:41 2023
Apr 17 04:23:34 rdma06 opafmd[1140340]:                            COUNTER: THIS SWEEP LAST SWEEP      TOTAL
Apr 17 04:23:34 rdma06 opafmd[1140340]: ------------------------------------ ---------- ---------- ----------
Apr 17 04:23:34 rdma06 opafmd[1140340]:   SM State transition to DISCOVERY:          1          0          1
Apr 17 04:23:34 rdma06 opafmd[1140340]:      SM State transition to MASTER:          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:     SM State transition to STANDBY:          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:    SM State transition to INACTIVE:          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:      Total transmitted SMA Packets:       1284          0       1284
Apr 17 04:23:34 rdma06 opafmd[1140340]:          Direct Routed SMA Packets:       1283          0       1283
Apr 17 04:23:34 rdma06 opafmd[1140340]:             LID Routed SMA Packets:          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:              SMA Query Retransmits:          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:    SMA Query Retransmits Exhausted:          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:                  SM TX GET(Notice):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:                  SM TX SET(Notice):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:           Total SM RX TRAP(Notice):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:                 SM RX TRAP(PortUp):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:               SM RX TRAP(PortDown):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:            SM RX TRAP(McGrpCreate):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:            SM RX TRAP(McGrpDelete):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:                 SM RX TRAP(UnPath):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:                 SM RX TRAP(RePath):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:           SM RX TRAP(PortStateChg):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:          SM RX TRAP(LinkIntegrity):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:          SM RX TRAP(BufferOverrun):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:            SM RX TRAP(FlowControl):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:            SM RX TRAP(LocalChange):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:            SM RX TRAP(SysImageChg):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:                SM RX TRAP(BadMKey):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:                SM RX TRAP(BadPkey):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:                SM RX TRAP(BadQkey):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:          SM RX TRAP(BadPkeySwPort):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:     SM RX TRAP(LinkWidthDowngrade):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:       SM RX TRAP(CostMatrixChange):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:          SM TX TRAPREPRESS(Notice):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:         SM TX GET(NodeDescription):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:                SM TX GET(NodeInfo):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:              SM TX GET(SwitchInfo):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:              SM TX SET(SwitchInfo):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:                SM TX GET(PortInfo):       1283          0       1283
Apr 17 04:23:34 rdma06 opafmd[1140340]:                SM TX SET(PortInfo):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:           SM TX GET(PortStateInfo):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:           SM TX SET(PortStateInfo):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:               SM TX GET(PKeyTable):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:               SM TX SET(PKeyTable):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:       SM TX GET(SL2SCMappingTable):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:       SM TX SET(SL2SCMappingTable):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:       SM TX GET(SC2SLMappingTable):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:       SM TX SET(SC2SLMappingTable):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:      SM TX GET(SC2VLtMappingTable):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:      SM TX SET(SC2VLtMappingTable):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:     SM TX GET(SC2VLntMappingTable):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:     SM TX SET(SC2VLntMappingTable):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:      SM TX GET(SC2VLrMappingTable):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:      SM TX SET(SC2VLrMappingTable):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:       SM TX GET(SC2SCMappingTable):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:       SM TX SET(SC2SCMappingTable):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:           SM TX SET(SC2SCMultiSet):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:      SM TX GET(VLArbitrationTable):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:      SM TX SET(VLArbitrationTable):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:                     SM TX GET(LFT):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:                     SM TX SET(LFT):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:                     SM TX GET(MFT):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:                     SM TX SET(MFT):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:               SM TX GET(AGGREGATE):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:               SM TX SET(AGGREGATE):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:                 SM TX GET(LedInfo):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:                 SM TX SET(LedInfo):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:               SM TX GET(CableInfo):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:                  SM RX GET(SmInfo):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:                  SM TX GET(SmInfo):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:                  SM RX SET(SmInfo):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:                  SM TX SET(SmInfo):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:              SM TX GETRESP(SmInfo):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:               SM TX GET(PortGroup):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:               SM TX SET(PortGroup):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:                    SM TX GET(PGFT):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:                    SM TX SET(PGFT):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:      SM TX GET(BufferControlTable):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:      SM TX SET(BufferControlTable):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:               SM TX GET(ARLidMask):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:               SM TX SET(ARLidMask):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:          SM RX GET(CongestionInfo):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:       SM RX GET(HfiCongestSetting):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:       SM RX SET(HfiCongestSetting):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:       SM RX GET(HfiCongestControl):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:       SM RX SET(HfiCongestControl):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:        SM RX GET(SwCongestSetting):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:        SM RX SET(SwCongestSetting):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:  SM RX GET(SwPortCongestionSetting:          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:                   SM RX GETRESP(*):       2566          0       2566
Apr 17 04:23:34 rdma06 opafmd[1140340]:                  SM RX STATUS BUSY:          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:              SM RX STATUS REDIRECT:          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:              SM RX STATUS BADCLASS:          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:             SM RX STATUS BADMETHOD:          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:         SM RX STATUS BADMETHODATTR:          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:              SM RX STATUS BADFIELD:          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:               SM RX STATUS UNKNOWN:          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:           SA RX GET(ClassPortInfo):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:               SA TX REPORT(Notice):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:              SA RX SET(InformInfo):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:              SA RX GET(NodeRecord):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:           SA RX GETTBL(NodeRecord):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:                SA RX GET(PortInfo):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:             SA RX GETTBL(PortInfo):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:                SA RX GET(SC2SCMAP):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:             SA RX GETTBL(SC2SCMap):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:                SA RX GET(SL2SCMAP):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:             SA RX GETTBL(SL2SCMap):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:                SA RX GET(SC2SLMap):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:             SA RX GETTBL(SC2SLMap):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:               SA RX GET(SC2VLtMap):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:            SA RX GETTBL(SC2VLtMap):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:              SA RX GET(SC2VLntMap):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:           SA RX GETTBL(SC2VLntMap):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:               SA RX GET(SC2VLrMap):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:            SA RX GETTBL(SC2VLrMap):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:              SA RX GET(SwitchInfo):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:          SA RX GETTBLE(SwitchInfo):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:                     SA RX GET(LFT):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:                  SA RX GETTBL(LFT):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:                     SA RX GET(MFT):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:                  SA RX GETTBL(MFT):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:           SA RX GET(VLArbitration):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:        SA RX GETTBL(VLArbitration):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:                  SA RX GET(SmInfo):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:               SA RX GETTBL(SmInfo):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:              SA RX GET(InformInfo):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:           SA RX GETTBL(InformInfo):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:              SA RX GET(LinkRecord):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:           SA RX GETTBL(LinkRecord):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:           SA RX GET(ServiceRecord):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:           SA RX SET(ServiceRecord):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:        SA RX GETTBL(ServiceRecord):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:        SA RX DELETE(ServiceRecord):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:              SA RX GET(PKeyRecord):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:           SA RX GETTBL(PKeyRecord):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:              SA RX GET(PathRecord):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:           SA RX GETTBL(PathRecord):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:          SA RX GET(McMemberRecord):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:          SA RX SET(McMemberRecord):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:       SA RX GETTBL(McMemberRecord):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:       SA RX DELETE(McMemberRecord):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:        SA RX GETTRACE(TraceRecord):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:    SA RX GETMULTI(MultiPathRecord):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:           SA RX GET(vFabricRecord):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:        SA RX GETTBL(vFabricRecord):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:        SA RX GET(FabricInfoRecord):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:   SA RX GET(QuarantinedNodeRecord):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]: SA RX GETTBL(QuarantinedNodeRecord):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:          SA RX GET(CongInfoRecord):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:       SA RX GETTBL(CongInfoRecord):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:        SA RX GET(SwitchCongRecord):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:     SA RX GETTBL(SwitchCongRecord):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:    SA RX GET(SwitchPortCongRecord):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]: SA RX GETTBL(SwitchPortCongRecord):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:           SA RX GET(HFICongRecord):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:           SA RX GET(HFICongRecord):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:       SA RX GET(HFICongCtrlRecord):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:       SA RX GET(HFICongCtrlRecord):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:           SA RX GET(BfrCtrlRecord):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:        SA RX GETTBL(BfrCtrlRecord):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:               SA RX GET(CableInfo):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:            SA RX GETTBL(CableInfo):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:         SA RX GET(PortGroupRecord):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:      SA RX GETTBL(PortGroupRecord):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:      SA RX GET(PortGroupFwdRecord):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:   SA RX GETTBL(PortGroupFwdRecord):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:                             (null):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:                             (null):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:           SA RX REPORTRESP(Notice):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]: SA RX GET(DeviceGroupMemberRecord):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]: SA RX GETTBL(DeviceGroupMemberRecord):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:   SA RX GET(DeviceGroupNameRecord):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]: SA RX GETTBL(DeviceGroupNameRecord):          0          0          0
Apr 17 04:23:34 rdma06 opafmd[1140340]:  SA RX GET(DeviceTreeMemberRecord):          0          0          0
Apr 17 04:23:34 rdma06 fm0_sm[1140340]: [RT-Conf server]: Setting SM LogLevel to 7
Apr 17 04:23:38 rdma06 opafmd[1140340]: SAibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:23:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:23:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:23:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:23:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:23:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:23:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:23:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:23:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:23:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:23:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:23:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:23:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:23:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:23:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:23:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:23:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:23:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:23:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:23:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:23:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:23:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:23:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:23:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:23:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:23:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:23:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:23:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:23:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:23:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:23:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:23:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:23:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:23:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:23:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:23:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:23:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:23:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:23:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:23:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:24:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:24:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:24:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:24:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:24:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:24:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:24:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:24:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:24:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:24:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:24:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:24:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:24:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:24:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:24:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:24:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:24:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:24:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:24:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:24:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:24:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:24:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:24:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:24:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:24:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:24:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:24:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:24:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:24:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:24:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:24:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:24:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:24:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:24:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:24:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:24:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:24:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:24:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:24:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:24:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:24:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:24:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:24:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:24:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:24:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:24:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:24:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:24:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:24:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:24:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:24:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:24:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:24:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:24:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:24:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:24:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:24:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:24:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:24:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:24:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:24:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:24:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:24:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:24:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:24:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:24:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:24:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:24:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:24:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:24:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:24:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:24:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:24:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:24:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:24:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:24:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:24:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:24:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:24:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:24:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:24:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:24:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:24:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:24:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:24:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:24:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:24:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:24:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:24:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:24:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:24:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:24:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:24:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:24:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:24:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:24:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:24:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:24:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:24:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:24:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:24:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:24:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:24:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:24:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:24:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:24:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:24:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:24:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:24:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:24:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:24:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:24:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:24:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:24:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:24:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:24:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:24:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:24:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:24:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:24:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:24:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:24:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:24:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:24:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:24:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:24:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:24:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:24:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:24:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:24:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:24:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:24:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:24:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:24:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:24:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:24:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:24:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:24:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:24:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:24:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:24:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:24:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:24:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:24:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:24:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:24:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:24:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:24:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:24:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:24:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:24:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:24:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:24:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:24:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:24:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:24:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:24:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:24:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:24:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:24:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:24:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:24:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:24:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:24:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:24:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:24:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:24:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:24:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:24:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:24:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:24:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:24:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:24:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:24:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:24:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:24:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:24:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:24:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:24:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:24:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:24:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:24:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:24:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:24:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:24:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:24:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:24:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:24:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:24:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:24:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:24:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:24:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:24:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:24:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:24:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:24:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:25:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:25:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:25:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:25:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:25:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:25:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:25:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:25:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:25:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:25:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:25:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:25:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:25:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:25:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:25:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:25:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:25:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:25:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:25:04 rdma06 systemd-logind[1064]: Session 17 logged out. Waiting for processes to exit.
Apr 17 04:25:04 rdma06 systemd[1]: session-17.scope: Deactivated successfully.
Apr 17 04:25:04 rdma06 systemd[1]: session-17.scope: Consumed 18.542s CPU time.
Apr 17 04:25:04 rdma06 systemd-logind[1064]: Removed session 17.
Apr 17 04:25:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:25:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:25:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:25:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:25:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:25:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:25:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:25:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:25:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:25:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:25:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:25:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:25:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:25:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:25:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:25:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:25:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:25:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:25:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:25:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:25:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:25:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:25:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:25:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:25:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:25:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:25:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:25:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:25:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:25:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:25:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:25:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:25:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:25:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:25:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:25:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:25:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:25:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:25:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:25:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:25:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:25:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:25:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:25:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:25:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:25:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:25:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:25:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:25:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:25:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:25:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:25:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:25:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:25:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:25:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:25:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:25:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:25:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:25:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:25:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:25:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:25:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:25:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:25:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:25:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:25:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:25:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:25:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:25:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:25:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:25:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:25:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:25:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:25:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:25:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:25:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:25:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:25:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:25:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:25:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:25:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:25:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:25:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:25:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:25:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:25:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:25:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:25:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:25:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:25:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:25:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:25:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:25:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:25:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:25:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:25:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:25:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:25:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:25:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:25:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:25:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:25:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:25:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:25:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:25:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:25:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:25:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:25:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:25:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:25:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:25:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:25:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:25:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:25:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:25:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:25:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:25:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:25:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:25:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:25:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:25:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:25:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:25:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:25:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:25:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:25:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:25:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:25:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:25:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:25:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:25:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:25:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:25:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:25:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:25:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:25:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:25:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:25:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:25:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:25:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:25:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:25:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:25:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:25:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:25:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:25:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:25:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:25:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:26:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:26:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:26:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:26:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:26:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:26:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:26:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:26:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:26:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:26:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:26:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:26:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:26:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:26:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:26:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:26:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:26:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:26:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:26:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:26:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:26:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:26:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:26:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:26:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:26:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:26:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:26:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:26:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:26:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:26:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:26:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:26:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:26:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:26:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:26:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:26:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:26:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:26:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:26:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:26:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:26:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:26:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:26:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:26:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:26:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:26:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:26:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:26:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:26:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:26:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:26:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:26:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:26:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:26:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:26:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:26:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:26:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:26:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:26:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:26:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:26:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:26:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:26:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:26:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:26:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:26:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:26:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:26:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:26:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:26:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:26:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:26:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:26:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:26:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:26:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:26:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:26:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:26:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:26:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:26:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:26:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:26:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:26:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:26:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:26:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:26:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:26:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:26:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:26:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:26:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:26:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:26:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:26:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:26:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:26:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:26:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:26:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:26:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:26:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:26:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:26:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:26:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:26:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:26:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:26:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:26:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:26:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:26:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:26:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:26:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:26:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:26:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:26:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:26:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:26:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:26:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:26:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:26:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:26:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:26:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:26:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:26:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:26:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:26:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:26:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:26:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:26:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:26:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:26:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:26:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:26:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:26:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:26:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:26:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:26:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:26:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:26:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:26:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:26:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:26:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:26:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:26:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:26:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:26:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:26:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:26:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:26:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:26:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:26:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:26:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:26:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:26:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:26:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:26:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:26:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:26:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:26:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:26:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:26:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:26:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:26:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:26:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:26:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:26:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:26:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:26:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:26:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:26:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:26:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:26:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:26:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:26:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:26:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:26:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:26:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:26:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:26:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:26:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:26:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:26:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:26:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:26:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:26:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:26:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:26:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:26:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:26:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:26:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:26:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:26:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:26:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:26:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:26:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:26:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:26:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:26:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:26:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:26:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:26:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:26:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:26:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:26:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:26:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:26:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:26:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:26:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:26:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:26:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:26:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:26:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:26:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:26:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:26:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:26:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:26:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:26:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:27:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:27:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:27:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:27:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:27:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:27:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:27:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:27:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:27:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:27:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:27:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:27:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:27:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:27:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:27:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:27:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:27:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:27:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:27:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:27:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:27:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:27:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:27:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:27:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:27:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:27:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:27:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:27:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:27:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:27:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:27:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:27:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:27:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:27:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:27:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:27:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:27:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:27:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:27:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:27:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:27:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:27:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:27:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:27:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:27:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:27:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:27:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:27:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:27:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:27:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:27:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:27:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:27:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:27:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:27:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:27:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:27:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:27:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:27:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:27:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:27:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:27:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:27:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:27:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:27:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:27:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:27:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:27:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:27:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:27:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:27:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:27:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:27:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:27:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:27:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:27:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:27:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:27:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:27:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:27:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:27:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:27:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:27:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:27:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:27:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:27:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:27:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:27:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:27:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:27:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:27:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:27:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:27:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:27:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:27:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:27:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:27:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:27:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:27:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:27:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:27:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:27:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:27:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:27:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:27:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:27:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:27:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:27:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:27:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:27:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:27:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:27:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:27:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:27:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:27:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:27:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:27:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:27:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:27:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:27:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:27:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:27:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:27:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:27:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:27:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:27:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:27:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:27:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:27:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:27:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:27:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:27:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:27:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:27:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:27:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:27:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:27:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:27:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:27:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:27:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:27:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:27:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:27:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:27:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:27:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:27:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:27:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:27:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:27:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:27:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:27:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:27:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:27:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:27:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:27:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:27:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:27:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:27:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:27:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:27:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:27:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:27:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:27:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:27:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:27:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:27:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:27:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:27:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:27:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:27:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:27:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:27:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:27:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:27:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:27:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:27:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:27:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:27:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:27:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:27:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:27:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:27:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:27:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:27:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:27:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:27:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:27:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:27:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:27:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:27:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:27:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:27:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:27:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:27:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:27:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:27:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:27:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:27:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:27:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:27:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:27:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:27:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:27:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:27:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:27:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:27:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:27:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:27:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:27:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:27:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:27:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:27:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:27:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:27:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:27:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:27:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:28:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:28:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:28:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:28:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:28:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:28:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:28:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:28:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:28:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:28:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:28:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:28:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:28:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:28:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:28:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:28:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:28:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:28:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:28:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:28:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:28:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:28:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:28:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:28:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:28:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:28:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:28:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:28:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:28:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:28:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:28:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:28:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:28:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:28:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:28:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:28:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:28:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:28:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:28:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:28:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:28:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:28:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:28:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:28:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:28:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:28:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:28:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:28:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:28:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:28:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:28:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:28:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:28:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:28:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:28:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:28:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:28:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:28:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:28:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:28:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:28:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:28:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:28:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:28:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:28:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:28:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:28:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:28:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:28:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:28:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:28:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:28:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:28:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:28:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:28:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:28:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:28:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:28:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:28:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:28:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:28:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:28:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:28:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:28:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:28:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:28:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:28:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:28:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:28:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:28:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:28:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:28:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:28:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:28:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:28:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:28:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:28:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:28:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:28:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:28:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:28:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:28:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:28:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:28:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:28:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:28:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:28:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:28:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:28:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:28:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:28:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:28:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:28:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:28:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:28:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:28:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:28:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:28:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:28:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:28:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:28:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:28:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:28:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:28:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:28:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:28:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:28:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:28:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:28:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:28:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:28:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:28:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:28:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:28:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:28:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:28:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:28:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:28:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:28:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:28:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:28:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:28:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:28:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:28:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:28:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:28:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:28:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:28:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:28:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:28:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:28:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:28:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:28:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:28:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:28:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:28:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:28:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:28:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:28:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:28:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:28:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:28:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:28:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:28:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:28:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:28:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:28:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:28:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:28:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:28:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:28:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:28:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:28:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:28:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:28:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:28:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:28:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:28:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:28:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:28:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:28:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:28:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:28:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:28:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:28:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:28:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:28:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:28:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:28:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:28:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:28:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:28:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:28:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:28:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:28:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:28:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:28:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:28:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:28:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:28:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:28:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:28:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:28:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:28:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:28:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:28:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:28:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:28:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:28:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:28:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:28:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:28:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:28:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:28:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:28:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:28:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:29:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:29:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:29:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:29:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:29:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:29:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:29:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:29:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:29:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:29:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:29:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:29:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:29:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:29:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:29:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:29:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:29:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:29:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:29:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:29:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:29:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:29:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:29:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:29:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:29:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:29:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:29:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:29:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:29:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:29:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:29:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:29:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:29:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:29:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:29:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:29:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:29:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:29:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:29:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:29:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:29:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:29:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:29:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:29:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:29:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:29:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:29:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:29:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:29:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:29:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:29:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:29:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:29:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:29:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:29:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:29:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:29:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:29:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:29:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:29:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:29:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:29:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:29:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:29:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:29:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:29:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:29:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:29:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:29:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:29:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:29:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:29:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:29:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:29:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:29:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:29:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:29:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:29:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:29:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:29:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:29:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:29:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:29:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:29:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:29:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:29:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:29:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:29:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:29:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:29:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:29:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:29:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:29:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:29:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:29:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:29:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:29:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:29:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:29:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:29:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:29:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:29:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:29:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:29:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:29:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:29:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:29:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:29:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:29:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:29:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:29:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:29:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:29:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:29:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:29:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:29:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:29:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:29:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:29:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:29:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:29:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:29:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:29:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:29:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:29:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:29:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:29:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:29:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:29:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:29:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:29:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:29:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:29:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:29:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:29:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:29:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:29:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:29:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:29:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:29:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:29:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:29:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:29:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:29:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:29:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:29:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:29:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:29:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:29:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:29:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:29:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:29:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:29:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:29:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:29:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:29:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:29:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:29:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:29:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:29:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:29:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:29:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:29:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:29:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:29:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:29:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:29:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:29:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:29:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:29:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:29:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:29:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:29:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:29:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:29:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:29:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:29:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:29:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:29:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:29:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:29:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:29:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:29:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:29:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:29:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:29:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:29:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:29:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:29:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:29:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:29:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:29:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:29:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:29:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:29:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:29:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:29:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:29:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:29:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:29:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:29:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:29:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:29:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:29:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:29:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:29:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:29:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:29:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:29:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:29:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:29:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:29:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:29:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:29:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:29:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:29:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:30:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:30:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:30:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:30:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:30:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:30:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:30:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:30:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:30:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:30:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:30:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:30:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:30:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:30:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:30:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:30:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:30:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:30:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:30:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:30:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:30:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:30:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:30:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:30:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:30:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:30:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:30:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:30:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:30:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:30:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:30:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:30:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:30:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:30:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:30:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:30:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:30:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:30:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:30:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:30:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:30:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:30:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:30:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:30:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:30:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:30:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:30:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:30:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:30:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:30:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:30:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:30:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:30:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:30:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:30:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:30:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:30:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:30:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:30:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:30:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:30:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:30:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:30:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:30:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:30:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:30:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:30:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:30:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:30:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:30:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:30:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:30:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:30:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:30:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:30:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:30:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:30:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:30:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:30:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:30:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:30:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:30:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:30:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:30:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:30:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:30:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:30:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:30:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:30:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:30:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:30:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:30:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:30:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:30:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:30:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:30:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:30:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:30:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:30:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:30:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:30:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:30:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:30:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:30:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:30:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:30:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:30:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:30:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:30:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:30:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:30:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:30:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:30:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:30:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:30:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:30:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:31:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:31:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:31:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:31:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:31:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:31:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:31:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:31:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:31:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:31:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:31:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:31:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:31:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:31:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:31:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:31:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:31:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:31:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:31:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:31:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:31:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:31:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:31:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:31:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:31:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:31:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:31:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:31:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:31:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:31:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:31:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:31:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:31:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:31:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:31:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:31:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:31:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:31:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:31:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:31:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:31:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:31:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:31:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:31:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:31:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:31:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:31:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:31:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:31:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:31:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:31:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:31:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:31:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:31:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:31:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:31:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:31:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:31:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:31:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:31:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:31:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:31:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:31:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:31:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:31:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:31:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:31:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:31:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:31:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:31:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:31:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:31:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:31:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:31:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:31:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:31:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:31:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:31:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:31:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:31:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:31:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:31:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:31:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:31:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:31:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:31:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:31:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:31:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:31:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:31:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:31:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:31:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:31:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:31:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:31:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:31:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:31:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:31:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:31:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:31:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:31:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:31:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:31:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:31:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:31:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:31:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:31:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:31:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:33:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:33:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:33:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:33:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:33:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:33:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:33:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:33:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:33:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:33:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:33:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:33:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:33:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:33:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:33:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:33:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:33:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:33:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:33:39 rdma06 rsyslogd[1041]: imjournal: 5266 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 04:33:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:33:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:33:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:33:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:33:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:33:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:33:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:33:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:33:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:33:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:33:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:33:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:33:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:33:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:33:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:33:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:33:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:33:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:33:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:33:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:33:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:33:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:33:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:33:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:33:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:33:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:33:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:33:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:33:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:33:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:33:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:33:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:33:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:33:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:33:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:33:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:33:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:33:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:33:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:33:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:33:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:33:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:33:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:33:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:33:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:33:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:33:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:33:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:33:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:33:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:33:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:33:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:33:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:33:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:33:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:33:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:33:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:33:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:33:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:33:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:33:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:33:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:33:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:33:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:33:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:33:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:33:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:33:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:33:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:33:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:33:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:33:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:34:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:34:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:34:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:34:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:34:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:34:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:34:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:34:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:34:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:34:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:34:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:34:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:34:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:34:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:34:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:34:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:34:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:34:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:34:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:34:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:34:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:34:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:34:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:34:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:34:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:34:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:34:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:34:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:34:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:34:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:34:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:34:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:34:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:34:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:34:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:34:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:34:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:34:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:34:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:34:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:34:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:34:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:34:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:34:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:34:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:34:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:34:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:34:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:34:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:34:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:34:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:34:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:34:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:34:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:34:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:34:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:34:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:34:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:34:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:34:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:34:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:34:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:34:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:34:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:34:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:34:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:34:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:34:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:34:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:34:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:34:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:34:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:34:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:34:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:34:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:34:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:34:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:34:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:34:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:34:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:34:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:34:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:34:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:34:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:34:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:34:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:34:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:34:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:34:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:34:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:34:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:34:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:34:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:34:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:34:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:34:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:34:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:34:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:34:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:34:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:34:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:34:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:34:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:34:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:34:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:34:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:34:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:34:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:34:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:34:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:34:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:34:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:34:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:34:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:34:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:34:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:34:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:34:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:34:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:34:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:34:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:34:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:34:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:34:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:34:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:34:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:34:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:34:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:34:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:34:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:34:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:34:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:34:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:34:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:34:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:34:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:34:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:34:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:34:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:34:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:34:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:34:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:34:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:34:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:34:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:34:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:34:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:34:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:34:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:34:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:34:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:34:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:34:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:34:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:34:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:34:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:34:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:34:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:34:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:34:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:34:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:34:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:34:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:34:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:34:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:34:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:34:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:34:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:34:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:34:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:34:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:34:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:34:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:34:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:34:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:34:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:34:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:34:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:34:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:34:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:34:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:34:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:34:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:34:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:34:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:34:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:34:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:34:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:34:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:34:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:34:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:34:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:34:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:34:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:34:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:34:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:34:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:34:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:34:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:34:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:34:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:34:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:34:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:34:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:34:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:34:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:34:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:34:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:34:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:34:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:34:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:34:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:34:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:34:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:34:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:34:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:35:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:35:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:35:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:35:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:35:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:35:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:35:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:35:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:35:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:35:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:35:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:35:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:35:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:35:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:35:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:35:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:35:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:35:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:35:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:35:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:35:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:35:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:35:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:35:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:35:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:35:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:35:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:35:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:35:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:35:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:35:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:35:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:35:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:35:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:35:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:35:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:35:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:35:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:35:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:35:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:35:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:35:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:35:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:35:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:35:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:35:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:35:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:35:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:35:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:35:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:35:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:35:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:35:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:35:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:35:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:35:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:35:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:35:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:35:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:35:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:35:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:35:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:35:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:35:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:35:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:35:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:35:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:35:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:35:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:35:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:35:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:35:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:35:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:35:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:35:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:35:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:35:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:35:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:35:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:35:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:35:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:35:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:35:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:35:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:35:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:35:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:35:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:35:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:35:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:35:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:35:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:35:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:35:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:35:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:35:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:35:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:35:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:35:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:35:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:35:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:35:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:35:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:35:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:35:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:35:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:35:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:35:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:35:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:35:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:35:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:35:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:35:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:35:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:35:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:35:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:35:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:35:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:35:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:35:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:35:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:35:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:35:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:35:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:35:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:35:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:35:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:35:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:35:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:35:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:35:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:35:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:35:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:35:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:35:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:35:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:35:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:35:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:35:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:35:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:35:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:35:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:35:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:35:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:35:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:35:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:35:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:35:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:35:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:35:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:35:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:35:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:35:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:35:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:35:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:35:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:35:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:35:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:35:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:35:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:35:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:35:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:35:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:35:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:35:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:35:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:35:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:35:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:35:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:35:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:35:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:35:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:35:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:35:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:35:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:35:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:35:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:35:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:35:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:35:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:35:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:35:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:35:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:35:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:35:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:35:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:35:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:35:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:35:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:35:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:35:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:35:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:35:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:35:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:35:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:35:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:35:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:35:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:35:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:35:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:35:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:35:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:35:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:35:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:35:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:35:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:35:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:35:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:35:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:35:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:35:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:35:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:35:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:35:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:35:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:35:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:35:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:36:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:36:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:36:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:36:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:36:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:36:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:36:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:36:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:36:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:36:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:36:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:36:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:36:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:36:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:36:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:36:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:36:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:36:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:36:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:36:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:36:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:36:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:36:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:36:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:36:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:36:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:36:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:36:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:36:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:36:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:36:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:36:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:36:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:36:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:36:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:36:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:36:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:36:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:36:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:36:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:36:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:36:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:36:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:36:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:36:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:36:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:36:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:36:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:36:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:36:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:36:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:36:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:36:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:36:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:36:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:36:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:36:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:36:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:36:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:36:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:36:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:36:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:36:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:36:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:36:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:36:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:36:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:36:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:36:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:36:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:36:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:36:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:36:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:36:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:36:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:36:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:36:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:36:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:36:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:36:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:36:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:36:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:36:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:36:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:36:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:36:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:36:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:36:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:36:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:36:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:36:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:36:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:36:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:36:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:36:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:36:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:36:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:36:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:36:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:36:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:36:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:36:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:36:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:36:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:36:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:36:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:36:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:36:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:36:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:36:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:36:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:36:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:36:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:36:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:36:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:36:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:36:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:36:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:36:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:36:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:36:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:36:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:36:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:36:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:36:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:36:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:36:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:36:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:36:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:36:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:36:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:36:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:36:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:36:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:36:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:36:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:36:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:36:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:36:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:36:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:36:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:36:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:36:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:36:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:36:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:36:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:36:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:36:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:36:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:36:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:36:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:36:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:36:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:36:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:36:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:36:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:36:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:36:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:36:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:36:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:36:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:36:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:36:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:36:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:36:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:36:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:36:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:36:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:36:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:36:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:36:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:36:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:36:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:36:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:36:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:36:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:36:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:36:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:36:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:36:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:36:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:36:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:36:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:36:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:36:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:36:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:36:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:36:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:36:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:36:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:36:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:36:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:36:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:36:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:36:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:36:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:36:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:36:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:36:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:36:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:36:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:36:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:36:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:36:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:36:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:36:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:36:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:36:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:36:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:36:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:36:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:36:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:36:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:36:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:36:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:36:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:37:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:37:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:37:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:37:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:37:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:37:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:37:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:37:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:37:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:37:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:37:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:37:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:37:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:37:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:37:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:37:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:37:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:37:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:37:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:37:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:37:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:37:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:37:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:37:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:37:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:37:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:37:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:37:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:37:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:37:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:37:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:37:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:37:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:37:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:37:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:37:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:37:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:37:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:37:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:37:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:37:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:37:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:37:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:37:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:37:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:37:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:37:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:37:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:37:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:37:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:37:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:37:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:37:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:37:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:37:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:37:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:37:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:37:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:37:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:37:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:37:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:37:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:37:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:37:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:37:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:37:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:37:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:37:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:37:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:37:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:37:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:37:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:37:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:37:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:37:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:37:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:37:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:37:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:37:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:37:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:37:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:37:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:37:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:37:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:37:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:37:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:37:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:37:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:37:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:37:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:37:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:37:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:37:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:37:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:37:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:37:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:37:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:37:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:37:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:37:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:37:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:37:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:37:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:37:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:37:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:37:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:37:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:37:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:37:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:37:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:37:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:37:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:37:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:37:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:37:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:37:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:37:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:37:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:37:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:37:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:37:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:37:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:37:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:37:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:37:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:37:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:37:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:37:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:37:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:37:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:37:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:37:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:37:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:37:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:37:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:37:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:37:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:37:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:37:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:37:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:37:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:37:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:37:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:37:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:37:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:37:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:37:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:37:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:37:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:37:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:37:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:37:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:37:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:37:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:37:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:37:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:37:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:37:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:37:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:37:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:37:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:37:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:37:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:37:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:37:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:37:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:37:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:37:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:37:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:37:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:37:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:37:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:37:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:37:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:37:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:37:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:37:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:37:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:37:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:37:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:37:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:37:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:37:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:37:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:37:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:37:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:37:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:37:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:37:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:37:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:37:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:37:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:37:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:37:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:37:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:37:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:37:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:37:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:37:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:37:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:37:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:37:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:37:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:37:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:37:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:37:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:37:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:37:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:37:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:37:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:37:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:37:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:37:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:37:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:37:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:37:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:38:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:38:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:38:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:38:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:38:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:38:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:38:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:38:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:38:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:38:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:38:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:38:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:38:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:38:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:38:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:38:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:38:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:38:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:38:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:38:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:38:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:38:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:38:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:38:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:38:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:38:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:38:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:38:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:38:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:38:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:38:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:38:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:38:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:38:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:38:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:38:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:38:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:38:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:38:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:38:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:38:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:38:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:38:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:38:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:38:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:38:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:38:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:38:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:38:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:38:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:38:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:38:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:38:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:38:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:38:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:38:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:38:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:38:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:38:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:38:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:38:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:38:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:38:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:38:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:38:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:38:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:38:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:38:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:38:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:38:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:38:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:38:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:38:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:38:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:38:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:38:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:38:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:38:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:38:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:38:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:38:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:38:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:38:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:38:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:38:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:38:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:38:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:38:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:38:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:38:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:38:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:38:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:38:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:38:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:38:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:38:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:38:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:38:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:38:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:38:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:38:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:38:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:38:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:38:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:38:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:38:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:38:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:38:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:38:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:38:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:38:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:38:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:38:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:38:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:38:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:38:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:38:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:38:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:38:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:38:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:38:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:38:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:38:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:38:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:38:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:38:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:38:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:38:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:38:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:38:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:38:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:38:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:38:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:38:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:38:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:38:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:38:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:38:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:38:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:38:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:38:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:38:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:38:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:38:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:38:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:38:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:38:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:38:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:38:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:38:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:38:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:38:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:38:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:38:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:38:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:38:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:38:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:38:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:38:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:38:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:38:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:38:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:38:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:38:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:38:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:38:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:38:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:38:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:38:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:38:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:38:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:38:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:38:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:38:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:38:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:38:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:38:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:38:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:38:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:38:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:38:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:38:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:38:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:38:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:38:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:38:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:38:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:38:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:38:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:38:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:38:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:38:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:38:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:38:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:38:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:38:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:38:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:38:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:38:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:38:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:38:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:38:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:38:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:38:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:38:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:38:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:38:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:38:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:38:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:38:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:38:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:38:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:38:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:38:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:38:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:38:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:39:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:39:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:39:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:39:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:39:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:39:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:39:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:39:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:39:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:39:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:39:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:39:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:39:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:39:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:39:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:39:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:39:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:39:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:39:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:39:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:39:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:39:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:39:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:39:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:39:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:39:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:39:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:39:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:39:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:39:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:39:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:39:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:39:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:39:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:39:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:39:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:39:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:39:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:39:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:39:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:39:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:39:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:39:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:39:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:39:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:39:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:39:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:39:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:39:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:39:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:39:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:39:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:39:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:39:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:39:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:39:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:39:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:39:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:39:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:39:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:39:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:39:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:39:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:39:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:39:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:39:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:39:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:39:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:39:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:39:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:39:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:39:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:39:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:39:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:39:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:39:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:39:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:39:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:39:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:39:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:39:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:39:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:39:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:39:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:39:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:39:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:39:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:39:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:39:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:39:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:39:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:39:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:39:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:39:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:39:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:39:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:39:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:39:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:39:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:39:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:39:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:39:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:39:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:39:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:39:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:39:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:39:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:39:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:39:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:39:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:39:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:39:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:39:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:39:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:39:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:39:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:39:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:39:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:39:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:39:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:39:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:39:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:39:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:39:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:39:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:39:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:39:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:39:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:39:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:39:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:39:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:39:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:39:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:39:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:39:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:39:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:39:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:39:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:39:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:39:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:39:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:39:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:39:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:39:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:39:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:39:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:39:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:39:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:39:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:39:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:39:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:39:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:39:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:39:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:39:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:39:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:40:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:40:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:40:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:40:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:40:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:40:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:40:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:40:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:40:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:40:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:40:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:40:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:40:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:40:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:40:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:40:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:40:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:40:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:40:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:40:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:40:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:40:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:40:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:40:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:40:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:40:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:40:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:40:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:40:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:40:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:40:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:40:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:40:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:40:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:40:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:40:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:40:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:40:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:40:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:40:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:40:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:40:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:40:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:40:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:40:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:40:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:40:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:40:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:40:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:40:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:40:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:40:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:40:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:40:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:40:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:40:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:40:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:40:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:40:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:40:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:40:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:40:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:40:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:40:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:40:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:40:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:40:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:40:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:40:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:40:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:40:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:40:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:40:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:40:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:40:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:40:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:40:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:40:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:40:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:40:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:40:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:40:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:40:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:40:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:40:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:40:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:40:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:40:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:40:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:40:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:40:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:40:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:40:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:40:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:40:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:40:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:40:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:40:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:40:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:40:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:40:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:40:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:40:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:40:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:40:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:40:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:40:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:40:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:40:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:40:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:40:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:40:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:40:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:40:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:40:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:40:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:40:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:40:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:40:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:40:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:40:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:40:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:40:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:40:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:40:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:40:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:40:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:40:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:40:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:40:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:40:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:40:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:40:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:40:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:40:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:40:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:40:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:40:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:40:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:40:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:40:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:40:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:40:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:40:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:40:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:40:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:40:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:40:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:40:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:40:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:40:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:40:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:40:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:40:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:40:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:40:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:40:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:40:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:40:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:40:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:40:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:40:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:40:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:40:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:40:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:40:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:40:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:40:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:40:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:40:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:40:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:40:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:40:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:40:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:40:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:40:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:40:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:40:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:40:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:40:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:40:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:40:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:40:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:40:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:40:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:40:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:40:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:40:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:40:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:40:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:40:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:40:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:40:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:40:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:40:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:40:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:40:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:40:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:40:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:40:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:40:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:40:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:40:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:40:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:40:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:40:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:40:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:40:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:40:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:40:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:40:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:40:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:40:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:40:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:40:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:40:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:41:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:41:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:41:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:41:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:41:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:41:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:41:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:41:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:41:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:41:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:41:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:41:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:41:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:41:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:41:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:41:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:41:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:41:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:41:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:41:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:41:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:41:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:41:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:41:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:41:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:41:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:41:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:41:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:41:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:41:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:41:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:41:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:41:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:41:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:41:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:41:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:41:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:41:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:41:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:41:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:41:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:41:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:41:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:41:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:41:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:41:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:41:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:41:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:41:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:41:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:41:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:41:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:41:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:41:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:41:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:41:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:41:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:41:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:41:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:41:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:41:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:41:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:41:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:41:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:41:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:41:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:41:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:41:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:41:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:41:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:41:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:41:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:41:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:41:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:41:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:41:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:41:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:41:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:41:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:41:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:41:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:41:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:41:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:41:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:41:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:41:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:41:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:41:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:41:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:41:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:41:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:41:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:41:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:41:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:41:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:41:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:41:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:41:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:41:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:41:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:41:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:41:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:41:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:41:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:41:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:41:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:41:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:41:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:41:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:41:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:41:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:41:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:41:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:41:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:41:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:41:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:41:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:41:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:41:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:41:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:41:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:41:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:41:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:41:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:41:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:41:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:43:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:43:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:43:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:43:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:43:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:43:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:43:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:43:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:43:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:43:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:43:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:43:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:43:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:43:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:43:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:43:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:43:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:43:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:43:44 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 04:43:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:43:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:43:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:43:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:43:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:43:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:43:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:43:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:43:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:43:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:43:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:43:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:43:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:43:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:43:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:43:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:43:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:43:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:43:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:43:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:43:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:43:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:43:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:43:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:43:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:43:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:43:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:43:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:43:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:43:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:43:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:43:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:43:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:43:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:43:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:43:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:43:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:43:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:43:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:43:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:43:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:43:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:43:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:43:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:43:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:43:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:43:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:43:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:43:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:43:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:43:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:43:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:43:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:43:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:44:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:44:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:44:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:44:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:44:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:44:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:44:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:44:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:44:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:44:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:44:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:44:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:44:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:44:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:44:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:44:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:44:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:44:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:44:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:44:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:44:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:44:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:44:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:44:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:44:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:44:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:44:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:44:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:44:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:44:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:44:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:44:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:44:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:44:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:44:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:44:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:44:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:44:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:44:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:44:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:44:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:44:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:44:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:44:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:44:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:44:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:44:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:44:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:44:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:44:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:44:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:44:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:44:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:44:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:44:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:44:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:44:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:44:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:44:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:44:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:44:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:44:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:44:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:44:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:44:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:44:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:44:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:44:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:44:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:44:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:44:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:44:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:44:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:44:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:44:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:44:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:44:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:44:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:44:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:44:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:44:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:44:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:44:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:44:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:44:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:44:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:44:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:44:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:44:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:44:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:44:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:44:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:44:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:44:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:44:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:44:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:44:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:44:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:44:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:44:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:44:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:44:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:44:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:44:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:44:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:44:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:44:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:44:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:44:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:44:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:44:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:44:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:44:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:44:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:44:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:44:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:44:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:44:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:44:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:44:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:44:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:44:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:44:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:44:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:44:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:44:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:44:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:44:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:44:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:44:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:44:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:44:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:44:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:44:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:44:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:44:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:44:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:44:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:44:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:44:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:44:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:44:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:44:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:44:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:44:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:44:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:44:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:44:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:44:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:44:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:44:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:44:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:44:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:44:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:44:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:44:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:44:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:44:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:44:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:44:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:44:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:44:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:44:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:44:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:44:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:44:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:44:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:44:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:44:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:44:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:44:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:44:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:44:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:44:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:44:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:44:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:44:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:44:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:44:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:44:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:44:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:44:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:44:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:44:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:44:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:44:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:44:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:44:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:44:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:44:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:44:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:44:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:44:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:44:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:44:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:44:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:44:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:44:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:44:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:44:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:44:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:44:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:44:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:44:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:44:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:44:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:44:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:44:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:44:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:44:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:44:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:44:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:44:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:44:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:44:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:44:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:45:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:45:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:45:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:45:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:45:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:45:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:45:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:45:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:45:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:45:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:45:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:45:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:45:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:45:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:45:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:45:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:45:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:45:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:45:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:45:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:45:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:45:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:45:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:45:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:45:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:45:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:45:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:45:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:45:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:45:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:45:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:45:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:45:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:45:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:45:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:45:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:45:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:45:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:45:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:45:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:45:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:45:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:45:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:45:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:45:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:45:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:45:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:45:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:45:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:45:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:45:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:45:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:45:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:45:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:45:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:45:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:45:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:45:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:45:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:45:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:45:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:45:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:45:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:45:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:45:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:45:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:45:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:45:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:45:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:45:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:45:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:45:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:45:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:45:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:45:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:45:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:45:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:45:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:45:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:45:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:45:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:45:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:45:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:45:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:45:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:45:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:45:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:45:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:45:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:45:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:45:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:45:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:45:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:45:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:45:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:45:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:45:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:45:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:45:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:45:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:45:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:45:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:45:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:45:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:45:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:45:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:45:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:45:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:45:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:45:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:45:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:45:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:45:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:45:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:45:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:45:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:45:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:45:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:45:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:45:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:45:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:45:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:45:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:45:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:45:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:45:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:45:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:45:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:45:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:45:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:45:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:45:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:45:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:45:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:45:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:45:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:45:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:45:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:45:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:45:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:45:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:45:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:45:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:45:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:45:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:45:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:45:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:45:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:45:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:45:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:45:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:45:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:45:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:45:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:45:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:45:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:45:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:45:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:45:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:45:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:45:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:45:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:45:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:45:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:45:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:45:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:45:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:45:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:45:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:45:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:45:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:45:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:45:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:45:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:45:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:45:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:45:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:45:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:45:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:45:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:45:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:45:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:45:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:45:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:45:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:45:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:45:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:45:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:45:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:45:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:45:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:45:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:45:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:45:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:45:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:45:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:45:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:45:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:45:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:45:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:45:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:45:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:45:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:45:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:45:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:45:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:45:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:45:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:45:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:45:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:45:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:45:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:45:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:45:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:45:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:45:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:46:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:46:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:46:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:46:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:46:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:46:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:46:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:46:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:46:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:46:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:46:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:46:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:46:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:46:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:46:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:46:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:46:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:46:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:46:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:46:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:46:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:46:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:46:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:46:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:46:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:46:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:46:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:46:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:46:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:46:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:46:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:46:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:46:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:46:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:46:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:46:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:46:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:46:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:46:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:46:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:46:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:46:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:46:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:46:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:46:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:46:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:46:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:46:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:46:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:46:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:46:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:46:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:46:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:46:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:46:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:46:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:46:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:46:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:46:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:46:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:46:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:46:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:46:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:46:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:46:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:46:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:46:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:46:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:46:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:46:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:46:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:46:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:46:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:46:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:46:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:46:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:46:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:46:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:46:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:46:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:46:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:46:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:46:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:46:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:46:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:46:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:46:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:46:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:46:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:46:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:46:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:46:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:46:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:46:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:46:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:46:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:46:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:46:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:46:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:46:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:46:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:46:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:46:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:46:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:46:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:46:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:46:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:46:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:46:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:46:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:46:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:46:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:46:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:46:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:46:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:46:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:46:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:46:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:46:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:46:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:46:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:46:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:46:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:46:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:46:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:46:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:46:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:46:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:46:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:46:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:46:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:46:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:46:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:46:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:46:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:46:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:46:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:46:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:46:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:46:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:46:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:46:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:46:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:46:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:46:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:46:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:46:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:46:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:46:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:46:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:46:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:46:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:46:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:46:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:46:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:46:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:46:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:46:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:46:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:46:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:46:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:46:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:46:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:46:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:46:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:46:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:46:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:46:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:46:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:46:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:46:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:46:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:46:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:46:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:46:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:46:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:46:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:46:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:46:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:46:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:46:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:46:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:46:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:46:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:46:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:46:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:46:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:46:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:46:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:46:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:46:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:46:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:46:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:46:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:46:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:46:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:46:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:46:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:46:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:46:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:46:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:46:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:46:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:46:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:46:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:46:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:46:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:46:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:46:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:46:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:46:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:46:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:46:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:46:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:46:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:46:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:47:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:47:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:47:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:47:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:47:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:47:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:47:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:47:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:47:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:47:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:47:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:47:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:47:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:47:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:47:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:47:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:47:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:47:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:47:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:47:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:47:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:47:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:47:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:47:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:47:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:47:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:47:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:47:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:47:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:47:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:47:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:47:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:47:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:47:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:47:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:47:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:47:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:47:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:47:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:47:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:47:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:47:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:47:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:47:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:47:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:47:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:47:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:47:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:47:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:47:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:47:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:47:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:47:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:47:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:47:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:47:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:47:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:47:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:47:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:47:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:47:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:47:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:47:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:47:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:47:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:47:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:47:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:47:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:47:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:47:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:47:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:47:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:47:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:47:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:47:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:47:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:47:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:47:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:47:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:47:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:47:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:47:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:47:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:47:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:47:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:47:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:47:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:47:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:47:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:47:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:47:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:47:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:47:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:47:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:47:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:47:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:47:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:47:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:47:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:47:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:47:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:47:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:47:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:47:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:47:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:47:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:47:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:47:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:47:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:47:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:47:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:47:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:47:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:47:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:47:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:47:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:47:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:47:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:47:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:47:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:47:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:47:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:47:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:47:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:47:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:47:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:47:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:47:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:47:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:47:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:47:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:47:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:47:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:47:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:47:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:47:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:47:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:47:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:47:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:47:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:47:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:47:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:47:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:47:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:47:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:47:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:47:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:47:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:47:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:47:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:47:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:47:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:47:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:47:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:47:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:47:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:47:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:47:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:47:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:47:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:47:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:47:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:47:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:47:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:47:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:47:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:47:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:47:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:47:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:47:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:47:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:47:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:47:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:47:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:47:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:47:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:48:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:48:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:48:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:48:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:48:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:48:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:48:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:48:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:48:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:48:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:48:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:48:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:48:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:48:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:48:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:48:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:48:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:48:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:48:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:48:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:48:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:48:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:48:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:48:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:48:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:48:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:48:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:48:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:48:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:48:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:48:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:48:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:48:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:48:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:48:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:48:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:48:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:48:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:48:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:48:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:48:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:48:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:48:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:48:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:48:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:48:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:48:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:48:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:48:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:48:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:48:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:48:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:48:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:48:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:48:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:48:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:48:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:48:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:48:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:48:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:48:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:48:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:48:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:48:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:48:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:48:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:48:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:48:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:48:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:48:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:48:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:48:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:48:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:48:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:48:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:48:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:48:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:48:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:48:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:48:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:48:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:48:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:48:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:48:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:48:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:48:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:48:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:48:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:48:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:48:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:48:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:48:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:48:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:48:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:48:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:48:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:48:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:48:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:48:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:48:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:48:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:48:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:48:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:48:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:48:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:48:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:48:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:48:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:48:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:48:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:48:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:48:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:48:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:48:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:48:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:48:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:48:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:48:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:48:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:48:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:48:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:48:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:48:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:48:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:48:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:48:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:48:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:48:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:48:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:48:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:48:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:48:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:48:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:48:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:48:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:48:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:48:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:48:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:48:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:48:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:48:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:48:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:48:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:48:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:48:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:48:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:48:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:48:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:48:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:48:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:48:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:48:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:48:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:48:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:48:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:48:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:48:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:48:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:48:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:48:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:48:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:48:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:48:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:48:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:48:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:48:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:48:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:48:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:48:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:48:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:48:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:48:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:48:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:48:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:48:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:48:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:48:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:48:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:48:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:48:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:48:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:48:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:48:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:48:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:48:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:48:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:48:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:48:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:48:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:48:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:48:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:48:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:48:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:48:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:48:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:48:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:48:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:48:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:48:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:48:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:48:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:48:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:48:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:48:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:48:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:48:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:48:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:48:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:48:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:48:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:48:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:48:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:48:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:48:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:48:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:48:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:49:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:49:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:49:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:49:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:49:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:49:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:49:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:49:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:49:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:49:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:49:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:49:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:49:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:49:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:49:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:49:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:49:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:49:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:49:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:49:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:49:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:49:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:49:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:49:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:49:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:49:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:49:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:49:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:49:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:49:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:49:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:49:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:49:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:49:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:49:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:49:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:49:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:49:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:49:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:49:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:49:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:49:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:49:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:49:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:49:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:49:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:49:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:49:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:49:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:49:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:49:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:49:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:49:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:49:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:49:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:49:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:49:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:49:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:49:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:49:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:49:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:49:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:49:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:49:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:49:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:49:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:49:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:49:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:49:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:49:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:49:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:49:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:49:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:49:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:49:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:49:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:49:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:49:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:49:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:49:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:49:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:49:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:49:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:49:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:49:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:49:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:49:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:49:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:49:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:49:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:49:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:49:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:49:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:49:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:49:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:49:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:49:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:49:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:49:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:49:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:49:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:49:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:49:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:49:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:49:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:49:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:49:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:49:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:49:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:49:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:49:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:49:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:49:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:49:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:49:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:49:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:49:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:49:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:49:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:49:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:49:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:49:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:49:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:49:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:49:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:49:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:49:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:49:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:49:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:49:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:49:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:49:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:49:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:49:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:49:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:49:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:49:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:49:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:49:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:49:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:49:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:49:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:49:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:49:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:49:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:49:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:49:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:49:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:49:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:49:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:49:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:49:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:49:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:49:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:49:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:49:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:49:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:49:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:49:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:49:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:49:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:49:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:49:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:49:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:49:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:49:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:49:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:49:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:49:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:49:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:49:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:49:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:49:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:49:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:49:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:49:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:49:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:49:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:49:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:49:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:49:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:49:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:49:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:49:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:49:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:49:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:49:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:49:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:49:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:49:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:49:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:49:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:49:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:49:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:49:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:49:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:49:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:49:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:49:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:49:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:49:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:49:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:49:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:49:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:49:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:49:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:50:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:50:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:50:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:50:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:50:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:50:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:50:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:50:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:50:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:50:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:50:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:50:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:50:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:50:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:50:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:50:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:50:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:50:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:50:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:50:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:50:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:50:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:50:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:50:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:50:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:50:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:50:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:50:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:50:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:50:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:50:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:50:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:50:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:50:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:50:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:50:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:50:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:50:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:50:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:50:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:50:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:50:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:50:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:50:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:50:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:50:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:50:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:50:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:50:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:50:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:50:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:50:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:50:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:50:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:50:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:50:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:50:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:50:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:50:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:50:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:50:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:50:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:50:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:50:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:50:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:50:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:50:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:50:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:50:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:50:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:50:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:50:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:50:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:50:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:50:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:50:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:50:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:50:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:50:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:50:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:50:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:50:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:50:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:50:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:50:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:50:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:50:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:50:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:50:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:50:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:50:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:50:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:50:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:50:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:50:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:50:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:50:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:50:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:50:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:50:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:50:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:50:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:50:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:50:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:50:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:50:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:50:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:50:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:50:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:50:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:50:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:50:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:50:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:50:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:50:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:50:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:50:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:50:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:50:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:50:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:50:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:50:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:50:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:50:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:50:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:50:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:50:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:50:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:50:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:50:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:50:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:50:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:50:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:50:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:50:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:50:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:50:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:50:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:50:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:50:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:50:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:50:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:50:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:50:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:50:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:50:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:50:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:50:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:50:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:50:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:50:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:50:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:50:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:50:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:50:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:50:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:50:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:50:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:50:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:50:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:50:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:50:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:50:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:50:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:50:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:50:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:50:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:50:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:50:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:50:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:50:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:50:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:50:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:50:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:50:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:50:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:50:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:50:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:50:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:50:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:50:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:50:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:50:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:50:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:50:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:50:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:50:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:50:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:50:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:50:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:50:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:50:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:50:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:50:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:50:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:50:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:50:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:50:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:50:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:50:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:50:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:50:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:50:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:50:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:50:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:50:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:50:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:50:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:50:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:50:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:50:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:50:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:50:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:50:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:50:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:50:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:51:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:51:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:51:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:51:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:51:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:51:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:51:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:51:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:51:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:51:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:51:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:51:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:51:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:51:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:51:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:51:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:51:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:51:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:51:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:51:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:51:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:51:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:51:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:51:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:51:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:51:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:51:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:51:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:51:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:51:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:51:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:51:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:51:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:51:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:51:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:51:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:51:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:51:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:51:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:51:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:51:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:51:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:51:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:51:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:51:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:51:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:51:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:51:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:51:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:51:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:51:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:51:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:51:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:51:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:51:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:51:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:51:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:51:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:51:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:51:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:51:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:51:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:51:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:51:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:51:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:51:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:51:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:51:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:51:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:51:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:51:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:51:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:51:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:51:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:51:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:51:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:51:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:51:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:51:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:51:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:51:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:51:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:51:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:51:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:51:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:51:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:51:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:51:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:51:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:51:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:51:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:51:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:51:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:51:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:51:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:51:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:51:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:51:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:51:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:51:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:51:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:51:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:51:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:51:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:51:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:51:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:51:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:51:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:51:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:51:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:51:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:51:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:51:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:51:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:51:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:51:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:51:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:51:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:51:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:51:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:51:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:51:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:51:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:51:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:51:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:51:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:51:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:51:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:51:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:51:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:51:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:51:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:51:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:51:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:51:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:51:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:51:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:51:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:51:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:51:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:51:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:51:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:51:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:51:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:53:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:53:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:53:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:53:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:53:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:53:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:53:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:53:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:53:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:53:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:53:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:53:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:53:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:53:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:53:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:53:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:53:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:53:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:53:45 rdma06 rsyslogd[1041]: imjournal: 5082 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 04:53:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:53:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:53:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:53:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:53:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:53:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:53:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:53:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:53:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:53:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:53:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:53:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:53:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:53:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:53:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:53:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:53:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:53:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:53:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:53:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:53:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:53:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:53:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:53:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:54:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:54:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:54:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:54:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:54:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:54:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:54:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:54:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:54:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:54:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:54:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:54:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:54:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:54:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:54:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:54:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:54:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:54:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:54:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:54:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:54:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:54:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:54:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:54:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:54:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:54:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:54:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:54:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:54:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:54:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:54:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:54:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:54:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:54:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:54:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:54:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:54:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:54:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:54:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:54:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:54:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:54:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:54:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:54:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:54:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:54:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:54:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:54:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:54:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:54:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:54:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:54:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:54:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:54:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:54:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:54:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:54:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:54:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:54:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:54:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:54:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:54:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:54:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:54:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:54:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:54:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:54:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:54:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:54:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:54:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:54:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:54:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:54:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:54:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:54:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:54:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:54:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:54:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:54:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:54:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:54:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:54:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:54:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:54:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:54:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:54:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:54:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:54:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:54:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:54:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:54:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:54:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:54:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:54:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:54:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:54:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:54:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:54:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:54:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:54:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:54:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:54:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:54:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:54:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:54:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:54:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:54:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:54:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:54:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:54:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:54:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:54:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:54:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:54:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:54:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:54:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:54:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:54:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:54:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:54:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:54:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:54:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:54:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:54:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:54:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:54:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:54:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:54:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:54:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:54:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:54:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:54:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:54:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:54:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:54:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:54:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:54:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:54:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:54:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:54:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:54:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:54:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:54:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:54:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:54:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:54:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:54:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:54:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:54:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:54:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:54:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:54:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:54:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:54:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:54:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:54:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:54:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:54:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:54:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:54:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:54:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:54:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:54:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:54:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:54:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:54:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:54:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:54:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:54:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:54:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:54:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:54:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:54:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:54:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:54:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:54:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:55:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:55:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:55:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:55:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:55:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:55:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:55:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:55:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:55:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:55:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:55:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:55:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:55:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:55:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:55:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:55:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:55:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:55:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:55:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:55:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:55:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:55:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:55:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:55:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:55:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:55:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:55:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:55:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:55:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:55:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:55:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:55:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:55:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:55:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:55:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:55:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:55:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:55:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:55:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:55:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:55:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:55:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:55:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:55:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:55:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:55:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:55:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:55:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:55:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:55:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:55:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:55:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:55:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:55:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:55:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:55:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:55:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:55:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:55:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:55:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:55:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:55:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:55:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:55:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:55:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:55:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:55:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:55:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:55:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:55:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:55:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:55:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:55:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:55:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:55:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:55:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:55:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:55:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:55:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:55:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:55:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:55:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:55:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:55:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:55:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:55:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:55:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:55:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:55:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:55:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:55:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:55:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:55:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:55:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:55:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:55:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:55:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:55:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:55:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:55:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:55:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:55:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:55:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:55:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:55:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:55:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:55:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:55:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:55:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:55:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:55:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:55:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:55:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:55:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:55:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:55:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:55:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:55:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:55:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:55:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:55:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:55:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:55:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:55:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:55:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:55:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:55:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:55:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:55:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:55:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:55:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:55:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:55:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:55:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:55:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:55:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:55:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:55:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:55:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:55:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:55:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:55:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:55:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:55:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:55:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:55:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:55:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:55:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:55:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:55:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:55:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:55:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:55:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:55:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:55:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:55:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:55:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:55:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:55:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:55:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:55:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:55:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:55:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:55:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:55:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:55:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:55:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:55:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:55:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:55:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:55:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:55:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:55:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:55:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:55:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:55:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:55:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:55:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:55:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:55:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:55:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:55:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:55:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:55:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:55:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:55:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:55:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:55:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:55:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:55:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:55:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:55:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:55:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:55:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:55:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:55:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:55:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:55:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:55:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:55:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:55:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:55:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:55:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:55:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:55:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:55:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:55:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:55:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:55:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:55:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:55:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:55:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:55:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:55:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:55:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:55:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:56:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:56:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:56:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:56:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:56:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:56:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:56:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:56:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:56:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:56:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:56:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:56:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:56:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:56:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:56:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:56:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:56:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:56:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:56:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:56:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:56:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:56:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:56:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:56:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:56:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:56:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:56:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:56:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:56:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:56:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:56:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:56:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:56:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:56:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:56:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:56:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:56:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:56:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:56:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:56:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:56:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:56:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:56:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:56:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:56:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:56:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:56:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:56:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:56:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:56:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:56:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:56:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:56:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:56:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:56:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:56:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:56:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:56:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:56:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:56:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:56:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:56:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:56:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:56:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:56:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:56:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:56:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:56:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:56:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:56:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:56:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:56:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:56:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:56:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:56:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:56:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:56:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:56:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:56:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:56:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:56:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:56:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:56:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:56:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:56:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:56:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:56:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:56:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:56:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:56:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:56:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:56:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:56:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:56:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:56:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:56:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:56:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:56:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:56:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:56:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:56:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:56:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:56:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:56:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:56:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:56:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:56:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:56:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:56:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:56:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:56:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:56:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:56:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:56:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:56:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:56:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:56:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:56:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:56:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:56:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:56:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:56:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:56:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:56:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:56:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:56:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:56:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:56:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:56:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:56:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:56:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:56:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:56:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:56:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:56:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:56:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:56:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:56:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:56:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:56:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:56:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:56:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:56:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:56:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:56:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:56:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:56:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:56:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:56:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:56:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:56:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:56:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:56:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:56:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:56:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:56:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:56:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:56:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:56:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:56:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:56:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:56:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:56:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:56:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:56:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:56:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:56:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:56:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:56:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:56:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:56:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:56:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:56:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:56:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:56:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:56:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:56:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:56:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:56:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:56:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:56:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:56:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:56:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:56:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:56:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:56:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:56:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:56:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:56:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:56:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:56:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:56:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:56:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:56:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:56:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:56:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:56:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:56:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:56:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:56:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:56:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:56:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:56:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:56:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:56:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:56:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:56:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:56:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:56:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:56:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:56:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:56:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:56:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:56:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:56:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:56:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:57:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:57:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:57:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:57:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:57:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:57:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:57:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:57:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:57:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:57:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:57:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:57:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:57:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:57:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:57:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:57:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:57:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:57:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:57:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:57:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:57:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:57:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:57:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:57:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:57:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:57:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:57:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:57:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:57:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:57:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:57:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:57:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:57:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:57:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:57:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:57:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:57:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:57:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:57:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:57:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:57:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:57:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:57:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:57:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:57:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:57:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:57:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:57:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:57:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:57:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:57:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:57:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:57:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:57:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:57:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:57:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:57:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:57:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:57:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:57:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:57:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:57:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:57:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:57:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:57:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:57:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:57:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:57:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:57:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:57:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:57:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:57:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:57:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:57:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:57:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:57:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:57:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:57:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:57:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:57:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:57:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:57:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:57:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:57:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:57:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:57:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:57:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:57:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:57:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:57:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:57:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:57:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:57:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:57:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:57:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:57:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:57:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:57:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:57:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:57:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:57:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:57:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:57:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:57:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:57:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:57:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:57:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:57:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:57:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:57:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:57:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:57:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:57:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:57:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:57:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:57:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:57:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:57:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:57:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:57:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:57:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:57:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:57:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:57:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:57:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:57:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:57:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:57:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:57:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:57:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:57:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:57:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:57:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:57:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:57:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:57:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:57:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:57:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:57:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:57:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:57:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:57:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:57:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:57:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:57:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:57:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:57:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:57:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:57:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:57:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:57:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:57:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:57:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:57:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:57:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:57:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:57:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:57:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:57:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:57:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:57:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:57:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:57:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:57:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:57:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:57:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:57:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:57:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:57:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:57:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:57:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:57:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:57:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:57:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:57:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:57:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:57:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:57:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:57:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:57:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:57:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:57:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:57:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:57:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:57:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:57:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:57:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:57:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:57:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:57:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:57:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:57:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:57:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:57:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:57:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:57:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:57:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:57:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:57:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:57:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:57:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:57:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:57:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:57:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:57:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:57:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:57:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:57:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:57:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:57:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:57:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:57:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:57:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:57:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:57:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:57:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:58:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:58:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:58:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:58:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:58:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:58:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:58:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:58:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:58:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:58:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:58:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:58:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:58:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:58:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:58:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:58:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:58:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:58:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:58:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:58:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:58:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:58:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:58:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:58:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:58:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:58:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:58:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:58:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:58:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:58:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:58:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:58:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:58:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:58:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:58:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:58:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:58:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:58:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:58:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:58:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:58:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:58:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:58:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:58:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:58:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:58:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:58:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:58:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:58:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:58:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:58:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:58:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:58:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:58:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:58:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:58:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:58:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:58:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:58:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:58:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:58:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:58:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:58:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:58:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:58:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:58:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:58:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:58:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:58:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:58:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:58:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:58:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:58:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:58:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:58:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:58:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:58:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:58:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:58:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:58:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:58:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:58:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:58:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:58:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:58:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:58:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:58:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:58:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:58:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:58:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:58:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:58:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:58:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:58:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:58:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:58:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:58:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:58:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:58:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:58:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:58:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:58:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:58:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:58:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:58:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:58:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:58:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:58:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:58:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:58:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:58:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:58:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:58:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:58:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:58:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:58:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:58:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:58:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:58:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:58:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:58:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:58:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:58:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:58:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:58:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:58:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:58:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:58:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:58:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:58:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:58:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:58:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:58:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:58:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:58:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:58:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:58:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:58:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:58:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:58:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:58:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:58:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:58:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:58:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:58:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:58:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:58:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:58:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:58:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:58:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:58:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:58:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:58:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:58:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:58:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:58:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:58:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:58:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:58:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:58:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:58:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:58:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:58:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:58:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:58:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:58:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:58:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:58:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:58:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:58:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:58:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:58:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:58:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:58:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:58:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:58:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:58:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:58:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:58:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:58:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:58:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:58:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:58:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:58:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:58:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:58:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:58:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:58:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:58:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:58:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:58:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:58:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:58:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:58:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:58:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:58:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:58:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:58:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:58:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:58:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:58:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:58:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:58:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:58:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:58:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:58:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:58:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:58:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:58:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:58:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:58:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:58:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:58:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:58:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:58:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:58:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:59:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:59:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:59:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:59:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:59:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:59:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:59:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:59:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:59:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:59:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:59:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:59:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:59:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:59:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:59:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:59:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:59:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:59:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:59:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:59:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:59:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:59:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:59:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:59:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:59:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:59:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:59:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:59:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:59:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:59:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:59:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:59:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:59:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:59:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:59:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:59:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:59:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:59:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:59:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:59:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:59:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:59:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:59:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:59:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:59:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:59:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:59:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:59:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:59:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:59:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:59:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:59:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:59:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:59:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:59:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:59:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:59:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:59:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:59:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:59:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:59:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:59:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:59:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:59:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:59:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:59:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:59:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:59:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:59:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:59:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:59:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:59:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:59:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:59:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:59:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:59:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:59:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:59:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:59:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:59:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:59:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:59:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:59:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:59:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:59:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:59:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:59:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:59:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:59:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:59:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:59:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:59:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:59:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:59:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:59:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:59:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:59:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:59:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:59:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:59:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:59:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:59:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:59:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:59:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:59:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:59:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:59:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:59:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:59:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:59:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:59:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:59:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:59:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:59:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:59:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:59:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:59:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:59:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:59:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:59:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:59:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:59:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:59:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:59:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:59:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:59:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:59:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:59:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:59:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:59:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:59:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:59:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:59:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:59:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:59:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:59:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:59:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:59:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 04:59:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 04:59:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:59:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:59:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:59:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 04:59:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 04:59:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 04:59:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 04:59:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 04:59:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 04:59:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 04:59:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 04:59:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 04:59:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 04:59:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 04:59:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 04:59:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 04:59:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:00:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:00:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:00:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:00:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:00:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:00:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:00:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:00:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:00:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:00:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:00:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:00:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:00:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:00:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:00:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:00:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:00:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:00:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:00:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:00:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:00:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:00:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:00:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:00:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:00:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:00:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:00:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:00:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:00:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:00:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:00:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:00:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:00:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:00:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:00:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:00:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:00:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:00:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:00:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:00:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:00:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:00:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:00:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:00:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:00:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:00:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:00:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:00:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:00:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:00:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:00:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:00:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:00:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:00:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:00:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:00:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:00:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:00:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:00:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:00:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:00:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:00:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:00:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:00:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:00:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:00:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:00:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:00:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:00:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:00:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:00:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:00:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:00:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:00:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:00:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:00:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:00:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:00:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:00:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:00:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:00:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:00:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:00:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:00:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:00:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:00:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:00:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:00:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:00:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:00:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:00:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:00:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:00:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:00:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:00:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:00:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:00:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:00:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:00:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:00:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:00:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:00:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:00:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:00:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:00:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:00:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:00:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:00:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:00:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:00:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:00:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:00:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:00:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:00:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:00:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:00:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:00:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:00:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:00:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:00:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:00:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:00:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:00:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:00:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:00:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:00:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:00:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:00:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:00:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:00:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:00:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:00:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:00:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:00:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:00:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:00:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:00:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:00:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:00:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:00:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:00:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:00:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:00:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:00:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:00:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:00:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:00:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:00:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:00:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:00:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:00:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:00:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:00:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:00:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:00:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:00:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:00:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:00:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:00:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:00:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:00:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:00:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:00:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:00:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:00:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:00:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:00:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:00:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:00:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:00:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:00:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:00:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:00:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:00:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:00:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:00:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:00:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:00:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:00:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:00:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:00:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:00:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:00:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:00:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:00:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:00:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:01:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:01:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:01:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:01:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:01:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:01:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:01:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:01:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:01:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:01:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:01:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:01:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:01:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:01:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:01:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:01:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:01:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:01:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:01:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:01:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:01:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:01:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:01:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:01:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:01:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:01:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:01:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:01:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:01:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:01:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:01:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:01:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:01:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:01:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:01:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:01:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:01:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:01:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:01:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:01:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:01:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:01:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:01:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:01:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:01:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:01:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:01:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:01:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:01:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:01:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:01:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:01:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:01:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:01:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:01:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:01:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:01:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:01:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:01:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:01:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:01:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:01:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:01:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:01:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:01:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:01:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:01:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:01:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:01:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:01:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:01:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:01:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:01:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:01:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:01:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:01:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:01:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:01:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:01:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:01:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:01:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:01:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:01:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:01:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:01:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:01:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:01:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:01:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:01:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:01:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:01:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:01:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:01:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:01:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:01:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:01:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:01:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:01:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:01:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:01:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:01:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:01:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:01:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:01:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:01:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:01:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:01:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:01:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:01:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:01:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:01:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:01:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:01:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:01:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:01:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:01:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:01:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:01:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:01:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:01:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:01:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:01:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:01:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:01:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:01:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:01:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:01:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:01:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:01:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:01:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:01:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:01:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:01:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:01:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:01:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:01:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:01:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:01:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:01:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:01:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:01:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:01:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:01:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:01:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:03:45 rdma06 dnf[1144218]: Updating Subscription Management repositories.
Apr 17 05:03:45 rdma06 dnf[1144218]: Unable to read consumer identity
Apr 17 05:03:45 rdma06 dnf[1144218]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 17 05:03:45 rdma06 dnf[1144218]: Metadata cache refreshed recently.
Apr 17 05:03:45 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 17 05:03:45 rdma06 systemd[1]: Finished dnf makecache.
Apr 17 05:03:46 rdma06 rsyslogd[1041]: imjournal: 5302 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 05:03:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:03:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:03:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:03:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:03:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:03:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:03:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:03:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:03:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:03:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:03:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:03:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:03:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:03:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:03:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:03:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:03:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:03:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:03:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:03:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:03:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:03:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:03:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:03:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:03:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:03:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:03:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:03:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:03:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:03:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:03:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:03:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:03:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:03:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:03:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:03:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:03:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:03:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:03:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:03:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:03:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:03:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:03:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:03:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:03:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:03:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:03:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:03:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:03:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:03:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:03:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:03:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:03:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:03:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:04:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:04:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:04:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:04:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:04:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:04:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:04:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:04:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:04:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:04:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:04:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:04:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:04:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:04:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:04:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:04:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:04:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:04:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:04:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:04:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:04:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:04:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:04:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:04:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:04:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:04:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:04:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:04:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:04:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:04:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:04:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:04:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:04:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:04:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:04:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:04:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:04:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:04:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:04:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:04:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:04:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:04:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:04:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:04:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:04:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:04:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:04:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:04:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:04:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:04:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:04:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:04:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:04:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:04:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:04:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:04:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:04:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:04:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:04:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:04:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:04:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:04:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:04:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:04:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:04:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:04:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:04:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:04:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:04:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:04:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:04:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:04:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:04:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:04:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:04:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:04:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:04:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:04:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:04:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:04:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:04:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:04:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:04:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:04:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:04:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:04:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:04:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:04:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:04:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:04:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:04:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:04:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:04:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:04:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:04:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:04:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:04:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:04:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:04:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:04:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:04:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:04:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:04:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:04:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:04:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:04:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:04:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:04:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:04:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:04:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:04:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:04:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:04:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:04:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:04:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:04:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:04:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:04:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:04:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:04:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:04:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:04:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:04:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:04:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:04:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:04:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:04:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:04:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:04:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:04:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:04:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:04:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:04:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:04:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:04:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:04:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:04:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:04:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:04:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:04:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:04:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:04:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:04:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:04:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:04:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:04:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:04:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:04:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:04:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:04:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:04:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:04:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:04:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:04:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:04:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:04:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:04:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:04:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:04:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:04:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:04:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:04:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:04:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:04:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:04:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:04:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:04:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:04:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:04:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:04:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:04:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:04:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:04:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:04:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:04:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:04:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:04:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:04:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:04:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:04:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:04:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:04:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:04:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:04:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:04:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:04:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:04:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:04:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:04:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:04:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:04:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:04:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:04:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:04:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:04:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:04:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:04:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:04:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:04:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:04:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:04:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:04:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:04:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:04:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:04:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:04:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:04:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:04:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:04:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:04:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:04:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:04:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:04:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:04:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:04:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:04:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:05:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:05:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:05:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:05:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:05:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:05:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:05:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:05:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:05:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:05:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:05:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:05:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:05:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:05:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:05:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:05:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:05:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:05:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:05:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:05:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:05:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:05:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:05:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:05:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:05:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:05:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:05:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:05:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:05:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:05:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:05:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:05:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:05:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:05:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:05:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:05:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:05:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:05:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:05:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:05:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:05:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:05:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:05:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:05:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:05:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:05:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:05:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:05:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:05:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:05:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:05:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:05:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:05:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:05:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:05:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:05:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:05:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:05:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:05:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:05:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:05:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:05:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:05:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:05:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:05:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:05:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:05:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:05:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:05:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:05:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:05:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:05:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:05:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:05:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:05:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:05:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:05:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:05:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:05:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:05:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:05:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:05:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:05:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:05:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:05:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:05:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:05:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:05:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:05:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:05:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:05:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:05:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:05:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:05:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:05:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:05:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:05:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:05:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:05:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:05:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:05:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:05:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:05:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:05:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:05:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:05:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:05:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:05:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:05:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:05:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:05:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:05:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:05:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:05:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:05:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:05:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:05:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:05:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:05:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:05:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:05:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:05:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:05:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:05:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:05:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:05:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:05:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:05:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:05:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:05:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:05:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:05:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:05:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:05:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:05:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:05:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:05:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:05:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:05:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:05:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:05:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:05:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:05:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:05:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:05:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:05:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:05:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:05:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:05:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:05:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:05:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:05:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:05:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:05:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:05:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:05:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:05:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:05:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:05:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:05:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:05:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:05:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:05:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:05:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:05:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:05:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:05:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:05:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:05:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:05:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:05:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:05:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:05:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:05:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:05:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:05:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:05:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:05:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:05:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:05:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:05:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:05:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:05:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:05:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:05:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:05:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:05:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:05:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:05:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:05:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:05:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:05:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:05:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:05:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:05:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:05:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:05:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:05:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:05:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:05:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:05:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:05:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:05:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:05:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:05:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:05:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:05:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:05:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:05:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:05:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:05:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:05:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:05:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:05:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:05:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:05:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:06:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:06:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:06:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:06:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:06:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:06:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:06:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:06:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:06:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:06:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:06:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:06:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:06:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:06:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:06:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:06:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:06:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:06:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:06:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:06:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:06:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:06:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:06:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:06:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:06:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:06:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:06:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:06:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:06:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:06:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:06:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:06:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:06:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:06:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:06:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:06:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:06:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:06:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:06:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:06:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:06:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:06:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:06:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:06:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:06:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:06:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:06:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:06:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:06:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:06:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:06:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:06:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:06:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:06:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:06:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:06:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:06:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:06:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:06:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:06:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:06:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:06:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:06:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:06:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:06:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:06:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:06:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:06:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:06:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:06:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:06:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:06:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:06:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:06:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:06:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:06:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:06:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:06:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:06:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:06:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:06:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:06:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:06:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:06:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:06:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:06:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:06:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:06:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:06:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:06:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:06:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:06:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:06:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:06:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:06:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:06:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:06:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:06:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:06:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:06:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:06:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:06:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:06:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:06:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:06:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:06:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:06:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:06:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:06:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:06:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:06:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:06:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:06:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:06:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:06:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:06:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:06:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:06:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:06:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:06:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:06:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:06:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:06:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:06:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:06:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:06:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:06:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:06:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:06:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:06:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:06:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:06:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:06:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:06:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:06:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:06:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:06:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:06:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:06:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:06:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:06:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:06:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:06:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:06:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:06:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:06:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:06:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:06:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:06:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:06:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:06:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:06:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:06:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:06:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:06:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:06:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:06:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:06:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:06:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:06:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:06:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:06:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:06:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:06:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:06:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:06:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:06:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:06:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:06:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:06:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:06:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:06:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:06:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:06:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:06:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:06:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:06:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:06:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:06:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:06:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:06:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:06:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:06:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:06:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:06:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:06:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:06:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:06:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:06:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:06:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:06:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:06:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:06:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:06:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:06:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:06:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:06:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:06:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:07:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:07:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:07:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:07:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:07:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:07:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:07:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:07:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:07:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:07:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:07:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:07:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:07:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:07:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:07:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:07:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:07:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:07:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:07:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:07:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:07:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:07:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:07:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:07:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:07:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:07:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:07:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:07:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:07:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:07:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:07:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:07:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:07:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:07:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:07:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:07:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:07:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:07:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:07:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:07:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:07:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:07:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:07:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:07:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:07:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:07:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:07:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:07:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:07:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:07:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:07:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:07:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:07:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:07:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:07:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:07:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:07:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:07:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:07:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:07:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:07:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:07:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:07:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:07:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:07:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:07:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:07:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:07:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:07:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:07:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:07:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:07:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:07:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:07:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:07:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:07:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:07:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:07:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:07:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:07:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:07:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:07:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:07:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:07:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:07:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:07:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:07:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:07:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:07:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:07:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:07:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:07:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:07:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:07:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:07:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:07:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:07:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:07:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:07:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:07:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:07:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:07:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:07:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:07:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:07:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:07:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:07:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:07:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:07:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:07:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:07:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:07:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:07:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:07:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:07:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:07:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:07:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:07:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:07:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:07:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:07:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:07:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:07:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:07:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:07:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:07:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:07:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:07:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:07:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:07:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:07:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:07:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:07:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:07:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:07:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:07:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:07:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:07:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:07:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:07:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:07:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:07:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:07:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:07:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:07:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:07:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:07:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:07:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:07:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:07:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:07:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:07:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:07:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:07:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:07:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:07:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:07:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:07:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:07:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:07:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:07:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:07:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:07:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:07:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:07:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:07:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:07:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:07:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:07:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:07:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:07:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:07:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:07:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:07:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:07:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:07:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:07:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:07:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:07:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:07:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:07:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:07:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:07:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:07:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:07:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:07:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:07:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:07:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:07:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:07:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:07:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:07:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:07:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:07:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:07:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:07:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:07:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:07:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:07:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:07:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:07:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:07:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:07:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:07:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:07:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:07:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:07:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:07:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:07:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:07:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:07:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:07:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:07:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:07:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:07:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:07:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:08:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:08:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:08:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:08:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:08:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:08:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:08:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:08:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:08:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:08:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:08:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:08:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:08:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:08:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:08:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:08:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:08:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:08:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:08:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:08:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:08:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:08:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:08:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:08:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:08:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:08:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:08:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:08:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:08:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:08:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:08:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:08:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:08:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:08:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:08:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:08:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:08:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:08:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:08:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:08:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:08:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:08:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:08:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:08:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:08:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:08:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:08:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:08:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:08:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:08:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:08:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:08:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:08:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:08:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:08:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:08:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:08:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:08:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:08:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:08:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:08:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:08:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:08:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:08:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:08:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:08:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:08:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:08:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:08:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:08:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:08:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:08:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:08:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:08:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:08:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:08:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:08:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:08:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:08:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:08:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:08:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:08:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:08:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:08:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:08:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:08:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:08:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:08:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:08:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:08:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:08:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:08:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:08:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:08:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:08:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:08:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:08:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:08:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:08:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:08:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:08:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:08:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:08:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:08:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:08:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:08:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:08:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:08:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:08:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:08:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:08:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:08:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:08:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:08:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:08:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:08:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:08:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:08:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:08:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:08:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:08:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:08:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:08:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:08:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:08:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:08:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:08:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:08:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:08:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:08:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:08:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:08:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:08:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:08:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:08:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:08:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:08:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:08:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:08:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:08:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:08:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:08:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:08:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:08:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:08:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:08:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:08:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:08:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:08:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:08:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:08:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:08:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:08:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:08:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:08:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:08:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:08:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:08:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:08:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:08:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:08:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:08:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:08:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:08:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:08:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:08:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:08:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:08:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:08:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:08:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:08:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:08:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:08:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:08:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:08:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:08:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:08:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:08:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:08:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:08:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:08:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:08:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:08:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:08:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:08:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:08:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:08:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:08:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:08:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:08:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:08:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:08:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:08:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:08:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:08:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:08:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:08:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:08:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:08:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:08:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:08:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:08:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:08:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:08:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:08:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:08:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:08:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:08:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:08:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:08:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:08:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:08:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:08:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:08:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:08:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:08:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:09:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:09:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:09:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:09:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:09:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:09:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:09:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:09:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:09:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:09:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:09:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:09:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:09:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:09:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:09:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:09:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:09:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:09:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:09:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:09:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:09:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:09:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:09:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:09:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:09:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:09:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:09:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:09:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:09:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:09:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:09:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:09:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:09:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:09:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:09:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:09:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:09:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:09:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:09:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:09:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:09:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:09:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:09:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:09:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:09:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:09:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:09:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:09:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:09:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:09:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:09:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:09:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:09:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:09:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:09:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:09:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:09:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:09:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:09:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:09:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:09:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:09:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:09:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:09:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:09:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:09:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:09:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:09:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:09:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:09:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:09:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:09:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:09:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:09:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:09:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:09:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:09:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:09:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:09:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:09:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:09:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:09:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:09:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:09:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:09:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:09:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:09:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:09:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:09:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:09:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:09:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:09:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:09:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:09:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:09:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:09:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:09:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:09:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:09:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:09:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:09:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:09:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:09:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:09:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:09:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:09:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:09:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:09:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:09:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:09:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:09:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:09:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:09:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:09:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:09:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:09:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:09:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:09:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:09:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:09:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:09:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:09:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:09:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:09:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:09:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:09:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:09:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:09:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:09:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:09:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:09:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:09:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:09:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:09:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:09:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:09:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:09:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:09:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:09:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:09:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:09:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:09:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:09:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:09:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:09:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:09:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:09:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:09:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:09:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:09:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:09:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:09:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:09:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:09:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:09:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:09:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:09:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:09:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:09:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:09:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:09:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:09:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:09:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:09:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:09:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:09:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:09:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:09:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:09:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:09:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:09:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:09:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:09:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:09:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:09:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:09:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:09:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:09:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:09:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:09:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:09:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:09:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:09:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:09:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:09:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:09:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:09:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:09:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:09:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:09:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:09:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:09:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:09:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:09:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:09:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:09:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:09:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:09:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:09:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:09:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:09:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:09:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:09:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:09:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:09:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:09:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:09:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:09:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:09:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:09:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:09:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:09:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:09:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:09:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:09:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:09:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:10:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:10:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:10:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:10:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:10:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:10:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:10:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:10:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:10:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:10:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:10:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:10:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:10:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:10:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:10:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:10:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:10:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:10:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:10:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:10:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:10:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:10:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:10:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:10:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:10:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:10:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:10:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:10:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:10:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:10:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:10:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:10:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:10:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:10:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:10:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:10:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:10:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:10:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:10:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:10:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:10:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:10:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:10:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:10:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:10:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:10:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:10:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:10:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:10:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:10:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:10:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:10:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:10:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:10:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:10:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:10:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:10:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:10:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:10:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:10:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:10:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:10:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:10:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:10:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:10:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:10:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:10:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:10:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:10:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:10:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:10:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:10:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:10:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:10:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:10:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:10:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:10:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:10:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:10:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:10:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:10:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:10:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:10:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:10:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:10:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:10:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:10:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:10:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:10:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:10:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:10:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:10:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:10:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:10:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:10:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:10:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:10:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:10:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:10:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:10:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:10:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:10:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:10:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:10:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:10:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:10:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:10:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:10:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:10:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:10:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:10:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:10:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:10:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:10:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:10:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:10:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:10:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:10:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:10:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:10:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:10:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:10:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:10:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:10:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:10:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:10:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:10:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:10:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:10:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:10:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:10:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:10:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:10:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:10:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:10:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:10:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:10:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:10:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:10:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:10:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:10:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:10:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:10:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:10:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:10:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:10:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:10:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:10:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:10:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:10:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:10:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:10:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:10:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:10:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:10:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:10:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:10:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:10:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:10:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:10:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:10:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:10:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:10:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:10:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:10:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:10:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:10:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:10:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:10:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:10:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:10:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:10:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:10:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:10:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:10:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:10:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:10:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:10:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:10:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:10:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:10:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:10:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:10:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:10:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:10:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:10:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:10:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:10:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:10:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:10:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:10:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:10:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:10:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:10:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:10:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:10:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:10:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:10:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:10:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:10:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:10:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:10:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:10:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:10:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:10:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:10:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:10:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:10:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:10:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:10:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:10:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:10:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:10:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:10:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:10:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:10:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:11:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:11:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:11:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:11:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:11:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:11:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:11:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:11:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:11:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:11:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:11:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:11:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:11:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:11:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:11:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:11:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:11:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:11:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:11:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:11:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:11:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:11:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:11:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:11:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:11:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:11:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:11:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:11:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:11:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:11:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:11:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:11:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:11:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:11:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:11:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:11:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:11:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:11:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:11:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:11:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:11:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:11:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:11:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:11:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:11:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:11:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:11:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:11:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:11:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:11:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:11:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:11:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:11:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:11:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:11:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:11:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:11:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:11:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:11:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:11:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:11:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:11:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:11:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:11:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:11:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:11:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:11:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:11:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:11:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:11:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:11:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:11:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:11:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:11:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:11:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:11:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:11:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:11:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:11:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:11:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:11:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:11:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:11:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:11:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:11:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:11:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:11:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:11:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:11:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:11:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:11:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:11:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:11:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:11:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:11:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:11:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:11:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:11:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:11:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:11:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:11:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:11:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:11:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:11:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:11:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:11:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:11:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:11:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:11:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:11:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:11:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:11:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:11:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:11:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:11:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:11:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:11:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:11:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:11:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:11:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:11:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:11:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:11:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:11:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:11:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:11:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:11:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:11:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:11:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:11:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:11:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:11:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:11:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:11:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:11:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:11:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:11:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:11:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:11:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:11:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:11:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:11:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:11:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:11:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:11:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:11:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:11:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:11:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:11:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:11:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:13:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:13:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:13:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:13:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:13:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:13:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:13:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:13:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:13:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:13:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:13:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:13:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:13:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:13:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:13:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:13:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:13:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:13:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:13:50 rdma06 rsyslogd[1041]: imjournal: 5088 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 05:13:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:13:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:13:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:13:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:13:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:13:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:13:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:13:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:13:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:13:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:13:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:13:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:13:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:13:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:13:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:13:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:13:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:13:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:14:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:14:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:14:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:14:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:14:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:14:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:14:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:14:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:14:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:14:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:14:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:14:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:14:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:14:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:14:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:14:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:14:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:14:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:14:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:14:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:14:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:14:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:14:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:14:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:14:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:14:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:14:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:14:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:14:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:14:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:14:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:14:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:14:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:14:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:14:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:14:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:14:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:14:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:14:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:14:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:14:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:14:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:14:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:14:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:14:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:14:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:14:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:14:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:14:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:14:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:14:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:14:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:14:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:14:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:14:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:14:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:14:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:14:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:14:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:14:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:14:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:14:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:14:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:14:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:14:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:14:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:14:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:14:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:14:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:14:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:14:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:14:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:14:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:14:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:14:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:14:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:14:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:14:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:14:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:14:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:14:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:14:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:14:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:14:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:14:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:14:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:14:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:14:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:14:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:14:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:14:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:14:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:14:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:14:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:14:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:14:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:14:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:14:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:14:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:14:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:14:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:14:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:14:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:14:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:14:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:14:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:14:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:14:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:14:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:14:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:14:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:14:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:14:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:14:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:14:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:14:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:14:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:14:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:14:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:14:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:14:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:14:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:14:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:14:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:14:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:14:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:14:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:14:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:14:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:14:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:14:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:14:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:14:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:14:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:14:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:14:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:14:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:14:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:14:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:14:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:14:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:14:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:14:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:14:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:14:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:14:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:14:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:14:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:14:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:14:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:14:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:14:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:14:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:14:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:14:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:14:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:14:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:14:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:14:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:14:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:14:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:14:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:14:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:14:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:14:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:14:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:14:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:14:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:14:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:14:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:14:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:14:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:14:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:14:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:14:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:14:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:14:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:14:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:14:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:14:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:14:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:14:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:14:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:14:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:14:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:14:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:14:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:14:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:14:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:14:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:14:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:14:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:14:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:14:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:14:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:14:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:14:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:14:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:14:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:14:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:14:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:14:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:14:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:14:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:14:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:14:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:14:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:14:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:14:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:14:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:14:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:14:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:14:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:14:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:14:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:14:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:15:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:15:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:15:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:15:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:15:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:15:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:15:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:15:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:15:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:15:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:15:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:15:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:15:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:15:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:15:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:15:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:15:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:15:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:15:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:15:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:15:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:15:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:15:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:15:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:15:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:15:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:15:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:15:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:15:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:15:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:15:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:15:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:15:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:15:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:15:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:15:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:15:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:15:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:15:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:15:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:15:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:15:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:15:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:15:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:15:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:15:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:15:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:15:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:15:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:15:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:15:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:15:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:15:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:15:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:15:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:15:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:15:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:15:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:15:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:15:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:15:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:15:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:15:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:15:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:15:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:15:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:15:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:15:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:15:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:15:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:15:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:15:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:15:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:15:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:15:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:15:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:15:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:15:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:15:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:15:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:15:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:15:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:15:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:15:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:15:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:15:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:15:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:15:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:15:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:15:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:15:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:15:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:15:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:15:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:15:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:15:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:15:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:15:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:15:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:15:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:15:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:15:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:15:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:15:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:15:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:15:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:15:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:15:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:15:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:15:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:15:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:15:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:15:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:15:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:15:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:15:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:15:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:15:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:15:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:15:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:15:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:15:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:15:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:15:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:15:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:15:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:15:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:15:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:15:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:15:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:15:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:15:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:15:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:15:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:15:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:15:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:15:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:15:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:15:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:15:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:15:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:15:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:15:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:15:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:15:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:15:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:15:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:15:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:15:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:15:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:15:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:15:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:15:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:15:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:15:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:15:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:15:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:15:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:15:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:15:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:15:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:15:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:15:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:15:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:15:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:15:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:15:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:15:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:15:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:15:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:15:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:15:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:15:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:15:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:15:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:15:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:15:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:15:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:15:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:15:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:15:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:15:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:15:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:15:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:15:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:15:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:15:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:15:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:15:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:15:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:15:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:15:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:15:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:15:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:15:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:15:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:15:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:15:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:15:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:15:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:15:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:15:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:15:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:15:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:15:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:15:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:15:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:15:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:15:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:15:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:15:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:15:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:15:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:15:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:15:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:15:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:16:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:16:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:16:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:16:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:16:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:16:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:16:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:16:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:16:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:16:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:16:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:16:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:16:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:16:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:16:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:16:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:16:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:16:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:16:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:16:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:16:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:16:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:16:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:16:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:16:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:16:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:16:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:16:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:16:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:16:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:16:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:16:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:16:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:16:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:16:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:16:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:16:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:16:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:16:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:16:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:16:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:16:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:16:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:16:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:16:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:16:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:16:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:16:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:16:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:16:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:16:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:16:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:16:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:16:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:16:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:16:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:16:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:16:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:16:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:16:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:16:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:16:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:16:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:16:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:16:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:16:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:16:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:16:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:16:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:16:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:16:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:16:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:16:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:16:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:16:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:16:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:16:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:16:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:16:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:16:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:16:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:16:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:16:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:16:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:16:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:16:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:16:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:16:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:16:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:16:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:16:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:16:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:16:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:16:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:16:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:16:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:16:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:16:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:16:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:16:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:16:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:16:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:16:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:16:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:16:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:16:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:16:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:16:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:16:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:16:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:16:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:16:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:16:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:16:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:16:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:16:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:16:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:16:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:16:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:16:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:16:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:16:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:16:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:16:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:16:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:16:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:16:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:16:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:16:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:16:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:16:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:16:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:16:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:16:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:16:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:16:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:16:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:16:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:16:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:16:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:16:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:16:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:16:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:16:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:16:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:16:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:16:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:16:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:16:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:16:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:16:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:16:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:16:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:16:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:16:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:16:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:16:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:16:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:16:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:16:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:16:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:16:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:16:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:16:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:16:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:16:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:16:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:16:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:16:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:16:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:16:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:16:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:16:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:16:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:16:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:16:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:16:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:16:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:16:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:16:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:16:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:16:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:16:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:16:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:16:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:16:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:16:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:16:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:16:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:16:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:16:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:16:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:16:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:16:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:16:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:16:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:16:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:16:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:16:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:16:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:16:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:16:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:16:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:16:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:16:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:16:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:16:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:16:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:16:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:16:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:16:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:16:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:16:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:16:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:16:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:16:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:17:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:17:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:17:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:17:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:17:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:17:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:17:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:17:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:17:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:17:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:17:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:17:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:17:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:17:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:17:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:17:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:17:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:17:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:17:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:17:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:17:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:17:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:17:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:17:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:17:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:17:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:17:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:17:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:17:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:17:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:17:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:17:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:17:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:17:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:17:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:17:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:17:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:17:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:17:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:17:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:17:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:17:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:17:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:17:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:17:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:17:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:17:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:17:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:17:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:17:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:17:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:17:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:17:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:17:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:17:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:17:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:17:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:17:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:17:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:17:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:17:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:17:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:17:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:17:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:17:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:17:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:17:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:17:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:17:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:17:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:17:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:17:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:17:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:17:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:17:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:17:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:17:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:17:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:17:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:17:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:17:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:17:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:17:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:17:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:17:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:17:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:17:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:17:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:17:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:17:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:17:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:17:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:17:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:17:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:17:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:17:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:17:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:17:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:17:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:17:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:17:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:17:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:17:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:17:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:17:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:17:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:17:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:17:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:17:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:17:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:17:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:17:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:17:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:17:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:17:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:17:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:17:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:17:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:17:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:17:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:17:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:17:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:17:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:17:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:17:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:17:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:17:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:17:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:17:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:17:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:17:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:17:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:17:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:17:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:17:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:17:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:17:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:17:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:17:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:17:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:17:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:17:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:17:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:17:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:17:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:17:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:17:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:17:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:17:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:17:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:17:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:17:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:17:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:17:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:17:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:17:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:17:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:17:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:17:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:17:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:17:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:17:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:17:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:17:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:17:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:17:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:17:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:17:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:17:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:17:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:17:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:17:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:17:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:17:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:17:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:17:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:17:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:17:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:17:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:17:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:17:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:17:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:17:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:17:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:17:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:17:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:17:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:17:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:17:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:17:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:17:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:17:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:17:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:17:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:17:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:17:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:17:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:17:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:17:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:17:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:17:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:17:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:17:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:17:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:17:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:17:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:17:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:17:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:17:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:17:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:17:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:17:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:17:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:17:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:17:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:17:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:18:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:18:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:18:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:18:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:18:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:18:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:18:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:18:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:18:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:18:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:18:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:18:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:18:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:18:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:18:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:18:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:18:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:18:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:18:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:18:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:18:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:18:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:18:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:18:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:18:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:18:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:18:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:18:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:18:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:18:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:18:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:18:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:18:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:18:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:18:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:18:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:18:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:18:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:18:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:18:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:18:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:18:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:18:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:18:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:18:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:18:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:18:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:18:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:18:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:18:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:18:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:18:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:18:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:18:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:18:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:18:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:18:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:18:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:18:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:18:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:18:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:18:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:18:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:18:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:18:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:18:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:18:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:18:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:18:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:18:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:18:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:18:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:18:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:18:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:18:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:18:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:18:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:18:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:18:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:18:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:18:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:18:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:18:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:18:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:18:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:18:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:18:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:18:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:18:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:18:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:18:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:18:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:18:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:18:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:18:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:18:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:18:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:18:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:18:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:18:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:18:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:18:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:18:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:18:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:18:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:18:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:18:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:18:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:18:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:18:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:18:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:18:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:18:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:18:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:18:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:18:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:18:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:18:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:18:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:18:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:18:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:18:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:18:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:18:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:18:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:18:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:18:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:18:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:18:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:18:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:18:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:18:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:18:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:18:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:18:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:18:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:18:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:18:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:18:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:18:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:18:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:18:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:18:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:18:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:18:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:18:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:18:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:18:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:18:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:18:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:18:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:18:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:18:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:18:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:18:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:18:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:18:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:18:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:18:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:18:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:18:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:18:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:18:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:18:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:18:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:18:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:18:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:18:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:18:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:18:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:18:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:18:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:18:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:18:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:18:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:18:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:18:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:18:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:18:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:18:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:18:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:18:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:18:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:18:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:18:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:18:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:18:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:18:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:18:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:18:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:18:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:18:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:18:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:18:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:18:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:18:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:18:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:18:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:18:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:18:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:18:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:18:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:18:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:18:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:18:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:18:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:18:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:18:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:18:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:18:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:18:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:18:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:18:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:18:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:18:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:18:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:19:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:19:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:19:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:19:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:19:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:19:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:19:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:19:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:19:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:19:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:19:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:19:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:19:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:19:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:19:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:19:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:19:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:19:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:19:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:19:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:19:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:19:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:19:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:19:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:19:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:19:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:19:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:19:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:19:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:19:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:19:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:19:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:19:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:19:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:19:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:19:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:19:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:19:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:19:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:19:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:19:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:19:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:19:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:19:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:19:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:19:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:19:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:19:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:19:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:19:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:19:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:19:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:19:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:19:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:19:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:19:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:19:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:19:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:19:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:19:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:19:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:19:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:19:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:19:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:19:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:19:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:19:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:19:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:19:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:19:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:19:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:19:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:19:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:19:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:19:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:19:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:19:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:19:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:19:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:19:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:19:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:19:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:19:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:19:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:19:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:19:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:19:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:19:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:19:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:19:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:19:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:19:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:19:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:19:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:19:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:19:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:19:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:19:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:19:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:19:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:19:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:19:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:19:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:19:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:19:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:19:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:19:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:19:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:19:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:19:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:19:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:19:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:19:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:19:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:19:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:19:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:19:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:19:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:19:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:19:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:19:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:19:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:19:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:19:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:19:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:19:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:19:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:19:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:19:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:19:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:19:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:19:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:19:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:19:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:19:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:19:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:19:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:19:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:19:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:19:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:19:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:19:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:19:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:19:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:19:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:19:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:19:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:19:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:19:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:19:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:19:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:19:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:19:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:19:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:19:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:19:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:19:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:19:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:19:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:19:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:19:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:19:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:19:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:19:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:19:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:19:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:19:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:19:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:19:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:19:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:19:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:19:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:19:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:19:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:19:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:19:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:19:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:19:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:19:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:19:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:19:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:19:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:19:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:19:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:19:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:19:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:19:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:19:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:19:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:19:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:19:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:19:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:19:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:19:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:19:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:19:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:19:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:19:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:19:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:19:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:19:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:19:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:19:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:19:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:19:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:19:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:19:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:19:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:19:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:19:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:19:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:19:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:19:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:19:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:19:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:19:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:20:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:20:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:20:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:20:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:20:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:20:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:20:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:20:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:20:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:20:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:20:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:20:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:20:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:20:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:20:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:20:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:20:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:20:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:20:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:20:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:20:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:20:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:20:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:20:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:20:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:20:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:20:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:20:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:20:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:20:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:20:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:20:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:20:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:20:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:20:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:20:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:20:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:20:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:20:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:20:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:20:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:20:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:20:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:20:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:20:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:20:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:20:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:20:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:20:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:20:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:20:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:20:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:20:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:20:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:20:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:20:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:20:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:20:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:20:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:20:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:20:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:20:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:20:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:20:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:20:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:20:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:20:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:20:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:20:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:20:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:20:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:20:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:20:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:20:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:20:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:20:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:20:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:20:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:20:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:20:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:20:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:20:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:20:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:20:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:20:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:20:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:20:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:20:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:20:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:20:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:20:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:20:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:20:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:20:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:20:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:20:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:20:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:20:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:20:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:20:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:20:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:20:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:20:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:20:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:20:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:20:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:20:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:20:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:20:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:20:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:20:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:20:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:20:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:20:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:20:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:20:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:20:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:20:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:20:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:20:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:20:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:20:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:20:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:20:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:20:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:20:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:20:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:20:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:20:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:20:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:20:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:20:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:20:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:20:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:20:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:20:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:20:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:20:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:20:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:20:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:20:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:20:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:20:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:20:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:20:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:20:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:20:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:20:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:20:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:20:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:20:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:20:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:20:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:20:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:20:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:20:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:20:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:20:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:20:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:20:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:20:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:20:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:20:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:20:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:20:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:20:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:20:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:20:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:20:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:20:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:20:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:20:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:20:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:20:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:20:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:20:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:20:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:20:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:20:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:20:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:20:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:20:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:20:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:20:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:20:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:20:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:20:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:20:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:20:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:20:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:20:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:20:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:20:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:20:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:20:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:20:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:20:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:20:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:20:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:20:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:20:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:20:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:20:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:20:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:20:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:20:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:20:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:20:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:20:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:20:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:20:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:20:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:20:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:20:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:20:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:20:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:21:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:21:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:21:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:21:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:21:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:21:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:21:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:21:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:21:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:21:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:21:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:21:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:21:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:21:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:21:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:21:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:21:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:21:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:21:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:21:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:21:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:21:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:21:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:21:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:21:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:21:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:21:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:21:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:21:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:21:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:21:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:21:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:21:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:21:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:21:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:21:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:21:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:21:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:21:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:21:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:21:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:21:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:21:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:21:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:21:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:21:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:21:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:21:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:21:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:21:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:21:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:21:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:21:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:21:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:21:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:21:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:21:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:21:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:21:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:21:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:21:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:21:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:21:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:21:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:21:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:21:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:21:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:21:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:21:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:21:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:21:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:21:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:21:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:21:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:21:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:21:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:21:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:21:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:21:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:21:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:21:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:21:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:21:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:21:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:21:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:21:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:21:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:21:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:21:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:21:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:21:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:21:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:21:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:21:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:21:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:21:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:21:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:21:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:21:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:21:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:21:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:21:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:21:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:21:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:21:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:21:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:21:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:21:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:21:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:21:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:21:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:21:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:21:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:21:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:21:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:21:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:21:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:21:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:21:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:21:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:21:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:21:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:21:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:21:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:21:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:21:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:21:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:21:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:21:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:21:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:21:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:21:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:21:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:21:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:21:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:21:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:21:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:21:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:21:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:21:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:21:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:21:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:21:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:21:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:21:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:21:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:21:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:21:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:21:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:21:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:21:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:21:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:21:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:21:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:21:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:21:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:21:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:21:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:21:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:21:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:21:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:21:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:21:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:21:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:21:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:21:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:21:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:21:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:21:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:21:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:21:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:21:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:21:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:21:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:21:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:21:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:21:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:21:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:21:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:21:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:23:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:23:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:23:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:23:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:23:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:23:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:23:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:23:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:23:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:23:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:23:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:23:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:23:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:23:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:23:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:23:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:23:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:23:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:23:51 rdma06 rsyslogd[1041]: imjournal: 5082 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 05:23:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:23:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:23:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:23:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:23:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:23:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:23:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:23:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:23:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:23:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:23:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:23:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:23:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:23:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:23:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:23:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:23:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:23:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:24:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:24:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:24:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:24:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:24:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:24:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:24:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:24:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:24:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:24:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:24:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:24:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:24:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:24:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:24:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:24:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:24:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:24:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:24:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:24:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:24:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:24:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:24:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:24:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:24:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:24:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:24:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:24:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:24:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:24:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:24:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:24:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:24:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:24:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:24:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:24:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:24:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:24:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:24:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:24:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:24:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:24:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:24:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:24:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:24:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:24:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:24:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:24:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:24:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:24:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:24:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:24:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:24:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:24:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:24:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:24:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:24:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:24:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:24:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:24:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:24:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:24:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:24:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:24:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:24:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:24:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:24:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:24:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:24:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:24:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:24:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:24:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:24:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:24:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:24:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:24:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:24:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:24:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:24:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:24:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:24:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:24:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:24:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:24:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:24:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:24:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:24:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:24:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:24:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:24:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:24:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:24:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:24:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:24:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:24:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:24:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:24:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:24:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:24:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:24:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:24:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:24:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:24:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:24:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:24:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:24:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:24:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:24:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:24:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:24:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:24:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:24:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:24:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:24:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:24:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:24:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:24:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:24:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:24:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:24:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:24:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:24:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:24:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:24:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:24:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:24:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:24:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:24:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:24:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:24:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:24:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:24:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:24:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:24:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:24:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:24:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:24:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:24:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:24:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:24:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:24:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:24:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:24:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:24:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:24:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:24:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:24:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:24:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:24:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:24:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:24:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:24:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:24:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:24:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:24:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:24:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:24:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:24:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:24:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:24:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:24:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:24:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:24:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:24:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:24:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:24:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:24:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:24:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:24:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:24:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:24:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:24:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:24:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:24:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:24:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:24:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:24:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:24:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:24:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:24:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:24:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:24:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:24:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:24:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:24:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:24:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:24:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:24:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:24:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:24:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:24:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:24:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:24:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:24:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:24:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:24:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:24:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:24:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:24:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:24:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:24:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:24:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:24:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:24:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:24:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:24:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:24:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:24:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:24:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:24:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:24:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:24:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:24:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:24:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:24:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:24:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:25:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:25:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:25:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:25:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:25:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:25:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:25:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:25:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:25:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:25:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:25:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:25:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:25:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:25:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:25:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:25:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:25:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:25:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:25:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:25:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:25:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:25:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:25:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:25:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:25:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:25:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:25:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:25:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:25:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:25:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:25:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:25:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:25:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:25:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:25:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:25:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:25:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:25:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:25:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:25:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:25:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:25:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:25:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:25:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:25:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:25:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:25:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:25:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:25:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:25:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:25:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:25:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:25:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:25:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:25:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:25:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:25:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:25:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:25:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:25:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:25:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:25:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:25:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:25:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:25:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:25:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:25:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:25:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:25:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:25:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:25:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:25:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:25:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:25:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:25:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:25:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:25:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:25:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:25:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:25:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:25:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:25:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:25:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:25:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:25:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:25:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:25:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:25:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:25:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:25:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:25:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:25:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:25:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:25:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:25:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:25:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:25:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:25:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:25:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:25:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:25:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:25:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:25:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:25:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:25:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:25:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:25:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:25:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:25:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:25:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:25:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:25:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:25:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:25:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:25:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:25:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:25:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:25:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:25:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:25:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:25:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:25:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:25:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:25:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:25:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:25:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:25:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:25:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:25:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:25:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:25:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:25:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:25:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:25:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:25:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:25:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:25:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:25:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:25:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:25:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:25:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:25:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:25:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:25:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:25:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:25:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:25:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:25:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:25:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:25:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:25:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:25:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:25:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:25:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:25:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:25:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:25:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:25:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:25:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:25:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:25:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:25:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:25:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:25:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:25:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:25:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:25:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:25:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:25:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:25:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:25:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:25:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:25:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:25:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:25:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:25:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:25:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:25:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:25:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:25:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:25:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:25:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:25:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:25:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:25:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:25:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:25:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:25:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:25:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:25:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:25:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:25:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:25:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:25:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:25:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:25:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:25:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:25:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:25:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:25:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:25:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:25:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:25:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:25:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:25:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:25:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:25:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:25:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:25:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:25:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:25:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:25:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:25:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:25:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:25:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:25:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:26:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:26:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:26:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:26:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:26:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:26:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:26:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:26:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:26:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:26:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:26:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:26:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:26:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:26:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:26:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:26:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:26:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:26:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:26:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:26:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:26:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:26:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:26:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:26:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:26:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:26:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:26:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:26:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:26:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:26:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:26:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:26:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:26:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:26:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:26:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:26:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:26:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:26:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:26:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:26:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:26:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:26:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:26:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:26:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:26:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:26:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:26:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:26:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:26:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:26:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:26:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:26:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:26:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:26:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:26:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:26:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:26:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:26:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:26:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:26:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:26:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:26:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:26:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:26:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:26:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:26:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:26:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:26:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:26:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:26:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:26:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:26:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:26:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:26:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:26:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:26:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:26:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:26:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:26:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:26:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:26:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:26:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:26:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:26:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:26:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:26:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:26:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:26:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:26:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:26:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:26:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:26:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:26:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:26:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:26:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:26:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:26:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:26:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:26:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:26:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:26:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:26:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:26:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:26:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:26:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:26:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:26:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:26:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:26:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:26:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:26:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:26:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:26:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:26:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:26:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:26:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:26:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:26:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:26:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:26:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:26:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:26:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:26:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:26:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:26:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:26:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:26:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:26:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:26:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:26:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:26:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:26:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:26:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:26:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:26:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:26:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:26:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:26:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:26:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:26:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:26:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:26:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:26:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:26:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:26:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:26:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:26:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:26:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:26:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:26:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:26:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:26:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:26:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:26:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:26:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:26:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:26:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:26:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:26:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:26:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:26:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:26:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:26:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:26:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:26:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:26:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:26:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:26:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:26:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:26:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:26:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:26:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:26:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:26:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:26:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:26:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:26:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:26:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:26:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:26:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:26:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:26:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:26:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:26:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:26:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:26:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:26:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:26:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:26:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:26:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:26:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:26:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:26:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:26:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:26:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:26:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:26:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:26:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:26:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:26:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:26:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:26:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:26:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:26:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:26:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:26:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:26:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:26:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:26:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:26:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:26:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:26:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:26:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:26:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:26:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:26:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:27:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:27:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:27:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:27:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:27:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:27:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:27:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:27:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:27:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:27:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:27:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:27:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:27:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:27:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:27:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:27:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:27:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:27:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:27:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:27:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:27:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:27:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:27:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:27:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:27:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:27:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:27:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:27:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:27:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:27:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:27:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:27:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:27:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:27:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:27:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:27:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:27:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:27:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:27:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:27:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:27:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:27:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:27:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:27:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:27:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:27:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:27:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:27:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:27:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:27:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:27:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:27:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:27:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:27:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:27:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:27:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:27:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:27:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:27:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:27:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:27:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:27:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:27:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:27:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:27:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:27:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:27:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:27:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:27:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:27:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:27:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:27:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:27:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:27:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:27:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:27:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:27:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:27:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:27:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:27:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:27:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:27:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:27:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:27:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:27:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:27:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:27:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:27:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:27:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:27:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:27:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:27:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:27:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:27:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:27:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:27:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:27:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:27:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:27:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:27:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:27:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:27:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:27:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:27:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:27:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:27:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:27:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:27:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:27:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:27:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:27:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:27:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:27:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:27:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:27:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:27:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:27:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:27:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:27:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:27:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:27:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:27:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:27:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:27:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:27:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:27:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:27:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:27:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:27:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:27:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:27:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:27:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:27:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:27:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:27:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:27:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:27:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:27:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:27:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:27:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:27:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:27:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:27:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:27:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:27:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:27:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:27:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:27:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:27:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:27:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:27:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:27:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:27:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:27:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:27:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:27:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:27:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:27:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:27:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:27:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:27:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:27:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:27:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:27:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:27:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:27:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:27:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:27:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:27:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:27:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:27:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:27:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:27:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:27:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:27:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:27:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:27:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:27:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:27:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:27:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:27:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:27:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:27:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:27:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:27:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:27:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:27:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:27:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:27:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:27:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:27:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:27:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:27:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:27:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:27:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:27:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:27:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:27:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:27:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:27:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:27:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:27:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:27:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:27:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:27:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:27:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:27:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:27:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:27:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:27:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:27:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:27:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:27:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:27:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:27:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:27:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:28:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:28:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:28:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:28:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:28:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:28:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:28:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:28:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:28:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:28:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:28:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:28:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:28:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:28:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:28:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:28:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:28:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:28:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:28:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:28:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:28:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:28:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:28:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:28:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:28:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:28:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:28:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:28:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:28:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:28:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:28:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:28:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:28:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:28:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:28:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:28:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:28:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:28:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:28:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:28:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:28:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:28:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:28:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:28:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:28:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:28:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:28:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:28:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:28:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:28:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:28:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:28:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:28:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:28:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:28:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:28:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:28:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:28:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:28:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:28:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:28:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:28:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:28:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:28:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:28:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:28:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:28:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:28:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:28:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:28:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:28:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:28:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:28:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:28:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:28:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:28:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:28:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:28:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:28:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:28:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:28:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:28:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:28:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:28:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:28:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:28:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:28:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:28:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:28:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:28:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:28:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:28:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:28:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:28:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:28:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:28:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:28:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:28:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:28:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:28:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:28:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:28:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:28:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:28:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:28:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:28:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:28:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:28:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:28:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:28:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:28:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:28:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:28:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:28:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:28:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:28:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:28:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:28:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:28:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:28:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:28:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:28:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:28:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:28:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:28:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:28:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:28:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:28:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:28:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:28:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:28:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:28:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:28:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:28:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:28:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:28:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:28:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:28:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:28:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:28:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:28:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:28:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:28:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:28:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:28:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:28:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:28:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:28:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:28:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:28:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:28:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:28:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:28:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:28:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:28:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:28:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:28:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:28:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:28:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:28:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:28:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:28:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:28:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:28:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:28:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:28:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:28:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:28:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:28:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:28:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:28:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:28:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:28:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:28:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:28:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:28:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:28:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:28:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:28:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:28:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:28:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:28:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:28:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:28:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:28:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:28:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:28:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:28:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:28:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:28:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:28:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:28:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:28:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:28:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:28:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:28:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:28:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:28:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:28:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:28:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:28:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:28:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:28:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:28:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:28:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:28:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:28:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:28:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:28:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:28:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:28:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:28:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:28:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:28:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:28:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:28:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:29:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:29:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:29:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:29:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:29:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:29:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:29:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:29:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:29:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:29:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:29:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:29:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:29:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:29:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:29:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:29:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:29:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:29:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:29:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:29:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:29:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:29:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:29:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:29:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:29:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:29:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:29:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:29:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:29:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:29:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:29:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:29:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:29:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:29:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:29:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:29:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:29:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:29:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:29:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:29:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:29:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:29:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:29:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:29:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:29:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:29:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:29:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:29:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:29:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:29:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:29:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:29:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:29:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:29:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:29:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:29:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:29:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:29:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:29:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:29:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:29:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:29:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:29:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:29:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:29:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:29:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:29:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:29:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:29:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:29:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:29:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:29:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:29:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:29:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:29:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:29:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:29:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:29:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:29:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:29:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:29:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:29:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:29:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:29:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:29:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:29:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:29:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:29:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:29:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:29:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:29:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:29:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:29:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:29:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:29:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:29:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:29:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:29:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:29:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:29:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:29:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:29:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:29:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:29:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:29:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:29:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:29:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:29:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:29:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:29:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:29:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:29:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:29:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:29:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:29:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:29:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:29:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:29:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:29:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:29:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:29:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:29:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:29:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:29:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:29:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:29:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:29:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:29:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:29:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:29:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:29:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:29:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:29:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:29:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:29:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:29:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:29:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:29:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:29:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:29:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:29:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:29:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:29:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:29:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:29:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:29:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:29:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:29:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:29:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:29:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:29:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:29:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:29:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:29:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:29:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:29:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:29:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:29:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:29:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:29:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:29:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:29:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:29:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:29:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:29:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:29:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:29:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:29:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:29:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:29:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:29:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:29:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:29:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:29:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:29:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:29:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:29:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:29:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:29:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:29:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:29:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:29:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:29:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:29:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:29:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:29:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:29:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:29:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:29:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:29:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:29:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:29:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:29:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:29:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:29:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:29:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:29:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:29:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:29:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:29:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:29:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:29:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:29:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:29:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:29:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:29:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:29:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:29:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:29:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:29:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:29:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:29:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:29:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:29:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:29:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:29:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:30:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:30:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:30:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:30:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:30:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:30:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:30:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:30:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:30:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:30:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:30:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:30:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:30:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:30:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:30:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:30:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:30:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:30:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:30:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:30:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:30:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:30:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:30:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:30:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:30:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:30:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:30:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:30:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:30:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:30:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:30:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:30:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:30:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:30:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:30:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:30:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:30:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:30:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:30:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:30:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:30:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:30:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:30:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:30:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:30:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:30:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:30:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:30:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:30:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:30:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:30:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:30:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:30:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:30:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:30:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:30:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:30:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:30:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:30:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:30:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:30:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:30:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:30:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:30:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:30:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:30:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:30:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:30:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:30:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:30:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:30:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:30:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:30:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:30:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:30:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:30:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:30:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:30:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:30:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:30:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:30:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:30:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:30:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:30:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:30:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:30:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:30:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:30:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:30:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:30:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:30:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:30:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:30:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:30:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:30:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:30:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:30:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:30:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:30:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:30:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:30:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:30:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:30:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:30:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:30:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:30:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:30:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:30:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:30:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:30:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:30:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:30:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:30:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:30:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:30:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:30:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:30:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:30:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:30:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:30:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:30:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:30:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:30:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:30:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:30:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:30:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:30:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:30:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:30:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:30:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:30:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:30:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:30:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:30:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:30:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:30:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:30:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:30:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:30:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:30:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:30:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:30:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:30:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:30:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:30:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:30:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:30:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:30:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:30:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:30:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:30:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:30:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:30:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:30:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:30:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:30:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:30:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:30:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:30:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:30:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:30:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:30:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:30:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:30:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:30:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:30:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:30:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:30:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:30:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:30:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:30:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:30:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:30:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:30:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:30:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:30:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:30:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:30:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:30:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:30:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:30:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:30:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:30:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:30:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:30:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:30:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:30:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:30:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:30:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:30:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:30:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:30:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:30:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:30:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:30:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:30:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:30:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:30:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:30:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:30:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:30:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:30:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:30:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:30:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:30:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:30:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:30:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:30:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:30:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:30:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:30:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:30:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:30:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:30:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:30:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:30:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:31:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:31:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:31:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:31:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:31:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:31:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:31:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:31:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:31:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:31:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:31:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:31:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:31:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:31:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:31:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:31:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:31:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:31:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:31:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:31:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:31:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:31:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:31:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:31:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:31:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:31:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:31:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:31:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:31:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:31:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:31:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:31:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:31:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:31:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:31:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:31:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:31:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:31:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:31:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:31:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:31:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:31:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:31:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:31:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:31:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:31:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:31:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:31:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:31:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:31:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:31:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:31:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:31:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:31:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:31:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:31:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:31:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:31:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:31:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:31:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:31:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:31:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:31:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:31:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:31:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:31:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:31:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:31:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:31:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:31:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:31:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:31:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:31:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:31:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:31:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:31:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:31:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:31:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:31:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:31:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:31:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:31:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:31:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:31:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:31:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:31:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:31:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:31:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:31:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:31:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:31:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:31:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:31:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:31:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:31:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:31:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:31:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:31:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:31:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:31:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:31:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:31:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:31:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:31:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:31:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:31:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:31:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:31:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:31:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:31:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:31:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:31:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:31:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:31:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:31:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:31:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:31:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:31:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:31:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:31:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:31:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:31:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:31:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:31:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:31:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:31:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:31:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:31:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:31:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:31:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:31:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:31:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:31:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:31:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:31:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:31:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:31:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:31:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:31:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:31:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:31:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:31:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:31:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:31:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:31:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:31:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:31:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:31:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:31:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:31:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:31:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:31:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:31:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:31:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:31:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:31:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:31:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:31:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:31:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:31:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:31:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:31:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:31:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:31:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:31:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:31:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:31:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:31:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:31:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:31:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:31:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:31:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:31:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:31:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:31:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:31:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:31:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:31:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:31:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:31:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:33:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:33:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:33:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:33:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:33:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:33:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:33:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:33:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:33:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:33:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:33:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:33:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:33:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:33:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:33:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:33:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:33:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:33:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:33:56 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 05:34:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:34:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:34:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:34:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:34:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:34:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:34:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:34:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:34:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:34:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:34:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:34:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:34:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:34:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:34:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:34:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:34:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:34:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:34:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:34:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:34:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:34:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:34:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:34:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:34:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:34:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:34:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:34:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:34:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:34:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:34:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:34:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:34:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:34:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:34:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:34:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:34:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:34:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:34:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:34:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:34:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:34:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:34:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:34:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:34:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:34:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:34:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:34:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:34:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:34:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:34:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:34:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:34:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:34:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:34:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:34:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:34:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:34:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:34:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:34:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:34:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:34:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:34:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:34:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:34:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:34:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:34:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:34:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:34:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:34:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:34:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:34:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:34:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:34:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:34:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:34:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:34:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:34:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:34:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:34:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:34:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:34:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:34:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:34:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:34:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:34:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:34:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:34:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:34:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:34:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:34:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:34:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:34:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:34:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:34:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:34:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:34:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:34:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:34:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:34:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:34:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:34:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:34:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:34:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:34:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:34:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:34:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:34:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:34:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:34:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:34:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:34:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:34:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:34:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:34:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:34:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:34:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:34:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:34:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:34:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:34:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:34:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:34:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:34:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:34:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:34:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:34:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:34:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:34:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:34:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:34:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:34:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:34:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:34:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:34:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:34:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:34:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:34:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:34:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:34:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:34:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:34:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:34:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:34:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:34:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:34:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:34:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:34:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:34:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:34:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:34:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:34:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:34:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:34:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:34:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:34:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:34:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:34:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:34:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:34:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:34:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:34:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:34:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:34:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:34:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:34:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:34:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:34:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:34:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:34:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:34:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:34:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:34:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:34:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:34:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:34:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:34:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:34:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:34:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:34:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:34:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:34:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:34:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:34:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:34:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:34:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:34:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:34:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:34:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:34:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:34:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:34:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:34:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:34:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:34:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:34:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:34:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:34:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:34:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:34:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:34:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:34:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:34:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:34:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:34:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:34:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:34:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:34:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:34:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:34:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:34:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:34:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:34:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:34:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:34:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:34:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:35:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:35:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:35:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:35:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:35:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:35:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:35:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:35:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:35:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:35:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:35:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:35:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:35:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:35:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:35:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:35:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:35:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:35:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:35:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:35:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:35:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:35:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:35:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:35:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:35:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:35:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:35:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:35:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:35:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:35:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:35:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:35:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:35:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:35:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:35:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:35:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:35:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:35:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:35:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:35:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:35:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:35:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:35:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:35:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:35:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:35:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:35:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:35:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:35:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:35:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:35:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:35:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:35:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:35:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:35:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:35:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:35:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:35:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:35:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:35:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:35:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:35:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:35:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:35:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:35:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:35:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:35:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:35:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:35:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:35:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:35:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:35:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:35:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:35:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:35:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:35:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:35:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:35:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:35:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:35:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:35:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:35:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:35:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:35:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:35:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:35:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:35:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:35:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:35:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:35:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:35:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:35:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:35:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:35:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:35:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:35:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:35:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:35:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:35:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:35:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:35:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:35:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:35:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:35:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:35:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:35:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:35:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:35:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:35:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:35:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:35:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:35:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:35:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:35:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:35:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:35:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:35:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:35:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:35:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:35:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:35:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:35:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:35:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:35:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:35:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:35:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:35:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:35:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:35:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:35:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:35:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:35:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:35:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:35:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:35:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:35:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:35:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:35:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:35:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:35:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:35:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:35:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:35:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:35:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:35:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:35:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:35:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:35:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:35:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:35:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:35:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:35:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:35:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:35:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:35:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:35:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:35:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:35:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:35:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:35:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:35:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:35:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:35:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:35:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:35:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:35:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:35:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:35:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:35:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:35:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:35:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:35:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:35:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:35:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:35:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:35:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:35:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:35:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:35:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:35:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:35:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:35:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:35:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:35:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:35:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:35:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:35:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:35:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:35:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:35:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:35:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:35:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:35:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:35:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:35:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:35:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:35:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:35:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:35:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:35:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:35:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:35:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:35:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:35:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:35:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:35:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:35:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:35:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:35:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:35:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:35:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:35:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:35:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:35:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:35:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:35:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:36:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:36:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:36:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:36:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:36:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:36:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:36:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:36:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:36:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:36:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:36:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:36:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:36:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:36:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:36:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:36:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:36:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:36:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:36:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:36:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:36:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:36:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:36:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:36:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:36:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:36:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:36:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:36:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:36:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:36:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:36:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:36:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:36:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:36:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:36:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:36:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:36:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:36:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:36:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:36:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:36:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:36:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:36:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:36:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:36:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:36:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:36:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:36:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:36:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:36:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:36:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:36:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:36:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:36:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:36:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:36:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:36:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:36:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:36:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:36:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:36:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:36:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:36:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:36:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:36:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:36:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:36:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:36:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:36:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:36:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:36:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:36:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:36:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:36:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:36:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:36:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:36:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:36:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:36:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:36:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:36:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:36:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:36:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:36:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:36:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:36:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:36:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:36:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:36:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:36:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:36:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:36:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:36:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:36:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:36:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:36:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:36:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:36:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:36:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:36:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:36:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:36:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:36:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:36:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:36:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:36:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:36:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:36:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:36:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:36:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:36:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:36:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:36:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:36:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:36:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:36:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:36:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:36:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:36:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:36:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:36:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:36:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:36:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:36:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:36:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:36:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:36:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:36:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:36:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:36:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:36:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:36:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:36:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:36:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:36:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:36:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:36:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:36:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:36:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:36:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:36:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:36:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:36:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:36:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:36:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:36:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:36:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:36:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:36:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:36:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:36:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:36:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:36:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:36:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:36:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:36:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:36:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:36:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:36:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:36:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:36:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:36:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:36:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:36:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:36:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:36:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:36:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:36:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:36:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:36:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:36:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:36:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:36:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:36:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:36:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:36:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:36:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:36:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:36:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:36:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:36:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:36:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:36:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:36:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:36:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:36:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:36:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:36:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:36:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:36:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:36:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:36:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:36:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:36:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:36:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:36:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:36:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:36:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:36:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:36:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:36:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:36:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:36:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:36:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:36:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:36:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:36:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:36:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:36:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:36:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:36:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:36:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:36:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:36:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:36:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:36:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:37:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:37:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:37:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:37:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:37:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:37:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:37:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:37:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:37:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:37:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:37:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:37:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:37:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:37:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:37:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:37:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:37:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:37:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:37:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:37:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:37:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:37:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:37:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:37:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:37:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:37:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:37:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:37:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:37:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:37:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:37:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:37:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:37:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:37:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:37:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:37:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:37:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:37:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:37:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:37:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:37:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:37:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:37:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:37:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:37:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:37:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:37:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:37:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:37:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:37:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:37:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:37:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:37:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:37:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:37:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:37:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:37:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:37:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:37:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:37:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:37:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:37:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:37:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:37:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:37:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:37:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:37:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:37:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:37:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:37:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:37:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:37:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:37:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:37:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:37:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:37:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:37:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:37:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:37:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:37:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:37:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:37:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:37:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:37:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:37:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:37:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:37:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:37:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:37:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:37:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:37:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:37:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:37:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:37:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:37:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:37:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:37:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:37:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:37:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:37:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:37:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:37:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:37:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:37:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:37:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:37:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:37:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:37:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:37:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:37:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:37:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:37:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:37:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:37:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:37:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:37:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:37:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:37:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:37:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:37:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:37:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:37:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:37:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:37:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:37:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:37:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:37:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:37:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:37:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:37:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:37:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:37:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:37:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:37:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:37:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:37:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:37:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:37:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:37:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:37:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:37:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:37:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:37:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:37:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:37:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:37:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:37:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:37:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:37:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:37:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:37:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:37:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:37:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:37:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:37:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:37:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:37:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:37:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:37:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:37:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:37:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:37:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:37:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:37:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:37:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:37:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:37:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:37:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:37:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:37:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:37:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:37:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:37:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:37:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:37:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:37:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:37:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:37:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:37:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:37:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:37:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:37:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:37:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:37:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:37:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:37:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:37:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:37:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:37:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:37:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:37:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:37:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:37:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:37:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:37:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:37:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:37:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:37:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:37:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:37:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:37:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:37:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:37:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:37:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:37:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:37:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:37:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:37:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:37:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:37:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:37:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:37:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:37:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:37:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:37:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:37:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:38:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:38:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:38:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:38:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:38:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:38:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:38:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:38:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:38:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:38:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:38:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:38:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:38:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:38:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:38:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:38:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:38:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:38:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:38:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:38:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:38:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:38:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:38:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:38:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:38:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:38:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:38:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:38:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:38:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:38:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:38:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:38:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:38:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:38:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:38:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:38:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:38:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:38:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:38:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:38:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:38:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:38:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:38:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:38:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:38:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:38:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:38:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:38:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:38:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:38:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:38:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:38:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:38:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:38:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:38:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:38:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:38:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:38:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:38:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:38:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:38:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:38:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:38:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:38:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:38:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:38:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:38:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:38:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:38:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:38:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:38:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:38:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:38:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:38:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:38:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:38:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:38:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:38:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:38:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:38:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:38:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:38:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:38:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:38:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:38:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:38:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:38:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:38:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:38:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:38:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:38:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:38:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:38:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:38:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:38:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:38:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:38:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:38:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:38:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:38:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:38:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:38:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:38:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:38:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:38:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:38:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:38:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:38:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:38:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:38:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:38:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:38:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:38:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:38:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:38:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:38:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:38:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:38:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:38:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:38:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:38:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:38:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:38:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:38:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:38:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:38:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:38:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:38:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:38:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:38:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:38:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:38:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:38:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:38:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:38:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:38:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:38:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:38:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:38:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:38:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:38:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:38:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:38:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:38:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:38:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:38:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:38:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:38:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:38:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:38:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:38:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:38:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:38:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:38:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:38:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:38:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:38:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:38:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:38:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:38:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:38:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:38:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:38:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:38:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:38:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:38:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:38:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:38:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:38:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:38:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:38:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:38:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:38:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:38:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:38:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:38:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:38:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:38:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:38:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:38:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:38:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:38:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:38:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:38:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:38:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:38:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:38:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:38:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:38:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:38:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:38:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:38:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:38:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:38:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:38:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:38:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:38:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:38:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:38:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:38:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:38:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:38:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:38:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:38:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:38:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:38:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:38:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:38:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:38:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:38:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:38:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:38:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:38:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:38:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:38:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:38:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:39:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:39:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:39:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:39:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:39:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:39:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:39:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:39:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:39:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:39:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:39:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:39:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:39:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:39:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:39:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:39:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:39:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:39:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:39:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:39:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:39:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:39:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:39:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:39:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:39:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:39:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:39:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:39:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:39:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:39:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:39:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:39:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:39:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:39:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:39:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:39:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:39:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:39:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:39:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:39:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:39:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:39:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:39:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:39:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:39:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:39:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:39:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:39:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:39:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:39:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:39:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:39:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:39:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:39:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:39:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:39:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:39:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:39:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:39:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:39:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:39:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:39:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:39:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:39:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:39:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:39:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:39:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:39:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:39:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:39:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:39:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:39:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:39:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:39:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:39:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:39:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:39:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:39:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:39:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:39:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:39:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:39:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:39:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:39:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:39:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:39:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:39:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:39:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:39:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:39:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:39:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:39:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:39:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:39:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:39:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:39:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:39:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:39:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:39:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:39:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:39:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:39:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:39:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:39:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:39:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:39:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:39:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:39:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:39:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:39:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:39:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:39:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:39:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:39:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:39:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:39:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:39:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:39:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:39:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:39:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:39:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:39:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:39:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:39:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:39:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:39:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:39:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:39:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:39:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:39:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:39:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:39:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:39:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:39:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:39:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:39:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:39:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:39:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:39:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:39:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:39:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:39:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:39:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:39:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:39:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:39:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:39:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:39:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:39:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:39:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:39:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:39:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:39:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:39:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:39:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:39:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:39:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:39:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:39:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:39:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:39:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:39:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:39:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:39:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:39:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:39:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:39:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:39:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:39:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:39:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:39:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:39:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:39:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:39:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:39:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:39:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:39:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:39:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:39:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:39:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:39:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:39:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:39:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:39:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:39:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:39:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:39:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:39:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:39:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:39:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:39:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:39:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:39:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:39:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:39:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:39:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:39:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:39:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:39:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:39:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:39:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:39:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:39:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:39:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:39:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:39:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:39:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:39:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:39:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:39:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:39:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:39:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:39:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:39:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:39:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:39:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:40:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:40:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:40:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:40:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:40:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:40:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:40:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:40:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:40:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:40:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:40:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:40:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:40:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:40:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:40:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:40:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:40:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:40:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:40:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:40:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:40:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:40:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:40:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:40:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:40:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:40:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:40:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:40:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:40:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:40:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:40:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:40:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:40:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:40:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:40:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:40:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:40:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:40:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:40:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:40:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:40:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:40:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:40:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:40:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:40:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:40:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:40:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:40:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:40:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:40:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:40:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:40:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:40:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:40:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:40:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:40:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:40:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:40:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:40:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:40:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:40:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:40:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:40:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:40:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:40:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:40:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:40:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:40:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:40:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:40:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:40:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:40:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:40:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:40:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:40:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:40:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:40:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:40:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:40:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:40:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:40:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:40:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:40:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:40:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:40:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:40:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:40:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:40:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:40:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:40:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:40:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:40:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:40:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:40:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:40:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:40:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:40:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:40:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:40:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:40:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:40:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:40:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:40:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:40:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:40:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:40:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:40:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:40:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:40:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:40:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:40:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:40:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:40:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:40:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:40:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:40:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:40:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:40:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:40:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:40:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:40:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:40:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:40:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:40:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:40:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:40:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:40:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:40:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:40:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:40:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:40:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:40:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:40:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:40:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:40:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:40:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:40:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:40:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:40:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:40:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:40:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:40:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:40:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:40:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:40:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:40:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:40:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:40:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:40:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:40:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:40:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:40:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:40:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:40:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:40:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:40:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:40:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:40:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:40:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:40:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:40:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:40:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:40:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:40:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:40:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:40:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:40:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:40:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:40:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:40:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:40:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:40:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:40:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:40:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:40:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:40:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:40:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:40:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:40:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:40:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:40:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:40:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:40:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:40:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:40:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:40:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:40:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:40:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:40:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:40:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:40:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:40:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:40:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:40:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:40:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:40:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:40:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:40:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:40:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:40:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:40:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:40:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:40:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:40:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:40:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:40:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:40:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:40:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:40:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:40:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:40:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:40:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:40:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:40:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:40:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:40:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:41:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:41:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:41:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:41:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:41:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:41:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:41:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:41:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:41:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:41:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:41:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:41:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:41:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:41:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:41:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:41:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:41:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:41:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:41:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:41:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:41:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:41:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:41:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:41:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:41:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:41:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:41:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:41:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:41:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:41:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:41:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:41:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:41:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:41:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:41:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:41:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:41:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:41:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:41:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:41:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:41:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:41:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:41:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:41:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:41:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:41:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:41:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:41:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:41:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:41:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:41:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:41:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:41:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:41:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:41:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:41:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:41:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:41:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:41:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:41:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:41:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:41:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:41:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:41:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:41:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:41:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:41:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:41:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:41:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:41:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:41:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:41:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:41:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:41:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:41:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:41:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:41:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:41:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:41:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:41:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:41:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:41:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:41:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:41:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:41:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:41:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:41:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:41:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:41:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:41:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:41:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:41:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:41:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:41:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:41:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:41:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:41:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:41:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:41:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:41:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:41:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:41:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:41:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:41:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:41:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:41:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:41:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:41:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:41:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:41:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:41:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:41:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:41:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:41:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:41:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:41:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:41:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:41:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:41:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:41:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:41:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:41:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:41:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:41:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:41:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:41:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:41:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:41:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:41:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:41:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:41:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:41:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:41:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:41:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:41:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:41:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:41:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:41:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:41:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:41:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:41:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:41:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:41:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:41:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:41:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:41:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:41:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:41:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:41:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:41:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:41:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:41:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:41:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:41:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:41:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:41:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:41:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:41:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:41:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:41:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:41:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:41:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:41:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:41:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:41:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:41:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:41:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:41:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:41:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:41:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:41:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:41:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:41:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:41:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:41:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:41:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:41:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:41:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:41:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:41:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:41:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:41:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:41:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:41:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:41:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:41:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:41:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:41:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:41:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:41:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:41:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:41:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:41:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:41:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:41:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:41:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:41:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:41:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:44:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:44:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:44:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:44:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:44:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:44:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:44:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:44:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:44:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:44:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:44:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:44:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:44:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:44:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:44:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:44:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:44:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:44:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:44:01 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 05:44:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:44:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:44:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:44:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:44:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:44:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:44:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:44:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:44:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:44:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:44:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:44:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:44:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:44:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:44:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:44:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:44:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:44:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:44:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:44:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:44:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:44:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:44:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:44:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:44:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:44:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:44:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:44:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:44:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:44:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:44:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:44:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:44:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:44:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:44:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:44:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:44:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:44:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:44:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:44:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:44:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:44:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:44:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:44:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:44:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:44:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:44:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:44:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:44:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:44:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:44:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:44:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:44:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:44:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:44:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:44:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:44:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:44:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:44:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:44:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:44:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:44:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:44:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:44:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:44:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:44:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:44:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:44:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:44:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:44:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:44:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:44:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:44:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:44:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:44:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:44:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:44:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:44:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:44:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:44:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:44:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:44:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:44:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:44:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:44:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:44:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:44:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:44:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:44:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:44:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:44:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:44:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:44:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:44:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:44:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:44:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:44:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:44:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:44:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:44:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:44:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:44:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:44:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:44:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:44:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:44:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:44:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:44:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:44:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:44:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:44:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:44:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:44:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:44:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:44:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:44:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:44:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:44:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:44:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:44:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:44:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:44:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:44:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:44:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:44:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:44:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:44:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:44:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:44:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:44:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:44:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:44:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:44:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:44:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:44:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:44:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:44:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:44:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:44:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:44:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:44:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:44:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:44:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:44:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:44:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:44:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:44:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:44:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:44:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:44:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:44:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:44:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:44:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:44:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:44:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:44:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:44:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:44:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:44:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:44:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:44:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:44:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:44:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:44:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:44:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:44:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:44:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:44:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:44:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:44:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:44:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:44:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:44:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:44:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:44:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:44:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:44:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:44:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:44:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:44:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:44:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:44:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:44:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:44:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:44:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:44:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:44:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:44:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:44:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:44:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:44:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:44:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:44:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:44:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:44:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:44:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:44:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:44:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:45:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:45:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:45:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:45:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:45:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:45:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:45:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:45:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:45:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:45:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:45:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:45:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:45:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:45:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:45:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:45:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:45:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:45:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:45:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:45:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:45:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:45:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:45:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:45:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:45:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:45:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:45:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:45:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:45:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:45:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:45:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:45:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:45:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:45:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:45:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:45:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:45:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:45:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:45:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:45:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:45:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:45:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:45:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:45:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:45:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:45:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:45:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:45:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:45:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:45:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:45:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:45:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:45:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:45:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:45:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:45:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:45:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:45:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:45:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:45:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:45:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:45:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:45:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:45:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:45:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:45:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:45:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:45:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:45:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:45:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:45:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:45:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:45:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:45:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:45:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:45:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:45:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:45:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:45:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:45:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:45:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:45:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:45:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:45:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:45:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:45:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:45:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:45:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:45:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:45:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:45:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:45:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:45:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:45:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:45:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:45:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:45:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:45:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:45:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:45:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:45:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:45:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:45:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:45:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:45:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:45:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:45:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:45:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:45:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:45:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:45:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:45:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:45:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:45:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:45:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:45:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:45:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:45:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:45:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:45:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:45:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:45:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:45:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:45:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:45:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:45:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:45:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:45:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:45:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:45:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:45:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:45:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:45:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:45:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:45:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:45:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:45:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:45:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:45:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:45:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:45:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:45:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:45:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:45:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:45:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:45:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:45:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:45:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:45:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:45:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:45:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:45:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:45:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:45:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:45:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:45:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:45:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:45:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:45:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:45:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:45:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:45:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:45:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:45:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:45:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:45:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:45:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:45:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:45:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:45:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:45:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:45:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:45:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:45:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:45:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:45:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:45:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:45:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:45:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:45:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:45:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:45:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:45:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:45:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:45:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:45:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:45:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:45:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:45:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:45:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:45:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:45:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:45:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:45:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:45:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:45:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:45:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:45:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:45:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:45:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:45:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:45:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:45:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:45:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:45:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:45:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:45:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:45:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:45:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:45:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:45:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:45:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:45:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:45:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:45:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:45:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:46:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:46:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:46:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:46:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:46:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:46:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:46:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:46:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:46:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:46:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:46:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:46:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:46:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:46:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:46:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:46:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:46:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:46:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:46:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:46:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:46:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:46:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:46:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:46:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:46:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:46:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:46:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:46:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:46:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:46:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:46:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:46:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:46:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:46:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:46:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:46:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:46:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:46:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:46:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:46:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:46:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:46:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:46:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:46:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:46:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:46:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:46:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:46:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:46:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:46:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:46:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:46:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:46:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:46:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:46:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:46:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:46:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:46:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:46:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:46:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:46:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:46:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:46:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:46:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:46:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:46:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:46:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:46:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:46:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:46:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:46:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:46:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:46:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:46:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:46:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:46:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:46:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:46:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:46:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:46:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:46:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:46:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:46:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:46:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:46:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:46:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:46:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:46:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:46:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:46:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:46:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:46:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:46:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:46:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:46:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:46:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:46:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:46:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:46:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:46:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:46:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:46:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:46:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:46:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:46:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:46:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:46:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:46:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:46:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:46:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:46:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:46:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:46:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:46:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:46:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:46:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:46:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:46:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:46:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:46:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:46:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:46:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:46:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:46:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:46:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:46:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:46:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:46:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:46:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:46:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:46:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:46:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:46:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:46:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:46:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:46:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:46:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:46:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:46:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:46:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:46:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:46:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:46:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:46:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:46:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:46:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:46:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:46:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:46:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:46:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:46:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:46:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:46:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:46:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:46:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:46:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:46:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:46:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:46:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:46:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:46:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:46:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:46:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:46:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:46:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:46:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:46:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:46:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:46:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:46:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:46:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:46:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:46:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:46:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:46:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:46:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:46:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:46:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:46:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:46:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:46:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:46:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:46:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:46:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:46:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:46:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:46:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:46:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:46:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:46:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:46:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:46:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:46:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:46:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:46:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:46:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:46:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:46:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:46:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:46:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:46:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:46:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:46:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:46:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:46:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:46:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:46:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:46:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:46:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:46:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:46:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:46:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:46:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:46:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:46:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:46:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:47:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:47:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:47:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:47:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:47:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:47:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:47:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:47:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:47:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:47:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:47:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:47:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:47:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:47:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:47:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:47:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:47:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:47:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:47:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:47:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:47:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:47:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:47:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:47:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:47:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:47:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:47:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:47:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:47:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:47:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:47:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:47:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:47:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:47:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:47:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:47:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:47:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:47:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:47:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:47:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:47:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:47:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:47:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:47:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:47:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:47:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:47:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:47:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:47:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:47:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:47:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:47:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:47:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:47:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:47:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:47:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:47:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:47:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:47:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:47:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:47:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:47:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:47:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:47:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:47:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:47:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:47:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:47:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:47:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:47:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:47:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:47:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:47:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:47:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:47:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:47:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:47:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:47:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:47:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:47:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:47:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:47:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:47:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:47:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:47:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:47:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:47:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:47:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:47:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:47:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:47:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:47:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:47:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:47:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:47:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:47:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:47:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:47:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:47:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:47:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:47:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:47:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:47:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:47:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:47:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:47:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:47:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:47:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:47:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:47:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:47:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:47:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:47:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:47:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:47:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:47:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:47:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:47:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:47:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:47:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:47:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:47:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:47:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:47:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:47:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:47:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:47:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:47:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:47:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:47:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:47:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:47:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:47:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:47:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:47:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:47:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:47:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:47:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:47:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:47:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:47:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:47:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:47:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:47:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:47:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:47:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:47:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:47:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:47:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:47:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:47:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:47:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:47:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:47:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:47:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:47:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:47:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:47:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:47:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:47:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:47:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:47:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:47:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:47:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:47:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:47:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:47:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:47:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:47:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:47:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:47:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:47:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:47:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:47:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:47:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:47:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:47:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:47:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:47:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:47:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:47:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:47:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:47:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:47:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:47:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:47:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:47:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:47:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:47:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:47:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:47:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:47:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:47:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:47:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:47:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:47:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:47:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:47:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:47:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:47:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:47:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:47:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:47:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:47:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:47:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:47:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:47:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:47:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:47:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:47:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:47:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:47:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:47:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:47:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:47:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:47:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:48:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:48:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:48:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:48:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:48:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:48:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:48:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:48:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:48:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:48:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:48:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:48:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:48:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:48:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:48:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:48:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:48:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:48:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:48:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:48:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:48:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:48:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:48:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:48:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:48:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:48:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:48:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:48:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:48:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:48:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:48:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:48:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:48:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:48:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:48:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:48:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:48:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:48:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:48:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:48:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:48:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:48:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:48:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:48:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:48:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:48:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:48:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:48:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:48:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:48:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:48:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:48:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:48:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:48:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:48:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:48:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:48:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:48:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:48:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:48:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:48:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:48:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:48:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:48:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:48:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:48:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:48:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:48:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:48:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:48:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:48:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:48:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:48:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:48:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:48:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:48:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:48:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:48:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:48:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:48:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:48:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:48:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:48:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:48:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:48:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:48:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:48:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:48:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:48:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:48:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:48:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:48:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:48:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:48:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:48:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:48:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:48:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:48:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:48:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:48:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:48:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:48:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:48:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:48:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:48:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:48:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:48:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:48:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:48:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:48:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:48:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:48:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:48:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:48:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:48:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:48:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:48:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:48:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:48:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:48:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:48:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:48:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:48:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:48:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:48:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:48:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:48:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:48:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:48:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:48:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:48:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:48:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:48:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:48:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:48:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:48:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:48:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:48:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:48:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:48:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:48:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:48:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:48:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:48:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:48:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:48:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:48:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:48:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:48:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:48:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:48:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:48:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:48:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:48:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:48:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:48:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:48:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:48:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:48:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:48:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:48:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:48:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:48:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:48:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:48:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:48:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:48:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:48:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:48:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:48:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:48:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:48:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:48:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:48:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:48:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:48:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:48:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:48:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:48:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:48:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:48:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:48:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:48:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:48:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:48:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:48:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:48:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:48:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:48:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:48:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:48:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:48:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:48:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:48:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:48:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:48:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:48:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:48:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:48:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:48:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:48:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:48:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:48:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:48:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:48:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:48:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:48:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:48:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:48:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:48:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:48:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:48:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:48:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:48:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:48:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:48:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:49:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:49:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:49:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:49:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:49:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:49:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:49:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:49:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:49:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:49:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:49:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:49:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:49:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:49:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:49:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:49:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:49:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:49:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:49:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:49:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:49:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:49:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:49:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:49:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:49:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:49:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:49:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:49:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:49:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:49:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:49:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:49:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:49:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:49:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:49:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:49:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:49:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:49:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:49:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:49:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:49:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:49:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:49:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:49:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:49:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:49:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:49:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:49:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:49:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:49:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:49:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:49:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:49:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:49:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:49:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:49:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:49:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:49:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:49:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:49:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:49:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:49:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:49:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:49:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:49:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:49:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:49:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:49:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:49:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:49:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:49:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:49:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:49:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:49:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:49:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:49:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:49:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:49:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:49:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:49:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:49:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:49:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:49:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:49:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:49:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:49:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:49:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:49:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:49:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:49:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:49:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:49:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:49:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:49:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:49:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:49:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:49:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:49:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:49:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:49:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:49:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:49:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:49:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:49:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:49:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:49:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:49:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:49:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:49:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:49:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:49:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:49:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:49:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:49:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:49:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:49:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:49:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:49:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:49:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:49:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:49:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:49:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:49:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:49:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:49:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:49:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:49:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:49:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:49:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:49:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:49:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:49:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:49:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:49:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:49:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:49:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:49:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:49:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:49:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:49:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:49:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:49:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:49:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:49:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:49:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:49:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:49:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:49:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:49:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:49:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:49:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:49:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:49:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:49:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:49:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:49:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:49:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:49:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:49:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:49:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:49:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:49:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:49:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:49:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:49:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:49:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:49:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:49:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:49:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:49:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:49:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:49:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:49:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:49:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:49:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:49:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:49:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:49:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:49:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:49:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:49:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:49:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:49:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:49:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:49:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:49:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:49:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:49:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:49:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:49:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:49:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:49:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:49:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:49:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:49:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:49:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:49:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:49:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:49:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:49:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:49:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:49:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:49:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:49:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:49:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:49:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:49:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:49:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:49:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:49:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:49:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:49:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:49:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:49:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:49:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:49:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:50:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:50:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:50:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:50:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:50:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:50:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:50:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:50:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:50:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:50:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:50:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:50:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:50:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:50:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:50:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:50:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:50:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:50:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:50:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:50:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:50:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:50:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:50:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:50:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:50:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:50:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:50:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:50:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:50:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:50:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:50:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:50:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:50:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:50:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:50:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:50:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:50:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:50:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:50:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:50:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:50:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:50:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:50:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:50:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:50:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:50:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:50:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:50:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:50:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:50:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:50:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:50:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:50:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:50:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:50:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:50:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:50:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:50:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:50:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:50:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:50:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:50:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:50:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:50:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:50:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:50:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:50:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:50:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:50:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:50:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:50:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:50:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:50:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:50:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:50:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:50:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:50:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:50:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:50:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:50:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:50:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:50:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:50:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:50:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:50:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:50:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:50:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:50:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:50:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:50:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:50:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:50:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:50:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:50:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:50:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:50:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:50:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:50:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:50:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:50:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:50:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:50:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:50:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:50:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:50:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:50:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:50:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:50:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:50:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:50:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:50:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:50:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:50:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:50:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:50:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:50:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:50:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:50:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:50:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:50:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:50:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:50:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:50:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:50:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:50:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:50:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:50:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:50:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:50:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:50:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:50:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:50:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:50:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:50:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:50:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:50:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:50:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:50:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:50:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:50:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:50:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:50:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:50:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:50:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:50:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:50:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:50:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:50:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:50:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:50:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:50:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:50:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:50:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:50:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:50:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:50:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:50:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:50:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:50:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:50:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:50:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:50:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:50:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:50:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:50:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:50:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:50:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:50:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:50:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:50:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:50:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:50:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:50:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:50:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:50:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:50:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:50:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:50:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:50:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:50:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:50:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:50:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:50:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:50:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:50:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:50:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:50:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:50:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:50:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:50:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:50:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:50:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:50:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:50:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:50:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:50:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:50:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:50:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:50:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:50:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:50:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:50:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:50:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:50:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:50:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:50:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:50:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:50:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:50:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:50:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:50:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:50:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:50:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:50:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:50:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:50:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:51:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:51:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:51:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:51:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:51:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:51:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:51:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:51:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:51:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:51:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:51:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:51:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:51:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:51:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:51:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:51:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:51:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:51:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:51:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:51:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:51:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:51:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:51:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:51:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:51:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:51:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:51:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:51:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:51:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:51:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:51:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:51:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:51:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:51:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:51:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:51:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:51:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:51:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:51:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:51:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:51:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:51:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:51:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:51:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:51:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:51:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:51:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:51:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:51:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:51:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:51:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:51:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:51:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:51:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:51:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:51:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:51:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:51:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:51:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:51:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:51:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:51:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:51:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:51:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:51:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:51:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:51:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:51:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:51:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:51:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:51:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:51:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:51:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:51:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:51:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:51:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:51:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:51:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:51:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:51:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:51:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:51:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:51:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:51:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:51:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:51:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:51:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:51:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:51:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:51:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:51:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:51:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:51:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:51:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:51:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:51:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:51:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:51:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:51:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:51:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:51:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:51:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:51:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:51:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:51:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:51:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:51:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:51:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:51:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:51:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:51:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:51:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:51:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:51:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:51:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:51:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:51:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:51:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:51:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:51:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:51:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:51:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:51:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:51:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:51:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:51:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:51:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:51:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:51:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:51:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:51:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:51:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:51:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:51:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:51:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:51:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:51:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:51:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:51:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:51:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:51:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:51:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:51:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:51:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:51:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:51:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:51:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:51:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:51:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:51:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:51:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:51:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:51:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:51:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:51:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:51:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:51:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:51:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:51:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:51:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:51:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:51:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:51:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:51:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:51:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:51:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:51:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:51:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:51:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:51:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:51:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:51:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:51:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:51:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:51:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:51:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:51:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:51:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:51:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:51:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:51:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:51:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:51:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:51:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:51:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:51:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:51:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:51:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:51:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:51:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:51:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:51:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:51:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:51:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:51:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:51:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:51:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:51:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:51:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:51:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:51:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:51:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:51:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:51:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:51:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:51:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:51:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:51:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:51:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:51:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:51:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:51:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:51:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:51:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:51:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:51:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:54:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:54:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:54:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:54:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:54:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:54:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:54:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:54:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:54:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:54:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:54:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:54:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:54:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:54:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:54:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:54:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:54:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:54:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:54:02 rdma06 rsyslogd[1041]: imjournal: 5082 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 05:54:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:54:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:54:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:54:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:54:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:54:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:54:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:54:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:54:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:54:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:54:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:54:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:54:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:54:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:54:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:54:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:54:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:54:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:54:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:54:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:54:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:54:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:54:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:54:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:54:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:54:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:54:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:54:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:54:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:54:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:54:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:54:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:54:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:54:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:54:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:54:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:54:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:54:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:54:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:54:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:54:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:54:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:54:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:54:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:54:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:54:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:54:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:54:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:54:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:54:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:54:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:54:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:54:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:54:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:54:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:54:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:54:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:54:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:54:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:54:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:54:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:54:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:54:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:54:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:54:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:54:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:54:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:54:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:54:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:54:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:54:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:54:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:54:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:54:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:54:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:54:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:54:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:54:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:54:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:54:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:54:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:54:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:54:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:54:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:54:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:54:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:54:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:54:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:54:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:54:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:54:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:54:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:54:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:54:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:54:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:54:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:54:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:54:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:54:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:54:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:54:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:54:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:54:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:54:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:54:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:54:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:54:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:54:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:54:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:54:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:54:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:54:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:54:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:54:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:54:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:54:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:54:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:54:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:54:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:54:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:54:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:54:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:54:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:54:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:54:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:54:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:54:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:54:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:54:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:54:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:54:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:54:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:54:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:54:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:54:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:54:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:54:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:54:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:54:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:54:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:54:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:54:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:54:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:54:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:54:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:54:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:54:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:54:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:54:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:54:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:54:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:54:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:54:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:54:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:54:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:54:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:54:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:54:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:54:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:54:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:54:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:54:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:54:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:54:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:54:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:54:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:54:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:54:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:54:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:54:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:54:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:54:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:54:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:54:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:54:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:54:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:54:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:54:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:54:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:54:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:54:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:54:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:54:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:54:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:54:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:54:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:54:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:54:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:54:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:54:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:54:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:54:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:54:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:54:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:54:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:54:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:54:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:54:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:55:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:55:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:55:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:55:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:55:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:55:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:55:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:55:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:55:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:55:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:55:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:55:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:55:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:55:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:55:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:55:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:55:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:55:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:55:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:55:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:55:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:55:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:55:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:55:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:55:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:55:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:55:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:55:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:55:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:55:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:55:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:55:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:55:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:55:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:55:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:55:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:55:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:55:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:55:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:55:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:55:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:55:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:55:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:55:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:55:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:55:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:55:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:55:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:55:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:55:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:55:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:55:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:55:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:55:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:55:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:55:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:55:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:55:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:55:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:55:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:55:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:55:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:55:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:55:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:55:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:55:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:55:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:55:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:55:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:55:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:55:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:55:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:55:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:55:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:55:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:55:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:55:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:55:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:55:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:55:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:55:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:55:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:55:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:55:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:55:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:55:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:55:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:55:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:55:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:55:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:55:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:55:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:55:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:55:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:55:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:55:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:55:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:55:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:55:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:55:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:55:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:55:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:55:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:55:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:55:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:55:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:55:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:55:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:55:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:55:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:55:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:55:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:55:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:55:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:55:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:55:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:55:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:55:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:55:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:55:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:55:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:55:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:55:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:55:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:55:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:55:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:55:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:55:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:55:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:55:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:55:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:55:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:55:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:55:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:55:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:55:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:55:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:55:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:55:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:55:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:55:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:55:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:55:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:55:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:55:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:55:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:55:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:55:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:55:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:55:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:55:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:55:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:55:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:55:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:55:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:55:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:55:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:55:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:55:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:55:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:55:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:55:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:55:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:55:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:55:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:55:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:55:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:55:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:55:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:55:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:55:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:55:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:55:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:55:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:55:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:55:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:55:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:55:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:55:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:55:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:55:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:55:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:55:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:55:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:55:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:55:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:55:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:55:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:55:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:55:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:55:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:55:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:55:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:55:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:55:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:55:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:55:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:55:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:55:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:55:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:55:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:55:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:55:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:55:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:55:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:55:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:55:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:55:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:55:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:55:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:55:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:55:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:55:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:55:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:55:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:55:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:56:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:56:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:56:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:56:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:56:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:56:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:56:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:56:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:56:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:56:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:56:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:56:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:56:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:56:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:56:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:56:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:56:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:56:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:56:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:56:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:56:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:56:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:56:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:56:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:56:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:56:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:56:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:56:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:56:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:56:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:56:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:56:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:56:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:56:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:56:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:56:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:56:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:56:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:56:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:56:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:56:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:56:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:56:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:56:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:56:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:56:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:56:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:56:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:56:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:56:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:56:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:56:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:56:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:56:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:56:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:56:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:56:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:56:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:56:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:56:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:56:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:56:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:56:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:56:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:56:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:56:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:56:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:56:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:56:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:56:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:56:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:56:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:56:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:56:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:56:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:56:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:56:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:56:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:56:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:56:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:56:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:56:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:56:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:56:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:56:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:56:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:56:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:56:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:56:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:56:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:56:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:56:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:56:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:56:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:56:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:56:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:56:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:56:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:56:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:56:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:56:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:56:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:56:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:56:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:56:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:56:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:56:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:56:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:56:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:56:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:56:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:56:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:56:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:56:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:56:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:56:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:56:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:56:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:56:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:56:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:56:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:56:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:56:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:56:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:56:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:56:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:56:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:56:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:56:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:56:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:56:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:56:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:56:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:56:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:56:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:56:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:56:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:56:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:56:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:56:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:56:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:56:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:56:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:56:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:56:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:56:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:56:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:56:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:56:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:56:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:56:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:56:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:56:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:56:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:56:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:56:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:56:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:56:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:56:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:56:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:56:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:56:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:56:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:56:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:56:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:56:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:56:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:56:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:56:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:56:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:56:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:56:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:56:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:56:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:56:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:56:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:56:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:56:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:56:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:56:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:56:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:56:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:56:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:56:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:56:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:56:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:56:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:56:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:56:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:56:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:56:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:56:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:56:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:56:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:56:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:56:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:56:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:56:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:56:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:56:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:56:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:56:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:56:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:56:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:56:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:56:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:56:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:56:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:56:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:56:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:56:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:56:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:56:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:56:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:56:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:56:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:57:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:57:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:57:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:57:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:57:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:57:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:57:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:57:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:57:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:57:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:57:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:57:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:57:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:57:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:57:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:57:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:57:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:57:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:57:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:57:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:57:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:57:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:57:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:57:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:57:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:57:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:57:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:57:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:57:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:57:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:57:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:57:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:57:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:57:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:57:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:57:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:57:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:57:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:57:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:57:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:57:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:57:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:57:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:57:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:57:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:57:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:57:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:57:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:57:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:57:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:57:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:57:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:57:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:57:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:57:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:57:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:57:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:57:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:57:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:57:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:57:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:57:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:57:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:57:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:57:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:57:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:57:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:57:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:57:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:57:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:57:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:57:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:57:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:57:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:57:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:57:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:57:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:57:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:57:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:57:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:57:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:57:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:57:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:57:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:57:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:57:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:57:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:57:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:57:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:57:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:57:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:57:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:57:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:57:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:57:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:57:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:57:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:57:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:57:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:57:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:57:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:57:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:57:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:57:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:57:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:57:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:57:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:57:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:57:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:57:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:57:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:57:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:57:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:57:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:57:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:57:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:57:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:57:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:57:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:57:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:57:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:57:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:57:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:57:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:57:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:57:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:57:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:57:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:57:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:57:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:57:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:57:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:57:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:57:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:57:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:57:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:57:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:57:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:57:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:57:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:57:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:57:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:57:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:57:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:57:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:57:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:57:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:57:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:57:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:57:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:57:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:57:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:57:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:57:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:57:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:57:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:57:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:57:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:57:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:57:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:57:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:57:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:57:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:57:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:57:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:57:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:57:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:57:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:57:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:57:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:57:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:57:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:57:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:57:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:57:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:57:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:57:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:57:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:57:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:57:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:57:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:57:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:57:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:57:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:57:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:57:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:57:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:57:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:57:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:57:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:57:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:57:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:57:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:57:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:57:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:57:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:57:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:57:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:57:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:57:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:57:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:57:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:57:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:57:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:57:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:57:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:57:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:57:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:57:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:57:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:57:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:57:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:57:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:57:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:57:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:57:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:58:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:58:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:58:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:58:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:58:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:58:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:58:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:58:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:58:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:58:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:58:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:58:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:58:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:58:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:58:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:58:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:58:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:58:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:58:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:58:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:58:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:58:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:58:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:58:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:58:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:58:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:58:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:58:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:58:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:58:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:58:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:58:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:58:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:58:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:58:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:58:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:58:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:58:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:58:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:58:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:58:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:58:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:58:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:58:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:58:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:58:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:58:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:58:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:58:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:58:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:58:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:58:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:58:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:58:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:58:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:58:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:58:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:58:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:58:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:58:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:58:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:58:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:58:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:58:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:58:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:58:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:58:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:58:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:58:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:58:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:58:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:58:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:58:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:58:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:58:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:58:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:58:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:58:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:58:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:58:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:58:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:58:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:58:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:58:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:58:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:58:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:58:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:58:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:58:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:58:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:58:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:58:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:58:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:58:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:58:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:58:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:58:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:58:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:58:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:58:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:58:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:58:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:58:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:58:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:58:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:58:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:58:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:58:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:58:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:58:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:58:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:58:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:58:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:58:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:58:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:58:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:58:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:58:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:58:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:58:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:58:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:58:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:58:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:58:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:58:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:58:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:58:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:58:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:58:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:58:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:58:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:58:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:58:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:58:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:58:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:58:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:58:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:58:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:58:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:58:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:58:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:58:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:58:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:58:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:58:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:58:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:58:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:58:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:58:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:58:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:58:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:58:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:58:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:58:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:58:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:58:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:58:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:58:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:58:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:58:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:58:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:58:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:58:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:58:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:58:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:58:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:58:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:58:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:58:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:58:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:58:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:58:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:58:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:58:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:58:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:58:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:58:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:58:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:58:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:58:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:58:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:58:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:58:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:58:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:58:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:58:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:58:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:58:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:58:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:58:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:58:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:58:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:58:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:58:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:58:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:58:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:58:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:58:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:58:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:58:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:58:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:58:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:58:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:58:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:58:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:58:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:58:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:58:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:58:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:58:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:58:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:58:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:58:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:58:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:58:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:58:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:59:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:59:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:59:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:59:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:59:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:59:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:59:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:59:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:59:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:59:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:59:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:59:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:59:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:59:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:59:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:59:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:59:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:59:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:59:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:59:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:59:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:59:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:59:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:59:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:59:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:59:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:59:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:59:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:59:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:59:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:59:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:59:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:59:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:59:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:59:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:59:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:59:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:59:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:59:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:59:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:59:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:59:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:59:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:59:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:59:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:59:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:59:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:59:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:59:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:59:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:59:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:59:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:59:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:59:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:59:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:59:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:59:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:59:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:59:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:59:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:59:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:59:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:59:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:59:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:59:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:59:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:59:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:59:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:59:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:59:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:59:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:59:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:59:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:59:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:59:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:59:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:59:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:59:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:59:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:59:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:59:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:59:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:59:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:59:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:59:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:59:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:59:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:59:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:59:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:59:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:59:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:59:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:59:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:59:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:59:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:59:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:59:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:59:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:59:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:59:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:59:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:59:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:59:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:59:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:59:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:59:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:59:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:59:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:59:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:59:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:59:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:59:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:59:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:59:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:59:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:59:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:59:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:59:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:59:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:59:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:59:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:59:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:59:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:59:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:59:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:59:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:59:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:59:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:59:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:59:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:59:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:59:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:59:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:59:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:59:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:59:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:59:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:59:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:59:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:59:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:59:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:59:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:59:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:59:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:59:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:59:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:59:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:59:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:59:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:59:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:59:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:59:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:59:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:59:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:59:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:59:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:59:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:59:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:59:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:59:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:59:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:59:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:59:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:59:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:59:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:59:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:59:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:59:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:59:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:59:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:59:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:59:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:59:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:59:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:59:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:59:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:59:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:59:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:59:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:59:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:59:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:59:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:59:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:59:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:59:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:59:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:59:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:59:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:59:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:59:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:59:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:59:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:59:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:59:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:59:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:59:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:59:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:59:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 05:59:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 05:59:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:59:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:59:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:59:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 05:59:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 05:59:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 05:59:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 05:59:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 05:59:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 05:59:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 05:59:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 05:59:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 05:59:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 05:59:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 05:59:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 05:59:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 05:59:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:00:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:00:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:00:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:00:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:00:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:00:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:00:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:00:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:00:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:00:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:00:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:00:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:00:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:00:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:00:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:00:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:00:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:00:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:00:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:00:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:00:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:00:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:00:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:00:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:00:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:00:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:00:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:00:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:00:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:00:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:00:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:00:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:00:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:00:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:00:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:00:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:00:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:00:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:00:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:00:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:00:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:00:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:00:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:00:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:00:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:00:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:00:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:00:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:00:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:00:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:00:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:00:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:00:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:00:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:00:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:00:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:00:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:00:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:00:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:00:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:00:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:00:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:00:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:00:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:00:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:00:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:00:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:00:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:00:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:00:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:00:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:00:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:00:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:00:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:00:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:00:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:00:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:00:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:00:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:00:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:00:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:00:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:00:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:00:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:00:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:00:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:00:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:00:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:00:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:00:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:00:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:00:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:00:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:00:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:00:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:00:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:00:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:00:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:00:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:00:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:00:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:00:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:00:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:00:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:00:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:00:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:00:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:00:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:00:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:00:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:00:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:00:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:00:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:00:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:00:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:00:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:00:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:00:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:00:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:00:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:00:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:00:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:00:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:00:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:00:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:00:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:00:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:00:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:00:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:00:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:00:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:00:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:00:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:00:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:00:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:00:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:00:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:00:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:00:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:00:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:00:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:00:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:00:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:00:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:00:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:00:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:00:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:00:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:00:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:00:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:00:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:00:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:00:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:00:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:00:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:00:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:00:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:00:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:00:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:00:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:00:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:00:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:00:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:00:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:00:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:00:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:00:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:00:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:00:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:00:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:00:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:00:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:00:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:00:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:00:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:00:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:00:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:00:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:00:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:00:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:00:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:00:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:00:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:00:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:00:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:00:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:00:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:00:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:00:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:00:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:00:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:00:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:00:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:00:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:00:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:00:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:00:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:00:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:00:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:00:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:00:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:00:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:00:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:00:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:00:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:00:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:00:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:00:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:00:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:00:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:00:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:00:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:00:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:00:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:00:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:00:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:01:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:01:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:01:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:01:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:01:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:01:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:01:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:01:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:01:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:01:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:01:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:01:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:01:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:01:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:01:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:01:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:01:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:01:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:01:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:01:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:01:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:01:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:01:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:01:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:01:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:01:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:01:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:01:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:01:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:01:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:01:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:01:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:01:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:01:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:01:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:01:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:01:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:01:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:01:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:01:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:01:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:01:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:01:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:01:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:01:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:01:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:01:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:01:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:01:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:01:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:01:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:01:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:01:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:01:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:01:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:01:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:01:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:01:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:01:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:01:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:01:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:01:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:01:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:01:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:01:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:01:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:01:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:01:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:01:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:01:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:01:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:01:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:01:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:01:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:01:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:01:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:01:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:01:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:01:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:01:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:01:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:01:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:01:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:01:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:01:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:01:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:01:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:01:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:01:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:01:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:01:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:01:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:01:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:01:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:01:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:01:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:01:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:01:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:01:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:01:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:01:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:01:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:01:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:01:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:01:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:01:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:01:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:01:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:01:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:01:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:01:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:01:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:01:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:01:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:01:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:01:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:01:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:01:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:01:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:01:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:01:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:01:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:01:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:01:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:01:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:01:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:01:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:01:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:01:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:01:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:01:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:01:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:01:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:01:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:01:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:01:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:01:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:01:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:01:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:01:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:01:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:01:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:01:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:01:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:01:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:01:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:01:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:01:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:01:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:01:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:01:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:01:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:01:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:01:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:01:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:01:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:01:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:01:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:01:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:01:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:01:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:01:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:01:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:01:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:01:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:01:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:01:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:01:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:01:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:01:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:01:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:01:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:01:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:01:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:01:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:01:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:01:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:01:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:01:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:01:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:01:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:01:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:01:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:01:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:01:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:01:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:01:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:01:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:01:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:01:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:01:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:01:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:01:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:01:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:01:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:01:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:01:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:01:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:01:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:01:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:01:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:01:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:01:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:01:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:01:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:01:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:01:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:01:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:01:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:01:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:01:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:01:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:01:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:01:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:01:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:01:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:04:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:04:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:04:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:04:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:04:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:04:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:04:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:04:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:04:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:04:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:04:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:04:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:04:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:04:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:04:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:04:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:04:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:04:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:04:07 rdma06 rsyslogd[1041]: imjournal: 5295 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 06:04:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:04:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:04:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:04:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:04:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:04:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:04:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:04:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:04:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:04:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:04:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:04:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:04:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:04:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:04:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:04:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:04:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:04:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:04:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:04:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:04:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:04:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:04:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:04:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:04:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:04:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:04:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:04:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:04:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:04:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:04:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:04:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:04:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:04:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:04:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:04:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:04:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:04:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:04:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:04:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:04:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:04:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:04:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:04:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:04:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:04:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:04:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:04:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:04:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:04:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:04:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:04:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:04:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:04:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:04:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:04:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:04:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:04:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:04:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:04:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:04:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:04:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:04:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:04:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:04:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:04:27 rdma06 systemd-journald[790]: Data hash table of /run/log/journal/95379d2f723847b68d9e976dee8967a9/system.journal has a fill level at 75.0 (163751 of 218332 items, 75497472 file size, 461 bytes per hash table item), suggesting rotation.
Apr 17 06:04:27 rdma06 systemd-journald[790]: /run/log/journal/95379d2f723847b68d9e976dee8967a9/system.journal: Journal header limits reached or header out-of-date, rotating.
Apr 17 06:04:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:04:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:04:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:04:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:04:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:04:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:04:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:04:27 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 17 06:04:27 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 17 06:04:27 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 17 06:04:27 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 17 06:04:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:04:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:04:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:04:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:04:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:04:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:04:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:04:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:04:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:04:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:04:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:04:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:04:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:04:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:04:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:04:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:04:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:04:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:04:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:04:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:04:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:04:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:04:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:04:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:04:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:04:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:04:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:04:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:04:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:04:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:04:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:04:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:04:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:04:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:04:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:04:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:04:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:04:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:04:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:04:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:04:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:04:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:04:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:04:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:04:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:04:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:04:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:04:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:04:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:04:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:04:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:04:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:04:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:04:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:04:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:04:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:04:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:04:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:04:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:04:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:04:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:04:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:04:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:04:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:04:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:04:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:04:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:04:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:04:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:04:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:04:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:04:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:04:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:04:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:04:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:04:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:04:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:04:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:04:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:04:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:04:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:04:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:04:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:04:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:04:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:04:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:04:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:04:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:04:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:04:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:04:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:04:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:04:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:04:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:04:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:04:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:04:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:04:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:04:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:04:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:04:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:04:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:04:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:04:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:04:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:04:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:04:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:04:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:05:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:05:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:05:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:05:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:05:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:05:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:05:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:05:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:05:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:05:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:05:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:05:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:05:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:05:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:05:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:05:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:05:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:05:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:05:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:05:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:05:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:05:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:05:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:05:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:05:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:05:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:05:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:05:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:05:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:05:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:05:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:05:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:05:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:05:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:05:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:05:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:05:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:05:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:05:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:05:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:05:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:05:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:05:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:05:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:05:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:05:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:05:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:05:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:05:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:05:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:05:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:05:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:05:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:05:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:05:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:05:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:05:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:05:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:05:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:05:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:05:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:05:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:05:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:05:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:05:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:05:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:05:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:05:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:05:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:05:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:05:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:05:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:05:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:05:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:05:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:05:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:05:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:05:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:05:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:05:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:05:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:05:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:05:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:05:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:05:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:05:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:05:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:05:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:05:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:05:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:05:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:05:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:05:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:05:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:05:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:05:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:05:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:05:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:05:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:05:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:05:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:05:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:05:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:05:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:05:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:05:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:05:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:05:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:05:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:05:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:05:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:05:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:05:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:05:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:05:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:05:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:05:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:05:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:05:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:05:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:05:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:05:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:05:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:05:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:05:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:05:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:05:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:05:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:05:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:05:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:05:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:05:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:05:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:05:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:05:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:05:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:05:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:05:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:05:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:05:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:05:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:05:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:05:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:05:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:05:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:05:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:05:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:05:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:05:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:05:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:05:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:05:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:05:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:05:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:05:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:05:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:05:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:05:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:05:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:05:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:05:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:05:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:05:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:05:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:05:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:05:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:05:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:05:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:05:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:05:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:05:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:05:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:05:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:05:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:05:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:05:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:05:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:05:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:05:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:05:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:05:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:05:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:05:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:05:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:05:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:05:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:05:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:05:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:05:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:05:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:05:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:05:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:05:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:05:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:05:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:05:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:05:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:05:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:05:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:05:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:05:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:05:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:05:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:05:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:05:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:05:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:05:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:05:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:05:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:05:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:05:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:05:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:05:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:05:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:05:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:05:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:06:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:06:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:06:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:06:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:06:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:06:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:06:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:06:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:06:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:06:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:06:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:06:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:06:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:06:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:06:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:06:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:06:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:06:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:06:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:06:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:06:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:06:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:06:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:06:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:06:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:06:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:06:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:06:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:06:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:06:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:06:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:06:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:06:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:06:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:06:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:06:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:06:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:06:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:06:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:06:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:06:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:06:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:06:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:06:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:06:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:06:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:06:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:06:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:06:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:06:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:06:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:06:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:06:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:06:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:06:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:06:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:06:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:06:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:06:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:06:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:06:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:06:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:06:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:06:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:06:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:06:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:06:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:06:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:06:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:06:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:06:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:06:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:06:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:06:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:06:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:06:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:06:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:06:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:06:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:06:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:06:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:06:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:06:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:06:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:06:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:06:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:06:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:06:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:06:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:06:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:06:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:06:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:06:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:06:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:06:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:06:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:06:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:06:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:06:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:06:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:06:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:06:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:06:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:06:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:06:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:06:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:06:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:06:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:06:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:06:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:06:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:06:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:06:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:06:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:06:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:06:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:06:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:06:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:06:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:06:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:06:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:06:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:06:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:06:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:06:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:06:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:06:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:06:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:06:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:06:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:06:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:06:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:06:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:06:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:06:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:06:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:06:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:06:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:06:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:06:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:06:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:06:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:06:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:06:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:06:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:06:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:06:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:06:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:06:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:06:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:06:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:06:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:06:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:06:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:06:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:06:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:06:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:06:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:06:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:06:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:06:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:06:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:06:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:06:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:06:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:06:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:06:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:06:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:06:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:06:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:06:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:06:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:06:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:06:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:06:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:06:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:06:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:06:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:06:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:06:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:06:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:06:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:06:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:06:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:06:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:06:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:06:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:06:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:06:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:06:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:06:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:06:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:06:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:06:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:06:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:06:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:06:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:06:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:06:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:06:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:06:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:06:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:06:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:06:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:06:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:06:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:06:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:06:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:06:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:06:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:06:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:06:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:06:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:06:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:06:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:06:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:07:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:07:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:07:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:07:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:07:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:07:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:07:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:07:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:07:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:07:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:07:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:07:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:07:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:07:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:07:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:07:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:07:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:07:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:07:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:07:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:07:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:07:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:07:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:07:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:07:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:07:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:07:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:07:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:07:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:07:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:07:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:07:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:07:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:07:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:07:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:07:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:07:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:07:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:07:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:07:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:07:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:07:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:07:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:07:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:07:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:07:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:07:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:07:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:07:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:07:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:07:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:07:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:07:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:07:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:07:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:07:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:07:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:07:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:07:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:07:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:07:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:07:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:07:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:07:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:07:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:07:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:07:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:07:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:07:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:07:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:07:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:07:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:07:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:07:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:07:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:07:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:07:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:07:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:07:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:07:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:07:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:07:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:07:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:07:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:07:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:07:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:07:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:07:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:07:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:07:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:07:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:07:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:07:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:07:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:07:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:07:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:07:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:07:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:07:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:07:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:07:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:07:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:07:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:07:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:07:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:07:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:07:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:07:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:07:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:07:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:07:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:07:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:07:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:07:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:07:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:07:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:07:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:07:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:07:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:07:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:07:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:07:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:07:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:07:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:07:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:07:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:07:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:07:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:07:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:07:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:07:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:07:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:07:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:07:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:07:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:07:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:07:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:07:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:07:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:07:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:07:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:07:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:07:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:07:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:07:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:07:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:07:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:07:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:07:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:07:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:07:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:07:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:07:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:07:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:07:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:07:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:07:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:07:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:07:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:07:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:07:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:07:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:07:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:07:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:07:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:07:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:07:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:07:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:07:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:07:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:07:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:07:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:07:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:07:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:07:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:07:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:07:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:07:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:07:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:07:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:07:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:07:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:07:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:07:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:07:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:07:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:07:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:07:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:07:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:07:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:07:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:07:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:07:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:07:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:07:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:07:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:07:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:07:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:07:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:07:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:07:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:07:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:07:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:07:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:07:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:07:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:07:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:07:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:07:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:07:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:07:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:07:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:07:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:07:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:07:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:07:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:08:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:08:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:08:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:08:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:08:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:08:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:08:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:08:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:08:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:08:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:08:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:08:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:08:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:08:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:08:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:08:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:08:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:08:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:08:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:08:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:08:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:08:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:08:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:08:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:08:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:08:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:08:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:08:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:08:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:08:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:08:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:08:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:08:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:08:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:08:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:08:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:08:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:08:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:08:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:08:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:08:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:08:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:08:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:08:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:08:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:08:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:08:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:08:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:08:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:08:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:08:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:08:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:08:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:08:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:08:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:08:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:08:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:08:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:08:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:08:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:08:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:08:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:08:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:08:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:08:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:08:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:08:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:08:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:08:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:08:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:08:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:08:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:08:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:08:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:08:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:08:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:08:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:08:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:08:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:08:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:08:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:08:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:08:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:08:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:08:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:08:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:08:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:08:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:08:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:08:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:08:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:08:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:08:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:08:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:08:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:08:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:08:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:08:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:08:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:08:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:08:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:08:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:08:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:08:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:08:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:08:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:08:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:08:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:08:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:08:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:08:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:08:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:08:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:08:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:08:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:08:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:08:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:08:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:08:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:08:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:08:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:08:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:08:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:08:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:08:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:08:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:08:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:08:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:08:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:08:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:08:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:08:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:08:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:08:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:08:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:08:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:08:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:08:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:08:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:08:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:08:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:08:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:08:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:08:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:08:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:08:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:08:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:08:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:08:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:08:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:08:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:08:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:08:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:08:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:08:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:08:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:08:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:08:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:08:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:08:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:08:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:08:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:08:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:08:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:08:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:08:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:08:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:08:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:08:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:08:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:08:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:08:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:08:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:08:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:08:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:08:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:08:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:08:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:08:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:08:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:08:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:08:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:08:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:08:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:08:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:08:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:08:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:08:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:08:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:08:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:08:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:08:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:08:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:08:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:08:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:08:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:08:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:08:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:08:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:08:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:08:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:08:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:08:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:08:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:08:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:08:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:08:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:08:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:08:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:08:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:08:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:08:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:08:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:08:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:08:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:08:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:09:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:09:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:09:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:09:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:09:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:09:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:09:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:09:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:09:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:09:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:09:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:09:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:09:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:09:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:09:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:09:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:09:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:09:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:09:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:09:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:09:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:09:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:09:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:09:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:09:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:09:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:09:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:09:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:09:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:09:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:09:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:09:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:09:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:09:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:09:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:09:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:09:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:09:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:09:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:09:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:09:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:09:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:09:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:09:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:09:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:09:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:09:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:09:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:09:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:09:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:09:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:09:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:09:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:09:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:09:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:09:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:09:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:09:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:09:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:09:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:09:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:09:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:09:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:09:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:09:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:09:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:09:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:09:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:09:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:09:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:09:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:09:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:09:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:09:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:09:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:09:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:09:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:09:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:09:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:09:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:09:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:09:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:09:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:09:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:09:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:09:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:09:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:09:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:09:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:09:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:09:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:09:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:09:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:09:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:09:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:09:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:09:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:09:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:09:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:09:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:09:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:09:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:09:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:09:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:09:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:09:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:09:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:09:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:09:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:09:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:09:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:09:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:09:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:09:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:09:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:09:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:09:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:09:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:09:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:09:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:09:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:09:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:09:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:09:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:09:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:09:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:09:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:09:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:09:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:09:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:09:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:09:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:09:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:09:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:09:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:09:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:09:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:09:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:09:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:09:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:09:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:09:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:09:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:09:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:09:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:09:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:09:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:09:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:09:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:09:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:09:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:09:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:09:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:09:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:09:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:09:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:09:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:09:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:09:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:09:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:09:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:09:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:09:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:09:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:09:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:09:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:09:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:09:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:09:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:09:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:09:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:09:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:09:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:09:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:09:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:09:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:09:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:09:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:09:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:09:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:09:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:09:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:09:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:09:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:09:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:09:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:09:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:09:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:09:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:09:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:09:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:09:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:09:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:09:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:09:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:09:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:09:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:09:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:09:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:09:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:09:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:09:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:09:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:09:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:09:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:09:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:09:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:09:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:09:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:09:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:09:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:09:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:09:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:09:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:09:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:09:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:10:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:10:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:10:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:10:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:10:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:10:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:10:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:10:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:10:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:10:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:10:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:10:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:10:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:10:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:10:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:10:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:10:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:10:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:10:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:10:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:10:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:10:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:10:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:10:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:10:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:10:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:10:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:10:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:10:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:10:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:10:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:10:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:10:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:10:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:10:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:10:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:10:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:10:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:10:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:10:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:10:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:10:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:10:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:10:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:10:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:10:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:10:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:10:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:10:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:10:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:10:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:10:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:10:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:10:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:10:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:10:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:10:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:10:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:10:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:10:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:10:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:10:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:10:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:10:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:10:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:10:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:10:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:10:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:10:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:10:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:10:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:10:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:10:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:10:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:10:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:10:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:10:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:10:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:10:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:10:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:10:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:10:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:10:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:10:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:10:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:10:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:10:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:10:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:10:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:10:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:10:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:10:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:10:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:10:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:10:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:10:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:10:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:10:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:10:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:10:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:10:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:10:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:10:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:10:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:10:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:10:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:10:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:10:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:10:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:10:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:10:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:10:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:10:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:10:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:10:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:10:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:10:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:10:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:10:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:10:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:10:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:10:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:10:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:10:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:10:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:10:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:10:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:10:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:10:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:10:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:10:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:10:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:10:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:10:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:10:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:10:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:10:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:10:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:10:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:10:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:10:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:10:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:10:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:10:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:10:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:10:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:10:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:10:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:10:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:10:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:10:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:10:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:10:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:10:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:10:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:10:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:10:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:10:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:10:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:10:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:10:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:10:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:10:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:10:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:10:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:10:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:10:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:10:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:10:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:10:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:10:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:10:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:10:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:10:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:10:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:10:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:10:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:10:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:10:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:10:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:10:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:10:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:10:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:10:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:10:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:10:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:10:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:10:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:10:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:10:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:10:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:10:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:10:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:10:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:10:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:10:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:10:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:10:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:10:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:10:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:10:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:10:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:10:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:10:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:10:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:10:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:10:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:10:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:10:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:10:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:10:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:10:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:10:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:10:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:10:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:10:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:11:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:11:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:11:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:11:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:11:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:11:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:11:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:11:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:11:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:11:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:11:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:11:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:11:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:11:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:11:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:11:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:11:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:11:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:11:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:11:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:11:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:11:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:11:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:11:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:11:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:11:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:11:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:11:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:11:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:11:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:11:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:11:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:11:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:11:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:11:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:11:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:11:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:11:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:11:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:11:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:11:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:11:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:11:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:11:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:11:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:11:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:11:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:11:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:11:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:11:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:11:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:11:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:11:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:11:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:11:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:11:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:11:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:11:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:11:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:11:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:11:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:11:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:11:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:11:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:11:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:11:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:11:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:11:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:11:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:11:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:11:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:11:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:11:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:11:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:11:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:11:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:11:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:11:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:11:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:11:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:11:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:11:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:11:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:11:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:11:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:11:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:11:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:11:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:11:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:11:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:11:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:11:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:11:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:11:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:11:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:11:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:11:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:11:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:11:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:11:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:11:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:11:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:11:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:11:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:11:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:11:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:11:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:11:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:11:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:11:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:11:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:11:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:11:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:11:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:11:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:11:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:11:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:11:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:11:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:11:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:11:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:11:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:11:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:11:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:11:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:11:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:11:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:11:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:11:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:11:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:11:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:11:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:11:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:11:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:11:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:11:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:11:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:11:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:11:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:11:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:11:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:11:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:11:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:11:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:11:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:11:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:11:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:11:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:11:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:11:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:11:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:11:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:11:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:11:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:11:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:11:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:11:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:11:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:11:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:11:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:11:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:11:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:11:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:11:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:11:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:11:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:11:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:11:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:11:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:11:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:11:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:11:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:11:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:11:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:11:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:11:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:11:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:11:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:11:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:11:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:11:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:11:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:11:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:11:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:11:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:11:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:11:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:11:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:11:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:11:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:11:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:11:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:11:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:11:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:11:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:11:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:11:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:11:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:11:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:11:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:11:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:11:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:11:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:11:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:11:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:11:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:11:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:11:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:11:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:11:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:11:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:11:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:11:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:11:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:11:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:11:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:12:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:12:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:12:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:12:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:12:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:12:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:12:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:12:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:12:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:12:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:12:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:12:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:12:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:12:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:12:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:12:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:12:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:12:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:14:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:14:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:14:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:14:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:14:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:14:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:14:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:14:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:14:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:14:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:14:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:14:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:14:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:14:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:14:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:14:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:14:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:14:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:14:12 rdma06 rsyslogd[1041]: imjournal: 5297 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 06:14:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:14:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:14:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:14:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:14:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:14:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:14:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:14:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:14:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:14:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:14:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:14:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:14:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:14:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:14:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:14:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:14:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:14:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:14:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:14:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:14:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:14:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:14:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:14:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:14:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:14:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:14:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:14:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:14:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:14:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:14:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:14:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:14:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:14:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:14:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:14:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:14:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:14:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:14:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:14:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:14:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:14:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:14:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:14:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:14:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:14:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:14:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:14:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:14:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:14:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:14:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:14:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:14:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:14:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:14:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:14:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:14:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:14:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:14:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:14:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:14:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:14:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:14:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:14:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:14:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:14:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:14:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:14:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:14:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:14:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:14:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:14:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:14:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:14:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:14:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:14:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:14:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:14:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:14:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:14:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:14:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:14:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:14:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:14:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:14:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:14:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:14:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:14:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:14:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:14:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:14:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:14:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:14:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:14:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:14:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:14:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:14:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:14:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:14:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:14:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:14:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:14:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:14:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:14:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:14:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:14:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:14:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:14:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:14:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:14:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:14:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:14:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:14:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:14:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:14:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:14:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:14:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:14:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:14:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:14:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:14:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:14:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:14:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:14:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:14:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:14:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:14:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:14:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:14:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:14:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:14:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:14:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:14:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:14:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:14:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:14:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:14:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:14:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:14:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:14:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:14:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:14:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:14:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:14:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:14:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:14:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:14:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:14:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:14:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:14:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:14:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:14:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:14:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:14:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:14:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:14:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:14:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:14:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:14:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:14:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:14:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:14:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:15:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:15:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:15:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:15:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:15:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:15:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:15:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:15:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:15:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:15:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:15:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:15:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:15:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:15:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:15:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:15:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:15:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:15:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:15:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:15:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:15:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:15:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:15:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:15:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:15:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:15:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:15:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:15:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:15:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:15:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:15:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:15:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:15:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:15:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:15:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:15:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:15:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:15:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:15:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:15:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:15:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:15:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:15:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:15:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:15:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:15:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:15:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:15:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:15:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:15:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:15:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:15:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:15:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:15:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:15:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:15:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:15:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:15:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:15:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:15:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:15:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:15:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:15:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:15:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:15:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:15:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:15:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:15:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:15:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:15:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:15:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:15:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:15:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:15:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:15:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:15:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:15:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:15:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:15:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:15:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:15:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:15:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:15:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:15:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:15:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:15:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:15:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:15:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:15:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:15:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:15:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:15:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:15:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:15:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:15:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:15:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:15:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:15:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:15:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:15:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:15:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:15:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:15:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:15:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:15:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:15:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:15:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:15:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:15:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:15:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:15:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:15:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:15:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:15:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:15:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:15:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:15:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:15:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:15:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:15:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:15:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:15:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:15:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:15:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:15:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:15:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:15:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:15:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:15:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:15:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:15:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:15:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:15:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:15:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:15:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:15:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:15:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:15:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:15:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:15:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:15:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:15:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:15:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:15:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:15:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:15:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:15:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:15:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:15:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:15:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:15:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:15:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:15:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:15:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:15:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:15:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:15:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:15:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:15:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:15:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:15:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:15:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:15:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:15:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:15:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:15:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:15:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:15:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:15:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:15:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:15:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:15:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:15:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:15:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:15:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:15:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:15:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:15:47 rdma06 systemd[1]: Starting dnf makecache...
Apr 17 06:15:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:15:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:15:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:15:47 rdma06 dnf[1145974]: Updating Subscription Management repositories.
Apr 17 06:15:47 rdma06 dnf[1145974]: Unable to read consumer identity
Apr 17 06:15:47 rdma06 dnf[1145974]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 17 06:15:49 rdma06 dnf[1145974]: Copr repo for fmf owned by psss                 2.5 kB/s | 3.3 kB     00:01
Apr 17 06:15:49 rdma06 dnf[1145974]: beaker-AppStream                                 52 kB/s | 168  B     00:00
Apr 17 06:15:49 rdma06 dnf[1145974]: Errors during downloading metadata for repository 'beaker-AppStream':
Apr 17 06:15:49 rdma06 dnf[1145974]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 06:15:49 rdma06 dnf[1145974]: Error: Failed to download metadata for repo 'beaker-AppStream': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 06:15:49 rdma06 dnf[1145974]: beaker-AppStream-debuginfo                       56 kB/s | 168  B     00:00
Apr 17 06:15:49 rdma06 dnf[1145974]: Errors during downloading metadata for repository 'beaker-AppStream-debuginfo':
Apr 17 06:15:49 rdma06 dnf[1145974]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 06:15:49 rdma06 dnf[1145974]: Error: Failed to download metadata for repo 'beaker-AppStream-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 06:15:49 rdma06 dnf[1145974]: beaker-BaseOS                                    56 kB/s | 168  B     00:00
Apr 17 06:15:49 rdma06 dnf[1145974]: Errors during downloading metadata for repository 'beaker-BaseOS':
Apr 17 06:15:49 rdma06 dnf[1145974]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 06:15:49 rdma06 dnf[1145974]: Error: Failed to download metadata for repo 'beaker-BaseOS': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 06:15:49 rdma06 dnf[1145974]: beaker-BaseOS-debuginfo                          48 kB/s | 168  B     00:00
Apr 17 06:15:49 rdma06 dnf[1145974]: Errors during downloading metadata for repository 'beaker-BaseOS-debuginfo':
Apr 17 06:15:49 rdma06 dnf[1145974]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 06:15:49 rdma06 dnf[1145974]: Error: Failed to download metadata for repo 'beaker-BaseOS-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 06:15:49 rdma06 dnf[1145974]: beaker-CRB                                       50 kB/s | 168  B     00:00
Apr 17 06:15:49 rdma06 dnf[1145974]: Errors during downloading metadata for repository 'beaker-CRB':
Apr 17 06:15:49 rdma06 dnf[1145974]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 06:15:49 rdma06 dnf[1145974]: Error: Failed to download metadata for repo 'beaker-CRB': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 06:15:49 rdma06 dnf[1145974]: beaker-CRB-debuginfo                             23 kB/s | 168  B     00:00
Apr 17 06:15:49 rdma06 dnf[1145974]: Errors during downloading metadata for repository 'beaker-CRB-debuginfo':
Apr 17 06:15:49 rdma06 dnf[1145974]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 06:15:49 rdma06 dnf[1145974]: Error: Failed to download metadata for repo 'beaker-CRB-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 06:15:49 rdma06 dnf[1145974]: beaker-HighAvailability                          58 kB/s | 168  B     00:00
Apr 17 06:15:49 rdma06 dnf[1145974]: Errors during downloading metadata for repository 'beaker-HighAvailability':
Apr 17 06:15:49 rdma06 dnf[1145974]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 06:15:49 rdma06 dnf[1145974]: Error: Failed to download metadata for repo 'beaker-HighAvailability': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 06:15:49 rdma06 dnf[1145974]: beaker-HighAvailability-debuginfo                57 kB/s | 168  B     00:00
Apr 17 06:15:49 rdma06 dnf[1145974]: Errors during downloading metadata for repository 'beaker-HighAvailability-debuginfo':
Apr 17 06:15:49 rdma06 dnf[1145974]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 06:15:49 rdma06 dnf[1145974]: Error: Failed to download metadata for repo 'beaker-HighAvailability-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 06:15:49 rdma06 dnf[1145974]: beaker-NFV                                       58 kB/s | 168  B     00:00
Apr 17 06:15:49 rdma06 dnf[1145974]: Errors during downloading metadata for repository 'beaker-NFV':
Apr 17 06:15:49 rdma06 dnf[1145974]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 06:15:49 rdma06 dnf[1145974]: Error: Failed to download metadata for repo 'beaker-NFV': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 06:15:49 rdma06 dnf[1145974]: beaker-NFV-debuginfo                             61 kB/s | 168  B     00:00
Apr 17 06:15:49 rdma06 dnf[1145974]: Errors during downloading metadata for repository 'beaker-NFV-debuginfo':
Apr 17 06:15:49 rdma06 dnf[1145974]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 06:15:49 rdma06 dnf[1145974]: Error: Failed to download metadata for repo 'beaker-NFV-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 06:15:49 rdma06 dnf[1145974]: beaker-RT                                        55 kB/s | 168  B     00:00
Apr 17 06:15:49 rdma06 dnf[1145974]: Errors during downloading metadata for repository 'beaker-RT':
Apr 17 06:15:49 rdma06 dnf[1145974]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 06:15:49 rdma06 dnf[1145974]: Error: Failed to download metadata for repo 'beaker-RT': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 06:15:49 rdma06 dnf[1145974]: beaker-RT-debuginfo                              57 kB/s | 168  B     00:00
Apr 17 06:15:49 rdma06 dnf[1145974]: Errors during downloading metadata for repository 'beaker-RT-debuginfo':
Apr 17 06:15:49 rdma06 dnf[1145974]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 06:15:49 rdma06 dnf[1145974]: Error: Failed to download metadata for repo 'beaker-RT-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 06:15:49 rdma06 dnf[1145974]: beaker-ResilientStorage                          53 kB/s | 168  B     00:00
Apr 17 06:15:49 rdma06 dnf[1145974]: Errors during downloading metadata for repository 'beaker-ResilientStorage':
Apr 17 06:15:49 rdma06 dnf[1145974]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 06:15:49 rdma06 dnf[1145974]: Error: Failed to download metadata for repo 'beaker-ResilientStorage': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 06:15:49 rdma06 dnf[1145974]: beaker-ResilientStorage-debuginfo                22 kB/s | 168  B     00:00
Apr 17 06:15:49 rdma06 dnf[1145974]: Errors during downloading metadata for repository 'beaker-ResilientStorage-debuginfo':
Apr 17 06:15:49 rdma06 dnf[1145974]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 06:15:49 rdma06 dnf[1145974]: Error: Failed to download metadata for repo 'beaker-ResilientStorage-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 06:15:49 rdma06 dnf[1145974]: beaker-SAP                                       34 kB/s | 168  B     00:00
Apr 17 06:15:49 rdma06 dnf[1145974]: Errors during downloading metadata for repository 'beaker-SAP':
Apr 17 06:15:49 rdma06 dnf[1145974]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 06:15:49 rdma06 dnf[1145974]: Error: Failed to download metadata for repo 'beaker-SAP': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 06:15:49 rdma06 dnf[1145974]: beaker-SAP-debuginfo                             58 kB/s | 168  B     00:00
Apr 17 06:15:49 rdma06 dnf[1145974]: Errors during downloading metadata for repository 'beaker-SAP-debuginfo':
Apr 17 06:15:49 rdma06 dnf[1145974]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 06:15:49 rdma06 dnf[1145974]: Error: Failed to download metadata for repo 'beaker-SAP-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 06:15:49 rdma06 dnf[1145974]: beaker-SAPHANA                                   57 kB/s | 168  B     00:00
Apr 17 06:15:49 rdma06 dnf[1145974]: Errors during downloading metadata for repository 'beaker-SAPHANA':
Apr 17 06:15:49 rdma06 dnf[1145974]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 06:15:49 rdma06 dnf[1145974]: Error: Failed to download metadata for repo 'beaker-SAPHANA': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 06:15:49 rdma06 dnf[1145974]: beaker-SAPHANA-debuginfo                         59 kB/s | 168  B     00:00
Apr 17 06:15:49 rdma06 dnf[1145974]: Errors during downloading metadata for repository 'beaker-SAPHANA-debuginfo':
Apr 17 06:15:49 rdma06 dnf[1145974]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 06:15:49 rdma06 dnf[1145974]: Error: Failed to download metadata for repo 'beaker-SAPHANA-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 06:15:50 rdma06 dnf[1145974]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr 17 06:15:50 rdma06 dnf[1145974]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr 17 06:15:50 rdma06 dnf[1145974]: Ignoring repositories: beaker-AppStream, beaker-AppStream-debuginfo, beaker-BaseOS, beaker-BaseOS-debuginfo, beaker-CRB, beaker-CRB-debuginfo, beaker-HighAvailability, beaker-HighAvailability-debuginfo, beaker-NFV, beaker-NFV-debuginfo, beaker-RT, beaker-RT-debuginfo, beaker-ResilientStorage, beaker-ResilientStorage-debuginfo, beaker-SAP, beaker-SAP-debuginfo, beaker-SAPHANA, beaker-SAPHANA-debuginfo
Apr 17 06:15:50 rdma06 dnf[1145974]: Metadata cache created.
Apr 17 06:15:50 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 17 06:15:50 rdma06 systemd[1]: Finished dnf makecache.
Apr 17 06:15:50 rdma06 systemd[1]: dnf-makecache.service: Consumed 1.002s CPU time.
Apr 17 06:15:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:15:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:15:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:15:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:15:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:15:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:15:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:15:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:15:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:15:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:15:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:15:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:15:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:15:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:15:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:15:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:15:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:15:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:15:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:15:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:15:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:15:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:15:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:15:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:15:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:15:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:15:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:15:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:15:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:15:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:15:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:15:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:15:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:15:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:15:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:15:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:16:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:16:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:16:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:16:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:16:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:16:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:16:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:16:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:16:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:16:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:16:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:16:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:16:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:16:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:16:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:16:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:16:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:16:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:16:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:16:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:16:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:16:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:16:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:16:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:16:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:16:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:16:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:16:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:16:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:16:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:16:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:16:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:16:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:16:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:16:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:16:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:16:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:16:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:16:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:16:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:16:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:16:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:16:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:16:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:16:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:16:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:16:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:16:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:16:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:16:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:16:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:16:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:16:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:16:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:16:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:16:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:16:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:16:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:16:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:16:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:16:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:16:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:16:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:16:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:16:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:16:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:16:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:16:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:16:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:16:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:16:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:16:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:16:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:16:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:16:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:16:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:16:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:16:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:16:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:16:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:16:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:16:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:16:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:16:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:16:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:16:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:16:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:16:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:16:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:16:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:16:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:16:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:16:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:16:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:16:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:16:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:16:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:16:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:16:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:16:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:16:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:16:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:16:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:16:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:16:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:16:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:16:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:16:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:16:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:16:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:16:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:16:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:16:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:16:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:16:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:16:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:16:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:16:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:16:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:16:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:16:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:16:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:16:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:16:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:16:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:16:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:16:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:16:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:16:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:16:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:16:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:16:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:16:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:16:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:16:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:16:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:16:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:16:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:16:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:16:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:16:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:16:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:16:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:16:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:16:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:16:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:16:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:16:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:16:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:16:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:16:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:16:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:16:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:16:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:16:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:16:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:16:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:16:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:16:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:16:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:16:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:16:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:16:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:16:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:16:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:16:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:16:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:16:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:16:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:16:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:16:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:16:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:16:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:16:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:16:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:16:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:16:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:16:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:16:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:16:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:16:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:16:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:16:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:16:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:16:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:16:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:16:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:16:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:16:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:16:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:16:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:16:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:16:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:16:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:16:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:16:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:16:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:16:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:16:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:16:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:16:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:16:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:16:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:16:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:16:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:16:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:16:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:16:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:16:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:16:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:16:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:16:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:16:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:16:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:16:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:16:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:17:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:17:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:17:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:17:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:17:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:17:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:17:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:17:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:17:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:17:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:17:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:17:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:17:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:17:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:17:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:17:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:17:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:17:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:17:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:17:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:17:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:17:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:17:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:17:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:17:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:17:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:17:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:17:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:17:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:17:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:17:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:17:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:17:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:17:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:17:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:17:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:17:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:17:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:17:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:17:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:17:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:17:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:17:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:17:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:17:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:17:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:17:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:17:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:17:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:17:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:17:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:17:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:17:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:17:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:17:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:17:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:17:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:17:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:17:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:17:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:17:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:17:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:17:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:17:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:17:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:17:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:17:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:17:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:17:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:17:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:17:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:17:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:17:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:17:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:17:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:17:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:17:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:17:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:17:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:17:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:17:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:17:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:17:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:17:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:17:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:17:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:17:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:17:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:17:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:17:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:17:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:17:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:17:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:17:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:17:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:17:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:17:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:17:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:17:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:17:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:17:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:17:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:17:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:17:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:17:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:17:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:17:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:17:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:17:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:17:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:17:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:17:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:17:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:17:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:17:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:17:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:17:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:17:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:17:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:17:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:17:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:17:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:17:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:17:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:17:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:17:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:17:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:17:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:17:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:17:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:17:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:17:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:17:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:17:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:17:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:17:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:17:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:17:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:17:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:17:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:17:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:17:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:17:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:17:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:17:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:17:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:17:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:17:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:17:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:17:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:17:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:17:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:17:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:17:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:17:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:17:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:17:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:17:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:17:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:17:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:17:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:17:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:17:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:17:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:17:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:17:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:17:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:17:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:17:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:17:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:17:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:17:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:17:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:17:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:17:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:17:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:17:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:17:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:17:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:17:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:17:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:17:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:17:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:17:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:17:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:17:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:17:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:17:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:17:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:17:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:17:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:17:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:17:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:17:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:17:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:17:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:17:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:17:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:17:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:17:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:17:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:17:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:17:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:17:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:17:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:17:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:17:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:17:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:17:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:17:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:17:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:17:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:17:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:17:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:17:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:17:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:18:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:18:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:18:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:18:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:18:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:18:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:18:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:18:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:18:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:18:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:18:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:18:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:18:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:18:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:18:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:18:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:18:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:18:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:18:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:18:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:18:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:18:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:18:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:18:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:18:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:18:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:18:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:18:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:18:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:18:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:18:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:18:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:18:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:18:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:18:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:18:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:18:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:18:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:18:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:18:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:18:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:18:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:18:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:18:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:18:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:18:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:18:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:18:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:18:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:18:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:18:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:18:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:18:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:18:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:18:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:18:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:18:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:18:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:18:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:18:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:18:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:18:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:18:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:18:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:18:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:18:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:18:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:18:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:18:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:18:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:18:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:18:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:18:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:18:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:18:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:18:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:18:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:18:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:18:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:18:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:18:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:18:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:18:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:18:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:18:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:18:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:18:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:18:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:18:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:18:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:18:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:18:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:18:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:18:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:18:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:18:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:18:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:18:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:18:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:18:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:18:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:18:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:18:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:18:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:18:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:18:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:18:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:18:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:18:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:18:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:18:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:18:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:18:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:18:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:18:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:18:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:18:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:18:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:18:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:18:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:18:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:18:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:18:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:18:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:18:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:18:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:18:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:18:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:18:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:18:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:18:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:18:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:18:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:18:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:18:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:18:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:18:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:18:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:18:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:18:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:18:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:18:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:18:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:18:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:18:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:18:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:18:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:18:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:18:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:18:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:18:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:18:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:18:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:18:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:18:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:18:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:18:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:18:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:18:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:18:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:18:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:18:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:18:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:18:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:18:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:18:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:18:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:18:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:18:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:18:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:18:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:18:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:18:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:18:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:18:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:18:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:18:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:18:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:18:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:18:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:18:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:18:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:18:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:18:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:18:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:18:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:18:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:18:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:18:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:18:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:18:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:18:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:18:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:18:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:18:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:18:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:18:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:18:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:18:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:18:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:18:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:18:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:18:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:18:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:18:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:18:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:18:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:18:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:18:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:18:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:18:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:18:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:18:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:18:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:18:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:18:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:19:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:19:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:19:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:19:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:19:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:19:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:19:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:19:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:19:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:19:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:19:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:19:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:19:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:19:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:19:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:19:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:19:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:19:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:19:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:19:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:19:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:19:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:19:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:19:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:19:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:19:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:19:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:19:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:19:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:19:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:19:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:19:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:19:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:19:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:19:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:19:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:19:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:19:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:19:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:19:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:19:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:19:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:19:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:19:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:19:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:19:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:19:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:19:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:19:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:19:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:19:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:19:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:19:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:19:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:19:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:19:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:19:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:19:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:19:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:19:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:19:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:19:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:19:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:19:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:19:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:19:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:19:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:19:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:19:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:19:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:19:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:19:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:19:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:19:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:19:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:19:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:19:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:19:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:19:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:19:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:19:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:19:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:19:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:19:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:19:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:19:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:19:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:19:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:19:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:19:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:19:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:19:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:19:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:19:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:19:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:19:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:19:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:19:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:19:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:19:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:19:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:19:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:19:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:19:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:19:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:19:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:19:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:19:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:19:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:19:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:19:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:19:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:19:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:19:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:19:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:19:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:19:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:19:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:19:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:19:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:19:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:19:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:19:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:19:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:19:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:19:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:19:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:19:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:19:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:19:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:19:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:19:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:19:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:19:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:19:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:19:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:19:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:19:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:19:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:19:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:19:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:19:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:19:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:19:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:19:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:19:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:19:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:19:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:19:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:19:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:19:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:19:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:19:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:19:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:19:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:19:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:19:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:19:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:19:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:19:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:19:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:19:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:19:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:19:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:19:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:19:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:19:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:19:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:19:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:19:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:19:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:19:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:19:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:19:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:19:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:19:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:19:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:19:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:19:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:19:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:19:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:19:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:19:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:19:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:19:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:19:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:19:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:19:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:19:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:19:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:19:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:19:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:19:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:19:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:19:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:19:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:19:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:19:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:19:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:19:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:19:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:19:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:19:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:19:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:19:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:19:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:19:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:19:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:19:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:19:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:19:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:19:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:19:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:19:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:19:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:19:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:20:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:20:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:20:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:20:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:20:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:20:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:20:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:20:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:20:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:20:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:20:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:20:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:20:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:20:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:20:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:20:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:20:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:20:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:20:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:20:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:20:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:20:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:20:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:20:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:20:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:20:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:20:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:20:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:20:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:20:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:20:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:20:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:20:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:20:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:20:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:20:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:20:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:20:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:20:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:20:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:20:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:20:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:20:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:20:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:20:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:20:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:20:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:20:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:20:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:20:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:20:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:20:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:20:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:20:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:20:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:20:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:20:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:20:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:20:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:20:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:20:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:20:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:20:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:20:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:20:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:20:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:20:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:20:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:20:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:20:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:20:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:20:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:20:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:20:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:20:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:20:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:20:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:20:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:20:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:20:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:20:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:20:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:20:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:20:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:20:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:20:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:20:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:20:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:20:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:20:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:20:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:20:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:20:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:20:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:20:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:20:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:20:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:20:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:20:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:20:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:20:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:20:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:20:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:20:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:20:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:20:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:20:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:20:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:20:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:20:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:20:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:20:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:20:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:20:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:20:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:20:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:20:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:20:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:20:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:20:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:20:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:20:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:20:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:20:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:20:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:20:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:20:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:20:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:20:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:20:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:20:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:20:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:20:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:20:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:20:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:20:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:20:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:20:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:20:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:20:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:20:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:20:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:20:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:20:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:20:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:20:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:20:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:20:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:20:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:20:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:20:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:20:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:20:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:20:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:20:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:20:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:20:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:20:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:20:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:20:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:20:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:20:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:20:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:20:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:20:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:20:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:20:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:20:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:20:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:20:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:20:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:20:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:20:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:20:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:20:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:20:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:20:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:20:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:20:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:20:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:20:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:20:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:20:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:20:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:20:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:20:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:20:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:20:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:20:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:20:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:20:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:20:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:20:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:20:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:20:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:20:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:20:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:20:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:20:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:20:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:20:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:20:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:20:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:20:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:20:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:20:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:20:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:20:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:20:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:20:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:20:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:20:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:20:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:20:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:20:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:20:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:21:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:21:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:21:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:21:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:21:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:21:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:21:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:21:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:21:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:21:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:21:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:21:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:21:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:21:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:21:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:21:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:21:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:21:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:21:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:21:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:21:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:21:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:21:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:21:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:21:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:21:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:21:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:21:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:21:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:21:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:21:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:21:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:21:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:21:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:21:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:21:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:21:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:21:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:21:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:21:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:21:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:21:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:21:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:21:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:21:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:21:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:21:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:21:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:21:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:21:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:21:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:21:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:21:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:21:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:21:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:21:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:21:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:21:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:21:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:21:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:21:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:21:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:21:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:21:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:21:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:21:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:21:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:21:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:21:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:21:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:21:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:21:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:21:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:21:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:21:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:21:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:21:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:21:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:21:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:21:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:21:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:21:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:21:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:21:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:21:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:21:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:21:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:21:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:21:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:21:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:21:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:21:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:21:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:21:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:21:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:21:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:21:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:21:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:21:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:21:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:21:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:21:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:21:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:21:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:21:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:21:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:21:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:21:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:21:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:21:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:21:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:21:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:21:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:21:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:21:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:21:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:21:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:21:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:21:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:21:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:21:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:21:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:21:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:21:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:21:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:21:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:21:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:21:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:21:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:21:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:21:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:21:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:21:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:21:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:21:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:21:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:21:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:21:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:21:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:21:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:21:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:21:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:21:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:21:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:21:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:21:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:21:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:21:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:21:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:21:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:21:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:21:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:21:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:21:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:21:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:21:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:21:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:21:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:21:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:21:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:21:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:21:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:21:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:21:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:21:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:21:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:21:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:21:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:21:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:21:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:21:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:21:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:21:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:21:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:21:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:21:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:21:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:21:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:21:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:21:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:21:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:21:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:21:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:21:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:21:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:21:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:21:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:21:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:21:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:21:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:21:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:21:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:21:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:21:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:21:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:21:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:21:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:21:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:21:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:21:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:21:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:21:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:21:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:21:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:21:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:21:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:22:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:22:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:22:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:22:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:22:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:22:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:22:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:22:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:22:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:22:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:22:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:22:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:22:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:22:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:22:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:22:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:24:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:24:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:24:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:24:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:24:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:24:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:24:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:24:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:24:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:24:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:24:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:24:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:24:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:24:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:24:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:24:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:24:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:24:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:24:13 rdma06 rsyslogd[1041]: imjournal: 5166 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 06:24:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:24:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:24:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:24:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:24:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:24:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:24:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:24:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:24:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:24:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:24:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:24:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:24:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:24:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:24:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:24:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:24:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:24:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:24:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:24:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:24:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:24:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:24:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:24:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:24:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:24:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:24:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:24:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:24:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:24:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:24:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:24:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:24:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:24:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:24:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:24:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:24:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:24:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:24:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:24:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:24:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:24:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:24:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:24:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:24:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:24:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:24:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:24:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:24:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:24:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:24:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:24:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:24:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:24:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:24:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:24:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:24:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:24:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:24:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:24:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:24:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:24:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:24:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:24:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:24:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:24:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:24:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:24:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:24:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:24:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:24:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:24:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:24:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:24:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:24:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:24:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:24:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:24:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:24:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:24:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:24:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:24:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:24:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:24:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:24:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:24:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:24:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:24:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:24:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:24:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:24:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:24:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:24:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:24:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:24:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:24:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:24:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:24:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:24:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:24:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:24:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:24:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:24:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:24:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:24:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:24:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:24:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:24:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:24:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:24:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:24:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:24:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:24:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:24:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:24:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:24:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:24:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:24:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:24:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:24:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:24:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:24:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:24:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:24:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:24:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:24:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:24:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:24:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:24:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:24:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:24:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:24:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:24:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:24:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:24:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:24:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:24:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:24:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:24:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:24:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:24:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:24:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:24:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:24:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:24:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:24:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:24:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:24:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:24:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:24:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:24:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:24:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:24:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:24:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:24:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:24:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:24:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:24:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:24:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:24:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:24:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:24:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:25:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:25:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:25:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:25:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:25:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:25:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:25:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:25:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:25:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:25:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:25:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:25:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:25:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:25:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:25:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:25:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:25:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:25:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:25:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:25:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:25:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:25:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:25:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:25:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:25:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:25:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:25:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:25:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:25:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:25:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:25:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:25:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:25:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:25:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:25:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:25:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:25:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:25:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:25:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:25:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:25:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:25:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:25:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:25:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:25:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:25:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:25:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:25:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:25:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:25:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:25:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:25:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:25:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:25:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:25:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:25:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:25:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:25:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:25:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:25:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:25:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:25:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:25:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:25:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:25:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:25:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:25:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:25:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:25:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:25:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:25:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:25:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:25:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:25:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:25:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:25:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:25:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:25:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:25:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:25:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:25:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:25:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:25:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:25:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:25:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:25:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:25:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:25:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:25:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:25:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:25:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:25:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:25:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:25:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:25:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:25:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:25:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:25:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:25:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:25:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:25:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:25:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:25:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:25:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:25:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:25:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:25:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:25:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:25:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:25:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:25:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:25:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:25:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:25:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:25:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:25:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:25:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:25:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:25:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:25:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:25:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:25:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:25:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:25:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:25:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:25:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:25:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:25:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:25:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:25:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:25:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:25:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:25:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:25:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:25:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:25:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:25:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:25:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:25:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:25:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:25:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:25:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:25:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:25:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:25:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:25:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:25:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:25:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:25:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:25:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:25:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:25:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:25:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:25:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:25:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:25:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:25:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:25:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:25:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:25:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:25:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:25:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:25:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:25:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:25:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:25:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:25:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:25:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:25:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:25:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:25:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:25:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:25:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:25:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:25:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:25:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:25:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:25:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:25:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:25:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:25:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:25:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:25:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:25:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:25:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:25:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:25:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:25:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:25:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:25:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:25:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:25:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:25:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:25:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:25:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:25:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:25:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:25:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:25:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:25:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:25:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:25:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:25:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:25:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:25:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:25:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:25:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:25:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:25:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:25:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:25:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:25:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:25:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:25:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:25:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:25:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:26:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:26:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:26:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:26:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:26:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:26:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:26:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:26:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:26:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:26:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:26:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:26:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:26:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:26:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:26:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:26:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:26:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:26:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:26:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:26:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:26:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:26:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:26:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:26:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:26:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:26:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:26:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:26:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:26:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:26:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:26:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:26:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:26:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:26:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:26:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:26:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:26:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:26:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:26:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:26:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:26:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:26:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:26:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:26:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:26:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:26:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:26:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:26:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:26:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:26:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:26:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:26:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:26:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:26:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:26:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:26:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:26:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:26:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:26:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:26:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:26:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:26:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:26:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:26:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:26:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:26:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:26:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:26:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:26:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:26:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:26:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:26:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:26:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:26:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:26:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:26:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:26:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:26:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:26:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:26:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:26:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:26:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:26:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:26:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:26:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:26:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:26:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:26:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:26:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:26:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:26:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:26:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:26:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:26:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:26:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:26:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:26:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:26:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:26:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:26:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:26:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:26:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:26:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:26:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:26:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:26:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:26:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:26:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:26:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:26:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:26:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:26:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:26:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:26:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:26:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:26:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:26:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:26:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:26:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:26:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:26:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:26:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:26:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:26:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:26:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:26:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:26:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:26:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:26:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:26:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:26:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:26:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:26:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:26:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:26:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:26:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:26:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:26:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:26:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:26:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:26:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:26:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:26:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:26:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:26:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:26:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:26:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:26:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:26:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:26:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:26:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:26:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:26:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:26:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:26:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:26:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:26:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:26:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:26:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:26:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:26:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:26:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:26:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:26:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:26:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:26:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:26:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:26:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:26:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:26:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:26:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:26:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:26:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:26:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:26:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:26:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:26:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:26:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:26:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:26:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:26:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:26:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:26:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:26:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:26:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:26:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:26:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:26:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:26:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:26:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:26:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:26:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:26:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:26:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:26:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:26:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:26:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:26:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:26:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:26:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:26:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:26:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:26:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:26:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:26:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:26:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:26:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:26:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:26:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:26:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:26:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:26:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:26:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:26:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:26:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:26:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:27:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:27:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:27:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:27:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:27:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:27:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:27:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:27:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:27:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:27:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:27:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:27:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:27:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:27:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:27:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:27:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:27:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:27:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:27:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:27:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:27:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:27:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:27:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:27:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:27:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:27:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:27:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:27:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:27:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:27:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:27:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:27:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:27:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:27:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:27:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:27:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:27:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:27:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:27:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:27:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:27:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:27:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:27:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:27:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:27:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:27:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:27:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:27:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:27:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:27:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:27:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:27:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:27:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:27:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:27:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:27:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:27:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:27:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:27:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:27:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:27:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:27:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:27:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:27:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:27:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:27:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:27:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:27:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:27:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:27:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:27:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:27:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:27:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:27:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:27:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:27:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:27:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:27:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:27:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:27:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:27:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:27:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:27:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:27:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:27:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:27:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:27:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:27:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:27:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:27:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:27:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:27:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:27:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:27:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:27:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:27:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:27:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:27:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:27:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:27:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:27:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:27:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:27:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:27:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:27:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:27:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:27:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:27:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:27:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:27:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:27:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:27:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:27:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:27:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:27:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:27:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:27:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:27:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:27:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:27:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:27:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:27:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:27:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:27:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:27:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:27:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:27:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:27:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:27:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:27:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:27:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:27:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:27:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:27:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:27:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:27:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:27:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:27:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:27:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:27:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:27:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:27:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:27:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:27:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:27:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:27:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:27:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:27:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:27:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:27:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:27:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:27:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:27:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:27:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:27:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:27:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:27:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:27:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:27:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:27:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:27:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:27:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:27:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:27:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:27:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:27:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:27:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:27:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:27:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:27:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:27:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:27:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:27:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:27:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:27:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:27:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:27:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:27:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:27:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:27:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:27:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:27:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:27:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:27:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:27:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:27:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:27:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:27:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:27:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:27:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:27:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:27:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:27:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:27:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:27:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:27:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:27:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:27:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:27:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:27:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:27:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:27:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:27:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:27:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:27:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:27:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:27:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:27:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:27:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:27:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:27:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:27:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:27:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:27:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:27:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:27:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:28:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:28:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:28:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:28:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:28:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:28:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:28:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:28:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:28:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:28:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:28:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:28:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:28:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:28:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:28:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:28:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:28:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:28:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:28:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:28:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:28:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:28:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:28:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:28:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:28:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:28:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:28:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:28:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:28:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:28:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:28:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:28:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:28:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:28:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:28:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:28:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:28:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:28:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:28:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:28:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:28:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:28:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:28:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:28:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:28:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:28:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:28:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:28:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:28:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:28:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:28:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:28:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:28:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:28:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:28:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:28:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:28:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:28:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:28:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:28:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:28:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:28:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:28:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:28:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:28:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:28:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:28:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:28:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:28:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:28:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:28:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:28:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:28:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:28:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:28:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:28:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:28:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:28:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:28:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:28:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:28:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:28:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:28:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:28:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:28:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:28:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:28:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:28:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:28:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:28:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:28:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:28:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:28:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:28:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:28:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:28:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:28:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:28:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:28:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:28:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:28:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:28:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:28:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:28:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:28:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:28:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:28:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:28:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:28:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:28:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:28:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:28:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:28:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:28:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:28:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:28:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:28:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:28:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:28:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:28:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:28:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:28:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:28:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:28:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:28:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:28:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:28:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:28:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:28:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:28:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:28:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:28:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:28:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:28:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:28:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:28:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:28:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:28:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:28:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:28:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:28:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:28:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:28:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:28:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:28:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:28:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:28:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:28:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:28:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:28:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:28:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:28:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:28:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:28:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:28:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:28:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:28:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:28:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:28:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:28:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:28:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:28:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:28:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:28:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:28:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:28:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:28:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:28:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:28:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:28:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:28:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:28:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:28:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:28:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:28:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:28:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:28:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:28:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:28:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:28:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:28:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:28:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:28:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:28:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:28:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:28:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:28:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:28:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:28:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:28:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:28:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:28:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:28:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:28:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:28:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:28:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:28:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:28:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:28:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:28:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:28:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:28:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:28:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:28:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:28:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:28:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:28:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:28:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:28:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:28:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:28:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:28:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:28:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:28:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:28:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:28:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:29:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:29:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:29:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:29:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:29:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:29:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:29:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:29:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:29:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:29:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:29:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:29:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:29:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:29:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:29:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:29:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:29:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:29:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:29:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:29:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:29:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:29:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:29:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:29:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:29:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:29:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:29:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:29:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:29:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:29:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:29:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:29:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:29:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:29:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:29:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:29:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:29:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:29:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:29:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:29:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:29:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:29:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:29:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:29:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:29:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:29:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:29:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:29:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:29:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:29:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:29:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:29:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:29:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:29:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:29:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:29:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:29:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:29:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:29:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:29:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:29:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:29:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:29:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:29:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:29:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:29:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:29:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:29:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:29:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:29:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:29:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:29:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:29:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:29:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:29:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:29:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:29:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:29:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:29:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:29:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:29:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:29:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:29:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:29:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:29:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:29:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:29:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:29:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:29:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:29:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:29:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:29:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:29:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:29:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:29:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:29:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:29:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:29:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:29:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:29:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:29:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:29:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:29:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:29:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:29:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:29:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:29:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:29:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:29:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:29:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:29:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:29:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:29:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:29:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:29:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:29:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:29:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:29:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:29:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:29:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:29:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:29:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:29:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:29:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:29:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:29:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:29:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:29:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:29:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:29:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:29:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:29:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:29:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:29:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:29:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:29:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:29:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:29:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:29:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:29:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:29:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:29:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:29:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:29:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:29:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:29:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:29:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:29:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:29:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:29:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:29:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:29:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:29:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:29:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:29:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:29:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:29:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:29:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:29:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:29:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:29:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:29:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:29:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:29:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:29:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:29:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:29:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:29:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:29:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:29:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:29:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:29:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:29:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:29:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:29:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:29:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:29:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:29:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:29:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:29:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:29:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:29:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:29:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:29:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:29:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:29:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:29:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:29:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:29:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:29:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:29:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:29:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:29:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:29:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:29:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:29:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:29:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:29:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:29:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:29:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:29:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:29:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:29:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:29:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:29:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:29:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:29:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:29:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:29:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:29:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:29:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:29:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:29:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:29:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:29:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:29:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:30:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:30:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:30:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:30:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:30:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:30:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:30:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:30:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:30:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:30:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:30:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:30:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:30:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:30:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:30:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:30:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:30:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:30:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:30:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:30:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:30:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:30:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:30:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:30:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:30:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:30:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:30:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:30:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:30:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:30:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:30:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:30:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:30:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:30:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:30:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:30:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:30:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:30:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:30:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:30:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:30:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:30:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:30:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:30:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:30:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:30:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:30:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:30:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:30:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:30:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:30:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:30:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:30:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:30:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:30:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:30:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:30:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:30:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:30:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:30:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:30:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:30:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:30:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:30:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:30:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:30:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:30:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:30:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:30:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:30:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:30:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:30:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:30:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:30:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:30:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:30:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:30:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:30:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:30:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:30:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:30:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:30:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:30:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:30:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:30:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:30:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:30:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:30:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:30:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:30:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:30:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:30:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:30:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:30:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:30:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:30:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:30:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:30:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:30:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:30:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:30:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:30:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:30:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:30:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:30:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:30:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:30:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:30:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:30:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:30:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:30:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:30:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:30:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:30:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:30:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:30:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:30:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:30:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:30:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:30:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:30:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:30:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:30:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:30:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:30:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:30:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:30:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:30:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:30:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:30:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:30:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:30:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:30:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:30:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:30:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:30:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:30:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:30:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:30:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:30:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:30:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:30:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:30:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:30:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:30:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:30:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:30:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:30:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:30:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:30:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:30:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:30:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:30:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:30:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:30:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:30:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:30:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:30:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:30:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:30:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:30:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:30:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:30:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:30:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:30:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:30:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:30:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:30:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:30:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:30:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:30:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:30:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:30:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:30:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:30:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:30:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:30:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:30:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:30:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:30:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:30:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:30:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:30:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:30:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:30:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:30:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:30:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:30:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:30:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:30:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:30:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:30:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:30:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:30:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:30:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:30:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:30:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:30:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:30:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:30:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:30:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:30:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:30:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:30:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:30:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:30:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:30:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:30:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:30:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:30:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:30:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:30:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:30:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:30:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:30:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:30:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:31:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:31:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:31:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:31:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:31:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:31:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:31:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:31:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:31:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:31:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:31:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:31:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:31:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:31:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:31:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:31:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:31:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:31:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:31:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:31:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:31:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:31:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:31:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:31:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:31:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:31:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:31:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:31:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:31:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:31:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:31:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:31:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:31:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:31:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:31:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:31:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:31:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:31:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:31:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:31:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:31:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:31:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:31:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:31:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:31:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:31:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:31:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:31:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:31:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:31:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:31:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:31:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:31:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:31:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:31:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:31:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:31:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:31:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:31:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:31:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:31:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:31:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:31:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:31:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:31:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:31:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:31:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:31:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:31:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:31:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:31:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:31:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:31:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:31:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:31:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:31:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:31:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:31:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:31:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:31:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:31:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:31:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:31:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:31:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:31:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:31:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:31:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:31:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:31:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:31:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:31:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:31:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:31:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:31:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:31:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:31:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:31:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:31:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:31:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:31:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:31:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:31:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:31:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:31:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:31:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:31:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:31:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:31:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:31:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:31:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:31:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:31:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:31:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:31:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:31:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:31:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:31:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:31:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:31:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:31:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:31:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:31:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:31:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:31:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:31:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:31:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:31:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:31:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:31:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:31:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:31:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:31:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:31:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:31:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:31:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:31:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:31:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:31:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:31:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:31:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:31:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:31:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:31:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:31:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:31:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:31:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:31:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:31:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:31:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:31:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:31:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:31:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:31:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:31:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:31:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:31:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:31:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:31:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:31:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:31:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:31:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:31:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:31:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:31:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:31:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:31:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:31:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:31:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:31:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:31:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:31:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:31:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:31:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:31:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:31:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:31:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:31:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:31:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:31:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:31:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:31:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:31:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:31:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:31:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:31:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:31:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:31:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:31:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:31:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:31:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:31:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:31:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:31:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:31:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:31:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:31:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:31:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:31:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:31:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:31:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:31:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:31:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:31:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:31:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:31:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:31:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:31:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:31:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:31:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:31:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:31:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:31:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:31:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:31:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:31:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:31:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:32:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:32:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:32:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:32:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:32:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:32:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:32:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:32:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:32:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:32:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:32:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:32:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:32:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:32:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:32:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:32:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:32:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:32:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:32:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:32:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:32:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:32:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:32:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:32:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:32:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:32:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:32:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:32:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:32:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:32:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:32:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:32:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:32:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:32:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:32:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:32:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:34:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:34:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:34:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:34:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:34:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:34:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:34:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:34:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:34:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:34:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:34:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:34:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:34:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:34:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:34:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:34:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:34:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:34:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:34:18 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 06:34:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:34:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:34:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:34:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:34:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:34:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:34:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:34:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:34:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:34:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:34:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:34:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:34:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:34:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:34:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:34:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:34:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:34:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:34:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:34:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:34:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:34:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:34:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:34:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:34:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:34:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:34:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:34:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:34:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:34:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:34:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:34:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:34:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:34:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:34:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:34:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:34:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:34:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:34:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:34:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:34:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:34:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:34:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:34:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:34:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:34:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:34:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:34:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:34:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:34:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:34:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:34:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:34:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:34:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:34:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:34:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:34:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:34:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:34:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:34:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:34:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:34:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:34:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:34:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:34:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:34:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:34:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:34:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:34:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:34:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:34:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:34:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:34:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:34:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:34:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:34:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:34:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:34:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:34:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:34:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:34:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:34:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:34:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:34:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:34:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:34:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:34:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:34:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:34:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:34:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:34:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:34:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:34:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:34:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:34:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:34:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:34:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:34:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:34:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:34:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:34:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:34:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:34:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:34:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:34:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:34:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:34:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:34:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:34:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:34:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:34:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:34:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:34:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:34:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:34:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:34:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:34:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:34:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:34:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:34:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:34:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:34:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:34:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:34:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:34:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:34:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:34:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:34:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:34:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:34:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:34:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:34:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:34:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:34:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:34:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:34:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:34:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:34:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:34:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:34:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:34:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:34:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:34:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:34:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:35:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:35:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:35:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:35:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:35:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:35:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:35:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:35:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:35:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:35:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:35:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:35:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:35:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:35:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:35:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:35:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:35:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:35:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:35:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:35:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:35:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:35:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:35:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:35:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:35:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:35:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:35:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:35:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:35:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:35:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:35:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:35:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:35:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:35:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:35:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:35:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:35:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:35:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:35:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:35:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:35:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:35:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:35:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:35:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:35:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:35:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:35:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:35:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:35:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:35:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:35:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:35:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:35:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:35:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:35:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:35:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:35:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:35:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:35:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:35:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:35:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:35:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:35:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:35:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:35:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:35:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:35:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:35:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:35:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:35:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:35:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:35:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:35:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:35:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:35:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:35:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:35:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:35:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:35:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:35:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:35:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:35:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:35:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:35:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:35:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:35:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:35:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:35:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:35:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:35:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:35:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:35:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:35:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:35:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:35:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:35:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:35:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:35:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:35:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:35:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:35:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:35:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:35:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:35:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:35:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:35:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:35:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:35:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:35:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:35:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:35:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:35:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:35:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:35:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:35:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:35:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:35:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:35:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:35:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:35:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:35:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:35:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:35:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:35:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:35:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:35:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:35:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:35:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:35:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:35:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:35:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:35:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:35:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:35:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:35:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:35:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:35:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:35:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:35:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:35:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:35:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:35:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:35:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:35:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:35:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:35:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:35:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:35:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:35:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:35:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:35:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:35:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:35:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:35:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:35:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:35:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:35:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:35:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:35:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:35:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:35:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:35:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:35:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:35:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:35:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:35:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:35:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:35:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:35:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:35:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:35:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:35:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:35:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:35:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:35:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:35:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:35:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:35:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:35:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:35:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:35:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:35:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:35:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:35:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:35:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:35:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:35:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:35:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:35:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:35:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:35:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:35:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:35:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:35:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:35:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:35:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:35:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:35:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:35:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:35:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:35:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:35:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:35:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:35:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:35:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:35:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:35:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:35:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:35:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:35:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:35:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:35:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:35:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:35:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:35:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:35:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:36:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:36:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:36:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:36:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:36:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:36:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:36:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:36:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:36:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:36:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:36:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:36:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:36:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:36:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:36:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:36:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:36:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:36:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:36:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:36:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:36:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:36:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:36:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:36:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:36:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:36:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:36:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:36:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:36:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:36:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:36:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:36:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:36:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:36:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:36:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:36:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:36:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:36:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:36:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:36:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:36:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:36:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:36:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:36:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:36:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:36:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:36:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:36:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:36:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:36:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:36:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:36:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:36:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:36:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:36:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:36:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:36:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:36:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:36:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:36:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:36:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:36:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:36:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:36:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:36:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:36:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:36:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:36:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:36:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:36:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:36:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:36:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:36:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:36:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:36:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:36:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:36:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:36:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:36:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:36:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:36:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:36:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:36:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:36:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:36:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:36:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:36:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:36:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:36:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:36:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:36:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:36:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:36:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:36:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:36:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:36:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:36:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:36:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:36:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:36:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:36:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:36:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:36:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:36:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:36:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:36:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:36:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:36:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:36:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:36:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:36:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:36:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:36:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:36:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:36:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:36:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:36:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:36:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:36:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:36:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:36:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:36:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:36:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:36:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:36:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:36:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:36:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:36:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:36:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:36:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:36:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:36:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:36:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:36:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:36:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:36:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:36:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:36:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:36:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:36:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:36:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:36:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:36:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:36:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:36:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:36:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:36:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:36:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:36:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:36:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:36:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:36:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:36:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:36:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:36:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:36:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:36:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:36:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:36:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:36:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:36:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:36:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:36:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:36:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:36:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:36:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:36:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:36:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:36:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:36:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:36:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:36:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:36:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:36:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:36:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:36:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:36:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:36:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:36:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:36:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:36:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:36:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:36:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:36:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:36:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:36:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:36:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:36:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:36:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:36:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:36:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:36:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:36:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:36:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:36:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:36:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:36:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:36:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:36:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:36:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:36:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:36:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:36:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:36:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:36:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:36:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:36:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:36:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:36:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:36:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:36:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:36:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:36:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:36:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:36:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:36:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:37:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:37:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:37:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:37:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:37:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:37:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:37:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:37:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:37:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:37:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:37:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:37:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:37:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:37:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:37:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:37:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:37:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:37:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:37:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:37:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:37:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:37:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:37:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:37:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:37:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:37:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:37:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:37:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:37:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:37:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:37:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:37:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:37:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:37:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:37:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:37:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:37:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:37:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:37:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:37:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:37:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:37:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:37:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:37:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:37:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:37:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:37:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:37:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:37:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:37:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:37:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:37:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:37:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:37:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:37:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:37:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:37:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:37:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:37:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:37:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:37:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:37:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:37:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:37:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:37:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:37:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:37:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:37:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:37:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:37:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:37:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:37:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:37:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:37:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:37:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:37:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:37:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:37:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:37:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:37:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:37:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:37:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:37:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:37:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:37:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:37:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:37:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:37:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:37:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:37:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:37:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:37:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:37:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:37:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:37:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:37:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:37:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:37:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:37:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:37:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:37:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:37:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:37:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:37:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:37:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:37:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:37:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:37:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:37:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:37:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:37:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:37:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:37:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:37:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:37:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:37:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:37:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:37:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:37:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:37:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:37:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:37:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:37:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:37:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:37:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:37:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:37:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:37:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:37:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:37:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:37:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:37:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:37:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:37:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:37:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:37:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:37:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:37:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:37:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:37:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:37:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:37:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:37:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:37:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:37:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:37:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:37:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:37:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:37:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:37:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:37:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:37:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:37:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:37:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:37:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:37:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:37:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:37:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:37:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:37:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:37:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:37:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:37:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:37:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:37:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:37:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:37:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:37:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:37:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:37:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:37:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:37:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:37:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:37:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:37:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:37:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:37:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:37:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:37:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:37:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:37:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:37:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:37:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:37:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:37:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:37:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:37:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:37:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:37:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:37:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:37:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:37:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:37:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:37:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:37:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:37:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:37:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:37:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:37:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:37:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:37:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:37:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:37:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:37:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:37:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:37:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:37:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:37:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:37:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:37:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:37:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:37:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:37:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:37:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:37:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:37:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:38:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:38:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:38:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:38:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:38:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:38:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:38:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:38:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:38:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:38:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:38:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:38:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:38:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:38:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:38:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:38:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:38:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:38:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:38:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:38:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:38:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:38:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:38:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:38:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:38:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:38:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:38:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:38:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:38:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:38:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:38:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:38:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:38:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:38:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:38:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:38:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:38:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:38:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:38:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:38:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:38:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:38:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:38:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:38:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:38:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:38:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:38:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:38:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:38:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:38:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:38:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:38:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:38:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:38:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:38:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:38:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:38:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:38:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:38:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:38:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:38:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:38:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:38:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:38:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:38:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:38:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:38:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:38:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:38:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:38:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:38:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:38:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:38:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:38:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:38:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:38:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:38:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:38:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:38:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:38:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:38:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:38:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:38:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:38:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:38:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:38:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:38:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:38:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:38:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:38:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:38:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:38:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:38:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:38:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:38:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:38:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:38:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:38:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:38:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:38:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:38:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:38:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:38:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:38:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:38:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:38:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:38:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:38:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:38:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:38:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:38:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:38:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:38:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:38:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:38:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:38:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:38:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:38:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:38:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:38:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:38:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:38:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:38:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:38:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:38:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:38:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:38:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:38:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:38:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:38:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:38:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:38:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:38:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:38:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:38:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:38:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:38:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:38:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:38:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:38:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:38:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:38:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:38:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:38:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:38:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:38:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:38:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:38:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:38:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:38:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:38:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:38:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:38:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:38:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:38:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:38:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:38:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:38:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:38:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:38:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:38:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:38:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:38:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:38:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:38:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:38:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:38:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:38:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:38:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:38:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:38:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:38:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:38:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:38:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:38:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:38:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:38:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:38:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:38:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:38:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:38:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:38:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:38:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:38:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:38:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:38:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:38:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:38:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:38:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:38:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:38:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:38:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:38:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:38:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:38:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:38:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:38:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:38:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:38:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:38:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:38:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:38:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:38:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:38:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:38:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:38:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:38:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:38:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:38:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:38:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:38:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:38:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:38:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:38:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:38:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:38:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:39:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:39:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:39:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:39:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:39:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:39:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:39:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:39:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:39:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:39:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:39:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:39:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:39:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:39:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:39:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:39:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:39:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:39:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:39:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:39:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:39:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:39:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:39:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:39:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:39:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:39:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:39:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:39:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:39:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:39:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:39:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:39:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:39:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:39:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:39:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:39:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:39:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:39:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:39:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:39:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:39:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:39:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:39:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:39:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:39:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:39:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:39:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:39:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:39:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:39:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:39:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:39:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:39:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:39:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:39:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:39:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:39:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:39:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:39:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:39:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:39:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:39:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:39:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:39:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:39:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:39:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:39:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:39:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:39:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:39:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:39:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:39:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:39:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:39:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:39:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:39:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:39:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:39:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:39:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:39:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:39:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:39:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:39:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:39:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:39:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:39:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:39:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:39:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:39:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:39:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:39:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:39:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:39:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:39:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:39:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:39:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:39:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:39:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:39:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:39:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:39:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:39:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:39:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:39:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:39:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:39:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:39:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:39:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:39:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:39:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:39:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:39:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:39:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:39:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:39:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:39:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:39:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:39:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:39:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:39:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:39:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:39:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:39:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:39:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:39:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:39:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:39:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:39:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:39:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:39:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:39:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:39:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:39:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:39:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:39:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:39:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:39:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:39:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:39:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:39:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:39:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:39:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:39:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:39:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:39:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:39:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:39:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:39:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:39:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:39:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:39:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:39:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:39:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:39:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:39:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:39:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:39:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:39:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:39:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:39:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:39:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:39:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:39:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:39:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:39:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:39:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:39:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:39:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:39:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:39:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:39:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:39:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:39:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:39:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:39:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:39:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:39:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:39:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:39:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:39:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:39:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:39:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:39:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:39:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:39:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:39:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:39:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:39:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:39:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:39:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:39:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:39:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:39:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:39:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:39:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:39:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:39:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:39:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:39:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:39:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:39:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:39:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:39:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:39:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:39:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:39:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:39:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:39:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:39:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:39:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:39:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:39:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:39:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:39:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:39:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:39:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:40:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:40:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:40:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:40:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:40:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:40:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:40:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:40:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:40:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:40:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:40:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:40:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:40:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:40:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:40:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:40:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:40:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:40:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:40:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:40:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:40:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:40:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:40:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:40:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:40:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:40:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:40:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:40:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:40:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:40:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:40:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:40:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:40:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:40:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:40:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:40:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:40:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:40:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:40:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:40:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:40:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:40:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:40:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:40:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:40:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:40:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:40:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:40:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:40:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:40:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:40:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:40:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:40:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:40:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:40:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:40:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:40:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:40:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:40:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:40:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:40:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:40:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:40:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:40:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:40:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:40:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:40:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:40:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:40:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:40:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:40:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:40:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:40:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:40:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:40:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:40:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:40:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:40:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:40:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:40:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:40:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:40:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:40:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:40:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:40:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:40:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:40:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:40:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:40:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:40:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:40:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:40:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:40:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:40:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:40:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:40:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:40:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:40:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:40:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:40:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:40:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:40:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:40:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:40:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:40:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:40:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:40:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:40:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:40:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:40:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:40:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:40:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:40:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:40:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:40:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:40:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:40:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:40:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:40:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:40:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:40:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:40:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:40:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:40:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:40:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:40:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:40:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:40:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:40:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:40:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:40:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:40:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:40:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:40:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:40:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:40:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:40:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:40:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:40:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:40:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:40:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:40:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:40:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:40:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:40:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:40:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:40:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:40:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:40:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:40:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:40:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:40:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:40:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:40:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:40:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:40:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:40:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:40:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:40:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:40:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:40:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:40:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:40:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:40:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:40:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:40:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:40:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:40:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:40:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:40:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:40:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:40:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:40:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:40:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:40:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:40:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:40:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:40:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:40:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:40:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:40:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:40:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:40:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:40:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:40:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:40:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:40:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:40:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:40:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:40:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:40:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:40:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:40:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:40:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:40:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:40:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:40:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:40:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:40:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:40:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:40:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:40:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:40:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:40:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:40:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:40:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:40:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:40:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:40:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:40:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:40:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:40:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:40:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:40:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:40:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:40:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:41:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:41:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:41:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:41:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:41:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:41:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:41:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:41:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:41:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:41:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:41:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:41:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:41:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:41:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:41:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:41:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:41:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:41:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:41:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:41:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:41:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:41:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:41:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:41:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:41:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:41:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:41:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:41:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:41:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:41:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:41:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:41:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:41:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:41:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:41:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:41:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:41:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:41:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:41:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:41:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:41:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:41:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:41:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:41:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:41:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:41:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:41:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:41:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:41:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:41:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:41:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:41:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:41:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:41:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:41:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:41:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:41:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:41:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:41:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:41:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:41:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:41:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:41:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:41:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:41:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:41:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:41:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:41:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:41:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:41:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:41:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:41:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:41:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:41:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:41:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:41:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:41:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:41:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:41:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:41:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:41:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:41:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:41:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:41:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:41:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:41:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:41:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:41:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:41:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:41:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:41:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:41:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:41:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:41:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:41:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:41:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:41:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:41:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:41:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:41:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:41:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:41:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:41:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:41:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:41:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:41:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:41:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:41:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:41:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:41:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:41:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:41:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:41:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:41:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:41:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:41:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:41:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:41:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:41:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:41:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:41:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:41:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:41:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:41:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:41:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:41:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:41:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:41:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:41:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:41:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:41:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:41:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:41:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:41:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:41:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:41:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:41:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:41:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:41:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:41:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:41:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:41:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:41:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:41:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:41:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:41:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:41:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:41:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:41:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:41:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:41:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:41:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:41:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:41:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:41:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:41:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:41:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:41:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:41:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:41:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:41:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:41:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:41:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:41:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:41:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:41:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:41:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:41:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:41:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:41:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:41:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:41:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:41:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:41:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:41:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:41:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:41:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:41:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:41:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:41:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:41:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:41:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:41:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:41:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:41:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:41:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:41:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:41:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:41:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:41:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:41:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:41:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:41:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:41:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:41:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:41:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:41:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:41:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:41:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:41:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:41:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:41:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:41:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:41:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:41:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:41:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:41:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:41:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:41:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:41:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:41:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:41:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:41:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:41:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:41:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:41:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:42:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:42:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:42:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:42:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:42:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:42:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:42:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:42:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:42:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:42:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:42:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:42:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:42:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:42:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:42:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:42:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:42:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:42:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:42:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:42:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:42:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:42:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:42:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:42:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:42:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:42:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:42:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:42:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:42:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:42:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:42:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:42:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:42:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:42:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:42:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:42:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:42:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:42:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:42:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:42:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:42:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:42:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:42:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:42:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:42:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:42:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:42:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:42:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:42:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:42:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:42:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:42:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:42:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:42:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:44:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:44:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:44:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:44:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:44:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:44:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:44:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:44:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:44:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:44:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:44:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:44:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:44:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:44:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:44:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:44:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:44:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:44:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:44:23 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 06:44:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:44:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:44:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:44:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:44:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:44:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:44:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:44:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:44:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:44:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:44:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:44:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:44:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:44:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:44:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:44:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:44:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:44:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:44:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:44:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:44:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:44:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:44:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:44:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:44:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:44:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:44:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:44:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:44:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:44:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:44:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:44:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:44:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:44:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:44:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:44:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:44:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:44:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:44:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:44:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:44:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:44:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:44:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:44:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:44:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:44:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:44:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:44:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:44:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:44:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:44:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:44:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:44:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:44:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:44:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:44:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:44:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:44:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:44:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:44:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:44:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:44:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:44:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:44:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:44:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:44:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:44:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:44:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:44:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:44:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:44:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:44:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:44:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:44:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:44:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:44:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:44:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:44:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:44:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:44:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:44:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:44:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:44:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:44:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:44:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:44:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:44:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:44:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:44:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:44:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:44:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:44:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:44:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:44:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:44:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:44:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:44:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:44:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:44:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:44:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:44:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:44:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:44:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:44:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:44:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:44:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:44:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:44:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:44:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:44:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:44:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:44:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:44:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:44:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:44:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:44:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:44:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:44:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:44:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:44:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:44:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:44:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:44:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:44:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:44:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:44:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:45:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:45:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:45:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:45:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:45:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:45:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:45:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:45:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:45:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:45:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:45:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:45:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:45:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:45:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:45:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:45:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:45:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:45:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:45:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:45:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:45:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:45:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:45:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:45:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:45:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:45:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:45:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:45:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:45:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:45:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:45:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:45:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:45:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:45:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:45:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:45:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:45:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:45:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:45:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:45:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:45:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:45:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:45:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:45:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:45:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:45:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:45:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:45:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:45:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:45:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:45:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:45:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:45:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:45:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:45:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:45:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:45:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:45:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:45:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:45:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:45:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:45:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:45:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:45:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:45:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:45:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:45:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:45:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:45:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:45:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:45:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:45:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:45:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:45:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:45:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:45:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:45:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:45:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:45:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:45:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:45:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:45:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:45:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:45:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:45:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:45:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:45:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:45:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:45:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:45:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:45:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:45:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:45:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:45:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:45:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:45:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:45:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:45:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:45:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:45:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:45:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:45:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:45:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:45:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:45:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:45:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:45:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:45:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:45:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:45:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:45:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:45:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:45:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:45:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:45:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:45:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:45:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:45:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:45:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:45:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:45:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:45:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:45:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:45:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:45:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:45:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:45:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:45:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:45:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:45:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:45:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:45:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:45:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:45:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:45:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:45:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:45:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:45:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:45:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:45:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:45:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:45:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:45:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:45:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:45:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:45:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:45:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:45:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:45:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:45:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:45:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:45:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:45:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:45:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:45:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:45:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:45:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:45:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:45:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:45:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:45:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:45:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:45:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:45:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:45:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:45:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:45:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:45:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:45:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:45:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:45:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:45:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:45:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:45:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:45:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:45:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:45:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:45:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:45:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:45:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:45:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:45:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:45:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:45:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:45:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:45:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:45:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:45:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:45:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:45:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:45:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:45:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:45:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:45:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:45:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:45:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:45:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:45:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:45:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:45:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:45:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:45:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:45:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:45:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:45:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:45:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:45:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:45:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:45:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:45:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:45:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:45:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:45:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:45:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:45:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:45:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:46:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:46:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:46:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:46:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:46:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:46:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:46:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:46:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:46:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:46:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:46:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:46:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:46:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:46:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:46:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:46:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:46:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:46:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:46:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:46:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:46:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:46:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:46:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:46:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:46:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:46:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:46:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:46:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:46:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:46:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:46:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:46:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:46:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:46:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:46:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:46:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:46:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:46:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:46:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:46:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:46:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:46:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:46:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:46:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:46:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:46:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:46:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:46:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:46:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:46:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:46:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:46:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:46:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:46:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:46:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:46:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:46:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:46:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:46:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:46:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:46:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:46:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:46:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:46:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:46:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:46:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:46:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:46:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:46:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:46:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:46:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:46:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:46:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:46:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:46:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:46:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:46:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:46:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:46:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:46:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:46:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:46:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:46:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:46:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:46:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:46:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:46:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:46:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:46:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:46:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:46:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:46:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:46:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:46:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:46:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:46:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:46:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:46:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:46:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:46:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:46:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:46:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:46:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:46:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:46:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:46:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:46:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:46:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:46:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:46:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:46:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:46:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:46:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:46:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:46:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:46:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:46:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:46:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:46:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:46:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:46:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:46:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:46:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:46:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:46:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:46:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:46:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:46:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:46:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:46:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:46:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:46:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:46:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:46:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:46:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:46:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:46:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:46:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:46:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:46:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:46:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:46:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:46:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:46:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:46:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:46:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:46:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:46:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:46:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:46:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:46:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:46:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:46:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:46:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:46:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:46:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:46:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:46:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:46:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:46:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:46:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:46:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:46:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:46:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:46:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:46:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:46:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:46:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:46:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:46:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:46:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:46:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:46:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:46:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:46:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:46:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:46:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:46:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:46:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:46:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:46:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:46:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:46:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:46:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:46:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:46:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:46:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:46:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:46:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:46:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:46:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:46:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:46:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:46:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:46:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:46:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:46:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:46:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:46:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:46:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:46:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:46:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:46:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:46:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:46:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:46:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:46:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:46:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:46:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:46:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:46:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:46:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:46:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:46:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:46:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:46:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:47:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:47:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:47:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:47:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:47:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:47:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:47:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:47:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:47:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:47:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:47:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:47:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:47:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:47:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:47:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:47:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:47:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:47:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:47:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:47:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:47:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:47:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:47:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:47:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:47:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:47:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:47:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:47:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:47:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:47:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:47:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:47:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:47:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:47:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:47:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:47:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:47:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:47:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:47:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:47:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:47:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:47:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:47:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:47:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:47:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:47:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:47:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:47:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:47:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:47:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:47:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:47:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:47:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:47:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:47:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:47:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:47:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:47:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:47:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:47:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:47:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:47:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:47:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:47:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:47:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:47:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:47:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:47:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:47:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:47:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:47:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:47:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:47:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:47:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:47:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:47:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:47:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:47:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:47:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:47:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:47:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:47:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:47:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:47:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:47:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:47:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:47:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:47:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:47:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:47:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:47:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:47:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:47:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:47:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:47:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:47:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:47:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:47:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:47:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:47:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:47:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:47:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:47:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:47:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:47:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:47:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:47:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:47:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:47:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:47:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:47:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:47:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:47:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:47:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:47:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:47:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:47:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:47:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:47:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:47:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:47:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:47:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:47:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:47:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:47:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:47:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:47:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:47:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:47:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:47:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:47:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:47:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:47:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:47:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:47:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:47:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:47:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:47:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:47:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:47:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:47:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:47:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:47:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:47:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:47:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:47:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:47:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:47:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:47:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:47:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:47:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:47:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:47:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:47:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:47:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:47:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:47:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:47:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:47:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:47:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:47:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:47:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:47:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:47:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:47:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:47:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:47:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:47:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:47:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:47:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:47:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:47:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:47:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:47:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:47:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:47:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:47:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:47:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:47:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:47:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:47:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:47:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:47:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:47:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:47:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:47:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:47:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:47:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:47:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:47:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:47:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:47:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:47:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:47:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:47:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:47:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:47:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:47:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:47:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:47:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:47:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:47:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:47:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:47:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:47:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:47:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:47:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:47:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:47:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:47:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:47:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:47:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:47:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:47:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:47:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:47:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:48:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:48:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:48:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:48:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:48:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:48:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:48:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:48:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:48:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:48:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:48:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:48:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:48:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:48:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:48:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:48:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:48:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:48:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:48:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:48:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:48:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:48:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:48:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:48:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:48:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:48:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:48:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:48:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:48:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:48:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:48:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:48:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:48:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:48:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:48:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:48:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:48:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:48:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:48:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:48:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:48:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:48:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:48:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:48:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:48:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:48:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:48:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:48:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:48:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:48:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:48:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:48:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:48:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:48:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:48:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:48:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:48:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:48:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:48:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:48:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:48:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:48:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:48:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:48:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:48:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:48:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:48:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:48:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:48:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:48:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:48:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:48:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:48:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:48:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:48:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:48:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:48:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:48:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:48:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:48:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:48:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:48:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:48:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:48:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:48:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:48:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:48:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:48:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:48:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:48:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:48:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:48:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:48:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:48:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:48:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:48:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:48:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:48:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:48:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:48:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:48:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:48:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:48:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:48:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:48:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:48:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:48:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:48:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:48:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:48:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:48:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:48:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:48:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:48:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:48:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:48:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:48:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:48:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:48:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:48:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:48:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:48:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:48:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:48:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:48:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:48:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:48:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:48:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:48:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:48:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:48:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:48:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:48:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:48:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:48:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:48:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:48:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:48:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:48:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:48:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:48:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:48:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:48:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:48:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:48:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:48:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:48:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:48:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:48:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:48:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:48:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:48:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:48:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:48:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:48:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:48:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:48:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:48:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:48:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:48:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:48:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:48:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:48:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:48:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:48:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:48:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:48:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:48:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:48:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:48:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:48:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:48:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:48:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:48:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:48:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:48:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:48:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:48:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:48:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:48:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:48:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:48:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:48:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:48:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:48:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:48:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:48:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:48:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:48:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:48:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:48:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:48:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:48:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:48:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:48:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:48:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:48:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:48:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:48:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:48:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:48:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:48:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:48:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:48:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:48:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:48:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:48:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:48:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:48:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:48:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:48:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:48:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:48:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:48:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:48:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:48:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:49:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:49:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:49:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:49:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:49:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:49:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:49:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:49:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:49:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:49:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:49:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:49:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:49:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:49:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:49:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:49:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:49:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:49:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:49:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:49:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:49:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:49:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:49:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:49:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:49:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:49:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:49:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:49:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:49:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:49:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:49:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:49:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:49:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:49:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:49:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:49:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:49:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:49:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:49:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:49:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:49:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:49:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:49:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:49:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:49:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:49:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:49:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:49:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:49:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:49:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:49:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:49:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:49:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:49:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:49:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:49:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:49:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:49:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:49:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:49:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:49:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:49:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:49:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:49:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:49:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:49:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:49:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:49:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:49:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:49:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:49:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:49:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:49:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:49:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:49:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:49:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:49:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:49:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:49:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:49:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:49:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:49:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:49:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:49:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:49:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:49:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:49:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:49:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:49:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:49:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:49:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:49:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:49:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:49:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:49:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:49:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:49:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:49:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:49:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:49:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:49:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:49:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:49:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:49:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:49:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:49:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:49:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:49:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:49:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:49:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:49:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:49:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:49:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:49:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:49:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:49:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:49:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:49:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:49:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:49:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:49:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:49:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:49:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:49:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:49:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:49:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:49:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:49:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:49:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:49:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:49:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:49:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:49:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:49:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:49:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:49:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:49:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:49:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:49:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:49:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:49:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:49:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:49:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:49:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:49:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:49:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:49:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:49:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:49:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:49:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:49:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:49:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:49:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:49:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:49:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:49:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:49:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:49:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:49:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:49:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:49:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:49:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:49:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:49:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:49:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:49:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:49:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:49:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:49:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:49:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:49:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:49:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:49:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:49:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:49:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:49:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:49:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:49:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:49:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:49:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:49:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:49:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:49:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:49:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:49:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:49:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:49:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:49:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:49:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:49:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:49:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:49:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:49:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:49:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:49:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:49:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:49:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:49:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:49:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:49:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:49:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:49:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:49:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:49:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:49:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:49:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:49:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:49:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:49:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:49:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:49:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:49:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:49:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:49:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:49:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:49:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:50:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:50:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:50:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:50:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:50:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:50:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:50:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:50:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:50:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:50:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:50:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:50:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:50:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:50:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:50:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:50:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:50:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:50:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:50:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:50:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:50:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:50:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:50:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:50:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:50:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:50:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:50:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:50:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:50:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:50:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:50:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:50:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:50:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:50:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:50:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:50:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:50:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:50:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:50:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:50:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:50:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:50:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:50:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:50:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:50:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:50:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:50:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:50:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:50:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:50:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:50:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:50:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:50:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:50:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:50:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:50:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:50:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:50:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:50:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:50:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:50:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:50:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:50:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:50:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:50:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:50:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:50:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:50:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:50:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:50:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:50:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:50:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:50:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:50:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:50:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:50:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:50:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:50:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:50:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:50:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:50:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:50:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:50:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:50:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:50:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:50:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:50:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:50:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:50:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:50:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:50:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:50:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:50:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:50:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:50:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:50:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:50:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:50:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:50:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:50:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:50:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:50:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:50:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:50:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:50:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:50:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:50:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:50:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:50:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:50:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:50:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:50:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:50:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:50:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:50:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:50:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:50:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:50:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:50:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:50:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:50:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:50:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:50:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:50:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:50:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:50:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:50:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:50:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:50:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:50:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:50:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:50:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:50:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:50:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:50:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:50:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:50:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:50:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:50:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:50:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:50:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:50:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:50:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:50:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:50:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:50:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:50:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:50:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:50:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:50:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:50:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:50:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:50:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:50:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:50:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:50:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:50:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:50:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:50:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:50:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:50:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:50:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:50:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:50:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:50:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:50:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:50:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:50:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:50:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:50:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:50:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:50:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:50:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:50:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:50:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:50:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:50:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:50:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:50:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:50:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:50:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:50:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:50:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:50:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:50:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:50:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:50:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:50:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:50:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:50:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:50:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:50:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:50:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:50:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:50:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:50:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:50:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:50:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:50:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:50:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:50:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:50:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:50:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:50:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:50:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:50:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:50:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:50:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:50:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:50:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:50:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:50:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:50:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:50:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:50:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:50:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:51:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:51:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:51:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:51:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:51:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:51:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:51:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:51:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:51:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:51:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:51:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:51:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:51:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:51:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:51:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:51:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:51:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:51:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:51:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:51:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:51:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:51:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:51:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:51:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:51:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:51:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:51:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:51:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:51:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:51:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:51:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:51:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:51:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:51:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:51:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:51:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:51:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:51:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:51:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:51:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:51:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:51:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:51:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:51:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:51:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:51:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:51:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:51:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:51:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:51:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:51:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:51:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:51:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:51:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:51:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:51:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:51:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:51:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:51:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:51:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:51:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:51:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:51:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:51:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:51:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:51:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:51:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:51:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:51:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:51:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:51:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:51:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:51:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:51:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:51:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:51:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:51:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:51:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:51:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:51:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:51:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:51:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:51:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:51:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:51:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:51:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:51:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:51:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:51:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:51:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:51:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:51:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:51:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:51:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:51:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:51:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:51:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:51:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:51:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:51:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:51:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:51:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:51:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:51:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:51:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:51:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:51:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:51:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:51:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:51:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:51:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:51:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:51:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:51:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:51:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:51:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:51:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:51:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:51:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:51:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:51:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:51:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:51:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:51:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:51:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:51:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:51:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:51:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:51:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:51:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:51:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:51:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:51:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:51:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:51:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:51:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:51:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:51:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:51:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:51:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:51:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:51:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:51:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:51:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:51:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:51:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:52:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:52:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:52:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:52:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:52:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:52:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:52:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:52:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:52:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:52:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:52:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:52:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:52:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:52:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:52:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:52:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:52:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:52:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:52:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:52:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:52:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:52:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:52:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:52:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:52:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:52:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:52:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:52:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:52:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:52:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:52:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:52:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:54:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:54:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:54:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:54:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:54:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:54:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:54:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:54:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:54:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:54:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:54:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:54:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:54:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:54:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:54:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:54:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:54:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:54:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:54:24 rdma06 rsyslogd[1041]: imjournal: 5082 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 06:54:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:54:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:54:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:54:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:54:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:54:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:54:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:54:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:54:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:54:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:54:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:54:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:54:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:54:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:54:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:54:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:54:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:54:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:54:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:54:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:54:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:54:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:54:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:54:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:54:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:54:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:54:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:54:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:54:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:54:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:54:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:54:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:54:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:54:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:54:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:54:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:54:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:54:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:54:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:54:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:54:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:54:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:54:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:54:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:54:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:54:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:54:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:54:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:54:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:54:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:54:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:54:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:54:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:54:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:54:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:54:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:54:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:54:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:54:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:54:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:54:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:54:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:54:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:54:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:54:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:54:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:54:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:54:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:54:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:54:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:54:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:54:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:54:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:54:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:54:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:54:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:54:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:54:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:54:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:54:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:54:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:54:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:54:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:54:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:54:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:54:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:54:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:54:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:54:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:54:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:54:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:54:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:54:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:54:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:54:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:54:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:54:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:54:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:54:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:54:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:54:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:54:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:54:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:54:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:54:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:54:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:54:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:54:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:54:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:54:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:54:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:54:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:54:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:54:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:54:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:54:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:54:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:54:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:54:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:54:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:54:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:54:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:54:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:54:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:54:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:54:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:55:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:55:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:55:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:55:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:55:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:55:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:55:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:55:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:55:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:55:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:55:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:55:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:55:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:55:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:55:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:55:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:55:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:55:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:55:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:55:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:55:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:55:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:55:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:55:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:55:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:55:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:55:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:55:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:55:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:55:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:55:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:55:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:55:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:55:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:55:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:55:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:55:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:55:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:55:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:55:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:55:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:55:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:55:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:55:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:55:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:55:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:55:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:55:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:55:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:55:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:55:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:55:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:55:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:55:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:55:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:55:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:55:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:55:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:55:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:55:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:55:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:55:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:55:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:55:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:55:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:55:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:55:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:55:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:55:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:55:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:55:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:55:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:55:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:55:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:55:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:55:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:55:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:55:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:55:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:55:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:55:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:55:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:55:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:55:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:55:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:55:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:55:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:55:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:55:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:55:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:55:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:55:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:55:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:55:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:55:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:55:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:55:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:55:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:55:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:55:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:55:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:55:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:55:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:55:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:55:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:55:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:55:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:55:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:55:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:55:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:55:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:55:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:55:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:55:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:55:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:55:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:55:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:55:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:55:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:55:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:55:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:55:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:55:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:55:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:55:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:55:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:55:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:55:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:55:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:55:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:55:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:55:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:55:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:55:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:55:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:55:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:55:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:55:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:55:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:55:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:55:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:55:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:55:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:55:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:55:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:55:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:55:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:55:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:55:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:55:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:55:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:55:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:55:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:55:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:55:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:55:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:55:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:55:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:55:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:55:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:55:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:55:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:55:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:55:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:55:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:55:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:55:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:55:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:55:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:55:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:55:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:55:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:55:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:55:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:55:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:55:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:55:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:55:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:55:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:55:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:55:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:55:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:55:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:55:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:55:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:55:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:55:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:55:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:55:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:55:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:55:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:55:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:55:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:55:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:55:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:55:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:55:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:55:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:55:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:55:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:55:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:55:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:55:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:55:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:55:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:55:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:55:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:55:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:55:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:55:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:55:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:55:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:55:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:55:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:55:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:55:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:56:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:56:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:56:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:56:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:56:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:56:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:56:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:56:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:56:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:56:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:56:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:56:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:56:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:56:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:56:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:56:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:56:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:56:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:56:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:56:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:56:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:56:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:56:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:56:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:56:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:56:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:56:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:56:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:56:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:56:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:56:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:56:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:56:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:56:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:56:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:56:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:56:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:56:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:56:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:56:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:56:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:56:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:56:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:56:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:56:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:56:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:56:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:56:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:56:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:56:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:56:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:56:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:56:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:56:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:56:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:56:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:56:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:56:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:56:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:56:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:56:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:56:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:56:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:56:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:56:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:56:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:56:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:56:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:56:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:56:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:56:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:56:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:56:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:56:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:56:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:56:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:56:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:56:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:56:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:56:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:56:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:56:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:56:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:56:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:56:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:56:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:56:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:56:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:56:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:56:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:56:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:56:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:56:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:56:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:56:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:56:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:56:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:56:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:56:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:56:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:56:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:56:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:56:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:56:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:56:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:56:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:56:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:56:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:56:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:56:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:56:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:56:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:56:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:56:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:56:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:56:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:56:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:56:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:56:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:56:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:56:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:56:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:56:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:56:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:56:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:56:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:56:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:56:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:56:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:56:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:56:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:56:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:56:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:56:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:56:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:56:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:56:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:56:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:56:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:56:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:56:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:56:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:56:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:56:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:56:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:56:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:56:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:56:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:56:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:56:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:56:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:56:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:56:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:56:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:56:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:56:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:56:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:56:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:56:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:56:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:56:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:56:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:56:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:56:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:56:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:56:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:56:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:56:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:56:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:56:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:56:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:56:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:56:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:56:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:56:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:56:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:56:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:56:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:56:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:56:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:56:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:56:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:56:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:56:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:56:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:56:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:56:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:56:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:56:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:56:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:56:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:56:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:56:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:56:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:56:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:56:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:56:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:56:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:56:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:56:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:56:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:56:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:56:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:56:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:56:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:56:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:56:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:56:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:56:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:56:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:56:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:56:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:56:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:56:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:56:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:56:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:57:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:57:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:57:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:57:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:57:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:57:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:57:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:57:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:57:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:57:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:57:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:57:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:57:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:57:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:57:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:57:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:57:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:57:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:57:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:57:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:57:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:57:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:57:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:57:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:57:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:57:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:57:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:57:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:57:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:57:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:57:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:57:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:57:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:57:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:57:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:57:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:57:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:57:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:57:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:57:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:57:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:57:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:57:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:57:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:57:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:57:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:57:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:57:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:57:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:57:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:57:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:57:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:57:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:57:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:57:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:57:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:57:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:57:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:57:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:57:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:57:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:57:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:57:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:57:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:57:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:57:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:57:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:57:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:57:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:57:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:57:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:57:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:57:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:57:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:57:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:57:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:57:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:57:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:57:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:57:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:57:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:57:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:57:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:57:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:57:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:57:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:57:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:57:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:57:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:57:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:57:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:57:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:57:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:57:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:57:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:57:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:57:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:57:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:57:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:57:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:57:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:57:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:57:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:57:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:57:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:57:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:57:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:57:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:57:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:57:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:57:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:57:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:57:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:57:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:57:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:57:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:57:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:57:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:57:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:57:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:57:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:57:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:57:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:57:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:57:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:57:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:57:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:57:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:57:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:57:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:57:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:57:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:57:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:57:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:57:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:57:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:57:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:57:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:57:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:57:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:57:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:57:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:57:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:57:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:57:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:57:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:57:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:57:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:57:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:57:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:57:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:57:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:57:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:57:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:57:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:57:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:57:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:57:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:57:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:57:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:57:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:57:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:57:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:57:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:57:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:57:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:57:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:57:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:57:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:57:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:57:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:57:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:57:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:57:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:57:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:57:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:57:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:57:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:57:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:57:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:57:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:57:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:57:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:57:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:57:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:57:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:57:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:57:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:57:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:57:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:57:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:57:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:57:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:57:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:57:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:57:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:57:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:57:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:57:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:57:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:57:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:57:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:57:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:57:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:57:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:57:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:57:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:57:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:57:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:57:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:57:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:57:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:57:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:57:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:57:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:57:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:58:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:58:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:58:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:58:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:58:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:58:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:58:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:58:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:58:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:58:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:58:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:58:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:58:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:58:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:58:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:58:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:58:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:58:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:58:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:58:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:58:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:58:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:58:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:58:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:58:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:58:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:58:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:58:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:58:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:58:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:58:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:58:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:58:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:58:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:58:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:58:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:58:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:58:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:58:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:58:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:58:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:58:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:58:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:58:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:58:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:58:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:58:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:58:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:58:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:58:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:58:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:58:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:58:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:58:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:58:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:58:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:58:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:58:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:58:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:58:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:58:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:58:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:58:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:58:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:58:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:58:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:58:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:58:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:58:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:58:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:58:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:58:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:58:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:58:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:58:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:58:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:58:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:58:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:58:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:58:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:58:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:58:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:58:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:58:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:58:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:58:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:58:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:58:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:58:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:58:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:58:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:58:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:58:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:58:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:58:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:58:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:58:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:58:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:58:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:58:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:58:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:58:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:58:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:58:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:58:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:58:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:58:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:58:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:58:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:58:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:58:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:58:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:58:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:58:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:58:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:58:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:58:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:58:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:58:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:58:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:58:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:58:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:58:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:58:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:58:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:58:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:58:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:58:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:58:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:58:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:58:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:58:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:58:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:58:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:58:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:58:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:58:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:58:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:58:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:58:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:58:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:58:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:58:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:58:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:58:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:58:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:58:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:58:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:58:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:58:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:58:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:58:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:58:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:58:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:58:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:58:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:58:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:58:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:58:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:58:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:58:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:58:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:58:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:58:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:58:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:58:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:58:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:58:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:58:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:58:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:58:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:58:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:58:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:58:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:58:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:58:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:58:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:58:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:58:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:58:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:58:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:58:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:58:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:58:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:58:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:58:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:58:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:58:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:58:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:58:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:58:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:58:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:58:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:58:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:58:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:58:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:58:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:58:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:58:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:58:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:58:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:58:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:58:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:58:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:58:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:58:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:58:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:58:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:58:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:58:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:58:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:58:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:58:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:58:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:58:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:58:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:59:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:59:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:59:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:59:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:59:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:59:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:59:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:59:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:59:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:59:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:59:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:59:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:59:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:59:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:59:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:59:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:59:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:59:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:59:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:59:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:59:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:59:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:59:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:59:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:59:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:59:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:59:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:59:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:59:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:59:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:59:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:59:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:59:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:59:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:59:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:59:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:59:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:59:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:59:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:59:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:59:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:59:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:59:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:59:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:59:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:59:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:59:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:59:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:59:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:59:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:59:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:59:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:59:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:59:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:59:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:59:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:59:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:59:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:59:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:59:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:59:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:59:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:59:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:59:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:59:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:59:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:59:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:59:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:59:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:59:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:59:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:59:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:59:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:59:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:59:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:59:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:59:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:59:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:59:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:59:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:59:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:59:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:59:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:59:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:59:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:59:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:59:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:59:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:59:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:59:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:59:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:59:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:59:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:59:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:59:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:59:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:59:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:59:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:59:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:59:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:59:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:59:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:59:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:59:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:59:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:59:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:59:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:59:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:59:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:59:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:59:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:59:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:59:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:59:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:59:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:59:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:59:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:59:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:59:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:59:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:59:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:59:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:59:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:59:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:59:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:59:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:59:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:59:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:59:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:59:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:59:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:59:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:59:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:59:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:59:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:59:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:59:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:59:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:59:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:59:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:59:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:59:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:59:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:59:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:59:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:59:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:59:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:59:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:59:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:59:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:59:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:59:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:59:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:59:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:59:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:59:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:59:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:59:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:59:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:59:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:59:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:59:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:59:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:59:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:59:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:59:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:59:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:59:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:59:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:59:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:59:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:59:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:59:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:59:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:59:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:59:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:59:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:59:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:59:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:59:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:59:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:59:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:59:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:59:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:59:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:59:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:59:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:59:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:59:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:59:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:59:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:59:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:59:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:59:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:59:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:59:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:59:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:59:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 06:59:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 06:59:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:59:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:59:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:59:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 06:59:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 06:59:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 06:59:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 06:59:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 06:59:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 06:59:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 06:59:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 06:59:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 06:59:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 06:59:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 06:59:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 06:59:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 06:59:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:00:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:00:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:00:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:00:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:00:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:00:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:00:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:00:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:00:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:00:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:00:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:00:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:00:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:00:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:00:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:00:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:00:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:00:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:00:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:00:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:00:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:00:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:00:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:00:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:00:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:00:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:00:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:00:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:00:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:00:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:00:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:00:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:00:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:00:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:00:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:00:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:00:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:00:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:00:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:00:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:00:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:00:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:00:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:00:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:00:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:00:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:00:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:00:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:00:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:00:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:00:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:00:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:00:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:00:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:00:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:00:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:00:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:00:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:00:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:00:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:00:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:00:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:00:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:00:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:00:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:00:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:00:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:00:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:00:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:00:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:00:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:00:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:00:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:00:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:00:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:00:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:00:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:00:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:00:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:00:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:00:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:00:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:00:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:00:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:00:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:00:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:00:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:00:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:00:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:00:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:00:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:00:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:00:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:00:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:00:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:00:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:00:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:00:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:00:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:00:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:00:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:00:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:00:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:00:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:00:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:00:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:00:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:00:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:00:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:00:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:00:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:00:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:00:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:00:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:00:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:00:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:00:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:00:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:00:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:00:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:00:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:00:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:00:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:00:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:00:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:00:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:00:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:00:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:00:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:00:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:00:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:00:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:00:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:00:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:00:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:00:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:00:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:00:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:00:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:00:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:00:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:00:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:00:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:00:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:00:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:00:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:00:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:00:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:00:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:00:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:00:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:00:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:00:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:00:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:00:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:00:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:00:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:00:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:00:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:00:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:00:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:00:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:00:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:00:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:00:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:00:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:00:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:00:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:00:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:00:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:00:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:00:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:00:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:00:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:00:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:00:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:00:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:00:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:00:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:00:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:00:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:00:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:00:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:00:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:00:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:00:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:00:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:00:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:00:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:00:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:00:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:00:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:00:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:00:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:00:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:00:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:00:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:00:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:00:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:00:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:00:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:00:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:00:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:00:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:00:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:00:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:00:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:00:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:00:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:00:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:00:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:00:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:00:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:00:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:00:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:00:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:01:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:01:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:01:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:01:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:01:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:01:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:01:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:01:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:01:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:01:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:01:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:01:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:01:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:01:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:01:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:01:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:01:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:01:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:01:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:01:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:01:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:01:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:01:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:01:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:01:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:01:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:01:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:01:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:01:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:01:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:01:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:01:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:01:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:01:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:01:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:01:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:01:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:01:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:01:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:01:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:01:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:01:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:01:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:01:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:01:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:01:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:01:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:01:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:01:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:01:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:01:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:01:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:01:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:01:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:01:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:01:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:01:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:01:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:01:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:01:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:01:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:01:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:01:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:01:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:01:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:01:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:01:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:01:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:01:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:01:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:01:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:01:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:01:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:01:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:01:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:01:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:01:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:01:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:01:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:01:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:01:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:01:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:01:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:01:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:01:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:01:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:01:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:01:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:01:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:01:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:01:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:01:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:01:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:01:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:01:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:01:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:01:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:01:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:01:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:01:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:01:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:01:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:01:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:01:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:01:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:01:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:01:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:01:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:01:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:01:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:01:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:01:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:01:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:01:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:01:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:01:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:01:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:01:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:01:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:01:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:01:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:01:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:01:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:01:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:01:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:01:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:01:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:01:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:01:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:01:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:01:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:01:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:01:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:01:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:01:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:01:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:01:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:01:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:01:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:01:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:01:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:01:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:01:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:01:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:01:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:01:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:01:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:01:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:01:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:01:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:01:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:01:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:01:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:01:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:01:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:01:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:01:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:01:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:01:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:01:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:01:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:01:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:01:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:01:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:01:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:01:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:01:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:01:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:01:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:01:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:01:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:01:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:01:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:01:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:01:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:01:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:01:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:01:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:01:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:01:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:01:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:01:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:01:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:01:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:01:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:01:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:01:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:01:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:01:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:01:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:01:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:01:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:01:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:01:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:01:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:01:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:01:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:01:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:01:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:01:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:01:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:01:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:01:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:01:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:01:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:01:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:01:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:01:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:01:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:01:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:01:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:01:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:01:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:01:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:01:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:01:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:02:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:02:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:02:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:02:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:02:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:02:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:02:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:02:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:02:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:02:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:02:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:02:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:02:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:02:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:02:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:02:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:02:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:02:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:02:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:02:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:02:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:02:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:02:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:02:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:02:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:02:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:02:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:02:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:02:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:02:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:02:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:02:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:02:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:02:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:02:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:02:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:02:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:02:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:02:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:02:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:02:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:02:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:02:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:02:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:02:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:02:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:02:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:02:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:02:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:02:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:02:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:02:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:02:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:02:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:02:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:02:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:02:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:02:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:02:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:02:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:02:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:02:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:02:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:02:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:02:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:02:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:02:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:02:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:02:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:02:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:02:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:02:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:04:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:04:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:04:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:04:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:04:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:04:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:04:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:04:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:04:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:04:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:04:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:04:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:04:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:04:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:04:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:04:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:04:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:04:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:04:29 rdma06 rsyslogd[1041]: imjournal: 5295 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 07:04:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:04:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:04:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:04:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:04:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:04:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:04:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:04:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:04:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:04:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:04:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:04:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:04:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:04:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:04:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:04:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:04:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:04:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:04:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:04:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:04:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:04:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:04:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:04:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:04:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:04:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:04:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:04:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:04:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:04:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:04:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:04:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:04:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:04:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:04:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:04:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:04:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:04:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:04:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:04:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:04:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:04:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:04:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:04:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:04:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:04:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:04:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:04:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:04:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:04:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:04:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:04:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:04:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:04:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:04:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:04:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:04:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:04:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:04:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:04:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:04:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:04:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:04:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:04:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:04:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:04:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:04:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:04:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:04:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:04:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:04:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:04:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:04:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:04:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:04:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:04:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:04:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:04:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:04:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:04:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:04:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:04:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:04:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:04:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:04:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:04:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:04:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:04:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:04:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:04:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:04:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:04:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:04:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:04:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:04:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:04:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:04:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:04:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:04:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:04:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:04:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:04:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:04:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:04:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:04:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:04:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:04:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:04:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:05:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:05:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:05:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:05:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:05:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:05:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:05:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:05:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:05:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:05:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:05:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:05:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:05:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:05:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:05:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:05:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:05:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:05:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:05:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:05:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:05:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:05:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:05:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:05:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:05:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:05:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:05:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:05:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:05:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:05:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:05:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:05:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:05:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:05:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:05:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:05:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:05:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:05:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:05:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:05:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:05:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:05:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:05:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:05:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:05:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:05:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:05:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:05:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:05:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:05:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:05:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:05:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:05:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:05:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:05:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:05:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:05:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:05:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:05:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:05:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:05:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:05:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:05:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:05:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:05:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:05:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:05:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:05:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:05:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:05:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:05:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:05:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:05:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:05:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:05:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:05:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:05:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:05:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:05:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:05:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:05:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:05:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:05:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:05:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:05:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:05:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:05:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:05:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:05:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:05:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:05:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:05:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:05:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:05:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:05:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:05:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:05:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:05:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:05:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:05:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:05:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:05:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:05:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:05:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:05:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:05:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:05:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:05:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:05:29 rdma06 systemd-journald[790]: Data hash table of /run/log/journal/95379d2f723847b68d9e976dee8967a9/system.journal has a fill level at 75.0 (163750 of 218332 items, 75497472 file size, 461 bytes per hash table item), suggesting rotation.
Apr 17 07:05:29 rdma06 systemd-journald[790]: /run/log/journal/95379d2f723847b68d9e976dee8967a9/system.journal: Journal header limits reached or header out-of-date, rotating.
Apr 17 07:05:29 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 17 07:05:29 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 17 07:05:29 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 17 07:05:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:05:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:05:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:05:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:05:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:05:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:05:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:05:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:05:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:05:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:05:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:05:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:05:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:05:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:05:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:05:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:05:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:05:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:05:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:05:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:05:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:05:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:05:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:05:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:05:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:05:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:05:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:05:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:05:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:05:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:05:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:05:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:05:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:05:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:05:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:05:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:05:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:05:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:05:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:05:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:05:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:05:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:05:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:05:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:05:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:05:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:05:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:05:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:05:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:05:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:05:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:05:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:05:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:05:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:05:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:05:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:05:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:05:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:05:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:05:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:05:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:05:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:05:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:05:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:05:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:05:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:05:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:05:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:05:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:05:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:05:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:05:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:05:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:05:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:05:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:05:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:05:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:05:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:05:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:05:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:05:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:05:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:05:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:05:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:05:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:05:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:05:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:05:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:05:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:05:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:05:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:05:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:05:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:05:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:05:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:05:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:05:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:05:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:05:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:05:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:05:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:05:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:05:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:05:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:05:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:05:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:05:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:05:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:06:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:06:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:06:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:06:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:06:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:06:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:06:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:06:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:06:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:06:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:06:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:06:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:06:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:06:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:06:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:06:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:06:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:06:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:06:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:06:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:06:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:06:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:06:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:06:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:06:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:06:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:06:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:06:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:06:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:06:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:06:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:06:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:06:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:06:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:06:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:06:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:06:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:06:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:06:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:06:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:06:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:06:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:06:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:06:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:06:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:06:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:06:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:06:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:06:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:06:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:06:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:06:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:06:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:06:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:06:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:06:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:06:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:06:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:06:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:06:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:06:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:06:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:06:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:06:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:06:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:06:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:06:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:06:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:06:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:06:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:06:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:06:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:06:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:06:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:06:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:06:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:06:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:06:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:06:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:06:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:06:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:06:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:06:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:06:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:06:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:06:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:06:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:06:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:06:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:06:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:06:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:06:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:06:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:06:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:06:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:06:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:06:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:06:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:06:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:06:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:06:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:06:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:06:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:06:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:06:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:06:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:06:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:06:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:06:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:06:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:06:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:06:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:06:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:06:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:06:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:06:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:06:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:06:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:06:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:06:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:06:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:06:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:06:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:06:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:06:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:06:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:06:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:06:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:06:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:06:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:06:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:06:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:06:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:06:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:06:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:06:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:06:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:06:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:06:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:06:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:06:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:06:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:06:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:06:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:06:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:06:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:06:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:06:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:06:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:06:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:06:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:06:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:06:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:06:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:06:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:06:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:06:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:06:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:06:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:06:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:06:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:06:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:06:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:06:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:06:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:06:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:06:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:06:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:06:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:06:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:06:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:06:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:06:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:06:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:06:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:06:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:06:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:06:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:06:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:06:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:06:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:06:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:06:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:06:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:06:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:06:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:06:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:06:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:06:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:06:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:06:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:06:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:06:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:06:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:06:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:06:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:06:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:06:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:06:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:06:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:06:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:06:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:06:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:06:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:06:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:06:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:06:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:06:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:06:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:06:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:06:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:06:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:06:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:06:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:06:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:06:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:07:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:07:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:07:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:07:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:07:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:07:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:07:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:07:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:07:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:07:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:07:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:07:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:07:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:07:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:07:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:07:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:07:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:07:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:07:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:07:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:07:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:07:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:07:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:07:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:07:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:07:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:07:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:07:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:07:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:07:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:07:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:07:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:07:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:07:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:07:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:07:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:07:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:07:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:07:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:07:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:07:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:07:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:07:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:07:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:07:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:07:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:07:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:07:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:07:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:07:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:07:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:07:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:07:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:07:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:07:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:07:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:07:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:07:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:07:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:07:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:07:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:07:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:07:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:07:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:07:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:07:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:07:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:07:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:07:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:07:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:07:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:07:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:07:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:07:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:07:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:07:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:07:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:07:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:07:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:07:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:07:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:07:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:07:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:07:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:07:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:07:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:07:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:07:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:07:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:07:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:07:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:07:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:07:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:07:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:07:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:07:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:07:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:07:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:07:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:07:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:07:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:07:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:07:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:07:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:07:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:07:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:07:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:07:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:07:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:07:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:07:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:07:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:07:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:07:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:07:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:07:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:07:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:07:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:07:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:07:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:07:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:07:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:07:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:07:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:07:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:07:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:07:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:07:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:07:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:07:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:07:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:07:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:07:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:07:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:07:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:07:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:07:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:07:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:07:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:07:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:07:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:07:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:07:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:07:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:07:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:07:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:07:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:07:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:07:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:07:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:07:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:07:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:07:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:07:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:07:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:07:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:07:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:07:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:07:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:07:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:07:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:07:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:07:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:07:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:07:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:07:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:07:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:07:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:07:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:07:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:07:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:07:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:07:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:07:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:07:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:07:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:07:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:07:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:07:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:07:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:07:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:07:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:07:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:07:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:07:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:07:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:07:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:07:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:07:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:07:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:07:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:07:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:07:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:07:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:07:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:07:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:07:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:07:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:07:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:07:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:07:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:07:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:07:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:07:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:07:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:07:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:07:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:07:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:07:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:07:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:07:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:07:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:07:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:07:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:07:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:07:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:08:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:08:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:08:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:08:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:08:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:08:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:08:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:08:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:08:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:08:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:08:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:08:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:08:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:08:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:08:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:08:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:08:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:08:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:08:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:08:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:08:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:08:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:08:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:08:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:08:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:08:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:08:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:08:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:08:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:08:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:08:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:08:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:08:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:08:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:08:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:08:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:08:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:08:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:08:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:08:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:08:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:08:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:08:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:08:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:08:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:08:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:08:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:08:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:08:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:08:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:08:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:08:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:08:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:08:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:08:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:08:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:08:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:08:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:08:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:08:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:08:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:08:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:08:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:08:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:08:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:08:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:08:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:08:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:08:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:08:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:08:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:08:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:08:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:08:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:08:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:08:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:08:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:08:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:08:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:08:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:08:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:08:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:08:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:08:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:08:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:08:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:08:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:08:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:08:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:08:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:08:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:08:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:08:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:08:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:08:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:08:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:08:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:08:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:08:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:08:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:08:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:08:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:08:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:08:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:08:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:08:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:08:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:08:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:08:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:08:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:08:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:08:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:08:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:08:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:08:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:08:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:08:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:08:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:08:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:08:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:08:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:08:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:08:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:08:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:08:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:08:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:08:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:08:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:08:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:08:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:08:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:08:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:08:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:08:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:08:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:08:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:08:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:08:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:08:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:08:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:08:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:08:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:08:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:08:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:08:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:08:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:08:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:08:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:08:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:08:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:08:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:08:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:08:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:08:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:08:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:08:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:08:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:08:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:08:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:08:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:08:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:08:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:08:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:08:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:08:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:08:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:08:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:08:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:08:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:08:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:08:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:08:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:08:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:08:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:08:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:08:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:08:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:08:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:08:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:08:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:08:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:08:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:08:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:08:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:08:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:08:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:08:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:08:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:08:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:08:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:08:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:08:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:08:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:08:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:08:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:08:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:08:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:08:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:08:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:08:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:08:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:08:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:08:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:08:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:08:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:08:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:08:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:08:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:08:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:08:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:08:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:08:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:08:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:08:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:08:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:08:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:09:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:09:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:09:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:09:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:09:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:09:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:09:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:09:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:09:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:09:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:09:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:09:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:09:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:09:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:09:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:09:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:09:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:09:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:09:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:09:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:09:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:09:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:09:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:09:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:09:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:09:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:09:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:09:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:09:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:09:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:09:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:09:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:09:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:09:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:09:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:09:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:09:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:09:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:09:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:09:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:09:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:09:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:09:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:09:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:09:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:09:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:09:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:09:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:09:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:09:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:09:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:09:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:09:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:09:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:09:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:09:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:09:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:09:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:09:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:09:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:09:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:09:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:09:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:09:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:09:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:09:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:09:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:09:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:09:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:09:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:09:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:09:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:09:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:09:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:09:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:09:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:09:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:09:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:09:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:09:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:09:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:09:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:09:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:09:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:09:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:09:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:09:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:09:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:09:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:09:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:09:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:09:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:09:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:09:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:09:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:09:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:09:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:09:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:09:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:09:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:09:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:09:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:09:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:09:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:09:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:09:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:09:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:09:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:09:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:09:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:09:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:09:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:09:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:09:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:09:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:09:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:09:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:09:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:09:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:09:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:09:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:09:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:09:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:09:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:09:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:09:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:09:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:09:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:09:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:09:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:09:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:09:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:09:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:09:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:09:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:09:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:09:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:09:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:09:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:09:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:09:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:09:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:09:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:09:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:09:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:09:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:09:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:09:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:09:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:09:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:09:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:09:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:09:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:09:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:09:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:09:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:09:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:09:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:09:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:09:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:09:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:09:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:09:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:09:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:09:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:09:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:09:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:09:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:09:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:09:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:09:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:09:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:09:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:09:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:09:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:09:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:09:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:09:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:09:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:09:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:09:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:09:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:09:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:09:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:09:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:09:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:09:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:09:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:09:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:09:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:09:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:09:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:09:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:09:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:09:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:09:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:09:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:09:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:09:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:09:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:09:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:09:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:09:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:09:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:09:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:09:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:09:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:09:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:09:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:09:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:09:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:09:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:09:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:09:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:09:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:09:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:10:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:10:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:10:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:10:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:10:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:10:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:10:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:10:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:10:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:10:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:10:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:10:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:10:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:10:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:10:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:10:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:10:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:10:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:10:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:10:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:10:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:10:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:10:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:10:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:10:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:10:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:10:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:10:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:10:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:10:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:10:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:10:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:10:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:10:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:10:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:10:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:10:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:10:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:10:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:10:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:10:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:10:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:10:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:10:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:10:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:10:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:10:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:10:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:10:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:10:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:10:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:10:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:10:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:10:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:10:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:10:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:10:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:10:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:10:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:10:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:10:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:10:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:10:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:10:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:10:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:10:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:10:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:10:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:10:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:10:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:10:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:10:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:10:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:10:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:10:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:10:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:10:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:10:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:10:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:10:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:10:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:10:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:10:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:10:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:10:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:10:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:10:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:10:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:10:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:10:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:10:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:10:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:10:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:10:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:10:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:10:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:10:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:10:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:10:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:10:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:10:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:10:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:10:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:10:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:10:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:10:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:10:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:10:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:10:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:10:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:10:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:10:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:10:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:10:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:10:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:10:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:10:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:10:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:10:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:10:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:10:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:10:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:10:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:10:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:10:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:10:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:10:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:10:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:10:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:10:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:10:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:10:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:10:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:10:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:10:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:10:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:10:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:10:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:10:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:10:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:10:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:10:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:10:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:10:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:10:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:10:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:10:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:10:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:10:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:10:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:10:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:10:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:10:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:10:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:10:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:10:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:10:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:10:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:10:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:10:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:10:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:10:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:10:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:10:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:10:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:10:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:10:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:10:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:10:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:10:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:10:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:10:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:10:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:10:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:10:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:10:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:10:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:10:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:10:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:10:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:10:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:10:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:10:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:10:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:10:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:10:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:10:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:10:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:10:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:10:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:10:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:10:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:10:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:10:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:10:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:10:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:10:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:10:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:10:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:10:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:10:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:10:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:10:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:10:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:10:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:10:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:10:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:10:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:10:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:10:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:10:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:10:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:10:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:10:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:10:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:10:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:11:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:11:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:11:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:11:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:11:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:11:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:11:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:11:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:11:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:11:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:11:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:11:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:11:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:11:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:11:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:11:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:11:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:11:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:11:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:11:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:11:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:11:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:11:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:11:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:11:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:11:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:11:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:11:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:11:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:11:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:11:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:11:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:11:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:11:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:11:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:11:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:11:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:11:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:11:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:11:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:11:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:11:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:11:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:11:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:11:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:11:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:11:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:11:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:11:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:11:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:11:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:11:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:11:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:11:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:11:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:11:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:11:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:11:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:11:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:11:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:11:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:11:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:11:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:11:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:11:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:11:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:11:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:11:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:11:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:11:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:11:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:11:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:11:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:11:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:11:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:11:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:11:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:11:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:11:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:11:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:11:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:11:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:11:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:11:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:11:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:11:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:11:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:11:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:11:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:11:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:11:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:11:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:11:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:11:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:11:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:11:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:11:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:11:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:11:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:11:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:11:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:11:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:11:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:11:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:11:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:11:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:11:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:11:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:11:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:11:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:11:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:11:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:11:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:11:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:11:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:11:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:11:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:11:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:11:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:11:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:11:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:11:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:11:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:11:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:11:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:11:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:11:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:11:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:11:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:11:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:11:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:11:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:11:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:11:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:11:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:11:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:11:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:11:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:11:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:11:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:11:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:11:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:11:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:11:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:11:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:11:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:11:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:11:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:11:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:11:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:11:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:11:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:11:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:11:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:11:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:11:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:11:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:11:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:11:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:11:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:11:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:11:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:11:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:11:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:11:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:11:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:11:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:11:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:11:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:11:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:11:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:11:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:11:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:11:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:11:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:11:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:11:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:11:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:11:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:11:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:11:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:11:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:11:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:11:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:11:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:11:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:11:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:11:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:11:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:11:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:11:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:11:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:11:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:11:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:11:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:11:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:11:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:11:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:11:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:11:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:11:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:11:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:11:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:11:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:11:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:11:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:11:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:11:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:11:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:11:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:11:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:11:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:11:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:11:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:11:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:11:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:12:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:12:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:12:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:12:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:12:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:12:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:12:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:12:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:12:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:12:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:12:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:12:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:12:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:12:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:12:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:12:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:12:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:12:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:12:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:12:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:12:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:12:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:12:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:12:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:12:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:12:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:12:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:12:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:12:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:12:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:12:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:12:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:12:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:12:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:12:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:12:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:12:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:12:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:12:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:12:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:12:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:12:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:12:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:12:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:12:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:12:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:12:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:12:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:12:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:12:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:12:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:12:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:12:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:12:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:12:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:12:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:12:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:12:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:12:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:12:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:12:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:12:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:12:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:12:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:12:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:12:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:12:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:12:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:12:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:12:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:12:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:12:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:12:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:12:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:12:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:12:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:12:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:12:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:12:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:12:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:12:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:12:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:12:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:12:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:12:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:12:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:12:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:12:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:12:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:12:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:14:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:14:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:14:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:14:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:14:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:14:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:14:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:14:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:14:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:14:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:14:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:14:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:14:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:14:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:14:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:14:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:14:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:14:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:14:34 rdma06 rsyslogd[1041]: imjournal: 5296 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 07:14:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:14:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:14:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:14:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:14:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:14:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:14:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:14:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:14:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:14:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:14:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:14:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:14:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:14:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:14:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:14:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:14:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:14:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:14:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:14:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:14:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:14:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:14:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:14:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:14:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:14:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:14:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:14:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:14:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:14:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:14:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:14:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:14:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:14:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:14:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:14:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:14:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:14:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:14:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:14:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:14:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:14:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:14:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:14:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:14:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:14:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:14:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:14:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:14:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:14:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:14:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:14:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:14:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:14:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:14:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:14:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:14:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:14:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:14:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:14:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:14:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:14:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:14:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:14:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:14:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:14:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:14:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:14:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:14:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:14:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:14:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:14:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:14:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:14:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:14:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:14:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:14:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:14:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:14:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:14:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:14:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:14:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:14:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:14:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:14:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:14:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:14:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:14:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:14:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:14:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:15:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:15:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:15:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:15:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:15:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:15:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:15:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:15:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:15:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:15:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:15:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:15:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:15:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:15:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:15:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:15:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:15:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:15:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:15:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:15:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:15:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:15:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:15:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:15:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:15:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:15:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:15:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:15:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:15:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:15:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:15:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:15:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:15:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:15:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:15:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:15:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:15:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:15:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:15:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:15:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:15:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:15:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:15:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:15:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:15:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:15:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:15:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:15:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:15:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:15:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:15:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:15:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:15:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:15:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:15:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:15:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:15:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:15:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:15:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:15:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:15:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:15:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:15:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:15:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:15:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:15:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:15:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:15:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:15:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:15:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:15:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:15:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:15:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:15:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:15:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:15:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:15:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:15:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:15:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:15:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:15:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:15:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:15:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:15:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:15:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:15:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:15:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:15:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:15:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:15:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:15:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:15:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:15:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:15:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:15:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:15:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:15:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:15:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:15:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:15:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:15:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:15:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:15:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:15:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:15:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:15:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:15:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:15:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:15:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:15:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:15:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:15:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:15:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:15:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:15:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:15:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:15:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:15:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:15:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:15:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:15:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:15:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:15:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:15:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:15:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:15:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:15:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:15:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:15:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:15:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:15:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:15:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:15:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:15:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:15:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:15:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:15:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:15:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:15:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:15:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:15:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:15:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:15:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:15:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:15:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:15:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:15:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:15:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:15:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:15:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:15:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:15:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:15:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:15:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:15:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:15:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:15:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:15:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:15:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:15:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:15:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:15:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:15:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:15:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:15:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:15:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:15:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:15:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:15:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:15:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:15:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:15:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:15:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:15:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:15:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:15:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:15:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:15:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:15:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:15:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:15:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:15:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:15:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:15:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:15:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:15:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:15:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:15:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:15:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:15:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:15:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:15:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:15:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:15:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:15:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:15:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:15:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:15:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:15:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:15:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:15:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:15:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:15:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:15:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:15:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:15:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:15:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:15:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:15:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:15:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:15:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:15:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:15:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:15:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:15:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:15:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:16:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:16:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:16:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:16:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:16:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:16:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:16:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:16:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:16:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:16:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:16:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:16:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:16:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:16:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:16:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:16:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:16:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:16:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:16:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:16:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:16:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:16:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:16:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:16:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:16:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:16:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:16:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:16:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:16:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:16:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:16:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:16:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:16:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:16:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:16:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:16:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:16:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:16:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:16:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:16:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:16:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:16:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:16:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:16:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:16:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:16:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:16:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:16:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:16:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:16:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:16:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:16:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:16:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:16:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:16:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:16:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:16:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:16:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:16:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:16:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:16:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:16:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:16:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:16:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:16:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:16:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:16:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:16:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:16:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:16:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:16:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:16:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:16:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:16:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:16:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:16:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:16:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:16:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:16:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:16:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:16:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:16:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:16:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:16:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:16:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:16:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:16:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:16:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:16:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:16:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:16:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:16:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:16:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:16:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:16:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:16:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:16:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:16:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:16:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:16:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:16:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:16:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:16:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:16:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:16:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:16:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:16:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:16:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:16:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:16:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:16:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:16:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:16:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:16:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:16:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:16:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:16:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:16:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:16:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:16:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:16:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:16:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:16:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:16:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:16:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:16:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:16:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:16:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:16:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:16:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:16:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:16:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:16:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:16:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:16:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:16:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:16:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:16:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:16:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:16:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:16:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:16:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:16:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:16:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:16:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:16:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:16:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:16:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:16:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:16:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:16:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:16:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:16:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:16:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:16:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:16:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:16:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:16:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:16:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:16:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:16:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:16:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:16:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:16:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:16:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:16:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:16:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:16:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:16:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:16:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:16:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:16:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:16:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:16:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:16:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:16:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:16:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:16:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:16:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:16:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:16:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:16:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:16:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:16:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:16:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:16:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:16:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:16:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:16:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:16:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:16:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:16:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:16:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:16:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:16:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:16:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:16:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:16:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:16:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:16:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:16:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:16:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:16:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:16:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:16:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:16:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:16:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:16:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:16:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:16:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:16:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:16:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:16:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:16:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:16:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:16:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:17:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:17:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:17:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:17:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:17:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:17:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:17:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:17:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:17:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:17:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:17:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:17:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:17:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:17:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:17:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:17:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:17:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:17:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:17:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:17:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:17:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:17:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:17:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:17:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:17:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:17:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:17:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:17:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:17:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:17:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:17:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:17:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:17:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:17:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:17:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:17:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:17:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:17:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:17:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:17:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:17:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:17:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:17:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:17:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:17:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:17:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:17:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:17:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:17:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:17:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:17:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:17:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:17:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:17:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:17:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:17:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:17:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:17:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:17:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:17:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:17:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:17:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:17:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:17:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:17:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:17:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:17:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:17:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:17:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:17:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:17:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:17:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:17:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:17:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:17:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:17:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:17:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:17:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:17:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:17:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:17:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:17:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:17:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:17:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:17:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:17:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:17:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:17:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:17:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:17:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:17:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:17:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:17:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:17:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:17:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:17:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:17:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:17:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:17:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:17:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:17:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:17:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:17:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:17:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:17:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:17:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:17:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:17:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:17:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:17:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:17:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:17:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:17:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:17:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:17:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:17:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:17:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:17:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:17:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:17:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:17:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:17:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:17:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:17:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:17:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:17:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:17:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:17:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:17:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:17:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:17:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:17:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:17:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:17:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:17:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:17:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:17:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:17:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:17:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:17:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:17:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:17:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:17:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:17:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:17:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:17:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:17:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:17:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:17:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:17:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:17:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:17:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:17:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:17:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:17:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:17:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:17:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:17:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:17:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:17:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:17:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:17:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:17:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:17:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:17:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:17:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:17:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:17:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:17:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:17:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:17:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:17:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:17:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:17:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:17:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:17:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:17:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:17:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:17:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:17:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:17:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:17:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:17:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:17:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:17:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:17:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:17:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:17:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:17:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:17:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:17:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:17:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:17:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:17:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:17:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:17:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:17:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:17:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:17:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:17:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:17:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:17:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:17:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:17:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:17:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:17:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:17:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:17:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:17:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:17:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:17:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:17:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:17:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:17:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:17:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:17:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:18:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:18:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:18:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:18:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:18:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:18:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:18:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:18:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:18:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:18:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:18:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:18:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:18:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:18:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:18:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:18:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:18:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:18:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:18:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:18:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:18:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:18:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:18:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:18:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:18:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:18:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:18:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:18:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:18:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:18:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:18:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:18:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:18:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:18:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:18:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:18:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:18:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:18:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:18:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:18:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:18:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:18:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:18:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:18:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:18:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:18:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:18:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:18:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:18:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:18:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:18:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:18:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:18:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:18:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:18:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:18:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:18:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:18:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:18:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:18:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:18:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:18:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:18:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:18:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:18:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:18:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:18:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:18:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:18:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:18:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:18:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:18:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:18:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:18:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:18:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:18:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:18:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:18:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:18:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:18:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:18:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:18:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:18:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:18:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:18:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:18:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:18:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:18:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:18:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:18:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:18:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:18:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:18:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:18:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:18:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:18:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:18:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:18:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:18:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:18:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:18:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:18:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:18:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:18:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:18:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:18:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:18:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:18:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:18:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:18:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:18:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:18:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:18:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:18:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:18:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:18:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:18:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:18:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:18:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:18:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:18:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:18:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:18:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:18:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:18:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:18:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:18:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:18:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:18:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:18:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:18:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:18:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:18:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:18:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:18:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:18:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:18:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:18:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:18:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:18:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:18:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:18:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:18:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:18:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:18:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:18:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:18:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:18:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:18:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:18:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:18:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:18:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:18:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:18:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:18:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:18:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:18:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:18:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:18:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:18:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:18:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:18:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:18:44 rdma06 systemd[1]: Starting dnf makecache...
Apr 17 07:18:44 rdma06 dnf[1147538]: Updating Subscription Management repositories.
Apr 17 07:18:45 rdma06 dnf[1147538]: Unable to read consumer identity
Apr 17 07:18:45 rdma06 dnf[1147538]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 17 07:18:45 rdma06 dnf[1147538]: Metadata cache refreshed recently.
Apr 17 07:18:45 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 17 07:18:45 rdma06 systemd[1]: Finished dnf makecache.
Apr 17 07:18:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:18:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:18:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:18:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:18:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:18:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:18:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:18:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:18:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:18:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:18:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:18:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:18:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:18:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:18:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:18:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:18:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:18:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:18:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:18:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:18:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:18:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:18:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:18:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:18:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:18:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:18:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:18:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:18:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:18:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:18:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:18:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:18:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:18:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:18:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:18:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:18:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:18:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:18:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:18:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:18:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:18:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:18:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:18:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:18:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:18:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:18:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:18:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:18:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:18:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:18:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:18:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:18:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:18:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:19:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:19:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:19:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:19:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:19:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:19:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:19:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:19:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:19:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:19:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:19:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:19:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:19:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:19:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:19:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:19:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:19:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:19:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:19:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:19:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:19:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:19:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:19:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:19:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:19:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:19:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:19:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:19:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:19:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:19:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:19:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:19:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:19:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:19:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:19:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:19:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:19:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:19:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:19:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:19:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:19:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:19:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:19:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:19:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:19:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:19:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:19:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:19:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:19:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:19:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:19:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:19:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:19:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:19:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:19:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:19:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:19:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:19:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:19:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:19:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:19:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:19:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:19:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:19:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:19:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:19:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:19:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:19:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:19:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:19:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:19:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:19:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:19:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:19:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:19:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:19:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:19:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:19:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:19:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:19:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:19:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:19:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:19:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:19:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:19:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:19:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:19:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:19:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:19:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:19:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:19:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:19:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:19:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:19:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:19:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:19:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:19:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:19:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:19:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:19:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:19:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:19:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:19:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:19:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:19:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:19:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:19:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:19:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:19:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:19:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:19:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:19:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:19:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:19:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:19:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:19:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:19:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:19:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:19:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:19:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:19:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:19:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:19:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:19:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:19:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:19:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:19:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:19:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:19:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:19:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:19:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:19:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:19:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:19:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:19:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:19:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:19:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:19:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:19:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:19:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:19:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:19:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:19:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:19:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:19:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:19:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:19:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:19:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:19:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:19:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:19:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:19:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:19:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:19:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:19:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:19:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:19:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:19:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:19:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:19:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:19:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:19:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:19:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:19:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:19:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:19:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:19:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:19:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:19:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:19:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:19:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:19:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:19:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:19:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:19:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:19:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:19:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:19:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:19:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:19:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:19:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:19:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:19:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:19:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:19:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:19:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:19:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:19:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:19:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:19:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:19:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:19:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:19:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:19:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:19:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:19:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:19:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:19:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:19:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:19:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:19:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:19:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:19:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:19:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:19:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:19:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:19:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:19:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:19:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:19:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:19:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:19:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:19:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:19:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:19:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:19:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:20:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:20:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:20:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:20:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:20:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:20:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:20:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:20:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:20:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:20:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:20:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:20:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:20:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:20:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:20:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:20:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:20:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:20:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:20:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:20:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:20:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:20:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:20:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:20:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:20:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:20:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:20:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:20:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:20:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:20:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:20:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:20:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:20:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:20:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:20:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:20:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:20:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:20:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:20:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:20:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:20:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:20:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:20:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:20:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:20:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:20:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:20:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:20:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:20:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:20:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:20:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:20:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:20:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:20:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:20:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:20:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:20:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:20:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:20:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:20:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:20:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:20:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:20:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:20:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:20:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:20:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:20:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:20:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:20:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:20:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:20:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:20:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:20:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:20:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:20:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:20:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:20:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:20:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:20:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:20:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:20:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:20:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:20:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:20:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:20:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:20:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:20:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:20:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:20:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:20:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:20:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:20:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:20:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:20:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:20:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:20:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:20:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:20:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:20:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:20:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:20:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:20:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:20:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:20:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:20:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:20:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:20:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:20:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:20:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:20:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:20:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:20:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:20:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:20:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:20:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:20:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:20:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:20:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:20:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:20:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:20:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:20:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:20:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:20:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:20:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:20:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:20:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:20:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:20:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:20:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:20:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:20:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:20:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:20:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:20:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:20:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:20:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:20:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:20:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:20:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:20:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:20:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:20:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:20:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:20:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:20:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:20:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:20:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:20:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:20:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:20:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:20:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:20:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:20:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:20:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:20:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:20:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:20:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:20:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:20:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:20:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:20:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:20:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:20:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:20:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:20:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:20:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:20:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:20:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:20:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:20:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:20:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:20:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:20:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:20:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:20:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:20:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:20:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:20:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:20:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:20:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:20:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:20:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:20:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:20:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:20:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:20:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:20:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:20:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:20:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:20:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:20:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:20:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:20:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:20:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:20:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:20:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:20:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:20:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:20:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:20:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:20:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:20:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:20:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:20:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:20:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:20:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:20:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:20:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:20:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:20:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:20:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:20:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:20:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:20:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:20:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:21:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:21:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:21:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:21:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:21:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:21:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:21:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:21:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:21:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:21:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:21:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:21:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:21:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:21:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:21:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:21:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:21:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:21:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:21:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:21:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:21:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:21:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:21:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:21:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:21:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:21:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:21:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:21:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:21:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:21:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:21:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:21:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:21:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:21:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:21:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:21:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:21:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:21:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:21:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:21:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:21:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:21:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:21:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:21:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:21:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:21:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:21:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:21:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:21:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:21:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:21:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:21:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:21:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:21:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:21:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:21:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:21:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:21:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:21:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:21:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:21:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:21:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:21:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:21:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:21:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:21:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:21:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:21:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:21:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:21:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:21:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:21:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:21:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:21:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:21:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:21:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:21:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:21:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:21:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:21:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:21:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:21:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:21:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:21:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:21:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:21:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:21:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:21:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:21:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:21:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:21:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:21:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:21:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:21:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:21:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:21:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:21:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:21:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:21:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:21:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:21:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:21:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:21:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:21:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:21:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:21:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:21:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:21:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:21:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:21:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:21:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:21:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:21:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:21:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:21:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:21:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:22:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:22:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:22:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:22:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:22:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:22:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:22:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:22:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:22:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:22:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:22:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:22:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:22:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:22:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:22:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:22:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:22:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:22:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:22:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:22:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:22:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:22:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:22:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:22:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:22:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:22:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:22:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:22:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:22:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:22:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:22:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:22:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:22:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:22:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:22:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:22:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:22:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:22:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:22:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:22:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:22:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:22:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:22:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:22:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:22:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:22:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:22:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:22:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:22:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:22:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:22:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:22:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:22:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:22:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:22:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:22:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:22:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:22:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:22:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:22:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:22:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:22:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:22:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:22:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:22:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:22:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:22:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:22:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:22:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:22:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:22:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:22:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:22:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:22:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:22:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:22:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:22:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:22:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:22:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:22:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:22:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:22:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:22:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:22:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:22:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:22:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:22:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:22:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:22:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:22:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:22:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:22:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:22:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:22:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:22:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:22:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:22:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:22:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:22:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:22:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:22:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:22:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:22:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:22:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:22:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:22:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:22:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:22:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:24:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:24:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:24:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:24:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:24:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:24:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:24:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:24:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:24:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:24:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:24:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:24:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:24:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:24:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:24:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:24:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:24:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:24:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:24:35 rdma06 rsyslogd[1041]: imjournal: 5089 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 07:24:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:24:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:24:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:24:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:24:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:24:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:24:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:24:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:24:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:24:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:24:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:24:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:24:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:24:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:24:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:24:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:24:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:24:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:24:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:24:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:24:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:24:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:24:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:24:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:24:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:24:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:24:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:24:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:24:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:24:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:24:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:24:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:24:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:24:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:24:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:24:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:24:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:24:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:24:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:24:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:24:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:24:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:24:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:24:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:24:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:24:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:24:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:24:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:24:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:24:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:24:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:24:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:24:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:24:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:24:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:24:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:24:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:24:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:24:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:24:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:24:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:24:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:24:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:24:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:24:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:24:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:24:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:24:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:24:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:24:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:24:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:24:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:24:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:24:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:24:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:24:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:24:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:24:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:24:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:24:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:24:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:24:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:24:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:24:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:24:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:24:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:24:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:24:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:24:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:24:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:25:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:25:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:25:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:25:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:25:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:25:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:25:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:25:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:25:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:25:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:25:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:25:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:25:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:25:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:25:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:25:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:25:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:25:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:25:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:25:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:25:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:25:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:25:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:25:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:25:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:25:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:25:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:25:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:25:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:25:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:25:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:25:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:25:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:25:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:25:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:25:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:25:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:25:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:25:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:25:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:25:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:25:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:25:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:25:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:25:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:25:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:25:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:25:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:25:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:25:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:25:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:25:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:25:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:25:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:25:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:25:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:25:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:25:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:25:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:25:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:25:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:25:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:25:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:25:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:25:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:25:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:25:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:25:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:25:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:25:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:25:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:25:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:25:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:25:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:25:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:25:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:25:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:25:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:25:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:25:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:25:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:25:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:25:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:25:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:25:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:25:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:25:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:25:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:25:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:25:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:25:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:25:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:25:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:25:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:25:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:25:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:25:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:25:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:25:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:25:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:25:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:25:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:25:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:25:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:25:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:25:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:25:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:25:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:25:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:25:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:25:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:25:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:25:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:25:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:25:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:25:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:25:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:25:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:25:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:25:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:25:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:25:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:25:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:25:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:25:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:25:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:25:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:25:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:25:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:25:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:25:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:25:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:25:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:25:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:25:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:25:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:25:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:25:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:25:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:25:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:25:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:25:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:25:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:25:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:25:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:25:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:25:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:25:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:25:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:25:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:25:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:25:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:25:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:25:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:25:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:25:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:25:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:25:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:25:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:25:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:25:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:25:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:25:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:25:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:25:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:25:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:25:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:25:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:25:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:25:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:25:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:25:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:25:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:25:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:25:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:25:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:25:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:25:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:25:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:25:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:25:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:25:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:25:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:25:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:25:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:25:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:25:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:25:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:25:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:25:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:25:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:25:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:25:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:25:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:25:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:25:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:25:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:25:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:25:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:25:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:25:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:25:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:25:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:25:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:25:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:25:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:25:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:25:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:25:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:25:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:25:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:25:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:25:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:25:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:25:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:25:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:26:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:26:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:26:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:26:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:26:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:26:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:26:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:26:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:26:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:26:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:26:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:26:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:26:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:26:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:26:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:26:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:26:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:26:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:26:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:26:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:26:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:26:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:26:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:26:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:26:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:26:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:26:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:26:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:26:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:26:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:26:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:26:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:26:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:26:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:26:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:26:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:26:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:26:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:26:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:26:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:26:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:26:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:26:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:26:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:26:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:26:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:26:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:26:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:26:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:26:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:26:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:26:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:26:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:26:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:26:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:26:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:26:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:26:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:26:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:26:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:26:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:26:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:26:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:26:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:26:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:26:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:26:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:26:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:26:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:26:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:26:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:26:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:26:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:26:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:26:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:26:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:26:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:26:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:26:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:26:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:26:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:26:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:26:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:26:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:26:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:26:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:26:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:26:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:26:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:26:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:26:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:26:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:26:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:26:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:26:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:26:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:26:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:26:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:26:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:26:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:26:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:26:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:26:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:26:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:26:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:26:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:26:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:26:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:26:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:26:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:26:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:26:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:26:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:26:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:26:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:26:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:26:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:26:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:26:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:26:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:26:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:26:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:26:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:26:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:26:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:26:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:26:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:26:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:26:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:26:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:26:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:26:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:26:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:26:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:26:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:26:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:26:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:26:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:26:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:26:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:26:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:26:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:26:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:26:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:26:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:26:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:26:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:26:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:26:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:26:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:26:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:26:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:26:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:26:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:26:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:26:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:26:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:26:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:26:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:26:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:26:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:26:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:26:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:26:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:26:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:26:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:26:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:26:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:26:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:26:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:26:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:26:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:26:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:26:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:26:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:26:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:26:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:26:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:26:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:26:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:26:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:26:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:26:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:26:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:26:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:26:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:26:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:26:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:26:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:26:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:26:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:26:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:26:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:26:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:26:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:26:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:26:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:26:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:26:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:26:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:26:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:26:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:26:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:26:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:26:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:26:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:26:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:26:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:26:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:26:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:26:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:26:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:26:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:26:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:26:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:26:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:27:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:27:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:27:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:27:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:27:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:27:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:27:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:27:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:27:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:27:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:27:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:27:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:27:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:27:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:27:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:27:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:27:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:27:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:27:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:27:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:27:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:27:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:27:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:27:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:27:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:27:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:27:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:27:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:27:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:27:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:27:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:27:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:27:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:27:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:27:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:27:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:27:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:27:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:27:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:27:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:27:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:27:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:27:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:27:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:27:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:27:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:27:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:27:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:27:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:27:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:27:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:27:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:27:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:27:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:27:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:27:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:27:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:27:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:27:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:27:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:27:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:27:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:27:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:27:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:27:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:27:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:27:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:27:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:27:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:27:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:27:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:27:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:27:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:27:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:27:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:27:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:27:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:27:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:27:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:27:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:27:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:27:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:27:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:27:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:27:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:27:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:27:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:27:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:27:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:27:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:27:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:27:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:27:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:27:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:27:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:27:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:27:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:27:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:27:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:27:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:27:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:27:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:27:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:27:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:27:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:27:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:27:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:27:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:27:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:27:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:27:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:27:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:27:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:27:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:27:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:27:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:27:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:27:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:27:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:27:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:27:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:27:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:27:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:27:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:27:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:27:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:27:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:27:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:27:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:27:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:27:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:27:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:27:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:27:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:27:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:27:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:27:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:27:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:27:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:27:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:27:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:27:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:27:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:27:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:27:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:27:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:27:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:27:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:27:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:27:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:27:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:27:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:27:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:27:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:27:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:27:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:27:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:27:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:27:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:27:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:27:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:27:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:27:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:27:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:27:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:27:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:27:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:27:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:27:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:27:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:27:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:27:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:27:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:27:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:27:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:27:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:27:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:27:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:27:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:27:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:27:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:27:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:27:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:27:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:27:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:27:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:27:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:27:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:27:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:27:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:27:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:27:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:27:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:27:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:27:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:27:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:27:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:27:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:27:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:27:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:27:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:27:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:27:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:27:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:27:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:27:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:27:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:27:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:27:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:27:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:27:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:27:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:27:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:27:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:27:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:27:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:28:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:28:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:28:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:28:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:28:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:28:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:28:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:28:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:28:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:28:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:28:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:28:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:28:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:28:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:28:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:28:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:28:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:28:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:28:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:28:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:28:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:28:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:28:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:28:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:28:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:28:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:28:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:28:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:28:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:28:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:28:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:28:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:28:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:28:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:28:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:28:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:28:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:28:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:28:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:28:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:28:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:28:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:28:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:28:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:28:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:28:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:28:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:28:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:28:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:28:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:28:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:28:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:28:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:28:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:28:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:28:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:28:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:28:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:28:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:28:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:28:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:28:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:28:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:28:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:28:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:28:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:28:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:28:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:28:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:28:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:28:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:28:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:28:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:28:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:28:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:28:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:28:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:28:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:28:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:28:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:28:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:28:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:28:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:28:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:28:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:28:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:28:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:28:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:28:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:28:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:28:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:28:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:28:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:28:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:28:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:28:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:28:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:28:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:28:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:28:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:28:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:28:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:28:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:28:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:28:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:28:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:28:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:28:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:28:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:28:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:28:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:28:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:28:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:28:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:28:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:28:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:28:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:28:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:28:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:28:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:28:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:28:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:28:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:28:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:28:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:28:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:28:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:28:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:28:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:28:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:28:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:28:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:28:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:28:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:28:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:28:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:28:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:28:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:28:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:28:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:28:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:28:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:28:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:28:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:28:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:28:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:28:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:28:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:28:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:28:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:28:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:28:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:28:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:28:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:28:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:28:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:28:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:28:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:28:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:28:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:28:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:28:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:28:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:28:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:28:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:28:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:28:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:28:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:28:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:28:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:28:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:28:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:28:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:28:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:28:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:28:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:28:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:28:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:28:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:28:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:28:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:28:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:28:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:28:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:28:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:28:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:28:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:28:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:28:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:28:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:28:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:28:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:28:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:28:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:28:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:28:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:28:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:28:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:28:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:28:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:28:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:28:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:28:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:28:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:28:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:28:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:28:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:28:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:28:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:28:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:28:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:28:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:28:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:28:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:28:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:28:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:29:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:29:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:29:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:29:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:29:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:29:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:29:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:29:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:29:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:29:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:29:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:29:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:29:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:29:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:29:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:29:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:29:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:29:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:29:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:29:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:29:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:29:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:29:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:29:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:29:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:29:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:29:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:29:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:29:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:29:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:29:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:29:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:29:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:29:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:29:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:29:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:29:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:29:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:29:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:29:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:29:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:29:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:29:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:29:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:29:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:29:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:29:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:29:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:29:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:29:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:29:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:29:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:29:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:29:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:29:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:29:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:29:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:29:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:29:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:29:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:29:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:29:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:29:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:29:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:29:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:29:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:29:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:29:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:29:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:29:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:29:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:29:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:29:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:29:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:29:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:29:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:29:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:29:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:29:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:29:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:29:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:29:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:29:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:29:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:29:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:29:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:29:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:29:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:29:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:29:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:29:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:29:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:29:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:29:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:29:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:29:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:29:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:29:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:29:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:29:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:29:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:29:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:29:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:29:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:29:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:29:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:29:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:29:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:29:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:29:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:29:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:29:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:29:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:29:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:29:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:29:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:29:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:29:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:29:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:29:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:29:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:29:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:29:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:29:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:29:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:29:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:29:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:29:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:29:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:29:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:29:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:29:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:29:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:29:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:29:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:29:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:29:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:29:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:29:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:29:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:29:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:29:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:29:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:29:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:29:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:29:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:29:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:29:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:29:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:29:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:29:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:29:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:29:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:29:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:29:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:29:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:29:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:29:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:29:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:29:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:29:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:29:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:29:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:29:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:29:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:29:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:29:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:29:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:29:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:29:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:29:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:29:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:29:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:29:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:29:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:29:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:29:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:29:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:29:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:29:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:29:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:29:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:29:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:29:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:29:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:29:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:29:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:29:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:29:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:29:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:29:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:29:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:29:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:29:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:29:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:29:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:29:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:29:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:29:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:29:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:29:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:29:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:29:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:29:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:29:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:29:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:29:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:29:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:29:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:29:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:29:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:29:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:29:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:29:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:29:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:29:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:30:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:30:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:30:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:30:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:30:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:30:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:30:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:30:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:30:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:30:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:30:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:30:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:30:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:30:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:30:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:30:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:30:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:30:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:30:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:30:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:30:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:30:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:30:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:30:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:30:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:30:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:30:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:30:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:30:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:30:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:30:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:30:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:30:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:30:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:30:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:30:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:30:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:30:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:30:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:30:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:30:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:30:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:30:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:30:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:30:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:30:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:30:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:30:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:30:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:30:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:30:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:30:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:30:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:30:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:30:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:30:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:30:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:30:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:30:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:30:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:30:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:30:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:30:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:30:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:30:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:30:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:30:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:30:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:30:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:30:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:30:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:30:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:30:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:30:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:30:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:30:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:30:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:30:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:30:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:30:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:30:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:30:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:30:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:30:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:30:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:30:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:30:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:30:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:30:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:30:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:30:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:30:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:30:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:30:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:30:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:30:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:30:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:30:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:30:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:30:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:30:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:30:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:30:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:30:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:30:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:30:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:30:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:30:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:30:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:30:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:30:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:30:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:30:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:30:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:30:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:30:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:30:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:30:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:30:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:30:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:30:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:30:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:30:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:30:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:30:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:30:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:30:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:30:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:30:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:30:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:30:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:30:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:30:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:30:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:30:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:30:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:30:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:30:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:30:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:30:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:30:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:30:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:30:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:30:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:30:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:30:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:30:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:30:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:30:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:30:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:30:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:30:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:30:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:30:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:30:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:30:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:30:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:30:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:30:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:30:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:30:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:30:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:30:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:30:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:30:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:30:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:30:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:30:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:30:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:30:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:30:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:30:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:30:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:30:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:30:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:30:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:30:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:30:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:30:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:30:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:30:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:30:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:30:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:30:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:30:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:30:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:30:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:30:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:30:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:30:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:30:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:30:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:30:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:30:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:30:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:30:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:30:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:30:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:30:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:30:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:30:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:30:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:30:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:30:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:30:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:30:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:30:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:30:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:30:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:30:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:30:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:30:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:30:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:30:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:30:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:30:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:31:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:31:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:31:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:31:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:31:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:31:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:31:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:31:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:31:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:31:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:31:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:31:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:31:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:31:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:31:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:31:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:31:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:31:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:31:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:31:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:31:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:31:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:31:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:31:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:31:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:31:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:31:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:31:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:31:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:31:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:31:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:31:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:31:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:31:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:31:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:31:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:31:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:31:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:31:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:31:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:31:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:31:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:31:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:31:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:31:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:31:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:31:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:31:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:31:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:31:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:31:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:31:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:31:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:31:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:31:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:31:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:31:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:31:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:31:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:31:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:31:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:31:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:31:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:31:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:31:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:31:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:31:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:31:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:31:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:31:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:31:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:31:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:31:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:31:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:31:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:31:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:31:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:31:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:31:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:31:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:31:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:31:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:31:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:31:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:31:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:31:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:31:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:31:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:31:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:31:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:31:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:31:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:31:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:31:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:31:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:31:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:31:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:31:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:31:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:31:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:31:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:31:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:31:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:31:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:31:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:31:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:31:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:31:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:31:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:31:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:31:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:31:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:31:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:31:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:31:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:31:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:31:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:31:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:31:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:31:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:31:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:31:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:31:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:31:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:31:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:31:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:31:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:31:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:31:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:31:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:31:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:31:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:31:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:31:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:31:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:31:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:31:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:31:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:31:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:31:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:31:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:31:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:31:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:31:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:31:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:31:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:31:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:31:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:31:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:31:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:31:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:31:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:31:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:31:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:31:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:31:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:31:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:31:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:31:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:31:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:31:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:31:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:31:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:31:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:31:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:31:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:31:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:31:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:31:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:31:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:31:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:31:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:31:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:31:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:31:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:31:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:31:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:31:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:31:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:31:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:31:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:31:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:31:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:31:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:31:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:31:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:31:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:31:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:31:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:31:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:31:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:31:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:31:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:31:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:31:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:31:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:31:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:31:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:31:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:31:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:31:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:31:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:31:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:31:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:31:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:31:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:32:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:32:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:32:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:32:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:32:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:32:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:32:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:32:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:32:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:32:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:32:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:32:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:32:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:32:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:32:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:32:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:32:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:32:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:32:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:32:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:32:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:32:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:32:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:32:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:32:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:32:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:32:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:32:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:32:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:32:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:32:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:32:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:32:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:32:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:32:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:32:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:32:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:32:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:32:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:32:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:32:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:32:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:32:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:32:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:32:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:32:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:32:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:32:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:32:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:32:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:32:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:32:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:32:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:32:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:32:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:32:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:32:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:32:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:32:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:32:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:32:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:32:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:32:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:32:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:32:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:32:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:32:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:32:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:32:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:32:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:32:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:32:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:32:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:32:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:32:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:32:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:32:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:32:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:32:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:32:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:32:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:32:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:32:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:32:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:32:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:32:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:32:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:32:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:32:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:32:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:32:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:32:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:32:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:32:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:32:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:32:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:32:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:32:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:32:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:32:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:32:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:32:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:32:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:32:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:32:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:32:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:32:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:32:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:34:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:34:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:34:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:34:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:34:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:34:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:34:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:34:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:34:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:34:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:34:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:34:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:34:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:34:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:34:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:34:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:34:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:34:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:34:40 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 07:34:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:34:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:34:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:34:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:34:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:34:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:34:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:34:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:34:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:34:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:34:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:34:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:34:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:34:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:34:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:34:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:34:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:34:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:34:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:34:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:34:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:34:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:34:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:34:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:34:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:34:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:34:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:34:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:34:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:34:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:34:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:34:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:34:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:34:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:34:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:34:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:34:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:34:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:34:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:34:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:34:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:34:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:34:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:34:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:34:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:34:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:34:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:34:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:34:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:34:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:34:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:34:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:34:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:34:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:35:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:35:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:35:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:35:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:35:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:35:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:35:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:35:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:35:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:35:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:35:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:35:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:35:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:35:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:35:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:35:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:35:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:35:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:35:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:35:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:35:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:35:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:35:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:35:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:35:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:35:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:35:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:35:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:35:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:35:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:35:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:35:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:35:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:35:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:35:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:35:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:35:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:35:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:35:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:35:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:35:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:35:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:35:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:35:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:35:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:35:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:35:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:35:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:35:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:35:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:35:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:35:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:35:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:35:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:35:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:35:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:35:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:35:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:35:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:35:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:35:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:35:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:35:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:35:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:35:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:35:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:35:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:35:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:35:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:35:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:35:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:35:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:35:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:35:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:35:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:35:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:35:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:35:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:35:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:35:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:35:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:35:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:35:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:35:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:35:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:35:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:35:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:35:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:35:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:35:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:35:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:35:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:35:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:35:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:35:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:35:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:35:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:35:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:35:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:35:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:35:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:35:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:35:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:35:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:35:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:35:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:35:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:35:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:35:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:35:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:35:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:35:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:35:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:35:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:35:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:35:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:35:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:35:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:35:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:35:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:35:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:35:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:35:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:35:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:35:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:35:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:35:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:35:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:35:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:35:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:35:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:35:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:35:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:35:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:35:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:35:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:35:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:35:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:35:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:35:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:35:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:35:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:35:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:35:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:35:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:35:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:35:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:35:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:35:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:35:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:35:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:35:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:35:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:35:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:35:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:35:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:35:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:35:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:35:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:35:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:35:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:35:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:35:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:35:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:35:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:35:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:35:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:35:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:35:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:35:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:35:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:35:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:35:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:35:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:35:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:35:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:35:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:35:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:35:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:35:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:35:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:35:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:35:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:35:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:35:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:35:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:35:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:35:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:35:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:35:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:35:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:35:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:35:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:35:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:35:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:35:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:35:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:35:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:35:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:35:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:35:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:35:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:35:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:35:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:35:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:35:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:35:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:35:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:35:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:35:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:35:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:35:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:35:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:35:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:35:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:35:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:36:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:36:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:36:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:36:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:36:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:36:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:36:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:36:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:36:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:36:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:36:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:36:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:36:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:36:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:36:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:36:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:36:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:36:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:36:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:36:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:36:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:36:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:36:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:36:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:36:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:36:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:36:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:36:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:36:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:36:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:36:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:36:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:36:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:36:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:36:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:36:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:36:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:36:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:36:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:36:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:36:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:36:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:36:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:36:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:36:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:36:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:36:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:36:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:36:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:36:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:36:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:36:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:36:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:36:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:36:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:36:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:36:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:36:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:36:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:36:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:36:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:36:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:36:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:36:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:36:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:36:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:36:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:36:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:36:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:36:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:36:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:36:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:36:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:36:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:36:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:36:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:36:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:36:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:36:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:36:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:36:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:36:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:36:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:36:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:36:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:36:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:36:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:36:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:36:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:36:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:36:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:36:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:36:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:36:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:36:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:36:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:36:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:36:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:36:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:36:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:36:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:36:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:36:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:36:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:36:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:36:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:36:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:36:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:36:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:36:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:36:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:36:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:36:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:36:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:36:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:36:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:36:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:36:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:36:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:36:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:36:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:36:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:36:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:36:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:36:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:36:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:36:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:36:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:36:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:36:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:36:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:36:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:36:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:36:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:36:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:36:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:36:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:36:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:36:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:36:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:36:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:36:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:36:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:36:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:36:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:36:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:36:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:36:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:36:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:36:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:36:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:36:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:36:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:36:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:36:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:36:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:36:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:36:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:36:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:36:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:36:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:36:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:36:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:36:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:36:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:36:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:36:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:36:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:36:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:36:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:36:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:36:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:36:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:36:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:36:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:36:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:36:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:36:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:36:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:36:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:36:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:36:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:36:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:36:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:36:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:36:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:36:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:36:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:36:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:36:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:36:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:36:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:36:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:36:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:36:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:36:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:36:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:36:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:36:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:36:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:36:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:36:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:36:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:36:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:36:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:36:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:36:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:36:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:36:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:36:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:36:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:36:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:36:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:36:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:36:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:36:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:37:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:37:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:37:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:37:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:37:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:37:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:37:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:37:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:37:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:37:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:37:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:37:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:37:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:37:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:37:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:37:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:37:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:37:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:37:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:37:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:37:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:37:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:37:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:37:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:37:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:37:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:37:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:37:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:37:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:37:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:37:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:37:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:37:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:37:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:37:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:37:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:37:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:37:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:37:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:37:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:37:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:37:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:37:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:37:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:37:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:37:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:37:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:37:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:37:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:37:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:37:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:37:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:37:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:37:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:37:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:37:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:37:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:37:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:37:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:37:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:37:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:37:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:37:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:37:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:37:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:37:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:37:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:37:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:37:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:37:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:37:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:37:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:37:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:37:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:37:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:37:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:37:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:37:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:37:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:37:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:37:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:37:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:37:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:37:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:37:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:37:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:37:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:37:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:37:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:37:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:37:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:37:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:37:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:37:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:37:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:37:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:37:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:37:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:37:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:37:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:37:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:37:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:37:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:37:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:37:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:37:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:37:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:37:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:37:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:37:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:37:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:37:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:37:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:37:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:37:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:37:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:37:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:37:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:37:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:37:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:37:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:37:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:37:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:37:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:37:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:37:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:37:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:37:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:37:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:37:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:37:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:37:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:37:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:37:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:37:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:37:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:37:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:37:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:37:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:37:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:37:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:37:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:37:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:37:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:37:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:37:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:37:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:37:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:37:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:37:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:37:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:37:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:37:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:37:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:37:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:37:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:37:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:37:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:37:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:37:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:37:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:37:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:37:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:37:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:37:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:37:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:37:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:37:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:37:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:37:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:37:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:37:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:37:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:37:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:37:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:37:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:37:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:37:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:37:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:37:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:37:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:37:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:37:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:37:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:37:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:37:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:37:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:37:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:37:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:37:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:37:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:37:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:37:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:37:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:37:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:37:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:37:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:37:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:37:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:37:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:37:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:37:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:37:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:37:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:37:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:37:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:37:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:37:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:37:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:37:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:37:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:37:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:37:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:37:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:37:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:37:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:38:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:38:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:38:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:38:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:38:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:38:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:38:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:38:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:38:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:38:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:38:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:38:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:38:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:38:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:38:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:38:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:38:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:38:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:38:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:38:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:38:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:38:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:38:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:38:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:38:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:38:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:38:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:38:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:38:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:38:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:38:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:38:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:38:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:38:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:38:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:38:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:38:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:38:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:38:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:38:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:38:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:38:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:38:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:38:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:38:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:38:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:38:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:38:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:38:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:38:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:38:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:38:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:38:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:38:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:38:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:38:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:38:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:38:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:38:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:38:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:38:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:38:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:38:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:38:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:38:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:38:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:38:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:38:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:38:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:38:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:38:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:38:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:38:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:38:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:38:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:38:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:38:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:38:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:38:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:38:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:38:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:38:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:38:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:38:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:38:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:38:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:38:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:38:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:38:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:38:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:38:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:38:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:38:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:38:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:38:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:38:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:38:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:38:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:38:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:38:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:38:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:38:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:38:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:38:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:38:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:38:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:38:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:38:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:38:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:38:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:38:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:38:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:38:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:38:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:38:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:38:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:38:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:38:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:38:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:38:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:38:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:38:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:38:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:38:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:38:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:38:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:38:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:38:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:38:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:38:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:38:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:38:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:38:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:38:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:38:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:38:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:38:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:38:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:38:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:38:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:38:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:38:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:38:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:38:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:38:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:38:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:38:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:38:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:38:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:38:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:38:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:38:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:38:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:38:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:38:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:38:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:38:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:38:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:38:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:38:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:38:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:38:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:38:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:38:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:38:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:38:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:38:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:38:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:38:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:38:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:38:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:38:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:38:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:38:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:38:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:38:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:38:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:38:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:38:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:38:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:38:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:38:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:38:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:38:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:38:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:38:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:38:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:38:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:38:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:38:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:38:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:38:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:38:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:38:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:38:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:38:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:38:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:38:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:38:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:38:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:38:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:38:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:38:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:38:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:38:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:38:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:38:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:38:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:38:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:38:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:38:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:38:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:38:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:38:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:38:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:38:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:39:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:39:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:39:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:39:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:39:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:39:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:39:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:39:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:39:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:39:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:39:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:39:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:39:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:39:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:39:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:39:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:39:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:39:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:39:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:39:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:39:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:39:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:39:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:39:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:39:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:39:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:39:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:39:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:39:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:39:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:39:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:39:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:39:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:39:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:39:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:39:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:39:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:39:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:39:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:39:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:39:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:39:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:39:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:39:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:39:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:39:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:39:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:39:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:39:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:39:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:39:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:39:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:39:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:39:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:39:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:39:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:39:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:39:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:39:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:39:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:39:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:39:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:39:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:39:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:39:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:39:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:39:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:39:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:39:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:39:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:39:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:39:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:39:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:39:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:39:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:39:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:39:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:39:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:39:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:39:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:39:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:39:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:39:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:39:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:39:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:39:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:39:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:39:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:39:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:39:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:39:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:39:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:39:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:39:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:39:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:39:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:39:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:39:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:39:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:39:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:39:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:39:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:39:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:39:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:39:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:39:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:39:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:39:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:39:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:39:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:39:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:39:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:39:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:39:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:39:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:39:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:39:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:39:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:39:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:39:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:39:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:39:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:39:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:39:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:39:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:39:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:39:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:39:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:39:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:39:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:39:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:39:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:39:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:39:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:39:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:39:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:39:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:39:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:39:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:39:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:39:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:39:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:39:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:39:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:39:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:39:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:39:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:39:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:39:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:39:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:39:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:39:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:39:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:39:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:39:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:39:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:39:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:39:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:39:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:39:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:39:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:39:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:39:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:39:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:39:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:39:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:39:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:39:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:39:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:39:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:39:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:39:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:39:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:39:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:39:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:39:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:39:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:39:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:39:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:39:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:39:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:39:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:39:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:39:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:39:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:39:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:39:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:39:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:39:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:39:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:39:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:39:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:39:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:39:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:39:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:39:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:39:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:39:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:39:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:39:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:39:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:39:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:39:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:39:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:39:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:39:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:39:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:39:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:39:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:39:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:39:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:39:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:39:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:39:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:39:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:39:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:40:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:40:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:40:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:40:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:40:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:40:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:40:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:40:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:40:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:40:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:40:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:40:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:40:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:40:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:40:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:40:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:40:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:40:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:40:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:40:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:40:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:40:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:40:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:40:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:40:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:40:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:40:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:40:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:40:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:40:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:40:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:40:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:40:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:40:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:40:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:40:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:40:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:40:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:40:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:40:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:40:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:40:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:40:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:40:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:40:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:40:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:40:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:40:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:40:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:40:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:40:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:40:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:40:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:40:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:40:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:40:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:40:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:40:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:40:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:40:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:40:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:40:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:40:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:40:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:40:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:40:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:40:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:40:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:40:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:40:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:40:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:40:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:40:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:40:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:40:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:40:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:40:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:40:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:40:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:40:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:40:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:40:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:40:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:40:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:40:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:40:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:40:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:40:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:40:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:40:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:40:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:40:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:40:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:40:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:40:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:40:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:40:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:40:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:40:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:40:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:40:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:40:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:40:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:40:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:40:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:40:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:40:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:40:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:40:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:40:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:40:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:40:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:40:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:40:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:40:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:40:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:40:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:40:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:40:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:40:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:40:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:40:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:40:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:40:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:40:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:40:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:40:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:40:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:40:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:40:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:40:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:40:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:40:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:40:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:40:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:40:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:40:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:40:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:40:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:40:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:40:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:40:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:40:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:40:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:40:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:40:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:40:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:40:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:40:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:40:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:40:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:40:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:40:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:40:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:40:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:40:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:40:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:40:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:40:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:40:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:40:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:40:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:40:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:40:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:40:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:40:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:40:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:40:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:40:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:40:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:40:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:40:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:40:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:40:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:40:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:40:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:40:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:40:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:40:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:40:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:40:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:40:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:40:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:40:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:40:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:40:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:40:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:40:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:40:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:40:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:40:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:40:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:40:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:40:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:40:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:40:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:40:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:40:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:40:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:40:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:40:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:40:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:40:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:40:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:40:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:40:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:40:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:40:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:40:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:40:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:40:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:40:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:40:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:40:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:40:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:40:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:41:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:41:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:41:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:41:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:41:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:41:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:41:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:41:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:41:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:41:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:41:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:41:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:41:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:41:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:41:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:41:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:41:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:41:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:41:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:41:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:41:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:41:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:41:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:41:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:41:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:41:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:41:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:41:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:41:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:41:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:41:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:41:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:41:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:41:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:41:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:41:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:41:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:41:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:41:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:41:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:41:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:41:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:41:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:41:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:41:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:41:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:41:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:41:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:41:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:41:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:41:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:41:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:41:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:41:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:41:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:41:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:41:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:41:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:41:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:41:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:41:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:41:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:41:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:41:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:41:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:41:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:41:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:41:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:41:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:41:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:41:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:41:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:41:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:41:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:41:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:41:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:41:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:41:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:41:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:41:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:41:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:41:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:41:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:41:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:41:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:41:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:41:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:41:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:41:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:41:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:41:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:41:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:41:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:41:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:41:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:41:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:41:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:41:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:41:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:41:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:41:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:41:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:41:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:41:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:41:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:41:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:41:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:41:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:41:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:41:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:41:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:41:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:41:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:41:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:41:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:41:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:41:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:41:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:41:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:41:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:41:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:41:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:41:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:41:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:41:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:41:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:41:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:41:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:41:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:41:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:41:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:41:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:41:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:41:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:41:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:41:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:41:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:41:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:41:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:41:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:41:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:41:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:41:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:41:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:41:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:41:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:41:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:41:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:41:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:41:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:41:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:41:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:41:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:41:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:41:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:41:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:41:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:41:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:41:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:41:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:41:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:41:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:41:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:41:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:41:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:41:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:41:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:41:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:41:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:41:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:41:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:41:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:41:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:41:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:41:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:41:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:41:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:41:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:41:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:41:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:41:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:41:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:41:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:41:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:41:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:41:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:41:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:41:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:41:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:41:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:41:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:41:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:41:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:41:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:41:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:41:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:41:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:41:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:41:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:41:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:41:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:41:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:41:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:41:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:41:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:41:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:41:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:41:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:41:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:41:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:41:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:41:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:41:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:41:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:41:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:41:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:42:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:42:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:42:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:42:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:42:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:42:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:42:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:42:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:42:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:42:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:42:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:42:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:42:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:42:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:42:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:42:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:42:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:42:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:42:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:42:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:42:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:42:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:42:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:42:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:42:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:42:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:42:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:42:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:42:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:42:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:42:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:42:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:42:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:42:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:42:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:42:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:42:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:42:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:42:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:42:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:42:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:42:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:42:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:42:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:42:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:42:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:42:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:42:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:42:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:42:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:42:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:42:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:42:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:42:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:42:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:42:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:42:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:42:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:42:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:42:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:42:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:42:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:42:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:42:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:42:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:42:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:42:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:42:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:42:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:42:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:42:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:42:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:42:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:42:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:42:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:42:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:42:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:42:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:42:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:42:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:42:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:42:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:42:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:42:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:42:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:42:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:42:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:42:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:42:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:42:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:42:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:42:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:42:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:42:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:42:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:42:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:42:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:42:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:42:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:42:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:42:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:42:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:42:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:42:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:42:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:42:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:42:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:42:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:42:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:42:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:42:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:42:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:42:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:42:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:42:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:42:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:42:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:42:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:42:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:42:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:42:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:42:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:42:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:42:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:42:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:42:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:42:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:42:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:42:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:42:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:42:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:42:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:42:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:42:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:42:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:42:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:42:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:42:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:42:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:42:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:42:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:42:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:42:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:42:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:44:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:44:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:44:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:44:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:44:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:44:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:44:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:44:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:44:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:44:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:44:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:44:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:44:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:44:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:44:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:44:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:44:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:44:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:44:45 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 07:44:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:44:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:44:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:44:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:44:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:44:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:44:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:44:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:44:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:44:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:44:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:44:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:44:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:44:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:44:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:44:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:44:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:44:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:44:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:44:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:44:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:44:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:44:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:44:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:44:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:44:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:44:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:44:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:44:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:44:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:44:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:44:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:44:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:44:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:44:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:44:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:45:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:45:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:45:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:45:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:45:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:45:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:45:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:45:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:45:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:45:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:45:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:45:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:45:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:45:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:45:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:45:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:45:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:45:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:45:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:45:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:45:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:45:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:45:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:45:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:45:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:45:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:45:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:45:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:45:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:45:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:45:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:45:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:45:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:45:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:45:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:45:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:45:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:45:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:45:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:45:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:45:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:45:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:45:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:45:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:45:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:45:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:45:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:45:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:45:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:45:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:45:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:45:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:45:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:45:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:45:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:45:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:45:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:45:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:45:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:45:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:45:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:45:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:45:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:45:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:45:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:45:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:45:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:45:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:45:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:45:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:45:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:45:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:45:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:45:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:45:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:45:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:45:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:45:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:45:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:45:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:45:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:45:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:45:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:45:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:45:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:45:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:45:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:45:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:45:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:45:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:45:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:45:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:45:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:45:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:45:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:45:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:45:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:45:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:45:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:45:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:45:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:45:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:45:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:45:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:45:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:45:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:45:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:45:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:45:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:45:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:45:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:45:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:45:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:45:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:45:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:45:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:45:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:45:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:45:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:45:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:45:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:45:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:45:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:45:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:45:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:45:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:45:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:45:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:45:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:45:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:45:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:45:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:45:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:45:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:45:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:45:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:45:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:45:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:45:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:45:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:45:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:45:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:45:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:45:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:45:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:45:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:45:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:45:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:45:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:45:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:45:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:45:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:45:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:45:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:45:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:45:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:45:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:45:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:45:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:45:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:45:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:45:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:45:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:45:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:45:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:45:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:45:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:45:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:45:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:45:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:45:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:45:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:45:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:45:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:45:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:45:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:45:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:45:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:45:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:45:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:45:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:45:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:45:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:45:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:45:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:45:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:45:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:45:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:45:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:45:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:45:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:45:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:45:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:45:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:45:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:45:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:45:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:45:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:45:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:45:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:45:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:45:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:45:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:45:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:45:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:45:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:45:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:45:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:45:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:45:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:45:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:45:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:45:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:45:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:45:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:45:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:46:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:46:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:46:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:46:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:46:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:46:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:46:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:46:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:46:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:46:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:46:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:46:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:46:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:46:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:46:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:46:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:46:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:46:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:46:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:46:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:46:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:46:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:46:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:46:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:46:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:46:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:46:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:46:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:46:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:46:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:46:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:46:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:46:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:46:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:46:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:46:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:46:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:46:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:46:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:46:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:46:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:46:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:46:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:46:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:46:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:46:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:46:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:46:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:46:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:46:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:46:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:46:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:46:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:46:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:46:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:46:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:46:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:46:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:46:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:46:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:46:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:46:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:46:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:46:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:46:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:46:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:46:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:46:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:46:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:46:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:46:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:46:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:46:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:46:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:46:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:46:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:46:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:46:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:46:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:46:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:46:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:46:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:46:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:46:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:46:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:46:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:46:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:46:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:46:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:46:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:46:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:46:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:46:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:46:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:46:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:46:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:46:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:46:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:46:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:46:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:46:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:46:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:46:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:46:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:46:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:46:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:46:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:46:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:46:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:46:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:46:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:46:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:46:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:46:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:46:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:46:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:46:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:46:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:46:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:46:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:46:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:46:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:46:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:46:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:46:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:46:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:46:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:46:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:46:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:46:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:46:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:46:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:46:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:46:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:46:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:46:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:46:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:46:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:46:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:46:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:46:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:46:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:46:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:46:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:46:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:46:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:46:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:46:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:46:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:46:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:46:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:46:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:46:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:46:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:46:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:46:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:46:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:46:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:46:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:46:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:46:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:46:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:46:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:46:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:46:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:46:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:46:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:46:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:46:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:46:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:46:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:46:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:46:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:46:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:46:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:46:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:46:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:46:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:46:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:46:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:46:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:46:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:46:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:46:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:46:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:46:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:46:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:46:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:46:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:46:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:46:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:46:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:46:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:46:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:46:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:46:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:46:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:46:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:46:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:46:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:46:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:46:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:46:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:46:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:46:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:46:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:46:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:46:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:46:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:46:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:46:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:46:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:46:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:46:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:46:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:46:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:47:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:47:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:47:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:47:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:47:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:47:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:47:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:47:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:47:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:47:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:47:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:47:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:47:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:47:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:47:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:47:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:47:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:47:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:47:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:47:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:47:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:47:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:47:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:47:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:47:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:47:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:47:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:47:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:47:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:47:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:47:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:47:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:47:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:47:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:47:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:47:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:47:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:47:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:47:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:47:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:47:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:47:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:47:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:47:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:47:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:47:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:47:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:47:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:47:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:47:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:47:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:47:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:47:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:47:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:47:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:47:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:47:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:47:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:47:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:47:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:47:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:47:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:47:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:47:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:47:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:47:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:47:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:47:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:47:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:47:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:47:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:47:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:47:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:47:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:47:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:47:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:47:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:47:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:47:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:47:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:47:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:47:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:47:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:47:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:47:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:47:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:47:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:47:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:47:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:47:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:47:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:47:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:47:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:47:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:47:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:47:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:47:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:47:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:47:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:47:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:47:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:47:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:47:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:47:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:47:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:47:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:47:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:47:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:47:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:47:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:47:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:47:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:47:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:47:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:47:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:47:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:47:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:47:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:47:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:47:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:47:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:47:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:47:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:47:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:47:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:47:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:47:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:47:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:47:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:47:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:47:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:47:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:47:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:47:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:47:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:47:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:47:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:47:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:47:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:47:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:47:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:47:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:47:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:47:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:47:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:47:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:47:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:47:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:47:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:47:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:47:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:47:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:47:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:47:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:47:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:47:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:47:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:47:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:47:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:47:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:47:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:47:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:47:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:47:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:47:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:47:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:47:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:47:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:47:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:47:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:47:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:47:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:47:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:47:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:47:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:47:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:47:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:47:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:47:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:47:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:47:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:47:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:47:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:47:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:47:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:47:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:47:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:47:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:47:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:47:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:47:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:47:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:47:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:47:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:47:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:47:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:47:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:47:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:47:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:47:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:47:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:47:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:47:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:47:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:47:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:47:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:47:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:47:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:47:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:47:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:47:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:47:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:47:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:47:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:47:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:47:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:48:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:48:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:48:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:48:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:48:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:48:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:48:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:48:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:48:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:48:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:48:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:48:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:48:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:48:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:48:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:48:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:48:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:48:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:48:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:48:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:48:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:48:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:48:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:48:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:48:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:48:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:48:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:48:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:48:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:48:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:48:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:48:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:48:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:48:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:48:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:48:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:48:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:48:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:48:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:48:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:48:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:48:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:48:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:48:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:48:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:48:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:48:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:48:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:48:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:48:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:48:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:48:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:48:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:48:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:48:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:48:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:48:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:48:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:48:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:48:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:48:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:48:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:48:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:48:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:48:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:48:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:48:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:48:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:48:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:48:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:48:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:48:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:48:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:48:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:48:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:48:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:48:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:48:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:48:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:48:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:48:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:48:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:48:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:48:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:48:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:48:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:48:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:48:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:48:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:48:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:48:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:48:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:48:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:48:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:48:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:48:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:48:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:48:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:48:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:48:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:48:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:48:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:48:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:48:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:48:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:48:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:48:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:48:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:48:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:48:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:48:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:48:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:48:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:48:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:48:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:48:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:48:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:48:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:48:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:48:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:48:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:48:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:48:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:48:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:48:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:48:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:48:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:48:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:48:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:48:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:48:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:48:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:48:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:48:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:48:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:48:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:48:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:48:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:48:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:48:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:48:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:48:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:48:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:48:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:48:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:48:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:48:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:48:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:48:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:48:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:48:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:48:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:48:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:48:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:48:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:48:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:48:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:48:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:48:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:48:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:48:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:48:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:48:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:48:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:48:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:48:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:48:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:48:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:48:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:48:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:48:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:48:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:48:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:48:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:48:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:48:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:48:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:48:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:48:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:48:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:48:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:48:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:48:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:48:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:48:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:48:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:48:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:48:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:48:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:48:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:48:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:48:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:48:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:48:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:48:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:48:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:48:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:48:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:48:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:48:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:48:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:48:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:48:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:48:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:48:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:48:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:48:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:48:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:48:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:48:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:48:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:48:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:48:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:48:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:48:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:48:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:49:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:49:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:49:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:49:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:49:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:49:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:49:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:49:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:49:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:49:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:49:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:49:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:49:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:49:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:49:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:49:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:49:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:49:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:49:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:49:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:49:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:49:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:49:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:49:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:49:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:49:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:49:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:49:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:49:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:49:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:49:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:49:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:49:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:49:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:49:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:49:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:49:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:49:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:49:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:49:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:49:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:49:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:49:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:49:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:49:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:49:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:49:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:49:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:49:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:49:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:49:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:49:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:49:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:49:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:49:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:49:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:49:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:49:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:49:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:49:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:49:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:49:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:49:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:49:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:49:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:49:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:49:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:49:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:49:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:49:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:49:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:49:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:49:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:49:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:49:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:49:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:49:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:49:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:49:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:49:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:49:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:49:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:49:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:49:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:49:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:49:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:49:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:49:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:49:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:49:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:49:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:49:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:49:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:49:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:49:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:49:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:49:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:49:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:49:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:49:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:49:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:49:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:49:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:49:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:49:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:49:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:49:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:49:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:49:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:49:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:49:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:49:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:49:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:49:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:49:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:49:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:49:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:49:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:49:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:49:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:49:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:49:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:49:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:49:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:49:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:49:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:49:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:49:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:49:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:49:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:49:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:49:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:49:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:49:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:49:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:49:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:49:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:49:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:49:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:49:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:49:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:49:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:49:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:49:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:49:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:49:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:49:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:49:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:49:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:49:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:49:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:49:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:49:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:49:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:49:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:49:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:49:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:49:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:49:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:49:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:49:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:49:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:49:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:49:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:49:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:49:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:49:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:49:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:49:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:49:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:49:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:49:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:49:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:49:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:49:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:49:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:49:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:49:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:49:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:49:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:49:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:49:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:49:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:49:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:49:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:49:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:49:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:49:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:49:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:49:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:49:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:49:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:49:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:49:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:49:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:49:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:49:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:49:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:49:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:49:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:49:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:49:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:49:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:49:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:49:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:49:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:49:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:49:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:49:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:49:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:49:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:49:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:49:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:49:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:49:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:49:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:50:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:50:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:50:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:50:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:50:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:50:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:50:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:50:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:50:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:50:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:50:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:50:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:50:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:50:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:50:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:50:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:50:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:50:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:50:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:50:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:50:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:50:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:50:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:50:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:50:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:50:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:50:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:50:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:50:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:50:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:50:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:50:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:50:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:50:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:50:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:50:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:50:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:50:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:50:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:50:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:50:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:50:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:50:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:50:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:50:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:50:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:50:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:50:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:50:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:50:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:50:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:50:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:50:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:50:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:50:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:50:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:50:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:50:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:50:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:50:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:50:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:50:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:50:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:50:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:50:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:50:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:50:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:50:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:50:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:50:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:50:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:50:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:50:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:50:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:50:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:50:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:50:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:50:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:50:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:50:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:50:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:50:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:50:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:50:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:50:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:50:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:50:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:50:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:50:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:50:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:50:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:50:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:50:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:50:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:50:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:50:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:50:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:50:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:50:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:50:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:50:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:50:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:50:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:50:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:50:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:50:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:50:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:50:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:50:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:50:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:50:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:50:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:50:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:50:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:50:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:50:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:50:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:50:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:50:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:50:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:50:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:50:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:50:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:50:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:50:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:50:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:50:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:50:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:50:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:50:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:50:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:50:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:50:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:50:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:50:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:50:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:50:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:50:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:50:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:50:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:50:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:50:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:50:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:50:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:50:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:50:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:50:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:50:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:50:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:50:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:50:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:50:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:50:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:50:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:50:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:50:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:50:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:50:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:50:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:50:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:50:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:50:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:50:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:50:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:50:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:50:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:50:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:50:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:50:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:50:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:50:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:50:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:50:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:50:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:50:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:50:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:50:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:50:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:50:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:50:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:50:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:50:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:50:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:50:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:50:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:50:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:50:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:50:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:50:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:50:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:50:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:50:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:50:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:50:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:50:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:50:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:50:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:50:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:50:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:50:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:50:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:50:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:50:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:50:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:50:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:50:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:50:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:50:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:50:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:50:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:50:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:50:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:50:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:50:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:50:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:50:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:51:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:51:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:51:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:51:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:51:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:51:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:51:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:51:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:51:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:51:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:51:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:51:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:51:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:51:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:51:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:51:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:51:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:51:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:51:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:51:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:51:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:51:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:51:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:51:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:51:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:51:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:51:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:51:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:51:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:51:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:51:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:51:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:51:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:51:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:51:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:51:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:51:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:51:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:51:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:51:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:51:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:51:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:51:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:51:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:51:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:51:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:51:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:51:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:51:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:51:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:51:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:51:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:51:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:51:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:51:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:51:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:51:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:51:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:51:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:51:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:51:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:51:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:51:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:51:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:51:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:51:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:51:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:51:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:51:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:51:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:51:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:51:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:51:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:51:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:51:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:51:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:51:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:51:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:51:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:51:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:51:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:51:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:51:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:51:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:51:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:51:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:51:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:51:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:51:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:51:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:51:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:51:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:51:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:51:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:51:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:51:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:51:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:51:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:51:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:51:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:51:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:51:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:51:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:51:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:51:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:51:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:51:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:51:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:51:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:51:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:51:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:51:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:51:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:51:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:51:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:51:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:51:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:51:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:51:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:51:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:51:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:51:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:51:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:51:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:51:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:51:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:51:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:51:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:51:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:51:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:51:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:51:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:51:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:51:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:51:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:51:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:51:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:51:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:51:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:51:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:51:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:51:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:51:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:51:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:51:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:51:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:51:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:51:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:51:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:51:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:51:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:51:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:51:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:51:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:51:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:51:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:51:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:51:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:51:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:51:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:51:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:51:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:51:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:51:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:51:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:51:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:51:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:51:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:51:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:51:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:51:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:51:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:51:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:51:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:51:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:51:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:51:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:51:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:51:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:51:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:51:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:51:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:51:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:51:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:51:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:51:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:52:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:52:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:52:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:52:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:52:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:52:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:52:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:52:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:52:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:52:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:52:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:52:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:52:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:52:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:52:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:52:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:52:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:52:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:52:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:52:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:52:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:52:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:52:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:52:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:52:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:52:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:52:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:52:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:52:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:52:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:52:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:52:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:52:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:52:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:52:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:52:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:52:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:52:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:52:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:52:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:52:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:52:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:52:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:52:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:52:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:52:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:52:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:52:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:52:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:52:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:52:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:52:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:52:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:52:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:52:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:52:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:52:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:52:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:52:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:52:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:52:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:52:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:52:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:52:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:52:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:52:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:52:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:52:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:52:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:52:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:52:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:52:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:52:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:52:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:52:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:52:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:52:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:52:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:52:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:52:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:52:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:52:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:52:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:52:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:52:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:52:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:52:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:52:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:52:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:52:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:52:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:52:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:52:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:52:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:52:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:52:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:52:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:52:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:52:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:52:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:52:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:52:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:52:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:52:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:52:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:52:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:52:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:52:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:52:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:52:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:52:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:52:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:52:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:52:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:52:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:52:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:52:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:52:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:52:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:52:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:52:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:52:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:52:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:52:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:52:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:52:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:52:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:52:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:52:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:52:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:52:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:52:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:52:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:52:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:52:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:52:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:52:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:52:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:52:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:52:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:52:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:52:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:52:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:52:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:52:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:52:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:52:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:52:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:52:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:52:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:52:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:52:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:52:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:52:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:52:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:52:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:52:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:52:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:52:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:52:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:52:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:52:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:54:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:54:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:54:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:54:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:54:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:54:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:54:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:54:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:54:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:54:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:54:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:54:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:54:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:54:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:54:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:54:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:54:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:54:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:54:46 rdma06 rsyslogd[1041]: imjournal: 5082 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 07:54:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:54:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:54:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:54:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:54:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:54:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:54:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:54:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:54:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:54:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:54:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:54:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:54:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:54:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:54:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:54:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:54:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:54:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:54:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:54:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:54:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:54:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:54:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:54:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:54:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:54:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:54:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:54:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:54:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:54:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:54:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:54:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:54:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:54:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:54:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:54:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:55:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:55:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:55:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:55:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:55:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:55:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:55:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:55:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:55:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:55:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:55:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:55:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:55:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:55:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:55:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:55:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:55:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:55:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:55:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:55:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:55:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:55:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:55:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:55:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:55:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:55:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:55:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:55:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:55:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:55:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:55:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:55:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:55:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:55:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:55:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:55:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:55:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:55:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:55:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:55:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:55:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:55:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:55:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:55:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:55:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:55:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:55:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:55:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:55:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:55:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:55:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:55:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:55:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:55:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:55:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:55:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:55:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:55:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:55:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:55:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:55:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:55:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:55:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:55:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:55:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:55:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:55:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:55:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:55:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:55:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:55:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:55:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:55:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:55:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:55:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:55:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:55:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:55:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:55:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:55:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:55:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:55:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:55:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:55:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:55:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:55:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:55:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:55:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:55:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:55:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:55:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:55:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:55:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:55:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:55:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:55:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:55:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:55:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:55:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:55:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:55:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:55:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:55:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:55:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:55:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:55:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:55:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:55:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:55:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:55:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:55:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:55:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:55:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:55:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:55:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:55:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:55:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:55:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:55:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:55:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:55:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:55:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:55:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:55:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:55:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:55:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:55:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:55:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:55:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:55:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:55:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:55:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:55:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:55:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:55:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:55:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:55:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:55:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:55:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:55:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:55:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:55:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:55:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:55:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:55:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:55:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:55:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:55:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:55:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:55:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:55:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:55:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:55:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:55:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:55:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:55:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:55:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:55:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:55:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:55:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:55:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:55:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:55:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:55:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:55:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:55:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:55:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:55:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:55:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:55:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:55:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:55:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:55:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:55:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:55:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:55:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:55:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:55:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:55:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:55:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:55:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:55:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:55:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:55:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:55:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:55:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:55:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:55:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:55:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:55:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:55:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:55:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:55:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:55:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:55:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:55:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:55:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:55:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:55:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:55:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:55:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:55:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:55:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:55:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:55:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:55:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:55:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:55:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:55:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:55:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:55:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:55:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:55:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:55:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:55:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:55:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:56:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:56:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:56:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:56:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:56:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:56:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:56:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:56:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:56:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:56:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:56:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:56:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:56:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:56:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:56:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:56:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:56:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:56:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:56:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:56:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:56:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:56:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:56:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:56:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:56:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:56:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:56:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:56:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:56:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:56:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:56:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:56:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:56:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:56:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:56:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:56:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:56:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:56:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:56:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:56:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:56:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:56:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:56:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:56:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:56:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:56:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:56:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:56:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:56:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:56:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:56:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:56:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:56:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:56:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:56:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:56:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:56:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:56:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:56:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:56:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:56:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:56:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:56:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:56:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:56:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:56:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:56:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:56:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:56:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:56:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:56:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:56:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:56:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:56:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:56:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:56:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:56:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:56:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:56:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:56:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:56:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:56:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:56:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:56:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:56:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:56:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:56:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:56:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:56:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:56:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:56:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:56:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:56:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:56:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:56:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:56:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:56:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:56:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:56:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:56:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:56:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:56:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:56:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:56:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:56:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:56:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:56:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:56:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:56:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:56:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:56:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:56:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:56:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:56:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:56:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:56:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:56:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:56:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:56:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:56:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:56:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:56:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:56:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:56:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:56:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:56:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:56:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:56:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:56:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:56:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:56:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:56:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:56:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:56:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:56:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:56:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:56:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:56:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:56:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:56:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:56:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:56:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:56:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:56:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:56:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:56:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:56:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:56:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:56:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:56:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:56:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:56:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:56:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:56:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:56:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:56:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:56:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:56:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:56:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:56:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:56:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:56:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:56:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:56:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:56:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:56:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:56:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:56:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:56:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:56:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:56:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:56:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:56:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:56:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:56:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:56:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:56:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:56:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:56:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:56:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:56:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:56:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:56:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:56:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:56:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:56:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:56:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:56:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:56:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:56:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:56:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:56:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:56:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:56:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:56:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:56:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:56:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:56:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:56:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:56:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:56:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:56:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:56:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:56:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:56:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:56:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:56:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:56:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:56:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:56:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:56:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:56:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:56:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:56:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:56:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:56:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:57:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:57:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:57:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:57:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:57:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:57:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:57:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:57:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:57:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:57:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:57:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:57:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:57:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:57:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:57:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:57:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:57:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:57:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:57:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:57:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:57:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:57:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:57:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:57:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:57:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:57:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:57:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:57:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:57:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:57:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:57:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:57:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:57:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:57:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:57:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:57:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:57:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:57:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:57:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:57:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:57:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:57:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:57:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:57:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:57:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:57:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:57:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:57:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:57:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:57:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:57:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:57:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:57:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:57:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:57:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:57:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:57:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:57:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:57:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:57:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:57:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:57:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:57:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:57:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:57:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:57:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:57:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:57:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:57:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:57:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:57:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:57:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:57:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:57:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:57:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:57:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:57:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:57:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:57:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:57:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:57:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:57:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:57:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:57:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:57:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:57:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:57:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:57:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:57:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:57:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:57:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:57:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:57:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:57:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:57:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:57:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:57:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:57:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:57:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:57:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:57:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:57:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:57:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:57:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:57:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:57:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:57:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:57:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:57:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:57:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:57:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:57:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:57:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:57:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:57:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:57:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:57:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:57:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:57:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:57:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:57:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:57:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:57:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:57:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:57:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:57:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:57:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:57:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:57:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:57:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:57:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:57:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:57:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:57:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:57:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:57:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:57:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:57:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:57:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:57:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:57:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:57:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:57:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:57:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:57:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:57:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:57:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:57:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:57:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:57:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:57:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:57:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:57:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:57:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:57:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:57:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:57:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:57:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:57:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:57:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:57:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:57:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:57:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:57:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:57:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:57:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:57:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:57:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:57:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:57:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:57:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:57:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:57:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:57:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:57:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:57:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:57:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:57:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:57:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:57:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:57:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:57:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:57:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:57:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:57:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:57:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:57:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:57:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:57:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:57:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:57:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:57:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:57:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:57:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:57:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:57:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:57:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:57:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:57:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:57:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:57:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:57:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:57:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:57:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:57:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:57:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:57:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:57:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:57:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:57:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:57:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:57:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:57:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:57:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:57:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:57:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:58:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:58:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:58:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:58:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:58:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:58:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:58:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:58:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:58:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:58:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:58:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:58:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:58:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:58:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:58:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:58:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:58:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:58:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:58:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:58:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:58:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:58:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:58:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:58:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:58:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:58:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:58:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:58:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:58:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:58:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:58:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:58:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:58:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:58:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:58:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:58:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:58:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:58:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:58:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:58:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:58:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:58:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:58:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:58:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:58:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:58:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:58:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:58:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:58:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:58:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:58:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:58:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:58:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:58:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:58:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:58:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:58:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:58:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:58:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:58:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:58:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:58:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:58:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:58:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:58:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:58:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:58:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:58:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:58:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:58:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:58:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:58:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:58:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:58:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:58:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:58:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:58:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:58:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:58:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:58:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:58:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:58:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:58:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:58:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:58:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:58:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:58:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:58:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:58:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:58:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:58:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:58:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:58:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:58:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:58:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:58:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:58:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:58:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:58:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:58:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:58:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:58:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:58:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:58:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:58:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:58:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:58:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:58:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:58:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:58:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:58:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:58:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:58:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:58:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:58:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:58:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:58:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:58:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:58:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:58:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:58:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:58:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:58:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:58:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:58:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:58:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:58:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:58:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:58:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:58:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:58:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:58:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:58:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:58:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:58:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:58:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:58:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:58:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:58:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:58:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:58:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:58:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:58:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:58:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:58:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:58:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:58:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:58:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:58:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:58:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:58:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:58:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:58:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:58:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:58:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:58:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:58:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:58:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:58:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:58:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:58:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:58:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:58:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:58:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:58:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:58:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:58:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:58:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:58:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:58:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:58:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:58:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:58:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:58:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:58:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:58:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:58:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:58:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:58:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:58:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:58:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:58:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:58:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:58:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:58:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:58:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:58:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:58:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:58:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:58:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:58:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:58:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:58:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:58:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:58:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:58:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:58:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:58:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:58:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:58:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:58:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:58:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:58:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:58:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:58:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:58:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:58:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:58:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:58:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:58:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:58:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:58:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:58:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:58:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:58:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:58:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:59:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:59:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:59:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:59:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:59:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:59:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:59:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:59:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:59:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:59:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:59:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:59:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:59:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:59:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:59:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:59:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:59:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:59:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:59:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:59:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:59:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:59:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:59:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:59:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:59:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:59:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:59:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:59:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:59:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:59:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:59:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:59:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:59:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:59:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:59:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:59:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:59:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:59:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:59:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:59:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:59:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:59:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:59:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:59:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:59:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:59:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:59:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:59:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:59:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:59:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:59:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:59:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:59:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:59:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:59:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:59:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:59:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:59:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:59:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:59:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:59:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:59:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:59:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:59:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:59:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:59:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:59:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:59:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:59:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:59:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:59:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:59:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:59:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:59:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:59:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:59:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:59:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:59:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:59:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:59:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:59:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:59:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:59:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:59:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:59:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:59:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:59:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:59:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:59:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:59:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:59:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:59:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:59:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:59:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:59:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:59:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:59:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:59:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:59:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:59:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:59:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:59:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:59:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:59:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:59:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:59:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:59:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:59:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:59:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:59:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:59:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:59:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:59:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:59:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:59:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:59:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:59:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:59:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:59:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:59:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:59:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:59:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:59:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:59:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:59:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:59:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:59:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:59:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:59:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:59:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:59:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:59:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:59:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:59:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:59:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:59:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:59:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:59:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:59:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:59:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:59:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:59:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:59:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:59:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:59:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:59:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:59:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:59:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:59:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:59:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:59:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:59:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:59:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:59:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:59:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:59:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:59:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:59:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:59:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:59:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:59:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:59:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:59:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:59:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:59:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:59:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:59:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:59:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:59:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:59:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:59:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:59:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:59:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:59:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:59:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:59:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:59:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:59:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:59:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:59:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:59:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:59:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:59:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:59:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:59:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:59:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:59:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:59:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:59:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:59:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:59:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:59:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:59:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:59:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:59:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:59:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:59:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:59:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 07:59:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 07:59:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:59:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:59:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:59:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 07:59:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 07:59:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 07:59:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 07:59:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 07:59:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 07:59:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 07:59:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 07:59:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 07:59:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 07:59:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 07:59:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 07:59:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 07:59:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:00:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:00:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:00:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:00:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:00:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:00:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:00:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:00:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:00:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:00:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:00:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:00:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:00:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:00:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:00:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:00:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:00:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:00:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:00:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:00:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:00:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:00:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:00:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:00:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:00:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:00:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:00:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:00:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:00:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:00:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:00:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:00:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:00:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:00:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:00:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:00:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:00:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:00:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:00:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:00:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:00:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:00:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:00:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:00:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:00:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:00:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:00:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:00:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:00:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:00:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:00:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:00:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:00:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:00:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:00:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:00:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:00:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:00:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:00:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:00:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:00:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:00:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:00:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:00:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:00:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:00:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:00:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:00:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:00:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:00:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:00:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:00:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:00:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:00:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:00:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:00:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:00:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:00:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:00:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:00:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:00:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:00:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:00:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:00:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:00:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:00:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:00:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:00:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:00:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:00:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:00:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:00:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:00:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:00:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:00:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:00:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:00:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:00:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:00:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:00:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:00:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:00:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:00:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:00:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:00:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:00:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:00:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:00:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:00:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:00:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:00:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:00:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:00:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:00:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:00:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:00:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:00:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:00:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:00:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:00:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:00:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:00:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:00:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:00:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:00:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:00:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:00:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:00:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:00:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:00:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:00:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:00:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:00:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:00:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:00:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:00:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:00:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:00:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:00:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:00:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:00:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:00:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:00:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:00:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:00:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:00:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:00:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:00:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:00:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:00:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:00:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:00:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:00:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:00:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:00:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:00:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:00:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:00:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:00:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:00:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:00:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:00:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:00:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:00:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:00:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:00:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:00:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:00:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:00:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:00:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:00:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:00:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:00:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:00:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:00:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:00:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:00:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:00:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:00:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:00:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:00:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:00:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:00:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:00:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:00:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:00:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:00:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:00:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:00:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:00:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:00:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:00:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:00:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:00:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:00:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:00:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:00:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:00:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:00:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:00:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:00:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:00:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:00:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:00:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:00:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:00:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:00:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:00:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:00:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:00:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:00:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:00:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:00:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:00:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:00:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:00:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:01:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:01:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:01:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:01:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:01:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:01:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:01:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:01:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:01:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:01:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:01:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:01:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:01:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:01:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:01:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:01:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:01:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:01:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:01:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:01:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:01:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:01:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:01:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:01:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:01:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:01:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:01:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:01:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:01:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:01:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:01:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:01:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:01:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:01:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:01:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:01:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:01:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:01:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:01:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:01:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:01:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:01:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:01:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:01:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:01:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:01:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:01:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:01:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:01:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:01:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:01:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:01:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:01:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:01:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:01:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:01:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:01:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:01:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:01:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:01:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:01:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:01:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:01:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:01:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:01:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:01:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:01:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:01:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:01:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:01:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:01:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:01:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:01:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:01:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:01:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:01:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:01:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:01:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:01:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:01:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:01:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:01:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:01:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:01:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:01:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:01:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:01:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:01:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:01:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:01:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:01:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:01:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:01:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:01:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:01:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:01:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:01:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:01:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:01:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:01:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:01:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:01:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:01:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:01:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:01:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:01:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:01:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:01:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:01:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:01:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:01:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:01:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:01:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:01:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:01:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:01:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:01:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:01:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:01:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:01:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:01:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:01:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:01:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:01:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:01:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:01:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:01:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:01:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:01:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:01:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:01:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:01:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:01:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:01:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:01:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:01:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:01:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:01:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:01:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:01:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:01:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:01:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:01:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:01:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:01:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:01:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:01:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:01:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:01:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:01:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:01:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:01:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:01:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:01:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:01:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:01:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:02:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:02:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:02:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:02:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:02:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:02:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:02:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:02:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:02:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:02:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:02:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:02:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:02:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:02:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:02:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:02:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:02:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:02:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:02:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:02:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:02:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:02:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:02:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:02:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:02:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:02:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:02:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:02:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:02:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:02:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:02:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:02:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:02:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:02:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:02:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:02:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:02:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:02:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:02:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:02:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:02:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:02:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:02:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:02:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:02:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:02:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:02:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:02:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:02:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:02:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:02:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:02:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:02:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:02:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:02:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:02:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:02:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:02:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:02:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:02:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:02:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:02:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:02:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:02:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:02:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:02:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:02:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:02:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:02:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:02:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:02:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:02:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:02:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:02:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:02:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:02:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:02:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:02:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:02:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:02:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:02:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:02:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:02:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:02:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:02:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:02:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:02:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:02:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:02:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:02:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:02:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:02:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:02:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:02:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:02:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:02:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:02:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:02:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:02:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:02:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:02:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:02:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:02:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:02:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:02:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:02:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:02:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:02:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:02:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:02:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:02:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:02:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:02:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:02:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:02:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:02:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:02:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:02:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:02:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:02:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:02:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:02:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:02:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:02:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:02:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:02:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:02:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:02:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:02:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:02:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:02:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:02:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:02:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:02:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:02:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:02:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:02:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:02:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:02:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:02:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:02:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:02:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:02:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:02:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:02:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:02:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:02:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:02:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:02:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:02:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:02:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:02:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:02:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:02:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:02:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:02:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:02:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:02:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:02:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:02:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:02:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:02:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:04:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:04:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:04:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:04:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:04:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:04:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:04:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:04:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:04:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:04:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:04:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:04:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:04:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:04:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:04:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:04:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:04:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:04:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:04:51 rdma06 rsyslogd[1041]: imjournal: 5295 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 08:04:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:04:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:04:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:04:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:04:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:04:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:04:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:04:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:04:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:04:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:04:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:04:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:04:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:04:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:04:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:04:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:04:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:04:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:05:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:05:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:05:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:05:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:05:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:05:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:05:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:05:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:05:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:05:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:05:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:05:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:05:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:05:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:05:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:05:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:05:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:05:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:05:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:05:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:05:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:05:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:05:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:05:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:05:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:05:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:05:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:05:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:05:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:05:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:05:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:05:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:05:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:05:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:05:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:05:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:05:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:05:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:05:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:05:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:05:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:05:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:05:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:05:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:05:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:05:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:05:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:05:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:05:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:05:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:05:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:05:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:05:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:05:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:05:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:05:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:05:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:05:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:05:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:05:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:05:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:05:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:05:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:05:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:05:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:05:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:05:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:05:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:05:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:05:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:05:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:05:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:05:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:05:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:05:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:05:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:05:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:05:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:05:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:05:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:05:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:05:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:05:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:05:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:05:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:05:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:05:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:05:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:05:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:05:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:05:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:05:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:05:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:05:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:05:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:05:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:05:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:05:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:05:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:05:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:05:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:05:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:05:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:05:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:05:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:05:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:05:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:05:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:05:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:05:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:05:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:05:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:05:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:05:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:05:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:05:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:05:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:05:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:05:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:05:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:05:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:05:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:05:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:05:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:05:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:05:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:05:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:05:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:05:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:05:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:05:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:05:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:05:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:05:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:05:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:05:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:05:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:05:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:05:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:05:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:05:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:05:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:05:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:05:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:05:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:05:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:05:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:05:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:05:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:05:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:05:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:05:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:05:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:05:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:05:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:05:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:05:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:05:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:05:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:05:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:05:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:05:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:05:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:05:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:05:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:05:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:05:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:05:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:05:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:05:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:05:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:05:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:05:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:05:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:05:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:05:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:05:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:05:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:05:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:05:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:05:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:05:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:05:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:05:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:05:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:05:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:05:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:05:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:05:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:05:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:05:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:05:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:05:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:05:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:05:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:05:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:05:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:05:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:05:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:05:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:05:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:05:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:05:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:05:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:05:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:05:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:05:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:05:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:05:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:05:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:05:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:05:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:05:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:05:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:05:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:05:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:06:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:06:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:06:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:06:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:06:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:06:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:06:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:06:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:06:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:06:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:06:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:06:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:06:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:06:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:06:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:06:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:06:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:06:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:06:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:06:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:06:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:06:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:06:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:06:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:06:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:06:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:06:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:06:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:06:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:06:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:06:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:06:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:06:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:06:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:06:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:06:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:06:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:06:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:06:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:06:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:06:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:06:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:06:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:06:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:06:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:06:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:06:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:06:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:06:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:06:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:06:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:06:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:06:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:06:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:06:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:06:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:06:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:06:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:06:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:06:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:06:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:06:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:06:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:06:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:06:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:06:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:06:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:06:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:06:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:06:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:06:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:06:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:06:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:06:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:06:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:06:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:06:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:06:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:06:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:06:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:06:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:06:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:06:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:06:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:06:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:06:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:06:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:06:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:06:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:06:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:06:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:06:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:06:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:06:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:06:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:06:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:06:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:06:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:06:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:06:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:06:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:06:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:06:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:06:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:06:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:06:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:06:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:06:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:06:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:06:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:06:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:06:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:06:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:06:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:06:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:06:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:06:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:06:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:06:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:06:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:06:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:06:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:06:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:06:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:06:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:06:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:06:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:06:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:06:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:06:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:06:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:06:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:06:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:06:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:06:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:06:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:06:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:06:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:06:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:06:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:06:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:06:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:06:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:06:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:06:36 rdma06 systemd-journald[790]: Data hash table of /run/log/journal/95379d2f723847b68d9e976dee8967a9/system.journal has a fill level at 75.0 (163750 of 218332 items, 75497472 file size, 461 bytes per hash table item), suggesting rotation.
Apr 17 08:06:36 rdma06 systemd-journald[790]: /run/log/journal/95379d2f723847b68d9e976dee8967a9/system.journal: Journal header limits reached or header out-of-date, rotating.
Apr 17 08:06:36 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 17 08:06:36 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 17 08:06:36 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 17 08:06:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:06:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:06:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:06:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:06:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:06:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:06:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:06:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:06:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:06:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:06:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:06:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:06:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:06:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:06:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:06:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:06:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:06:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:06:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:06:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:06:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:06:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:06:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:06:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:06:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:06:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:06:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:06:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:06:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:06:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:06:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:06:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:06:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:06:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:06:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:06:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:06:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:06:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:06:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:06:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:06:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:06:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:06:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:06:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:06:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:06:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:06:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:06:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:06:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:06:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:06:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:06:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:06:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:06:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:06:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:06:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:06:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:06:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:06:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:06:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:06:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:06:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:06:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:06:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:06:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:06:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:06:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:06:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:06:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:06:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:06:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:06:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:07:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:07:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:07:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:07:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:07:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:07:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:07:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:07:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:07:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:07:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:07:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:07:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:07:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:07:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:07:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:07:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:07:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:07:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:07:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:07:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:07:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:07:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:07:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:07:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:07:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:07:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:07:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:07:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:07:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:07:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:07:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:07:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:07:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:07:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:07:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:07:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:07:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:07:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:07:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:07:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:07:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:07:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:07:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:07:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:07:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:07:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:07:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:07:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:07:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:07:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:07:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:07:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:07:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:07:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:07:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:07:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:07:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:07:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:07:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:07:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:07:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:07:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:07:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:07:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:07:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:07:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:07:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:07:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:07:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:07:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:07:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:07:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:07:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:07:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:07:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:07:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:07:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:07:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:07:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:07:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:07:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:07:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:07:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:07:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:07:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:07:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:07:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:07:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:07:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:07:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:07:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:07:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:07:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:07:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:07:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:07:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:07:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:07:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:07:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:07:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:07:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:07:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:07:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:07:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:07:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:07:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:07:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:07:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:07:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:07:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:07:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:07:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:07:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:07:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:07:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:07:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:07:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:07:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:07:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:07:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:07:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:07:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:07:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:07:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:07:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:07:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:07:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:07:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:07:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:07:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:07:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:07:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:07:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:07:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:07:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:07:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:07:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:07:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:07:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:07:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:07:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:07:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:07:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:07:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:07:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:07:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:07:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:07:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:07:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:07:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:07:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:07:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:07:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:07:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:07:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:07:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:07:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:07:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:07:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:07:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:07:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:07:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:07:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:07:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:07:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:07:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:07:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:07:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:07:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:07:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:07:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:07:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:07:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:07:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:07:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:07:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:07:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:07:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:07:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:07:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:07:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:07:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:07:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:07:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:07:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:07:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:07:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:07:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:07:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:07:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:07:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:07:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:07:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:07:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:07:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:07:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:07:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:07:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:07:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:07:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:07:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:07:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:07:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:07:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:07:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:07:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:07:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:07:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:07:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:07:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:07:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:07:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:07:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:07:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:07:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:07:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:08:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:08:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:08:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:08:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:08:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:08:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:08:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:08:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:08:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:08:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:08:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:08:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:08:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:08:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:08:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:08:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:08:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:08:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:08:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:08:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:08:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:08:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:08:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:08:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:08:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:08:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:08:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:08:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:08:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:08:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:08:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:08:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:08:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:08:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:08:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:08:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:08:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:08:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:08:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:08:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:08:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:08:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:08:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:08:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:08:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:08:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:08:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:08:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:08:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:08:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:08:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:08:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:08:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:08:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:08:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:08:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:08:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:08:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:08:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:08:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:08:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:08:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:08:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:08:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:08:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:08:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:08:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:08:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:08:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:08:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:08:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:08:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:08:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:08:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:08:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:08:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:08:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:08:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:08:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:08:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:08:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:08:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:08:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:08:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:08:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:08:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:08:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:08:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:08:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:08:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:08:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:08:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:08:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:08:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:08:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:08:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:08:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:08:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:08:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:08:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:08:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:08:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:08:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:08:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:08:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:08:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:08:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:08:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:08:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:08:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:08:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:08:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:08:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:08:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:08:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:08:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:08:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:08:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:08:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:08:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:08:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:08:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:08:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:08:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:08:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:08:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:08:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:08:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:08:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:08:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:08:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:08:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:08:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:08:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:08:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:08:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:08:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:08:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:08:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:08:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:08:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:08:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:08:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:08:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:08:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:08:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:08:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:08:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:08:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:08:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:08:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:08:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:08:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:08:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:08:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:08:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:08:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:08:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:08:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:08:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:08:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:08:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:08:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:08:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:08:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:08:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:08:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:08:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:08:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:08:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:08:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:08:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:08:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:08:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:08:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:08:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:08:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:08:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:08:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:08:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:08:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:08:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:08:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:08:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:08:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:08:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:08:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:08:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:08:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:08:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:08:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:08:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:08:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:08:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:08:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:08:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:08:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:08:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:08:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:08:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:08:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:08:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:08:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:08:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:08:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:08:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:08:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:08:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:08:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:08:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:08:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:08:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:08:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:08:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:08:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:08:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:09:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:09:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:09:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:09:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:09:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:09:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:09:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:09:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:09:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:09:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:09:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:09:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:09:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:09:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:09:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:09:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:09:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:09:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:09:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:09:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:09:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:09:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:09:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:09:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:09:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:09:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:09:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:09:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:09:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:09:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:09:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:09:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:09:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:09:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:09:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:09:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:09:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:09:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:09:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:09:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:09:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:09:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:09:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:09:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:09:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:09:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:09:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:09:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:09:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:09:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:09:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:09:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:09:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:09:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:09:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:09:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:09:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:09:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:09:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:09:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:09:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:09:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:09:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:09:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:09:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:09:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:09:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:09:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:09:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:09:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:09:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:09:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:09:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:09:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:09:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:09:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:09:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:09:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:09:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:09:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:09:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:09:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:09:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:09:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:09:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:09:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:09:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:09:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:09:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:09:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:09:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:09:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:09:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:09:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:09:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:09:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:09:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:09:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:09:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:09:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:09:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:09:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:09:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:09:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:09:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:09:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:09:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:09:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:09:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:09:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:09:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:09:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:09:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:09:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:09:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:09:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:09:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:09:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:09:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:09:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:09:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:09:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:09:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:09:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:09:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:09:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:09:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:09:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:09:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:09:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:09:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:09:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:09:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:09:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:09:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:09:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:09:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:09:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:09:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:09:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:09:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:09:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:09:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:09:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:09:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:09:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:09:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:09:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:09:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:09:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:09:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:09:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:09:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:09:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:09:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:09:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:09:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:09:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:09:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:09:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:09:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:09:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:09:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:09:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:09:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:09:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:09:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:09:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:09:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:09:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:09:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:09:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:09:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:09:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:09:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:09:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:09:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:09:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:09:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:09:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:09:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:09:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:09:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:09:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:09:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:09:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:09:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:09:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:09:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:09:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:09:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:09:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:09:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:09:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:09:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:09:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:09:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:09:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:09:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:09:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:09:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:09:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:09:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:09:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:09:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:09:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:09:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:09:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:09:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:09:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:09:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:09:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:09:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:09:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:09:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:09:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:10:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:10:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:10:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:10:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:10:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:10:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:10:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:10:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:10:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:10:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:10:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:10:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:10:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:10:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:10:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:10:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:10:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:10:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:10:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:10:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:10:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:10:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:10:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:10:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:10:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:10:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:10:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:10:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:10:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:10:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:10:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:10:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:10:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:10:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:10:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:10:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:10:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:10:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:10:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:10:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:10:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:10:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:10:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:10:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:10:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:10:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:10:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:10:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:10:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:10:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:10:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:10:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:10:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:10:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:10:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:10:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:10:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:10:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:10:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:10:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:10:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:10:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:10:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:10:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:10:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:10:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:10:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:10:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:10:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:10:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:10:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:10:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:10:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:10:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:10:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:10:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:10:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:10:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:10:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:10:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:10:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:10:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:10:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:10:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:10:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:10:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:10:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:10:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:10:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:10:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:10:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:10:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:10:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:10:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:10:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:10:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:10:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:10:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:10:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:10:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:10:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:10:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:10:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:10:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:10:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:10:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:10:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:10:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:10:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:10:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:10:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:10:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:10:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:10:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:10:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:10:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:10:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:10:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:10:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:10:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:10:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:10:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:10:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:10:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:10:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:10:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:10:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:10:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:10:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:10:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:10:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:10:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:10:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:10:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:10:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:10:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:10:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:10:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:10:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:10:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:10:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:10:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:10:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:10:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:10:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:10:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:10:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:10:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:10:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:10:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:10:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:10:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:10:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:10:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:10:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:10:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:10:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:10:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:10:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:10:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:10:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:10:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:10:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:10:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:10:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:10:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:10:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:10:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:10:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:10:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:10:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:10:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:10:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:10:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:10:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:10:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:10:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:10:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:10:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:10:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:10:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:10:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:10:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:10:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:10:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:10:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:10:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:10:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:10:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:10:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:10:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:10:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:10:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:10:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:10:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:10:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:10:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:10:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:10:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:10:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:10:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:10:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:10:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:10:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:10:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:10:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:10:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:10:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:10:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:10:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:10:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:10:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:10:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:10:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:10:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:10:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:11:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:11:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:11:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:11:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:11:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:11:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:11:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:11:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:11:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:11:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:11:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:11:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:11:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:11:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:11:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:11:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:11:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:11:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:11:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:11:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:11:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:11:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:11:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:11:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:11:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:11:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:11:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:11:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:11:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:11:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:11:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:11:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:11:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:11:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:11:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:11:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:11:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:11:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:11:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:11:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:11:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:11:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:11:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:11:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:11:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:11:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:11:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:11:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:11:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:11:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:11:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:11:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:11:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:11:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:11:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:11:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:11:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:11:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:11:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:11:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:11:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:11:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:11:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:11:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:11:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:11:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:11:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:11:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:11:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:11:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:11:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:11:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:11:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:11:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:11:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:11:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:11:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:11:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:11:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:11:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:11:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:11:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:11:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:11:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:11:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:11:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:11:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:11:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:11:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:11:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:11:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:11:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:11:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:11:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:11:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:11:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:11:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:11:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:11:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:11:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:11:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:11:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:11:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:11:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:11:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:11:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:11:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:11:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:11:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:11:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:11:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:11:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:11:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:11:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:11:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:11:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:11:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:11:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:11:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:11:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:11:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:11:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:11:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:11:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:11:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:11:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:11:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:11:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:11:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:11:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:11:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:11:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:11:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:11:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:11:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:11:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:11:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:11:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:11:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:11:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:11:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:11:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:11:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:11:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:11:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:11:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:11:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:11:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:11:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:11:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:11:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:11:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:11:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:11:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:11:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:11:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:11:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:11:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:11:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:11:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:11:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:11:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:11:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:11:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:11:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:11:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:11:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:11:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:11:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:11:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:11:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:11:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:11:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:11:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:11:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:11:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:11:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:11:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:11:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:11:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:11:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:11:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:11:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:11:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:11:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:11:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:11:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:11:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:11:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:11:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:11:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:11:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:11:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:11:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:11:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:11:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:11:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:11:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:11:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:11:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:11:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:11:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:11:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:11:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:11:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:11:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:11:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:11:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:11:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:11:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:11:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:11:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:11:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:11:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:11:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:11:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:12:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:12:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:12:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:12:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:12:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:12:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:12:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:12:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:12:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:12:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:12:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:12:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:12:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:12:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:12:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:12:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:12:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:12:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:12:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:12:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:12:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:12:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:12:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:12:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:12:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:12:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:12:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:12:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:12:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:12:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:12:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:12:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:12:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:12:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:12:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:12:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:12:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:12:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:12:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:12:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:12:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:12:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:12:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:12:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:12:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:12:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:12:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:12:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:12:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:12:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:12:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:12:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:12:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:12:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:12:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:12:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:12:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:12:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:12:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:12:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:12:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:12:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:12:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:12:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:12:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:12:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:12:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:12:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:12:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:12:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:12:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:12:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:12:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:12:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:12:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:12:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:12:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:12:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:12:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:12:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:12:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:12:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:12:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:12:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:12:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:12:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:12:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:12:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:12:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:12:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:12:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:12:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:12:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:12:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:12:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:12:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:12:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:12:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:12:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:12:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:12:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:12:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:12:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:12:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:12:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:12:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:12:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:12:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:12:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:12:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:12:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:12:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:12:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:12:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:12:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:12:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:12:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:12:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:12:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:12:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:12:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:12:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:12:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:12:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:12:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:12:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:12:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:12:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:12:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:12:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:12:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:12:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:12:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:12:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:12:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:12:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:12:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:12:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:12:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:12:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:12:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:12:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:12:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:12:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:12:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:12:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:12:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:12:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:12:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:12:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:12:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:12:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:12:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:12:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:12:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:12:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:12:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:12:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:12:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:12:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:12:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:12:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:12:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:12:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:12:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:12:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:12:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:12:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:12:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:12:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:12:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:12:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:12:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:12:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:12:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:12:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:12:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:12:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:12:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:12:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:14:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:14:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:14:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:14:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:14:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:14:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:14:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:14:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:14:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:14:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:14:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:14:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:14:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:14:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:14:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:14:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:14:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:14:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:14:56 rdma06 rsyslogd[1041]: imjournal: 5296 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 08:15:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:15:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:15:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:15:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:15:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:15:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:15:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:15:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:15:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:15:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:15:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:15:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:15:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:15:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:15:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:15:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:15:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:15:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:15:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:15:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:15:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:15:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:15:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:15:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:15:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:15:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:15:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:15:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:15:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:15:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:15:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:15:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:15:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:15:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:15:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:15:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:15:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:15:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:15:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:15:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:15:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:15:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:15:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:15:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:15:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:15:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:15:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:15:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:15:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:15:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:15:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:15:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:15:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:15:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:15:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:15:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:15:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:15:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:15:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:15:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:15:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:15:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:15:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:15:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:15:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:15:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:15:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:15:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:15:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:15:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:15:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:15:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:15:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:15:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:15:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:15:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:15:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:15:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:15:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:15:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:15:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:15:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:15:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:15:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:15:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:15:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:15:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:15:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:15:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:15:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:15:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:15:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:15:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:15:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:15:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:15:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:15:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:15:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:15:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:15:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:15:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:15:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:15:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:15:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:15:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:15:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:15:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:15:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:15:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:15:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:15:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:15:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:15:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:15:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:15:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:15:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:15:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:15:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:15:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:15:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:15:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:15:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:15:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:15:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:15:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:15:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:15:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:15:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:15:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:15:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:15:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:15:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:15:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:15:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:15:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:15:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:15:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:15:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:15:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:15:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:15:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:15:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:15:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:15:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:15:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:15:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:15:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:15:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:15:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:15:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:15:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:15:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:15:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:15:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:15:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:15:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:15:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:15:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:15:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:15:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:15:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:15:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:15:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:15:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:15:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:15:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:15:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:15:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:15:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:15:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:15:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:15:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:15:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:15:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:15:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:15:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:15:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:15:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:15:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:15:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:15:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:15:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:15:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:15:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:15:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:15:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:15:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:15:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:15:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:15:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:15:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:15:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:15:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:15:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:15:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:15:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:15:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:15:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:15:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:15:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:15:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:15:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:15:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:15:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:15:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:15:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:15:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:15:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:15:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:15:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:15:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:15:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:15:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:15:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:15:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:15:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:16:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:16:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:16:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:16:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:16:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:16:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:16:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:16:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:16:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:16:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:16:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:16:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:16:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:16:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:16:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:16:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:16:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:16:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:16:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:16:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:16:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:16:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:16:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:16:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:16:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:16:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:16:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:16:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:16:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:16:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:16:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:16:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:16:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:16:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:16:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:16:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:16:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:16:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:16:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:16:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:16:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:16:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:16:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:16:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:16:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:16:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:16:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:16:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:16:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:16:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:16:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:16:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:16:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:16:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:16:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:16:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:16:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:16:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:16:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:16:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:16:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:16:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:16:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:16:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:16:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:16:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:16:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:16:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:16:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:16:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:16:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:16:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:16:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:16:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:16:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:16:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:16:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:16:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:16:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:16:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:16:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:16:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:16:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:16:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:16:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:16:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:16:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:16:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:16:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:16:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:16:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:16:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:16:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:16:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:16:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:16:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:16:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:16:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:16:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:16:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:16:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:16:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:16:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:16:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:16:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:16:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:16:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:16:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:16:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:16:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:16:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:16:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:16:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:16:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:16:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:16:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:16:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:16:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:16:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:16:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:16:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:16:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:16:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:16:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:16:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:16:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:16:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:16:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:16:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:16:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:16:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:16:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:16:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:16:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:16:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:16:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:16:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:16:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:16:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:16:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:16:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:16:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:16:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:16:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:16:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:16:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:16:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:16:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:16:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:16:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:16:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:16:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:16:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:16:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:16:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:16:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:16:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:16:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:16:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:16:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:16:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:16:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:16:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:16:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:16:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:16:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:16:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:16:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:16:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:16:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:16:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:16:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:16:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:16:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:16:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:16:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:16:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:16:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:16:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:16:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:16:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:16:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:16:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:16:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:16:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:16:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:16:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:16:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:16:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:16:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:16:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:16:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:16:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:16:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:16:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:16:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:16:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:16:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:16:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:16:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:16:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:16:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:16:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:16:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:16:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:16:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:16:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:16:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:16:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:16:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:16:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:16:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:16:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:16:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:16:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:16:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:17:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:17:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:17:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:17:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:17:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:17:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:17:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:17:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:17:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:17:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:17:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:17:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:17:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:17:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:17:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:17:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:17:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:17:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:17:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:17:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:17:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:17:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:17:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:17:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:17:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:17:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:17:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:17:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:17:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:17:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:17:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:17:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:17:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:17:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:17:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:17:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:17:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:17:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:17:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:17:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:17:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:17:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:17:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:17:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:17:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:17:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:17:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:17:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:17:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:17:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:17:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:17:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:17:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:17:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:17:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:17:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:17:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:17:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:17:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:17:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:17:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:17:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:17:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:17:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:17:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:17:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:17:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:17:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:17:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:17:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:17:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:17:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:17:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:17:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:17:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:17:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:17:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:17:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:17:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:17:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:17:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:17:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:17:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:17:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:17:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:17:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:17:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:17:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:17:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:17:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:17:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:17:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:17:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:17:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:17:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:17:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:17:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:17:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:17:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:17:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:17:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:17:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:17:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:17:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:17:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:17:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:17:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:17:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:17:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:17:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:17:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:17:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:17:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:17:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:17:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:17:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:17:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:17:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:17:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:17:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:17:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:17:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:17:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:17:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:17:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:17:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:17:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:17:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:17:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:17:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:17:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:17:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:17:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:17:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:17:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:17:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:17:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:17:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:17:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:17:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:17:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:17:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:17:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:17:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:17:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:17:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:17:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:17:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:17:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:17:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:17:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:17:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:17:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:17:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:17:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:17:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:17:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:17:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:17:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:17:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:17:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:17:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:17:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:17:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:17:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:17:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:17:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:17:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:17:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:17:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:17:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:17:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:17:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:17:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:17:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:17:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:17:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:17:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:17:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:17:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:17:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:17:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:17:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:17:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:17:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:17:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:17:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:17:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:17:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:17:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:17:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:17:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:17:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:17:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:17:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:17:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:17:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:17:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:17:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:17:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:17:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:17:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:17:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:17:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:17:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:17:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:17:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:17:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:17:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:17:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:17:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:17:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:17:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:17:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:17:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:17:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:18:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:18:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:18:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:18:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:18:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:18:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:18:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:18:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:18:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:18:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:18:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:18:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:18:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:18:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:18:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:18:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:18:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:18:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:18:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:18:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:18:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:18:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:18:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:18:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:18:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:18:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:18:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:18:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:18:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:18:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:18:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:18:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:18:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:18:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:18:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:18:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:18:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:18:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:18:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:18:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:18:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:18:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:18:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:18:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:18:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:18:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:18:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:18:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:18:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:18:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:18:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:18:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:18:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:18:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:18:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:18:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:18:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:18:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:18:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:18:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:18:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:18:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:18:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:18:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:18:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:18:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:18:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:18:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:18:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:18:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:18:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:18:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:18:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:18:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:18:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:18:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:18:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:18:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:18:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:18:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:18:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:18:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:18:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:18:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:18:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:18:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:18:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:18:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:18:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:18:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:18:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:18:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:18:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:18:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:18:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:18:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:18:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:18:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:18:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:18:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:18:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:18:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:18:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:18:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:18:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:18:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:18:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:18:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:18:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:18:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:18:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:18:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:18:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:18:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:18:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:18:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:18:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:18:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:18:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:18:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:18:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:18:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:18:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:18:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:18:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:18:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:18:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:18:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:18:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:18:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:18:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:18:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:18:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:18:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:18:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:18:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:18:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:18:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:18:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:18:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:18:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:18:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:18:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:18:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:18:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:18:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:18:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:18:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:18:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:18:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:18:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:18:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:18:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:18:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:18:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:18:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:18:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:18:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:18:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:18:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:18:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:18:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:18:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:18:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:18:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:18:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:18:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:18:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:18:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:18:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:18:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:18:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:18:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:18:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:18:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:18:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:18:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:18:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:18:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:18:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:18:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:18:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:18:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:18:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:18:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:18:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:18:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:18:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:18:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:18:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:18:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:18:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:18:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:18:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:18:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:18:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:18:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:18:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:18:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:18:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:18:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:18:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:18:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:18:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:18:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:18:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:18:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:18:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:18:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:18:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:18:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:18:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:18:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:18:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:18:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:18:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:19:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:19:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:19:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:19:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:19:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:19:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:19:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:19:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:19:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:19:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:19:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:19:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:19:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:19:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:19:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:19:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:19:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:19:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:19:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:19:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:19:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:19:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:19:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:19:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:19:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:19:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:19:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:19:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:19:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:19:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:19:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:19:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:19:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:19:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:19:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:19:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:19:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:19:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:19:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:19:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:19:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:19:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:19:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:19:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:19:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:19:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:19:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:19:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:19:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:19:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:19:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:19:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:19:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:19:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:19:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:19:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:19:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:19:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:19:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:19:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:19:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:19:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:19:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:19:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:19:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:19:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:19:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:19:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:19:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:19:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:19:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:19:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:19:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:19:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:19:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:19:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:19:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:19:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:19:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:19:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:19:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:19:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:19:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:19:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:19:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:19:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:19:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:19:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:19:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:19:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:19:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:19:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:19:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:19:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:19:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:19:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:19:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:19:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:19:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:19:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:19:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:19:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:19:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:19:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:19:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:19:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:19:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:19:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:19:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:19:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:19:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:19:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:19:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:19:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:19:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:19:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:19:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:19:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:19:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:19:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:19:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:19:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:19:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:19:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:19:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:19:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:19:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:19:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:19:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:19:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:19:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:19:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:19:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:19:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:19:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:19:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:19:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:19:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:19:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:19:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:19:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:19:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:19:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:19:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:19:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:19:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:19:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:19:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:19:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:19:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:19:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:19:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:19:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:19:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:19:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:19:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:19:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:19:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:19:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:19:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:19:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:19:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:19:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:19:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:19:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:19:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:19:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:19:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:19:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:19:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:19:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:19:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:19:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:19:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:19:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:19:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:19:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:19:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:19:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:19:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:19:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:19:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:19:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:19:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:19:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:19:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:19:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:19:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:19:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:19:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:19:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:19:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:19:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:19:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:19:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:19:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:19:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:19:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:19:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:19:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:19:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:19:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:19:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:19:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:19:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:19:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:19:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:19:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:19:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:19:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:19:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:19:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:19:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:19:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:19:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:19:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:20:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:20:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:20:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:20:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:20:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:20:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:20:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:20:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:20:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:20:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:20:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:20:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:20:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:20:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:20:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:20:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:20:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:20:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:20:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:20:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:20:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:20:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:20:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:20:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:20:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:20:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:20:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:20:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:20:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:20:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:20:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:20:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:20:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:20:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:20:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:20:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:20:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:20:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:20:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:20:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:20:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:20:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:20:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:20:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:20:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:20:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:20:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:20:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:20:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:20:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:20:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:20:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:20:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:20:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:20:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:20:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:20:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:20:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:20:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:20:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:20:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:20:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:20:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:20:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:20:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:20:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:20:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:20:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:20:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:20:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:20:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:20:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:20:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:20:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:20:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:20:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:20:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:20:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:20:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:20:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:20:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:20:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:20:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:20:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:20:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:20:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:20:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:20:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:20:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:20:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:20:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:20:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:20:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:20:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:20:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:20:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:20:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:20:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:20:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:20:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:20:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:20:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:20:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:20:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:20:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:20:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:20:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:20:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:20:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:20:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:20:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:20:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:20:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:20:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:20:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:20:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:20:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:20:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:20:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:20:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:20:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:20:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:20:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:20:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:20:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:20:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:20:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:20:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:20:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:20:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:20:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:20:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:20:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:20:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:20:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:20:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:20:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:20:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:20:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:20:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:20:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:20:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:20:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:20:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:20:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:20:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:20:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:20:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:20:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:20:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:20:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:20:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:20:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:20:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:20:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:20:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:20:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:20:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:20:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:20:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:20:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:20:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:20:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:20:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:20:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:20:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:20:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:20:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:20:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:20:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:20:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:20:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:20:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:20:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:20:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:20:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:20:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:20:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:20:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:20:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:20:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:20:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:20:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:20:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:20:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:20:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:20:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:20:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:20:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:20:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:20:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:20:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:20:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:20:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:20:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:20:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:20:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:20:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:20:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:20:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:20:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:20:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:20:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:20:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:20:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:20:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:20:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:20:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:20:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:20:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:20:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:20:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:20:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:20:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:20:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:20:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:21:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:21:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:21:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:21:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:21:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:21:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:21:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:21:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:21:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:21:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:21:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:21:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:21:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:21:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:21:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:21:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:21:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:21:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:21:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:21:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:21:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:21:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:21:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:21:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:21:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:21:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:21:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:21:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:21:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:21:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:21:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:21:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:21:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:21:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:21:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:21:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:21:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:21:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:21:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:21:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:21:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:21:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:21:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:21:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:21:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:21:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:21:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:21:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:21:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:21:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:21:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:21:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:21:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:21:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:21:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:21:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:21:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:21:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:21:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:21:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:21:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:21:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:21:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:21:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:21:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:21:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:21:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:21:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:21:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:21:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:21:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:21:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:21:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:21:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:21:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:21:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:21:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:21:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:21:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:21:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:21:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:21:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:21:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:21:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:21:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:21:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:21:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:21:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:21:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:21:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:21:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:21:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:21:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:21:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:21:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:21:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:21:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:21:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:21:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:21:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:21:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:21:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:21:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:21:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:21:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:21:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:21:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:21:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:21:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:21:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:21:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:21:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:21:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:21:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:21:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:21:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:21:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:21:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:21:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:21:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:21:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:21:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:21:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:21:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:21:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:21:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:21:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:21:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:21:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:21:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:21:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:21:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:21:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:21:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:21:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:21:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:21:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:21:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:21:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:21:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:21:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:21:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:21:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:21:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:21:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:21:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:21:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:21:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:21:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:21:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:21:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:21:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:21:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:21:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:21:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:21:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:21:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:21:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:21:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:21:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:21:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:21:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:21:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:21:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:21:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:21:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:21:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:21:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:21:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:21:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:21:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:21:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:21:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:21:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:21:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:21:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:21:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:21:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:21:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:21:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:21:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:21:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:21:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:21:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:21:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:21:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:21:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:21:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:21:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:21:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:21:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:21:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:21:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:21:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:21:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:21:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:21:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:21:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:21:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:21:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:21:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:21:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:21:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:21:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:21:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:21:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:21:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:21:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:21:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:21:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:21:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:21:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:21:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:21:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:21:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:21:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:22:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:22:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:22:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:22:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:22:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:22:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:22:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:22:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:22:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:22:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:22:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:22:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:22:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:22:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:22:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:22:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:22:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:22:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:22:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:22:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:22:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:22:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:22:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:22:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:22:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:22:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:22:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:22:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:22:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:22:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:22:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:22:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:22:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:22:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:22:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:22:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:22:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:22:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:22:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:22:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:22:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:22:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:22:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:22:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:22:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:22:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:22:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:22:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:22:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:22:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:22:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:22:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:22:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:22:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:22:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:22:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:22:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:22:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:22:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:22:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:22:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:22:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:22:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:22:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:22:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:22:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:22:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:22:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:22:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:22:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:22:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:22:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:22:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:22:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:22:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:22:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:22:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:22:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:22:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:22:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:22:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:22:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:22:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:22:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:22:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:22:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:22:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:22:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:22:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:22:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:22:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:22:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:22:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:22:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:22:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:22:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:22:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:22:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:22:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:22:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:22:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:22:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:22:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:22:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:22:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:22:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:22:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:22:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:22:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:22:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:22:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:22:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:22:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:22:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:22:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:22:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:22:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:22:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:22:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:22:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:22:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:22:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:22:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:22:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:22:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:22:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:22:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:22:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:22:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:22:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:22:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:22:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:22:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:22:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:22:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:22:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:22:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:22:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:22:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:22:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:22:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:22:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:22:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:22:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:22:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:22:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:22:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:22:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:22:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:22:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:22:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:22:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:22:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:22:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:22:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:22:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:22:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:22:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:22:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:22:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:22:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:22:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:22:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:22:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:22:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:22:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:22:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:22:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:22:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:22:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:22:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:22:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:22:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:22:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:22:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:22:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:22:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:22:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:22:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:22:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:22:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:22:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:22:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:22:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:22:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:22:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:22:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:22:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:22:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:22:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:22:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:22:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:22:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:22:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:22:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:22:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:22:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:22:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:24:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:24:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:24:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:24:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:24:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:24:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:24:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:24:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:24:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:24:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:24:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:24:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:24:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:24:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:24:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:24:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:24:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:24:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:24:57 rdma06 rsyslogd[1041]: imjournal: 5082 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 08:25:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:25:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:25:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:25:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:25:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:25:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:25:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:25:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:25:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:25:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:25:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:25:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:25:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:25:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:25:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:25:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:25:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:25:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:25:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:25:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:25:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:25:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:25:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:25:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:25:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:25:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:25:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:25:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:25:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:25:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:25:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:25:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:25:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:25:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:25:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:25:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:25:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:25:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:25:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:25:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:25:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:25:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:25:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:25:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:25:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:25:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:25:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:25:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:25:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:25:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:25:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:25:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:25:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:25:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:25:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:25:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:25:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:25:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:25:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:25:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:25:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:25:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:25:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:25:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:25:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:25:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:25:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:25:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:25:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:25:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:25:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:25:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:25:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:25:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:25:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:25:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:25:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:25:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:25:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:25:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:25:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:25:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:25:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:25:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:25:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:25:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:25:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:25:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:25:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:25:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:25:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:25:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:25:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:25:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:25:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:25:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:25:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:25:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:25:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:25:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:25:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:25:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:25:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:25:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:25:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:25:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:25:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:25:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:25:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:25:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:25:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:25:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:25:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:25:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:25:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:25:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:25:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:25:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:25:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:25:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:25:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:25:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:25:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:25:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:25:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:25:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:25:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:25:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:25:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:25:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:25:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:25:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:25:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:25:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:25:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:25:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:25:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:25:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:25:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:25:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:25:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:25:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:25:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:25:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:25:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:25:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:25:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:25:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:25:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:25:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:25:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:25:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:25:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:25:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:25:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:25:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:25:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:25:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:25:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:25:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:25:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:25:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:25:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:25:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:25:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:25:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:25:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:25:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:25:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:25:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:25:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:25:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:25:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:25:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:25:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:25:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:25:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:25:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:25:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:25:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:25:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:25:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:25:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:25:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:25:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:25:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:25:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:25:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:25:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:25:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:25:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:25:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:25:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:25:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:25:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:25:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:25:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:25:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:25:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:25:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:25:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:25:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:25:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:25:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:25:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:25:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:25:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:25:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:25:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:25:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:25:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:25:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:25:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:25:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:25:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:25:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:26:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:26:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:26:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:26:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:26:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:26:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:26:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:26:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:26:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:26:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:26:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:26:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:26:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:26:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:26:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:26:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:26:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:26:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:26:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:26:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:26:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:26:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:26:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:26:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:26:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:26:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:26:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:26:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:26:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:26:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:26:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:26:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:26:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:26:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:26:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:26:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:26:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:26:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:26:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:26:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:26:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:26:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:26:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:26:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:26:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:26:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:26:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:26:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:26:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:26:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:26:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:26:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:26:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:26:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:26:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:26:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:26:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:26:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:26:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:26:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:26:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:26:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:26:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:26:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:26:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:26:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:26:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:26:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:26:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:26:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:26:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:26:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:26:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:26:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:26:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:26:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:26:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:26:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:26:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:26:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:26:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:26:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:26:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:26:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:26:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:26:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:26:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:26:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:26:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:26:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:26:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:26:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:26:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:26:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:26:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:26:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:26:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:26:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:26:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:26:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:26:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:26:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:26:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:26:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:26:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:26:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:26:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:26:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:26:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:26:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:26:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:26:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:26:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:26:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:26:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:26:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:26:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:26:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:26:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:26:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:26:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:26:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:26:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:26:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:26:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:26:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:26:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:26:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:26:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:26:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:26:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:26:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:26:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:26:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:26:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:26:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:26:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:26:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:26:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:26:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:26:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:26:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:26:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:26:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:26:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:26:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:26:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:26:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:26:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:26:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:26:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:26:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:26:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:26:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:26:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:26:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:26:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:26:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:26:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:26:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:26:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:26:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:26:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:26:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:26:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:26:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:26:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:26:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:26:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:26:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:26:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:26:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:26:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:26:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:26:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:26:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:26:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:26:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:26:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:26:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:26:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:26:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:26:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:26:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:26:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:26:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:26:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:26:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:26:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:26:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:26:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:26:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:26:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:26:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:26:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:26:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:26:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:26:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:26:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:26:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:26:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:26:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:26:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:26:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:26:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:26:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:26:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:26:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:26:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:26:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:26:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:26:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:26:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:26:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:26:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:26:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:27:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:27:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:27:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:27:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:27:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:27:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:27:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:27:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:27:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:27:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:27:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:27:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:27:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:27:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:27:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:27:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:27:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:27:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:27:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:27:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:27:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:27:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:27:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:27:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:27:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:27:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:27:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:27:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:27:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:27:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:27:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:27:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:27:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:27:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:27:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:27:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:27:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:27:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:27:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:27:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:27:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:27:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:27:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:27:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:27:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:27:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:27:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:27:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:27:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:27:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:27:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:27:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:27:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:27:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:27:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:27:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:27:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:27:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:27:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:27:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:27:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:27:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:27:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:27:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:27:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:27:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:27:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:27:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:27:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:27:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:27:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:27:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:27:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:27:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:27:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:27:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:27:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:27:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:27:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:27:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:27:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:27:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:27:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:27:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:27:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:27:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:27:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:27:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:27:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:27:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:27:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:27:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:27:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:27:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:27:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:27:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:27:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:27:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:27:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:27:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:27:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:27:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:27:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:27:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:27:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:27:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:27:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:27:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:27:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:27:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:27:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:27:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:27:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:27:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:27:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:27:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:27:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:27:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:27:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:27:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:27:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:27:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:27:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:27:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:27:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:27:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:27:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:27:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:27:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:27:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:27:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:27:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:27:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:27:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:27:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:27:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:27:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:27:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:27:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:27:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:27:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:27:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:27:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:27:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:27:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:27:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:27:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:27:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:27:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:27:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:27:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:27:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:27:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:27:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:27:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:27:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:27:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:27:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:27:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:27:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:27:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:27:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:27:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:27:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:27:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:27:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:27:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:27:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:27:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:27:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:27:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:27:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:27:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:27:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:27:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:27:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:27:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:27:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:27:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:27:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:27:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:27:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:27:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:27:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:27:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:27:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:27:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:27:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:27:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:27:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:27:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:27:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:27:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:27:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:27:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:27:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:27:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:27:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:27:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:27:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:27:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:27:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:27:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:27:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:27:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:27:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:27:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:27:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:27:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:27:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:27:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:27:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:27:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:27:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:27:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:27:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:28:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:28:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:28:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:28:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:28:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:28:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:28:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:28:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:28:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:28:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:28:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:28:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:28:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:28:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:28:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:28:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:28:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:28:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:28:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:28:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:28:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:28:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:28:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:28:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:28:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:28:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:28:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:28:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:28:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:28:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:28:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:28:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:28:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:28:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:28:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:28:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:28:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:28:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:28:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:28:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:28:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:28:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:28:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:28:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:28:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:28:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:28:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:28:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:28:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:28:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:28:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:28:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:28:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:28:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:28:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:28:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:28:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:28:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:28:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:28:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:28:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:28:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:28:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:28:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:28:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:28:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:28:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:28:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:28:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:28:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:28:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:28:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:28:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:28:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:28:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:28:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:28:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:28:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:28:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:28:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:28:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:28:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:28:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:28:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:28:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:28:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:28:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:28:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:28:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:28:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:28:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:28:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:28:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:28:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:28:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:28:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:28:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:28:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:28:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:28:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:28:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:28:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:28:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:28:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:28:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:28:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:28:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:28:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:28:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:28:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:28:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:28:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:28:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:28:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:28:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:28:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:28:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:28:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:28:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:28:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:28:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:28:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:28:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:28:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:28:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:28:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:28:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:28:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:28:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:28:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:28:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:28:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:28:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:28:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:28:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:28:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:28:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:28:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:28:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:28:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:28:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:28:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:28:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:28:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:28:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:28:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:28:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:28:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:28:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:28:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:28:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:28:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:28:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:28:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:28:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:28:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:28:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:28:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:28:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:28:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:28:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:28:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:28:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:28:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:28:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:28:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:28:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:28:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:28:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:28:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:28:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:28:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:28:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:28:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:28:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:28:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:28:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:28:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:28:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:28:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:28:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:28:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:28:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:28:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:28:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:28:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:28:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:28:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:28:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:28:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:28:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:28:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:28:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:28:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:28:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:28:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:28:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:28:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:28:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:28:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:28:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:28:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:28:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:28:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:28:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:28:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:28:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:28:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:28:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:28:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:28:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:28:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:28:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:28:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:28:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:28:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:29:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:29:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:29:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:29:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:29:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:29:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:29:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:29:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:29:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:29:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:29:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:29:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:29:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:29:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:29:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:29:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:29:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:29:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:29:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:29:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:29:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:29:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:29:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:29:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:29:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:29:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:29:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:29:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:29:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:29:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:29:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:29:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:29:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:29:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:29:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:29:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:29:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:29:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:29:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:29:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:29:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:29:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:29:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:29:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:29:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:29:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:29:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:29:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:29:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:29:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:29:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:29:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:29:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:29:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:29:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:29:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:29:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:29:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:29:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:29:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:29:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:29:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:29:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:29:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:29:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:29:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:29:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:29:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:29:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:29:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:29:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:29:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:29:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:29:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:29:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:29:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:29:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:29:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:29:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:29:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:29:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:29:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:29:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:29:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:29:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:29:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:29:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:29:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:29:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:29:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:29:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:29:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:29:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:29:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:29:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:29:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:29:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:29:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:29:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:29:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:29:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:29:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:29:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:29:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:29:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:29:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:29:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:29:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:29:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:29:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:29:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:29:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:29:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:29:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:29:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:29:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:29:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:29:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:29:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:29:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:29:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:29:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:29:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:29:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:29:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:29:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:29:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:29:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:29:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:29:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:29:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:29:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:29:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:29:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:29:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:29:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:29:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:29:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:29:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:29:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:29:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:29:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:29:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:29:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:29:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:29:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:29:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:29:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:29:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:29:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:29:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:29:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:29:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:29:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:29:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:29:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:29:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:29:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:29:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:29:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:29:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:29:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:29:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:29:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:29:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:29:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:29:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:29:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:29:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:29:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:29:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:29:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:29:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:29:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:29:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:29:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:29:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:29:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:29:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:29:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:29:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:29:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:29:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:29:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:29:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:29:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:29:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:29:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:29:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:29:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:29:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:29:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:29:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:29:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:29:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:29:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:29:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:29:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:29:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:29:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:29:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:29:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:29:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:29:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:29:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:29:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:29:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:29:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:29:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:29:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:29:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:29:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:29:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:29:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:29:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:29:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:30:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:30:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:30:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:30:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:30:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:30:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:30:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:30:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:30:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:30:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:30:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:30:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:30:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:30:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:30:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:30:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:30:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:30:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:30:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:30:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:30:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:30:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:30:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:30:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:30:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:30:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:30:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:30:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:30:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:30:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:30:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:30:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:30:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:30:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:30:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:30:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:30:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:30:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:30:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:30:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:30:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:30:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:30:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:30:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:30:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:30:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:30:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:30:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:30:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:30:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:30:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:30:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:30:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:30:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:30:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:30:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:30:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:30:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:30:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:30:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:30:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:30:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:30:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:30:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:30:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:30:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:30:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:30:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:30:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:30:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:30:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:30:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:30:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:30:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:30:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:30:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:30:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:30:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:30:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:30:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:30:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:30:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:30:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:30:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:30:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:30:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:30:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:30:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:30:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:30:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:30:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:30:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:30:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:30:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:30:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:30:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:30:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:30:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:30:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:30:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:30:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:30:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:30:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:30:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:30:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:30:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:30:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:30:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:30:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:30:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:30:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:30:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:30:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:30:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:30:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:30:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:30:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:30:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:30:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:30:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:30:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:30:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:30:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:30:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:30:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:30:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:30:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:30:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:30:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:30:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:30:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:30:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:30:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:30:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:30:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:30:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:30:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:30:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:30:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:30:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:30:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:30:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:30:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:30:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:30:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:30:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:30:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:30:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:30:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:30:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:30:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:30:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:30:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:30:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:30:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:30:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:30:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:30:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:30:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:30:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:30:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:30:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:30:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:30:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:30:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:30:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:30:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:30:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:30:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:30:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:30:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:30:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:30:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:30:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:30:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:30:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:30:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:30:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:30:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:30:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:30:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:30:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:30:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:30:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:30:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:30:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:30:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:30:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:30:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:30:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:30:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:30:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:30:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:30:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:30:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:30:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:30:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:30:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:30:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:30:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:30:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:30:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:30:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:30:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:30:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:30:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:30:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:30:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:30:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:30:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:30:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:30:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:30:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:30:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:30:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:30:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:31:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:31:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:31:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:31:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:31:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:31:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:31:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:31:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:31:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:31:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:31:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:31:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:31:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:31:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:31:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:31:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:31:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:31:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:31:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:31:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:31:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:31:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:31:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:31:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:31:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:31:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:31:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:31:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:31:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:31:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:31:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:31:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:31:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:31:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:31:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:31:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:31:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:31:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:31:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:31:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:31:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:31:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:31:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:31:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:31:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:31:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:31:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:31:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:31:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:31:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:31:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:31:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:31:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:31:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:31:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:31:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:31:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:31:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:31:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:31:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:31:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:31:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:31:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:31:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:31:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:31:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:31:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:31:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:31:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:31:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:31:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:31:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:31:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:31:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:31:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:31:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:31:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:31:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:31:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:31:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:31:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:31:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:31:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:31:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:31:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:31:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:31:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:31:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:31:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:31:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:31:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:31:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:31:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:31:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:31:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:31:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:31:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:31:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:31:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:31:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:31:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:31:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:31:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:31:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:31:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:31:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:31:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:31:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:31:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:31:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:31:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:31:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:31:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:31:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:31:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:31:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:31:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:31:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:31:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:31:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:31:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:31:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:31:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:31:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:31:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:31:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:31:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:31:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:31:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:31:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:31:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:31:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:31:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:31:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:31:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:31:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:31:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:31:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:31:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:31:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:31:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:31:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:31:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:31:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:31:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:31:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:31:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:31:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:31:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:31:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:31:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:31:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:31:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:31:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:31:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:31:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:31:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:31:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:31:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:31:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:31:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:31:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:31:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:31:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:31:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:31:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:31:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:31:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:31:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:32:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:32:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:32:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:32:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:32:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:32:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:32:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:32:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:32:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:32:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:32:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:32:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:32:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:32:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:32:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:32:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:32:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:32:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:32:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:32:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:32:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:32:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:32:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:32:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:32:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:32:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:32:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:32:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:32:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:32:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:32:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:32:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:32:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:32:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:32:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:32:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:32:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:32:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:32:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:32:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:32:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:32:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:32:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:32:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:32:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:32:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:32:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:32:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:32:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:32:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:32:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:32:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:32:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:32:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:32:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:32:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:32:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:32:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:32:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:32:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:32:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:32:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:32:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:32:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:32:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:32:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:32:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:32:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:32:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:32:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:32:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:32:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:32:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:32:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:32:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:32:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:32:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:32:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:32:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:32:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:32:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:32:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:32:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:32:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:32:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:32:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:32:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:32:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:32:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:32:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:32:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:32:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:32:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:32:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:32:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:32:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:32:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:32:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:32:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:32:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:32:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:32:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:32:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:32:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:32:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:32:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:32:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:32:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:32:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:32:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:32:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:32:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:32:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:32:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:32:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:32:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:32:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:32:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:32:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:32:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:32:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:32:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:32:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:32:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:32:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:32:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:32:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:32:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:32:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:32:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:32:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:32:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:32:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:32:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:32:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:32:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:32:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:32:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:32:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:32:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:32:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:32:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:32:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:32:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:32:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:32:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:32:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:32:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:32:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:32:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:32:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:32:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:32:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:32:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:32:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:32:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:32:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:32:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:32:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:32:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:32:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:32:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:32:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:32:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:32:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:32:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:32:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:32:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:32:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:32:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:32:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:32:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:32:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:32:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:32:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:32:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:32:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:32:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:32:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:32:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:32:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:32:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:32:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:32:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:32:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:32:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:32:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:32:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:32:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:32:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:32:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:32:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:32:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:32:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:32:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:32:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:32:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:32:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:35:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:35:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:35:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:35:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:35:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:35:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:35:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:35:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:35:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:35:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:35:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:35:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:35:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:35:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:35:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:35:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:35:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:35:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:35:02 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 08:35:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:35:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:35:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:35:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:35:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:35:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:35:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:35:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:35:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:35:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:35:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:35:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:35:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:35:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:35:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:35:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:35:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:35:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:35:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:35:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:35:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:35:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:35:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:35:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:35:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:35:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:35:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:35:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:35:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:35:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:35:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:35:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:35:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:35:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:35:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:35:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:35:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:35:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:35:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:35:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:35:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:35:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:35:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:35:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:35:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:35:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:35:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:35:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:35:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:35:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:35:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:35:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:35:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:35:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:35:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:35:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:35:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:35:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:35:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:35:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:35:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:35:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:35:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:35:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:35:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:35:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:35:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:35:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:35:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:35:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:35:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:35:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:35:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:35:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:35:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:35:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:35:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:35:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:35:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:35:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:35:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:35:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:35:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:35:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:35:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:35:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:35:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:35:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:35:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:35:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:35:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:35:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:35:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:35:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:35:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:35:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:35:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:35:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:35:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:35:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:35:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:35:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:35:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:35:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:35:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:35:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:35:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:35:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:35:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:35:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:35:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:35:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:35:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:35:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:35:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:35:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:35:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:35:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:35:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:35:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:35:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:35:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:35:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:35:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:35:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:35:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:35:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:35:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:35:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:35:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:35:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:35:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:35:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:35:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:35:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:35:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:35:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:35:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:35:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:35:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:35:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:35:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:35:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:35:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:35:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:35:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:35:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:35:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:35:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:35:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:35:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:35:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:35:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:35:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:35:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:35:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:35:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:35:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:35:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:35:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:35:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:35:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:35:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:35:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:35:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:35:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:35:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:35:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:35:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:35:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:35:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:35:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:35:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:35:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:35:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:35:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:35:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:35:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:35:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:35:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:35:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:35:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:35:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:35:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:35:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:35:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:35:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:35:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:35:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:35:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:35:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:35:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:35:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:35:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:35:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:35:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:35:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:35:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:36:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:36:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:36:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:36:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:36:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:36:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:36:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:36:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:36:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:36:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:36:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:36:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:36:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:36:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:36:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:36:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:36:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:36:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:36:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:36:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:36:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:36:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:36:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:36:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:36:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:36:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:36:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:36:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:36:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:36:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:36:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:36:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:36:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:36:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:36:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:36:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:36:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:36:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:36:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:36:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:36:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:36:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:36:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:36:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:36:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:36:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:36:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:36:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:36:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:36:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:36:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:36:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:36:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:36:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:36:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:36:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:36:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:36:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:36:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:36:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:36:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:36:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:36:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:36:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:36:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:36:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:36:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:36:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:36:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:36:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:36:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:36:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:36:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:36:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:36:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:36:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:36:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:36:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:36:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:36:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:36:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:36:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:36:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:36:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:36:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:36:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:36:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:36:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:36:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:36:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:36:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:36:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:36:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:36:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:36:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:36:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:36:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:36:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:36:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:36:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:36:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:36:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:36:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:36:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:36:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:36:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:36:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:36:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:36:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:36:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:36:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:36:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:36:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:36:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:36:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:36:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:36:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:36:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:36:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:36:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:36:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:36:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:36:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:36:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:36:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:36:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:36:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:36:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:36:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:36:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:36:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:36:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:36:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:36:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:36:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:36:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:36:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:36:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:36:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:36:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:36:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:36:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:36:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:36:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:36:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:36:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:36:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:36:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:36:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:36:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:36:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:36:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:36:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:36:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:36:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:36:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:36:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:36:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:36:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:36:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:36:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:36:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:36:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:36:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:36:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:36:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:36:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:36:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:36:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:36:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:36:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:36:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:36:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:36:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:36:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:36:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:36:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:36:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:36:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:36:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:36:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:36:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:36:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:36:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:36:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:36:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:36:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:36:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:36:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:36:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:36:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:36:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:36:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:36:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:36:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:36:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:36:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:36:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:36:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:36:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:36:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:36:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:36:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:36:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:36:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:36:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:36:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:36:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:36:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:36:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:36:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:36:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:36:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:36:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:36:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:36:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:37:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:37:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:37:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:37:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:37:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:37:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:37:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:37:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:37:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:37:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:37:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:37:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:37:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:37:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:37:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:37:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:37:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:37:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:37:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:37:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:37:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:37:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:37:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:37:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:37:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:37:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:37:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:37:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:37:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:37:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:37:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:37:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:37:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:37:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:37:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:37:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:37:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:37:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:37:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:37:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:37:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:37:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:37:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:37:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:37:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:37:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:37:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:37:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:37:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:37:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:37:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:37:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:37:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:37:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:37:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:37:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:37:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:37:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:37:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:37:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:37:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:37:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:37:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:37:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:37:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:37:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:37:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:37:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:37:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:37:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:37:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:37:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:37:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:37:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:37:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:37:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:37:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:37:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:37:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:37:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:37:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:37:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:37:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:37:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:37:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:37:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:37:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:37:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:37:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:37:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:37:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:37:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:37:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:37:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:37:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:37:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:37:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:37:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:37:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:37:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:37:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:37:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:37:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:37:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:37:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:37:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:37:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:37:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:37:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:37:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:37:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:37:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:37:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:37:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:37:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:37:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:37:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:37:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:37:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:37:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:37:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:37:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:37:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:37:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:37:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:37:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:37:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:37:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:37:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:37:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:37:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:37:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:37:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:37:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:37:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:37:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:37:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:37:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:37:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:37:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:37:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:37:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:37:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:37:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:37:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:37:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:37:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:37:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:37:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:37:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:37:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:37:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:37:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:37:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:37:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:37:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:37:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:37:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:37:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:37:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:37:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:37:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:37:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:37:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:37:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:37:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:37:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:37:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:37:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:37:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:37:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:37:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:37:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:37:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:37:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:37:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:37:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:37:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:37:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:37:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:37:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:37:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:37:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:37:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:37:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:37:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:37:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:37:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:37:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:37:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:37:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:37:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:37:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:37:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:37:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:37:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:37:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:37:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:37:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:37:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:37:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:37:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:37:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:37:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:37:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:37:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:37:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:37:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:37:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:37:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:37:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:37:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:37:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:37:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:37:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:37:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:38:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:38:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:38:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:38:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:38:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:38:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:38:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:38:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:38:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:38:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:38:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:38:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:38:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:38:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:38:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:38:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:38:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:38:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:38:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:38:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:38:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:38:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:38:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:38:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:38:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:38:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:38:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:38:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:38:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:38:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:38:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:38:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:38:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:38:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:38:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:38:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:38:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:38:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:38:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:38:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:38:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:38:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:38:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:38:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:38:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:38:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:38:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:38:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:38:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:38:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:38:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:38:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:38:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:38:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:38:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:38:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:38:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:38:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:38:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:38:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:38:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:38:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:38:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:38:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:38:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:38:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:38:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:38:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:38:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:38:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:38:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:38:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:38:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:38:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:38:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:38:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:38:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:38:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:38:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:38:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:38:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:38:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:38:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:38:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:38:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:38:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:38:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:38:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:38:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:38:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:38:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:38:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:38:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:38:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:38:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:38:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:38:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:38:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:38:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:38:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:38:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:38:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:38:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:38:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:38:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:38:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:38:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:38:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:38:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:38:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:38:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:38:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:38:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:38:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:38:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:38:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:38:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:38:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:38:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:38:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:38:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:38:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:38:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:38:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:38:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:38:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:38:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:38:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:38:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:38:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:38:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:38:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:38:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:38:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:38:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:38:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:38:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:38:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:38:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:38:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:38:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:38:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:38:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:38:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:38:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:38:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:38:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:38:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:38:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:38:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:38:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:38:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:38:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:38:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:38:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:38:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:38:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:38:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:38:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:38:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:38:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:38:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:38:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:38:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:38:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:38:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:38:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:38:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:38:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:38:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:38:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:38:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:38:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:38:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:38:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:38:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:38:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:38:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:38:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:38:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:38:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:38:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:38:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:38:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:38:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:38:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:38:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:38:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:38:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:38:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:38:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:38:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:38:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:38:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:38:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:38:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:38:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:38:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:38:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:38:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:38:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:38:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:38:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:38:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:38:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:38:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:38:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:38:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:38:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:38:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:38:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:38:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:38:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:38:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:38:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:38:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:39:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:39:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:39:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:39:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:39:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:39:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:39:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:39:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:39:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:39:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:39:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:39:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:39:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:39:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:39:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:39:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:39:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:39:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:39:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:39:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:39:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:39:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:39:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:39:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:39:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:39:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:39:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:39:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:39:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:39:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:39:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:39:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:39:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:39:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:39:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:39:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:39:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:39:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:39:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:39:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:39:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:39:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:39:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:39:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:39:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:39:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:39:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:39:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:39:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:39:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:39:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:39:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:39:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:39:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:39:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:39:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:39:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:39:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:39:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:39:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:39:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:39:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:39:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:39:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:39:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:39:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:39:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:39:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:39:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:39:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:39:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:39:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:39:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:39:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:39:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:39:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:39:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:39:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:39:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:39:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:39:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:39:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:39:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:39:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:39:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:39:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:39:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:39:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:39:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:39:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:39:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:39:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:39:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:39:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:39:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:39:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:39:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:39:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:39:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:39:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:39:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:39:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:39:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:39:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:39:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:39:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:39:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:39:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:39:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:39:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:39:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:39:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:39:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:39:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:39:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:39:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:39:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:39:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:39:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:39:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:39:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:39:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:39:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:39:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:39:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:39:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:39:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:39:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:39:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:39:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:39:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:39:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:39:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:39:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:39:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:39:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:39:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:39:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:39:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:39:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:39:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:39:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:39:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:39:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:39:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:39:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:39:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:39:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:39:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:39:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:39:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:39:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:39:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:39:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:39:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:39:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:39:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:39:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:39:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:39:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:39:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:39:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:39:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:39:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:39:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:39:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:39:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:39:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:39:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:39:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:39:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:39:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:39:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:39:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:39:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:39:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:39:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:39:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:39:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:39:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:39:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:39:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:39:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:39:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:39:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:39:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:39:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:39:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:39:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:39:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:39:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:39:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:39:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:39:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:39:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:39:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:39:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:39:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:39:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:39:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:39:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:39:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:39:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:39:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:39:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:39:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:39:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:39:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:39:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:39:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:39:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:39:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:39:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:39:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:39:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:39:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:40:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:40:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:40:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:40:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:40:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:40:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:40:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:40:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:40:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:40:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:40:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:40:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:40:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:40:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:40:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:40:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:40:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:40:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:40:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:40:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:40:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:40:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:40:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:40:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:40:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:40:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:40:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:40:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:40:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:40:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:40:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:40:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:40:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:40:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:40:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:40:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:40:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:40:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:40:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:40:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:40:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:40:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:40:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:40:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:40:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:40:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:40:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:40:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:40:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:40:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:40:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:40:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:40:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:40:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:40:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:40:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:40:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:40:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:40:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:40:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:40:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:40:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:40:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:40:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:40:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:40:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:40:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:40:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:40:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:40:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:40:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:40:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:40:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:40:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:40:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:40:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:40:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:40:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:40:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:40:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:40:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:40:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:40:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:40:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:40:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:40:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:40:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:40:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:40:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:40:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:40:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:40:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:40:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:40:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:40:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:40:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:40:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:40:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:40:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:40:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:40:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:40:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:40:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:40:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:40:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:40:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:40:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:40:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:40:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:40:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:40:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:40:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:40:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:40:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:40:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:40:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:40:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:40:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:40:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:40:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:40:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:40:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:40:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:40:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:40:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:40:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:40:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:40:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:40:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:40:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:40:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:40:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:40:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:40:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:40:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:40:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:40:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:40:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:40:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:40:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:40:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:40:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:40:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:40:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:40:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:40:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:40:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:40:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:40:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:40:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:40:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:40:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:40:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:40:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:40:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:40:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:40:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:40:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:40:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:40:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:40:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:40:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:40:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:40:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:40:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:40:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:40:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:40:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:40:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:40:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:40:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:40:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:40:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:40:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:40:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:40:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:40:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:40:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:40:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:40:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:40:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:40:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:40:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:40:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:40:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:40:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:40:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:40:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:40:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:40:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:40:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:40:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:40:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:40:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:40:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:40:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:40:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:40:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:40:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:40:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:40:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:40:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:40:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:40:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:40:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:40:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:40:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:40:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:40:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:40:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:40:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:40:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:40:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:40:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:40:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:40:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:41:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:41:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:41:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:41:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:41:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:41:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:41:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:41:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:41:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:41:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:41:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:41:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:41:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:41:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:41:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:41:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:41:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:41:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:41:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:41:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:41:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:41:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:41:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:41:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:41:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:41:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:41:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:41:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:41:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:41:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:41:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:41:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:41:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:41:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:41:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:41:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:41:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:41:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:41:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:41:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:41:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:41:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:41:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:41:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:41:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:41:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:41:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:41:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:41:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:41:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:41:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:41:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:41:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:41:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:41:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:41:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:41:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:41:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:41:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:41:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:41:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:41:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:41:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:41:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:41:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:41:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:41:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:41:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:41:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:41:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:41:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:41:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:41:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:41:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:41:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:41:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:41:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:41:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:41:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:41:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:41:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:41:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:41:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:41:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:41:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:41:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:41:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:41:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:41:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:41:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:41:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:41:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:41:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:41:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:41:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:41:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:41:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:41:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:41:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:41:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:41:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:41:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:41:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:41:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:41:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:41:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:41:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:41:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:41:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:41:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:41:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:41:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:41:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:41:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:41:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:41:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:41:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:41:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:41:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:41:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:41:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:41:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:41:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:41:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:41:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:41:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:41:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:41:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:41:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:41:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:41:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:41:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:41:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:41:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:41:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:41:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:41:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:41:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:41:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:41:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:41:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:41:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:41:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:41:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:41:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:41:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:41:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:41:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:41:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:41:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:41:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:41:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:41:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:41:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:41:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:41:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:41:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:41:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:41:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:41:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:41:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:41:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:41:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:41:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:41:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:41:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:41:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:41:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:41:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:41:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:41:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:41:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:41:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:41:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:41:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:41:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:41:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:41:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:41:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:41:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:41:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:41:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:41:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:41:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:41:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:41:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:41:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:41:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:41:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:41:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:41:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:41:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:41:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:41:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:41:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:41:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:41:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:41:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:41:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:41:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:41:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:41:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:41:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:41:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:41:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:41:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:41:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:41:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:41:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:41:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:41:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:41:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:41:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:41:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:41:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:41:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:42:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:42:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:42:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:42:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:42:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:42:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:42:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:42:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:42:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:42:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:42:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:42:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:42:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:42:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:42:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:42:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:42:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:42:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:42:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:42:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:42:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:42:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:42:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:42:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:42:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:42:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:42:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:42:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:42:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:42:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:42:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:42:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:42:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:42:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:42:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:42:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:42:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:42:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:42:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:42:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:42:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:42:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:42:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:42:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:42:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:42:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:42:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:42:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:42:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:42:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:42:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:42:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:42:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:42:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:42:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:42:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:42:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:42:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:42:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:42:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:42:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:42:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:42:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:42:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:42:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:42:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:42:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:42:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:42:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:42:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:42:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:42:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:42:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:42:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:42:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:42:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:42:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:42:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:42:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:42:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:42:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:42:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:42:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:42:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:42:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:42:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:42:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:42:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:42:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:42:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:42:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:42:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:42:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:42:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:42:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:42:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:42:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:42:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:42:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:42:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:42:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:42:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:42:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:42:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:42:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:42:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:42:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:42:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:42:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:42:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:42:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:42:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:42:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:42:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:42:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:42:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:42:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:42:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:42:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:42:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:42:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:42:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:42:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:42:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:42:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:42:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:42:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:42:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:42:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:42:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:42:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:42:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:42:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:42:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:42:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:42:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:42:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:42:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:42:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:42:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:42:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:42:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:42:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:42:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:42:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:42:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:42:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:42:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:42:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:42:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:42:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:42:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:42:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:42:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:42:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:42:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:42:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:42:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:42:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:42:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:42:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:42:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:42:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:42:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:42:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:42:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:42:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:42:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:42:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:42:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:42:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:42:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:42:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:42:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:42:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:42:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:42:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:42:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:42:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:42:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:42:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:42:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:42:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:42:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:42:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:42:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:42:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:42:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:42:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:42:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:42:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:42:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:42:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:42:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:42:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:42:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:42:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:42:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:42:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:42:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:42:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:42:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:42:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:42:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:42:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:42:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:42:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:42:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:42:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:42:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:42:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:42:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:42:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:42:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:42:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:42:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:45:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:45:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:45:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:45:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:45:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:45:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:45:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:45:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:45:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:45:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:45:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:45:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:45:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:45:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:45:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:45:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:45:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:45:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:45:07 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 08:45:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:45:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:45:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:45:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:45:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:45:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:45:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:45:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:45:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:45:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:45:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:45:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:45:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:45:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:45:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:45:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:45:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:45:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:45:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:45:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:45:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:45:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:45:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:45:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:45:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:45:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:45:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:45:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:45:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:45:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:45:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:45:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:45:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:45:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:45:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:45:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:45:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:45:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:45:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:45:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:45:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:45:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:45:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:45:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:45:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:45:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:45:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:45:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:45:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:45:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:45:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:45:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:45:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:45:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:45:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:45:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:45:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:45:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:45:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:45:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:45:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:45:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:45:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:45:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:45:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:45:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:45:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:45:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:45:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:45:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:45:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:45:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:45:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:45:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:45:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:45:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:45:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:45:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:45:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:45:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:45:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:45:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:45:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:45:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:45:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:45:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:45:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:45:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:45:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:45:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:45:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:45:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:45:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:45:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:45:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:45:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:45:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:45:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:45:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:45:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:45:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:45:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:45:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:45:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:45:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:45:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:45:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:45:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:45:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:45:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:45:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:45:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:45:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:45:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:45:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:45:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:45:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:45:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:45:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:45:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:45:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:45:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:45:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:45:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:45:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:45:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:45:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:45:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:45:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:45:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:45:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:45:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:45:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:45:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:45:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:45:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:45:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:45:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:45:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:45:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:45:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:45:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:45:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:45:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:45:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:45:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:45:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:45:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:45:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:45:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:45:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:45:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:45:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:45:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:45:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:45:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:45:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:45:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:45:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:45:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:45:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:45:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:45:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:45:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:45:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:45:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:45:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:45:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:45:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:45:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:45:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:45:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:45:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:45:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:45:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:45:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:45:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:45:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:45:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:45:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:46:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:46:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:46:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:46:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:46:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:46:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:46:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:46:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:46:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:46:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:46:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:46:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:46:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:46:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:46:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:46:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:46:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:46:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:46:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:46:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:46:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:46:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:46:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:46:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:46:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:46:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:46:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:46:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:46:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:46:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:46:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:46:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:46:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:46:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:46:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:46:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:46:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:46:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:46:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:46:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:46:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:46:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:46:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:46:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:46:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:46:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:46:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:46:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:46:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:46:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:46:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:46:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:46:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:46:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:46:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:46:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:46:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:46:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:46:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:46:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:46:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:46:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:46:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:46:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:46:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:46:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:46:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:46:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:46:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:46:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:46:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:46:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:46:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:46:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:46:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:46:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:46:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:46:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:46:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:46:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:46:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:46:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:46:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:46:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:46:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:46:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:46:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:46:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:46:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:46:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:46:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:46:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:46:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:46:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:46:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:46:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:46:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:46:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:46:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:46:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:46:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:46:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:46:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:46:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:46:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:46:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:46:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:46:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:46:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:46:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:46:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:46:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:46:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:46:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:46:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:46:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:46:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:46:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:46:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:46:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:46:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:46:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:46:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:46:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:46:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:46:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:46:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:46:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:46:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:46:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:46:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:46:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:46:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:46:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:46:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:46:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:46:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:46:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:46:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:46:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:46:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:46:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:46:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:46:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:46:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:46:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:46:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:46:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:46:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:46:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:46:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:46:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:46:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:46:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:46:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:46:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:46:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:46:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:46:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:46:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:46:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:46:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:46:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:46:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:46:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:46:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:46:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:46:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:46:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:46:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:46:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:46:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:46:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:46:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:46:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:46:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:46:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:46:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:46:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:46:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:46:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:46:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:46:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:46:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:46:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:46:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:46:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:46:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:46:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:46:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:46:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:46:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:46:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:46:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:46:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:46:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:46:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:46:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:46:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:46:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:46:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:46:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:46:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:46:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:46:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:46:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:46:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:46:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:46:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:46:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:46:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:46:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:46:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:46:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:46:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:46:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:47:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:47:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:47:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:47:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:47:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:47:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:47:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:47:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:47:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:47:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:47:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:47:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:47:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:47:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:47:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:47:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:47:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:47:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:47:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:47:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:47:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:47:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:47:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:47:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:47:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:47:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:47:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:47:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:47:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:47:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:47:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:47:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:47:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:47:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:47:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:47:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:47:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:47:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:47:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:47:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:47:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:47:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:47:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:47:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:47:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:47:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:47:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:47:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:47:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:47:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:47:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:47:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:47:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:47:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:47:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:47:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:47:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:47:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:47:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:47:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:47:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:47:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:47:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:47:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:47:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:47:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:47:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:47:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:47:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:47:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:47:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:47:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:47:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:47:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:47:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:47:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:47:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:47:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:47:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:47:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:47:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:47:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:47:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:47:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:47:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:47:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:47:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:47:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:47:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:47:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:47:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:47:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:47:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:47:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:47:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:47:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:47:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:47:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:47:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:47:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:47:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:47:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:47:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:47:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:47:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:47:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:47:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:47:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:47:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:47:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:47:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:47:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:47:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:47:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:47:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:47:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:47:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:47:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:47:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:47:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:47:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:47:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:47:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:47:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:47:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:47:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:47:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:47:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:47:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:47:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:47:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:47:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:47:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:47:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:47:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:47:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:47:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:47:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:47:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:47:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:47:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:47:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:47:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:47:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:47:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:47:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:47:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:47:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:47:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:47:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:47:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:47:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:47:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:47:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:47:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:47:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:47:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:47:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:47:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:47:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:47:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:47:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:47:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:47:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:47:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:47:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:47:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:47:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:47:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:47:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:47:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:47:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:47:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:47:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:47:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:47:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:47:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:47:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:47:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:47:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:47:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:47:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:47:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:47:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:47:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:47:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:47:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:47:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:47:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:47:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:47:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:47:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:47:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:47:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:47:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:47:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:47:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:47:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:47:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:47:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:47:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:47:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:47:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:47:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:47:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:47:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:47:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:47:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:47:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:47:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:47:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:47:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:47:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:47:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:47:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:47:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:48:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:48:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:48:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:48:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:48:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:48:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:48:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:48:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:48:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:48:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:48:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:48:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:48:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:48:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:48:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:48:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:48:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:48:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:48:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:48:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:48:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:48:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:48:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:48:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:48:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:48:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:48:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:48:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:48:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:48:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:48:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:48:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:48:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:48:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:48:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:48:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:48:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:48:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:48:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:48:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:48:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:48:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:48:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:48:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:48:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:48:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:48:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:48:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:48:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:48:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:48:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:48:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:48:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:48:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:48:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:48:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:48:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:48:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:48:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:48:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:48:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:48:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:48:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:48:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:48:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:48:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:48:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:48:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:48:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:48:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:48:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:48:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:48:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:48:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:48:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:48:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:48:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:48:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:48:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:48:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:48:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:48:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:48:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:48:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:48:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:48:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:48:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:48:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:48:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:48:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:48:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:48:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:48:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:48:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:48:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:48:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:48:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:48:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:48:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:48:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:48:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:48:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:48:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:48:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:48:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:48:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:48:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:48:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:48:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:48:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:48:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:48:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:48:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:48:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:48:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:48:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:48:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:48:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:48:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:48:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:48:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:48:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:48:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:48:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:48:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:48:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:48:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:48:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:48:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:48:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:48:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:48:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:48:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:48:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:48:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:48:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:48:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:48:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:48:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:48:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:48:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:48:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:48:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:48:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:48:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:48:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:48:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:48:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:48:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:48:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:48:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:48:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:48:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:48:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:48:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:48:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:48:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:48:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:48:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:48:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:48:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:48:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:48:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:48:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:48:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:48:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:48:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:48:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:48:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:48:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:48:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:48:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:48:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:48:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:48:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:48:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:48:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:48:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:48:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:48:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:48:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:48:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:48:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:48:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:48:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:48:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:48:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:48:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:48:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:48:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:48:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:48:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:48:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:48:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:48:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:48:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:48:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:48:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:48:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:48:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:48:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:48:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:48:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:48:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:48:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:48:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:48:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:48:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:48:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:48:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:48:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:48:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:48:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:48:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:48:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:48:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:49:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:49:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:49:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:49:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:49:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:49:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:49:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:49:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:49:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:49:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:49:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:49:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:49:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:49:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:49:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:49:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:49:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:49:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:49:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:49:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:49:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:49:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:49:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:49:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:49:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:49:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:49:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:49:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:49:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:49:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:49:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:49:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:49:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:49:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:49:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:49:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:49:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:49:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:49:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:49:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:49:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:49:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:49:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:49:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:49:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:49:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:49:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:49:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:49:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:49:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:49:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:49:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:49:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:49:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:49:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:49:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:49:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:49:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:49:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:49:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:49:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:49:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:49:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:49:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:49:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:49:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:49:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:49:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:49:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:49:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:49:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:49:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:49:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:49:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:49:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:49:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:49:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:49:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:49:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:49:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:49:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:49:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:49:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:49:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:49:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:49:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:49:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:49:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:49:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:49:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:49:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:49:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:49:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:49:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:49:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:49:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:49:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:49:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:49:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:49:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:49:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:49:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:49:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:49:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:49:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:49:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:49:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:49:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:49:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:49:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:49:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:49:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:49:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:49:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:49:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:49:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:49:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:49:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:49:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:49:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:49:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:49:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:49:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:49:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:49:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:49:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:49:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:49:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:49:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:49:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:49:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:49:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:49:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:49:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:49:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:49:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:49:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:49:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:49:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:49:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:49:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:49:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:49:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:49:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:49:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:49:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:49:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:49:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:49:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:49:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:49:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:49:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:49:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:49:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:49:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:49:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:49:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:49:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:49:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:49:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:49:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:49:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:49:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:49:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:49:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:49:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:49:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:49:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:49:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:49:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:49:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:49:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:49:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:49:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:49:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:49:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:49:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:49:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:49:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:49:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:49:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:49:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:49:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:49:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:49:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:49:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:49:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:49:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:49:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:49:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:49:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:49:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:49:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:49:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:49:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:49:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:49:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:49:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:49:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:49:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:49:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:49:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:49:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:49:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:49:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:49:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:49:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:49:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:49:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:49:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:49:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:49:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:49:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:49:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:49:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:49:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:50:00 rdma06 systemd[1]: Starting dnf makecache...
Apr 17 08:50:01 rdma06 dnf[1149764]: Updating Subscription Management repositories.
Apr 17 08:50:01 rdma06 dnf[1149764]: Unable to read consumer identity
Apr 17 08:50:01 rdma06 dnf[1149764]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 17 08:50:01 rdma06 dnf[1149764]: Metadata cache refreshed recently.
Apr 17 08:50:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 17 08:50:01 rdma06 systemd[1]: Finished dnf makecache.
Apr 17 08:50:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:50:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:50:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:50:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:50:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:50:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:50:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:50:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:50:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:50:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:50:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:50:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:50:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:50:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:50:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:50:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:50:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:50:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:50:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:50:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:50:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:50:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:50:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:50:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:50:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:50:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:50:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:50:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:50:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:50:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:50:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:50:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:50:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:50:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:50:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:50:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:50:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:50:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:50:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:50:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:50:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:50:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:50:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:50:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:50:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:50:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:50:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:50:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:50:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:50:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:50:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:50:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:50:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:50:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:50:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:50:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:50:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:50:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:50:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:50:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:50:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:50:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:50:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:50:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:50:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:50:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:50:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:50:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:50:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:50:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:50:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:50:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:50:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:50:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:50:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:50:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:50:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:50:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:50:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:50:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:50:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:50:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:50:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:50:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:50:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:50:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:50:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:50:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:50:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:50:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:50:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:50:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:50:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:50:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:50:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:50:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:50:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:50:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:50:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:50:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:50:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:50:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:50:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:50:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:50:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:50:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:50:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:50:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:50:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:50:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:50:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:50:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:50:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:50:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:50:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:50:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:50:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:50:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:50:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:50:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:50:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:50:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:50:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:50:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:50:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:50:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:50:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:50:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:50:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:50:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:50:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:50:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:50:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:50:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:50:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:50:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:50:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:50:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:50:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:50:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:50:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:50:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:50:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:50:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:50:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:50:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:50:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:50:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:50:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:50:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:50:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:50:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:50:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:50:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:50:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:50:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:50:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:50:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:50:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:50:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:50:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:50:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:50:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:50:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:50:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:50:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:50:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:50:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:50:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:50:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:50:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:50:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:50:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:50:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:50:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:50:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:50:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:50:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:50:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:50:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:50:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:50:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:50:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:50:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:50:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:50:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:50:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:50:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:50:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:50:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:50:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:50:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:50:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:50:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:50:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:50:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:50:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:50:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:50:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:50:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:50:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:50:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:50:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:50:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:50:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:50:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:50:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:50:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:50:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:50:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:50:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:50:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:50:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:50:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:50:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:50:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:51:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:51:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:51:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:51:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:51:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:51:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:51:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:51:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:51:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:51:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:51:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:51:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:51:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:51:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:51:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:51:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:51:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:51:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:51:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:51:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:51:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:51:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:51:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:51:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:51:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:51:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:51:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:51:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:51:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:51:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:51:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:51:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:51:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:51:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:51:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:51:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:51:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:51:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:51:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:51:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:51:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:51:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:51:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:51:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:51:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:51:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:51:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:51:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:51:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:51:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:51:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:51:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:51:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:51:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:51:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:51:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:51:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:51:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:51:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:51:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:51:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:51:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:51:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:51:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:51:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:51:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:51:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:51:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:51:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:51:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:51:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:51:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:51:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:51:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:51:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:51:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:51:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:51:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:51:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:51:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:51:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:51:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:51:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:51:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:51:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:51:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:51:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:51:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:51:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:51:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:51:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:51:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:51:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:51:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:51:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:51:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:51:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:51:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:51:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:51:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:51:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:51:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:51:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:51:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:51:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:51:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:51:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:51:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:51:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:51:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:51:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:51:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:51:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:51:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:51:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:51:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:51:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:51:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:51:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:51:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:51:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:51:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:51:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:51:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:51:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:51:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:51:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:51:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:51:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:51:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:51:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:51:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:51:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:51:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:51:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:51:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:51:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:51:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:51:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:51:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:51:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:51:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:51:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:51:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:51:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:51:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:51:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:51:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:51:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:51:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:51:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:51:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:51:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:51:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:51:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:51:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:51:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:51:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:51:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:51:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:51:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:51:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:51:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:51:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:51:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:51:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:51:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:51:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:51:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:51:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:51:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:51:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:51:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:51:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:51:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:51:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:51:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:51:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:51:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:51:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:51:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:51:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:51:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:51:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:51:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:51:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:51:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:51:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:51:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:51:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:51:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:51:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:51:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:51:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:51:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:51:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:51:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:51:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:51:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:51:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:51:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:51:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:51:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:51:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:51:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:51:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:51:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:51:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:51:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:51:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:51:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:51:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:51:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:51:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:51:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:51:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:52:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:52:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:52:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:52:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:52:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:52:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:52:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:52:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:52:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:52:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:52:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:52:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:52:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:52:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:52:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:52:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:52:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:52:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:52:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:52:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:52:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:52:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:52:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:52:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:52:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:52:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:52:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:52:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:52:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:52:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:52:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:52:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:52:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:52:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:52:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:52:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:52:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:52:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:52:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:52:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:52:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:52:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:52:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:52:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:52:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:52:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:52:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:52:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:52:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:52:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:52:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:52:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:52:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:52:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:52:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:52:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:52:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:52:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:52:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:52:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:52:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:52:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:52:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:52:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:52:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:52:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:52:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:52:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:52:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:52:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:52:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:52:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:52:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:52:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:52:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:52:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:52:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:52:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:52:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:52:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:52:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:52:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:52:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:52:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:52:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:52:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:52:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:52:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:52:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:52:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:52:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:52:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:52:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:52:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:52:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:52:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:52:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:52:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:52:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:52:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:52:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:52:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:52:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:52:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:52:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:52:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:52:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:52:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:52:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:52:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:52:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:52:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:52:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:52:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:52:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:52:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:52:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:52:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:52:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:52:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:52:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:52:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:52:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:52:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:52:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:52:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:52:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:52:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:52:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:52:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:52:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:52:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:52:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:52:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:52:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:52:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:52:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:52:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:52:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:52:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:52:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:52:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:52:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:52:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:52:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:52:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:52:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:52:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:52:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:52:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:52:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:52:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:52:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:52:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:52:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:52:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:52:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:52:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:52:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:52:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:52:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:52:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:52:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:52:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:52:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:52:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:52:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:52:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:52:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:52:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:52:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:52:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:52:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:52:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:52:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:52:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:52:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:52:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:52:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:52:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:52:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:52:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:52:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:52:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:52:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:52:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:52:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:52:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:52:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:52:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:52:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:52:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:52:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:52:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:52:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:52:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:52:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:52:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:52:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:52:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:52:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:52:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:52:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:52:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:52:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:52:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:52:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:52:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:52:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:52:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:52:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:52:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:52:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:52:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:52:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:52:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:53:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:53:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:53:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:53:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:53:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:53:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:53:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:53:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:53:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:53:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:53:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:53:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:53:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:53:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:53:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:53:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:53:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:53:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:55:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:55:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:55:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:55:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:55:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:55:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:55:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:55:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:55:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:55:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:55:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:55:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:55:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:55:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:55:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:55:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:55:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:55:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:55:08 rdma06 rsyslogd[1041]: imjournal: 5089 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 08:55:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:55:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:55:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:55:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:55:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:55:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:55:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:55:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:55:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:55:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:55:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:55:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:55:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:55:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:55:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:55:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:55:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:55:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:55:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:55:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:55:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:55:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:55:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:55:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:55:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:55:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:55:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:55:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:55:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:55:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:55:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:55:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:55:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:55:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:55:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:55:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:55:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:55:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:55:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:55:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:55:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:55:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:55:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:55:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:55:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:55:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:55:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:55:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:55:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:55:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:55:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:55:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:55:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:55:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:55:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:55:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:55:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:55:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:55:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:55:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:55:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:55:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:55:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:55:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:55:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:55:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:55:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:55:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:55:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:55:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:55:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:55:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:55:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:55:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:55:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:55:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:55:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:55:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:55:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:55:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:55:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:55:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:55:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:55:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:55:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:55:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:55:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:55:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:55:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:55:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:55:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:55:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:55:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:55:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:55:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:55:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:55:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:55:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:55:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:55:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:55:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:55:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:55:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:55:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:55:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:55:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:55:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:55:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:55:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:55:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:55:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:55:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:55:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:55:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:55:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:55:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:55:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:55:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:55:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:55:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:55:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:55:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:55:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:55:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:55:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:55:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:55:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:55:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:55:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:55:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:55:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:55:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:55:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:55:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:55:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:55:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:55:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:55:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:55:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:55:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:55:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:55:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:55:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:55:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:55:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:55:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:55:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:55:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:55:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:55:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:55:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:55:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:55:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:55:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:55:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:55:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:55:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:55:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:55:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:55:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:55:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:55:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:55:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:55:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:55:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:55:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:55:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:55:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:55:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:55:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:55:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:55:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:55:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:55:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:55:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:55:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:55:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:55:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:55:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:55:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:56:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:56:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:56:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:56:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:56:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:56:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:56:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:56:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:56:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:56:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:56:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:56:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:56:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:56:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:56:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:56:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:56:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:56:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:56:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:56:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:56:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:56:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:56:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:56:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:56:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:56:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:56:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:56:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:56:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:56:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:56:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:56:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:56:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:56:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:56:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:56:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:56:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:56:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:56:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:56:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:56:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:56:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:56:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:56:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:56:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:56:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:56:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:56:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:56:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:56:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:56:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:56:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:56:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:56:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:56:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:56:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:56:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:56:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:56:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:56:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:56:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:56:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:56:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:56:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:56:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:56:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:56:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:56:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:56:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:56:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:56:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:56:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:56:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:56:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:56:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:56:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:56:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:56:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:56:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:56:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:56:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:56:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:56:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:56:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:56:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:56:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:56:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:56:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:56:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:56:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:56:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:56:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:56:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:56:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:56:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:56:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:56:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:56:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:56:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:56:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:56:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:56:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:56:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:56:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:56:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:56:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:56:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:56:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:56:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:56:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:56:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:56:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:56:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:56:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:56:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:56:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:56:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:56:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:56:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:56:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:56:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:56:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:56:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:56:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:56:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:56:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:56:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:56:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:56:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:56:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:56:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:56:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:56:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:56:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:56:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:56:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:56:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:56:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:56:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:56:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:56:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:56:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:56:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:56:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:56:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:56:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:56:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:56:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:56:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:56:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:56:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:56:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:56:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:56:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:56:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:56:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:56:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:56:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:56:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:56:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:56:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:56:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:56:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:56:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:56:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:56:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:56:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:56:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:56:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:56:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:56:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:56:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:56:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:56:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:56:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:56:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:56:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:56:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:56:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:56:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:56:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:56:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:56:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:56:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:56:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:56:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:56:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:56:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:56:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:56:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:56:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:56:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:56:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:56:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:56:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:56:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:56:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:56:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:56:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:56:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:56:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:56:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:56:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:56:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:56:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:56:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:56:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:56:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:56:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:56:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:56:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:56:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:56:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:56:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:56:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:56:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:57:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:57:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:57:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:57:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:57:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:57:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:57:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:57:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:57:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:57:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:57:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:57:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:57:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:57:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:57:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:57:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:57:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:57:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:57:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:57:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:57:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:57:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:57:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:57:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:57:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:57:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:57:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:57:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:57:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:57:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:57:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:57:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:57:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:57:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:57:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:57:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:57:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:57:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:57:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:57:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:57:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:57:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:57:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:57:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:57:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:57:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:57:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:57:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:57:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:57:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:57:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:57:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:57:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:57:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:57:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:57:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:57:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:57:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:57:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:57:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:57:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:57:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:57:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:57:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:57:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:57:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:57:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:57:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:57:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:57:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:57:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:57:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:57:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:57:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:57:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:57:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:57:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:57:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:57:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:57:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:57:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:57:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:57:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:57:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:57:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:57:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:57:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:57:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:57:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:57:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:57:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:57:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:57:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:57:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:57:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:57:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:57:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:57:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:57:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:57:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:57:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:57:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:57:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:57:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:57:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:57:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:57:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:57:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:57:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:57:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:57:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:57:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:57:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:57:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:57:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:57:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:57:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:57:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:57:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:57:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:57:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:57:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:57:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:57:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:57:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:57:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:57:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:57:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:57:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:57:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:57:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:57:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:57:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:57:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:57:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:57:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:57:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:57:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:57:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:57:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:57:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:57:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:57:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:57:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:57:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:57:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:57:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:57:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:57:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:57:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:57:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:57:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:57:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:57:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:57:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:57:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:57:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:57:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:57:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:57:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:57:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:57:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:57:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:57:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:57:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:57:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:57:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:57:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:57:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:57:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:57:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:57:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:57:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:57:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:57:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:57:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:57:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:57:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:57:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:57:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:57:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:57:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:57:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:57:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:57:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:57:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:57:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:57:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:57:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:57:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:57:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:57:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:57:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:57:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:57:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:57:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:57:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:57:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:57:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:57:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:57:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:57:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:57:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:57:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:57:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:57:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:57:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:57:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:57:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:57:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:57:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:57:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:57:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:57:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:57:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:57:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:58:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:58:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:58:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:58:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:58:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:58:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:58:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:58:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:58:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:58:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:58:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:58:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:58:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:58:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:58:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:58:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:58:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:58:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:58:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:58:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:58:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:58:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:58:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:58:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:58:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:58:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:58:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:58:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:58:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:58:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:58:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:58:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:58:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:58:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:58:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:58:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:58:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:58:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:58:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:58:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:58:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:58:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:58:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:58:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:58:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:58:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:58:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:58:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:58:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:58:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:58:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:58:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:58:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:58:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:58:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:58:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:58:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:58:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:58:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:58:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:58:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:58:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:58:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:58:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:58:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:58:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:58:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:58:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:58:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:58:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:58:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:58:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:58:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:58:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:58:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:58:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:58:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:58:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:58:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:58:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:58:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:58:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:58:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:58:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:58:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:58:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:58:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:58:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:58:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:58:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:58:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:58:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:58:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:58:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:58:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:58:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:58:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:58:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:58:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:58:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:58:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:58:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:58:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:58:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:58:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:58:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:58:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:58:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:58:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:58:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:58:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:58:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:58:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:58:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:58:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:58:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:58:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:58:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:58:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:58:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:58:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:58:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:58:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:58:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:58:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:58:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:58:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:58:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:58:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:58:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:58:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:58:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:58:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:58:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:58:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:58:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:58:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:58:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:58:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:58:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:58:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:58:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:58:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:58:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:58:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:58:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:58:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:58:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:58:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:58:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:58:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:58:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:58:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:58:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:58:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:58:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:58:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:58:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:58:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:58:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:58:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:58:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:58:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:58:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:58:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:58:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:58:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:58:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:58:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:58:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:58:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:58:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:58:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:58:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:58:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:58:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:58:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:58:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:58:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:58:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:58:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:58:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:58:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:58:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:58:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:58:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:58:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:58:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:58:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:58:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:58:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:58:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:58:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:58:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:58:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:58:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:58:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:58:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:58:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:58:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:58:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:58:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:58:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:58:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:58:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:58:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:58:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:58:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:58:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:58:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:58:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:58:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:58:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:58:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:58:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:58:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:59:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:59:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:59:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:59:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:59:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:59:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:59:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:59:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:59:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:59:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:59:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:59:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:59:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:59:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:59:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:59:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:59:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:59:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:59:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:59:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:59:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:59:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:59:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:59:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:59:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:59:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:59:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:59:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:59:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:59:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:59:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:59:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:59:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:59:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:59:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:59:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:59:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:59:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:59:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:59:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:59:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:59:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:59:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:59:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:59:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:59:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:59:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:59:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:59:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:59:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:59:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:59:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:59:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:59:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:59:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:59:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:59:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:59:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:59:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:59:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:59:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:59:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:59:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:59:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:59:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:59:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:59:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:59:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:59:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:59:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:59:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:59:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:59:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:59:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:59:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:59:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:59:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:59:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:59:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:59:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:59:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:59:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:59:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:59:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:59:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:59:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:59:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:59:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:59:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:59:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:59:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:59:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:59:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:59:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:59:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:59:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:59:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:59:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:59:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:59:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:59:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:59:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:59:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:59:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:59:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:59:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:59:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:59:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:59:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:59:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:59:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:59:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:59:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:59:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:59:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:59:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:59:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:59:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:59:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:59:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:59:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:59:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:59:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:59:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:59:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:59:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:59:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:59:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:59:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:59:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:59:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:59:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:59:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:59:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:59:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:59:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:59:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:59:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:59:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:59:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:59:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:59:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:59:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:59:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:59:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:59:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:59:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:59:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:59:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:59:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:59:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:59:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:59:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:59:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:59:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:59:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:59:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:59:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:59:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:59:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:59:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:59:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:59:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:59:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:59:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:59:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:59:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:59:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:59:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:59:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:59:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:59:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:59:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:59:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:59:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:59:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:59:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:59:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:59:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:59:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:59:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:59:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:59:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:59:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:59:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:59:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:59:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:59:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:59:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:59:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:59:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:59:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:59:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:59:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:59:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:59:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:59:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:59:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 08:59:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 08:59:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:59:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:59:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:59:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 08:59:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 08:59:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 08:59:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 08:59:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 08:59:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 08:59:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 08:59:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 08:59:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 08:59:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 08:59:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 08:59:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 08:59:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 08:59:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:00:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:00:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:00:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:00:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:00:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:00:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:00:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:00:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:00:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:00:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:00:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:00:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:00:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:00:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:00:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:00:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:00:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:00:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:00:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:00:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:00:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:00:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:00:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:00:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:00:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:00:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:00:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:00:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:00:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:00:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:00:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:00:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:00:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:00:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:00:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:00:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:00:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:00:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:00:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:00:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:00:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:00:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:00:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:00:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:00:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:00:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:00:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:00:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:00:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:00:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:00:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:00:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:00:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:00:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:00:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:00:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:00:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:00:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:00:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:00:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:00:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:00:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:00:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:00:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:00:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:00:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:00:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:00:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:00:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:00:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:00:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:00:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:00:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:00:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:00:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:00:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:00:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:00:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:00:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:00:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:00:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:00:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:00:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:00:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:00:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:00:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:00:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:00:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:00:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:00:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:00:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:00:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:00:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:00:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:00:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:00:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:00:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:00:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:00:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:00:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:00:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:00:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:00:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:00:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:00:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:00:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:00:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:00:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:00:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:00:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:00:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:00:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:00:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:00:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:00:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:00:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:00:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:00:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:00:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:00:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:00:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:00:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:00:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:00:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:00:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:00:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:00:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:00:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:00:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:00:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:00:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:00:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:00:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:00:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:00:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:00:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:00:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:00:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:00:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:00:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:00:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:00:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:00:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:00:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:00:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:00:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:00:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:00:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:00:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:00:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:00:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:00:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:00:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:00:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:00:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:00:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:00:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:00:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:00:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:00:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:00:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:00:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:00:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:00:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:00:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:00:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:00:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:00:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:00:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:00:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:00:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:00:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:00:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:00:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:00:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:00:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:00:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:00:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:00:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:00:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:00:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:00:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:00:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:00:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:00:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:00:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:00:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:00:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:00:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:00:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:00:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:00:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:00:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:00:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:00:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:00:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:00:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:00:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:00:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:00:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:00:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:00:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:00:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:00:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:00:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:00:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:00:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:00:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:00:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:00:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:00:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:00:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:00:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:00:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:00:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:00:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:01:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:01:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:01:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:01:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:01:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:01:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:01:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:01:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:01:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:01:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:01:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:01:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:01:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:01:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:01:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:01:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:01:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:01:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:01:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:01:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:01:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:01:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:01:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:01:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:01:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:01:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:01:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:01:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:01:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:01:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:01:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:01:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:01:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:01:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:01:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:01:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:01:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:01:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:01:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:01:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:01:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:01:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:01:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:01:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:01:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:01:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:01:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:01:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:01:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:01:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:01:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:01:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:01:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:01:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:01:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:01:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:01:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:01:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:01:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:01:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:01:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:01:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:01:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:01:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:01:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:01:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:01:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:01:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:01:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:01:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:01:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:01:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:01:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:01:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:01:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:01:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:01:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:01:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:01:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:01:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:01:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:01:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:01:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:01:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:01:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:01:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:01:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:01:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:01:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:01:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:01:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:01:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:01:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:01:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:01:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:01:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:01:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:01:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:01:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:01:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:01:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:01:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:01:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:01:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:01:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:01:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:01:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:01:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:01:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:01:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:01:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:01:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:01:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:01:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:01:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:01:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:01:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:01:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:01:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:01:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:01:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:01:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:01:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:01:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:01:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:01:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:01:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:01:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:01:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:01:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:01:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:01:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:01:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:01:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:01:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:01:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:01:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:01:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:01:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:01:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:01:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:01:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:01:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:01:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:01:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:01:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:01:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:01:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:01:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:01:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:01:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:01:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:01:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:01:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:01:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:01:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:01:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:01:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:01:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:01:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:01:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:01:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:01:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:01:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:01:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:01:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:01:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:01:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:01:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:01:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:01:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:01:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:01:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:01:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:01:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:01:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:01:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:01:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:01:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:01:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:01:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:01:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:01:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:01:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:01:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:01:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:01:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:01:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:01:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:01:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:01:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:01:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:01:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:01:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:01:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:01:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:01:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:01:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:01:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:01:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:01:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:01:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:01:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:01:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:01:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:01:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:01:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:01:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:01:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:01:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:01:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:01:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:01:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:01:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:01:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:01:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:02:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:02:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:02:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:02:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:02:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:02:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:02:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:02:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:02:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:02:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:02:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:02:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:02:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:02:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:02:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:02:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:02:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:02:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:02:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:02:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:02:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:02:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:02:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:02:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:02:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:02:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:02:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:02:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:02:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:02:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:02:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:02:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:02:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:02:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:02:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:02:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:02:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:02:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:02:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:02:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:02:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:02:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:02:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:02:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:02:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:02:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:02:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:02:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:02:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:02:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:02:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:02:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:02:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:02:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:02:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:02:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:02:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:02:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:02:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:02:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:02:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:02:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:02:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:02:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:02:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:02:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:02:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:02:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:02:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:02:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:02:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:02:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:02:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:02:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:02:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:02:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:02:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:02:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:02:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:02:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:02:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:02:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:02:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:02:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:02:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:02:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:02:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:02:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:02:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:02:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:02:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:02:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:02:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:02:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:02:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:02:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:02:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:02:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:02:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:02:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:02:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:02:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:02:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:02:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:02:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:02:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:02:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:02:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:02:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:02:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:02:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:02:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:02:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:02:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:02:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:02:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:02:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:02:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:02:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:02:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:02:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:02:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:02:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:02:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:02:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:02:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:02:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:02:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:02:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:02:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:02:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:02:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:02:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:02:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:02:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:02:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:02:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:02:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:02:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:02:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:02:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:02:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:02:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:02:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:02:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:02:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:02:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:02:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:02:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:02:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:02:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:02:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:02:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:02:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:02:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:02:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:02:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:02:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:02:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:02:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:02:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:02:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:02:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:02:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:02:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:02:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:02:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:02:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:02:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:02:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:02:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:02:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:02:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:02:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:02:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:02:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:02:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:02:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:02:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:02:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:02:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:02:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:02:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:02:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:02:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:02:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:02:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:02:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:02:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:02:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:02:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:02:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:02:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:02:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:02:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:02:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:02:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:02:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:02:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:02:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:02:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:02:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:02:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:02:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:02:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:02:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:02:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:02:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:02:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:02:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:02:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:02:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:02:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:02:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:02:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:02:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:03:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:03:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:03:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:03:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:03:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:03:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:03:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:03:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:03:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:03:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:03:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:03:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:03:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:03:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:03:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:03:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:03:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:03:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:05:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:05:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:05:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:05:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:05:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:05:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:05:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:05:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:05:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:05:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:05:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:05:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:05:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:05:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:05:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:05:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:05:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:05:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:05:13 rdma06 rsyslogd[1041]: imjournal: 5295 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 09:05:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:05:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:05:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:05:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:05:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:05:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:05:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:05:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:05:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:05:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:05:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:05:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:05:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:05:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:05:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:05:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:05:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:05:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:05:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:05:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:05:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:05:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:05:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:05:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:05:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:05:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:05:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:05:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:05:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:05:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:05:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:05:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:05:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:05:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:05:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:05:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:05:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:05:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:05:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:05:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:05:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:05:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:05:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:05:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:05:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:05:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:05:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:05:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:05:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:05:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:05:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:05:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:05:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:05:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:05:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:05:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:05:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:05:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:05:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:05:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:05:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:05:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:05:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:05:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:05:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:05:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:05:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:05:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:05:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:05:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:05:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:05:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:05:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:05:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:05:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:05:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:05:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:05:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:05:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:05:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:05:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:05:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:05:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:05:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:05:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:05:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:05:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:05:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:05:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:05:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:05:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:05:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:05:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:05:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:05:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:05:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:05:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:05:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:05:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:05:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:05:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:05:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:05:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:05:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:05:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:05:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:05:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:05:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:05:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:05:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:05:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:05:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:05:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:05:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:05:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:05:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:05:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:05:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:05:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:05:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:05:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:05:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:05:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:05:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:05:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:05:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:05:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:05:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:05:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:05:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:05:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:05:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:05:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:05:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:05:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:05:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:05:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:05:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:05:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:05:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:05:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:05:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:05:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:05:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:05:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:05:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:05:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:05:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:05:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:05:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:05:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:05:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:05:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:05:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:05:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:05:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:05:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:05:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:05:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:05:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:05:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:05:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:06:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:06:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:06:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:06:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:06:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:06:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:06:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:06:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:06:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:06:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:06:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:06:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:06:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:06:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:06:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:06:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:06:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:06:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:06:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:06:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:06:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:06:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:06:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:06:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:06:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:06:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:06:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:06:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:06:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:06:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:06:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:06:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:06:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:06:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:06:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:06:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:06:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:06:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:06:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:06:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:06:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:06:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:06:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:06:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:06:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:06:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:06:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:06:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:06:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:06:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:06:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:06:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:06:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:06:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:06:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:06:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:06:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:06:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:06:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:06:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:06:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:06:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:06:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:06:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:06:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:06:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:06:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:06:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:06:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:06:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:06:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:06:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:06:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:06:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:06:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:06:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:06:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:06:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:06:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:06:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:06:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:06:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:06:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:06:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:06:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:06:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:06:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:06:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:06:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:06:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:06:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:06:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:06:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:06:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:06:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:06:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:06:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:06:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:06:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:06:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:06:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:06:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:06:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:06:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:06:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:06:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:06:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:06:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:06:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:06:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:06:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:06:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:06:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:06:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:06:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:06:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:06:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:06:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:06:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:06:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:06:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:06:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:06:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:06:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:06:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:06:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:06:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:06:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:06:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:06:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:06:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:06:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:06:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:06:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:06:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:06:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:06:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:06:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:06:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:06:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:06:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:06:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:06:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:06:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:06:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:06:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:06:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:06:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:06:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:06:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:06:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:06:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:06:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:06:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:06:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:06:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:06:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:06:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:06:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:06:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:06:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:06:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:06:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:06:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:06:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:06:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:06:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:06:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:06:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:06:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:06:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:06:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:06:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:06:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:06:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:06:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:06:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:06:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:06:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:06:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:06:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:06:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:06:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:06:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:06:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:06:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:06:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:06:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:06:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:06:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:06:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:06:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:06:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:06:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:06:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:06:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:06:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:06:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:06:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:06:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:06:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:06:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:06:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:06:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:06:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:06:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:06:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:06:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:06:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:06:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:06:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:06:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:06:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:06:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:06:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:06:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:07:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:07:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:07:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:07:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:07:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:07:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:07:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:07:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:07:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:07:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:07:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:07:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:07:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:07:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:07:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:07:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:07:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:07:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:07:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:07:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:07:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:07:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:07:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:07:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:07:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:07:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:07:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:07:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:07:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:07:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:07:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:07:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:07:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:07:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:07:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:07:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:07:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:07:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:07:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:07:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:07:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:07:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:07:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:07:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:07:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:07:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:07:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:07:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:07:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:07:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:07:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:07:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:07:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:07:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:07:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:07:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:07:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:07:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:07:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:07:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:07:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:07:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:07:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:07:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:07:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:07:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:07:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:07:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:07:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:07:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:07:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:07:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:07:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:07:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:07:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:07:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:07:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:07:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:07:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:07:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:07:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:07:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:07:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:07:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:07:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:07:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:07:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:07:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:07:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:07:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:07:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:07:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:07:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:07:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:07:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:07:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:07:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:07:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:07:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:07:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:07:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:07:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:07:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:07:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:07:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:07:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:07:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:07:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:07:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:07:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:07:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:07:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:07:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:07:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:07:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:07:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:07:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:07:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:07:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:07:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:07:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:07:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:07:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:07:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:07:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:07:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:07:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:07:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:07:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:07:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:07:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:07:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:07:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:07:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:07:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:07:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:07:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:07:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:07:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:07:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:07:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:07:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:07:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:07:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:07:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:07:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:07:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:07:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:07:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:07:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:07:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:07:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:07:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:07:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:07:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:07:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:07:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:07:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:07:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:07:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:07:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:07:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:07:43 rdma06 systemd-journald[790]: Data hash table of /run/log/journal/95379d2f723847b68d9e976dee8967a9/system.journal has a fill level at 75.0 (163750 of 218332 items, 75497472 file size, 461 bytes per hash table item), suggesting rotation.
Apr 17 09:07:43 rdma06 systemd-journald[790]: /run/log/journal/95379d2f723847b68d9e976dee8967a9/system.journal: Journal header limits reached or header out-of-date, rotating.
Apr 17 09:07:43 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 17 09:07:43 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 17 09:07:43 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 17 09:07:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:07:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:07:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:07:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:07:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:07:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:07:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:07:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:07:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:07:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:07:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:07:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:07:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:07:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:07:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:07:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:07:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:07:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:07:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:07:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:07:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:07:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:07:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:07:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:07:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:07:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:07:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:07:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:07:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:07:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:07:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:07:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:07:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:07:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:07:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:07:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:07:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:07:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:07:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:07:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:07:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:07:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:07:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:07:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:07:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:07:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:07:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:07:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:07:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:07:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:07:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:07:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:07:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:07:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:08:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:08:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:08:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:08:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:08:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:08:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:08:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:08:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:08:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:08:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:08:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:08:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:08:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:08:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:08:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:08:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:08:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:08:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:08:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:08:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:08:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:08:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:08:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:08:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:08:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:08:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:08:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:08:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:08:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:08:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:08:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:08:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:08:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:08:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:08:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:08:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:08:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:08:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:08:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:08:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:08:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:08:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:08:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:08:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:08:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:08:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:08:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:08:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:08:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:08:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:08:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:08:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:08:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:08:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:08:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:08:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:08:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:08:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:08:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:08:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:08:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:08:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:08:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:08:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:08:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:08:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:08:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:08:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:08:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:08:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:08:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:08:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:08:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:08:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:08:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:08:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:08:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:08:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:08:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:08:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:08:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:08:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:08:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:08:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:08:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:08:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:08:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:08:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:08:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:08:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:08:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:08:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:08:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:08:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:08:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:08:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:08:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:08:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:08:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:08:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:08:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:08:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:08:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:08:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:08:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:08:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:08:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:08:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:08:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:08:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:08:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:08:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:08:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:08:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:08:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:08:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:08:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:08:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:08:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:08:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:08:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:08:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:08:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:08:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:08:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:08:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:08:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:08:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:08:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:08:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:08:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:08:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:08:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:08:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:08:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:08:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:08:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:08:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:08:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:08:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:08:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:08:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:08:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:08:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:08:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:08:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:08:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:08:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:08:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:08:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:08:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:08:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:08:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:08:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:08:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:08:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:08:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:08:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:08:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:08:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:08:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:08:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:08:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:08:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:08:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:08:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:08:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:08:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:08:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:08:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:08:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:08:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:08:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:08:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:08:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:08:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:08:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:08:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:08:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:08:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:08:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:08:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:08:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:08:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:08:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:08:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:08:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:08:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:08:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:08:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:08:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:08:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:08:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:08:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:08:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:08:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:08:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:08:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:08:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:08:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:08:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:08:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:08:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:08:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:08:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:08:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:08:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:08:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:08:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:08:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:08:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:08:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:08:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:08:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:08:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:08:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:09:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:09:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:09:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:09:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:09:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:09:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:09:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:09:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:09:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:09:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:09:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:09:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:09:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:09:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:09:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:09:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:09:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:09:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:09:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:09:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:09:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:09:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:09:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:09:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:09:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:09:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:09:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:09:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:09:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:09:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:09:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:09:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:09:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:09:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:09:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:09:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:09:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:09:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:09:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:09:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:09:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:09:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:09:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:09:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:09:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:09:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:09:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:09:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:09:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:09:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:09:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:09:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:09:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:09:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:09:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:09:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:09:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:09:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:09:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:09:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:09:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:09:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:09:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:09:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:09:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:09:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:09:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:09:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:09:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:09:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:09:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:09:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:09:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:09:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:09:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:09:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:09:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:09:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:09:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:09:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:09:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:09:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:09:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:09:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:09:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:09:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:09:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:09:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:09:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:09:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:09:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:09:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:09:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:09:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:09:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:09:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:09:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:09:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:09:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:09:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:09:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:09:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:09:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:09:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:09:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:09:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:09:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:09:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:09:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:09:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:09:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:09:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:09:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:09:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:09:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:09:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:09:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:09:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:09:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:09:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:09:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:09:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:09:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:09:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:09:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:09:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:09:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:09:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:09:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:09:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:09:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:09:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:09:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:09:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:09:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:09:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:09:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:09:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:09:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:09:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:09:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:09:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:09:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:09:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:09:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:09:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:09:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:09:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:09:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:09:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:09:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:09:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:09:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:09:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:09:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:09:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:09:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:09:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:09:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:09:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:09:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:09:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:09:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:09:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:09:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:09:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:09:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:09:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:09:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:09:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:09:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:09:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:09:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:09:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:09:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:09:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:09:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:09:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:09:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:09:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:09:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:09:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:09:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:09:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:09:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:09:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:09:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:09:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:09:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:09:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:09:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:09:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:09:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:09:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:09:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:09:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:09:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:09:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:09:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:09:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:09:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:09:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:09:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:09:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:09:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:09:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:09:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:09:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:09:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:09:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:09:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:09:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:09:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:09:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:09:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:09:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:10:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:10:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:10:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:10:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:10:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:10:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:10:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:10:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:10:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:10:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:10:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:10:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:10:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:10:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:10:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:10:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:10:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:10:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:10:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:10:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:10:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:10:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:10:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:10:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:10:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:10:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:10:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:10:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:10:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:10:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:10:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:10:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:10:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:10:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:10:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:10:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:10:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:10:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:10:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:10:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:10:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:10:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:10:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:10:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:10:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:10:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:10:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:10:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:10:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:10:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:10:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:10:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:10:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:10:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:10:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:10:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:10:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:10:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:10:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:10:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:10:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:10:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:10:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:10:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:10:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:10:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:10:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:10:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:10:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:10:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:10:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:10:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:10:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:10:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:10:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:10:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:10:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:10:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:10:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:10:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:10:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:10:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:10:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:10:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:10:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:10:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:10:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:10:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:10:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:10:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:10:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:10:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:10:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:10:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:10:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:10:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:10:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:10:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:10:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:10:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:10:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:10:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:10:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:10:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:10:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:10:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:10:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:10:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:10:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:10:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:10:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:10:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:10:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:10:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:10:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:10:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:10:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:10:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:10:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:10:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:10:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:10:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:10:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:10:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:10:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:10:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:10:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:10:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:10:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:10:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:10:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:10:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:10:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:10:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:10:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:10:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:10:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:10:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:10:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:10:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:10:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:10:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:10:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:10:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:10:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:10:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:10:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:10:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:10:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:10:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:10:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:10:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:10:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:10:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:10:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:10:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:10:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:10:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:10:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:10:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:10:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:10:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:10:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:10:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:10:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:10:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:10:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:10:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:10:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:10:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:10:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:10:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:10:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:10:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:10:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:10:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:10:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:10:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:10:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:10:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:10:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:10:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:10:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:10:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:10:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:10:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:10:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:10:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:10:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:10:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:10:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:10:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:10:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:10:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:10:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:10:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:10:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:10:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:10:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:10:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:10:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:10:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:10:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:10:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:10:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:10:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:10:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:10:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:10:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:10:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:10:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:10:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:10:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:10:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:10:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:10:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:11:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:11:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:11:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:11:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:11:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:11:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:11:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:11:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:11:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:11:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:11:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:11:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:11:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:11:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:11:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:11:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:11:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:11:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:11:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:11:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:11:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:11:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:11:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:11:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:11:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:11:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:11:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:11:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:11:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:11:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:11:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:11:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:11:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:11:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:11:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:11:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:11:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:11:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:11:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:11:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:11:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:11:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:11:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:11:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:11:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:11:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:11:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:11:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:11:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:11:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:11:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:11:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:11:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:11:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:11:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:11:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:11:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:11:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:11:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:11:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:11:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:11:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:11:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:11:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:11:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:11:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:11:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:11:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:11:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:11:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:11:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:11:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:11:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:11:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:11:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:11:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:11:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:11:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:11:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:11:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:11:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:11:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:11:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:11:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:11:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:11:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:11:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:11:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:11:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:11:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:11:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:11:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:11:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:11:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:11:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:11:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:11:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:11:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:11:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:11:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:11:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:11:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:11:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:11:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:11:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:11:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:11:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:11:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:11:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:11:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:11:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:11:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:11:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:11:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:11:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:11:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:11:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:11:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:11:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:11:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:11:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:11:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:11:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:11:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:11:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:11:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:11:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:11:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:11:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:11:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:11:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:11:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:11:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:11:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:11:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:11:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:11:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:11:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:11:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:11:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:11:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:11:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:11:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:11:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:11:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:11:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:11:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:11:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:11:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:11:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:11:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:11:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:11:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:11:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:11:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:11:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:11:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:11:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:11:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:11:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:11:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:11:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:11:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:11:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:11:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:11:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:11:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:11:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:11:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:11:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:11:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:11:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:11:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:11:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:11:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:11:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:11:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:11:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:11:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:11:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:11:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:11:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:11:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:11:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:11:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:11:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:11:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:11:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:11:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:11:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:11:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:11:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:11:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:11:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:11:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:11:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:11:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:11:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:11:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:11:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:11:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:11:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:11:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:11:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:11:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:11:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:11:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:11:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:11:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:11:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:11:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:11:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:11:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:11:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:11:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:11:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:12:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:12:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:12:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:12:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:12:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:12:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:12:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:12:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:12:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:12:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:12:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:12:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:12:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:12:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:12:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:12:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:12:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:12:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:12:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:12:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:12:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:12:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:12:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:12:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:12:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:12:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:12:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:12:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:12:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:12:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:12:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:12:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:12:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:12:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:12:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:12:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:12:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:12:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:12:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:12:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:12:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:12:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:12:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:12:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:12:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:12:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:12:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:12:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:12:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:12:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:12:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:12:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:12:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:12:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:12:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:12:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:12:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:12:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:12:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:12:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:12:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:12:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:12:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:12:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:12:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:12:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:12:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:12:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:12:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:12:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:12:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:12:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:12:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:12:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:12:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:12:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:12:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:12:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:12:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:12:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:12:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:12:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:12:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:12:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:12:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:12:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:12:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:12:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:12:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:12:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:12:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:12:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:12:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:12:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:12:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:12:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:12:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:12:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:12:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:12:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:12:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:12:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:12:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:12:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:12:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:12:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:12:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:12:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:12:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:12:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:12:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:12:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:12:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:12:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:12:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:12:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:12:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:12:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:12:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:12:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:12:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:12:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:12:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:12:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:12:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:12:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:12:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:12:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:12:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:12:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:12:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:12:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:12:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:12:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:12:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:12:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:12:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:12:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:12:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:12:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:12:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:12:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:12:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:12:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:12:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:12:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:12:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:12:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:12:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:12:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:12:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:12:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:12:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:12:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:12:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:12:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:12:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:12:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:12:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:12:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:12:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:12:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:12:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:12:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:12:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:12:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:12:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:12:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:12:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:12:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:12:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:12:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:12:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:12:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:12:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:12:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:12:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:12:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:12:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:12:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:12:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:12:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:12:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:12:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:12:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:12:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:12:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:12:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:12:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:12:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:12:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:12:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:12:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:12:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:12:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:12:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:12:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:12:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:12:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:12:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:12:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:12:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:12:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:12:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:12:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:12:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:12:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:12:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:12:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:12:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:12:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:12:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:12:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:12:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:12:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:12:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:13:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:13:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:13:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:13:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:13:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:13:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:13:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:13:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:13:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:13:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:13:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:13:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:13:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:13:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:13:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:13:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:13:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:13:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:13:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:13:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:13:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:13:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:13:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:13:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:13:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:13:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:13:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:13:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:13:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:13:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:13:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:13:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:13:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:13:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:13:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:13:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:15:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:15:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:15:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:15:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:15:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:15:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:15:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:15:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:15:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:15:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:15:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:15:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:15:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:15:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:15:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:15:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:15:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:15:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:15:18 rdma06 rsyslogd[1041]: imjournal: 5296 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 09:15:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:15:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:15:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:15:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:15:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:15:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:15:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:15:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:15:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:15:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:15:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:15:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:15:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:15:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:15:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:15:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:15:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:15:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:15:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:15:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:15:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:15:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:15:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:15:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:15:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:15:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:15:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:15:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:15:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:15:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:15:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:15:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:15:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:15:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:15:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:15:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:15:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:15:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:15:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:15:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:15:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:15:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:15:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:15:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:15:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:15:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:15:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:15:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:15:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:15:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:15:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:15:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:15:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:15:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:15:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:15:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:15:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:15:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:15:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:15:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:15:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:15:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:15:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:15:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:15:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:15:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:15:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:15:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:15:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:15:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:15:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:15:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:15:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:15:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:15:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:15:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:15:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:15:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:15:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:15:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:15:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:15:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:15:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:15:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:15:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:15:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:15:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:15:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:15:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:15:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:15:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:15:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:15:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:15:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:15:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:15:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:15:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:15:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:15:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:15:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:15:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:15:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:15:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:15:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:15:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:15:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:15:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:15:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:15:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:15:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:15:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:15:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:15:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:15:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:15:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:15:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:15:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:15:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:15:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:15:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:15:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:15:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:15:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:15:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:15:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:15:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:15:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:15:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:15:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:15:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:15:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:15:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:15:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:15:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:15:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:15:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:15:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:15:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:15:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:15:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:15:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:15:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:15:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:15:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:16:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:16:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:16:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:16:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:16:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:16:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:16:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:16:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:16:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:16:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:16:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:16:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:16:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:16:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:16:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:16:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:16:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:16:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:16:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:16:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:16:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:16:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:16:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:16:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:16:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:16:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:16:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:16:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:16:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:16:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:16:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:16:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:16:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:16:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:16:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:16:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:16:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:16:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:16:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:16:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:16:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:16:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:16:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:16:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:16:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:16:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:16:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:16:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:16:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:16:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:16:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:16:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:16:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:16:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:16:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:16:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:16:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:16:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:16:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:16:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:16:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:16:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:16:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:16:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:16:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:16:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:16:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:16:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:16:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:16:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:16:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:16:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:16:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:16:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:16:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:16:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:16:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:16:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:16:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:16:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:16:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:16:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:16:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:16:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:16:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:16:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:16:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:16:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:16:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:16:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:16:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:16:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:16:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:16:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:16:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:16:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:16:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:16:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:16:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:16:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:16:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:16:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:16:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:16:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:16:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:16:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:16:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:16:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:16:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:16:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:16:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:16:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:16:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:16:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:16:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:16:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:16:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:16:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:16:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:16:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:16:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:16:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:16:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:16:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:16:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:16:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:16:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:16:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:16:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:16:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:16:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:16:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:16:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:16:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:16:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:16:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:16:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:16:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:16:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:16:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:16:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:16:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:16:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:16:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:16:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:16:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:16:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:16:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:16:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:16:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:16:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:16:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:16:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:16:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:16:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:16:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:16:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:16:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:16:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:16:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:16:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:16:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:16:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:16:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:16:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:16:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:16:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:16:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:16:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:16:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:16:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:16:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:16:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:16:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:16:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:16:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:16:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:16:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:16:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:16:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:16:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:16:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:16:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:16:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:16:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:16:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:16:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:16:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:16:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:16:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:16:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:16:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:16:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:16:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:16:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:16:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:16:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:16:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:16:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:16:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:16:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:16:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:16:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:16:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:16:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:16:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:16:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:16:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:16:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:16:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:16:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:16:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:16:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:16:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:16:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:16:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:17:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:17:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:17:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:17:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:17:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:17:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:17:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:17:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:17:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:17:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:17:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:17:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:17:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:17:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:17:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:17:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:17:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:17:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:17:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:17:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:17:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:17:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:17:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:17:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:17:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:17:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:17:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:17:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:17:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:17:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:17:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:17:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:17:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:17:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:17:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:17:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:17:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:17:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:17:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:17:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:17:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:17:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:17:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:17:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:17:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:17:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:17:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:17:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:17:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:17:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:17:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:17:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:17:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:17:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:17:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:17:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:17:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:17:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:17:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:17:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:17:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:17:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:17:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:17:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:17:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:17:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:17:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:17:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:17:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:17:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:17:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:17:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:17:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:17:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:17:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:17:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:17:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:17:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:17:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:17:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:17:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:17:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:17:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:17:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:17:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:17:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:17:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:17:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:17:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:17:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:17:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:17:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:17:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:17:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:17:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:17:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:17:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:17:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:17:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:17:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:17:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:17:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:17:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:17:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:17:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:17:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:17:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:17:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:17:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:17:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:17:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:17:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:17:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:17:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:17:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:17:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:17:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:17:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:17:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:17:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:17:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:17:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:17:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:17:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:17:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:17:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:17:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:17:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:17:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:17:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:17:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:17:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:17:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:17:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:17:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:17:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:17:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:17:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:17:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:17:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:17:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:17:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:17:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:17:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:17:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:17:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:17:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:17:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:17:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:17:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:17:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:17:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:17:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:17:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:17:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:17:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:17:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:17:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:17:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:17:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:17:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:17:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:17:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:17:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:17:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:17:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:17:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:17:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:17:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:17:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:17:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:17:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:17:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:17:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:17:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:17:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:17:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:17:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:17:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:17:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:17:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:17:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:17:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:17:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:17:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:17:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:17:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:17:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:17:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:17:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:17:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:17:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:17:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:17:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:17:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:17:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:17:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:17:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:17:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:17:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:17:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:17:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:17:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:17:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:17:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:17:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:17:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:17:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:17:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:17:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:17:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:17:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:17:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:17:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:17:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:17:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:18:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:18:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:18:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:18:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:18:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:18:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:18:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:18:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:18:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:18:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:18:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:18:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:18:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:18:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:18:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:18:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:18:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:18:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:18:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:18:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:18:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:18:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:18:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:18:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:18:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:18:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:18:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:18:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:18:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:18:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:18:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:18:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:18:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:18:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:18:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:18:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:18:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:18:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:18:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:18:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:18:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:18:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:18:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:18:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:18:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:18:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:18:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:18:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:18:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:18:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:18:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:18:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:18:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:18:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:18:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:18:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:18:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:18:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:18:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:18:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:18:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:18:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:18:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:18:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:18:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:18:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:18:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:18:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:18:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:18:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:18:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:18:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:18:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:18:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:18:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:18:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:18:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:18:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:18:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:18:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:18:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:18:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:18:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:18:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:18:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:18:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:18:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:18:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:18:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:18:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:18:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:18:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:18:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:18:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:18:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:18:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:18:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:18:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:18:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:18:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:18:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:18:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:18:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:18:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:18:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:18:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:18:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:18:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:18:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:18:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:18:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:18:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:18:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:18:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:18:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:18:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:18:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:18:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:18:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:18:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:18:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:18:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:18:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:18:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:18:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:18:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:18:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:18:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:18:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:18:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:18:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:18:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:18:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:18:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:18:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:18:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:18:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:18:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:18:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:18:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:18:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:18:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:18:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:18:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:18:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:18:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:18:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:18:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:18:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:18:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:18:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:18:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:18:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:18:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:18:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:18:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:18:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:18:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:18:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:18:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:18:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:18:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:18:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:18:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:18:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:18:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:18:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:18:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:18:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:18:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:18:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:18:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:18:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:18:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:18:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:18:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:18:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:18:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:18:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:18:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:18:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:18:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:18:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:18:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:18:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:18:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:18:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:18:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:18:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:18:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:18:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:18:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:18:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:18:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:18:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:18:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:18:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:18:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:18:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:18:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:18:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:18:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:18:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:18:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:18:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:18:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:18:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:18:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:18:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:18:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:18:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:18:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:18:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:18:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:18:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:18:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:19:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:19:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:19:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:19:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:19:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:19:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:19:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:19:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:19:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:19:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:19:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:19:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:19:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:19:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:19:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:19:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:19:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:19:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:19:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:19:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:19:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:19:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:19:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:19:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:19:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:19:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:19:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:19:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:19:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:19:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:19:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:19:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:19:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:19:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:19:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:19:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:19:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:19:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:19:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:19:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:19:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:19:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:19:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:19:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:19:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:19:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:19:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:19:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:19:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:19:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:19:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:19:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:19:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:19:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:19:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:19:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:19:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:19:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:19:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:19:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:19:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:19:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:19:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:19:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:19:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:19:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:19:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:19:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:19:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:19:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:19:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:19:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:19:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:19:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:19:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:19:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:19:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:19:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:19:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:19:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:19:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:19:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:19:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:19:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:19:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:19:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:19:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:19:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:19:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:19:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:19:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:19:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:19:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:19:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:19:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:19:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:19:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:19:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:19:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:19:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:19:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:19:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:19:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:19:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:19:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:19:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:19:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:19:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:19:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:19:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:19:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:19:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:19:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:19:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:19:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:19:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:19:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:19:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:19:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:19:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:19:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:19:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:19:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:19:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:19:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:19:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:19:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:19:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:19:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:19:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:19:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:19:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:19:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:19:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:19:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:19:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:19:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:19:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:19:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:19:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:19:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:19:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:19:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:19:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:19:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:19:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:19:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:19:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:19:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:19:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:19:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:19:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:19:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:19:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:19:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:19:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:19:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:19:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:19:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:19:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:19:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:19:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:19:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:19:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:19:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:19:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:19:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:19:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:19:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:19:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:19:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:19:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:19:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:19:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:19:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:19:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:19:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:19:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:19:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:19:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:19:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:19:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:19:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:19:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:19:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:19:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:19:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:19:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:19:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:19:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:19:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:19:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:19:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:19:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:19:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:19:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:19:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:19:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:19:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:19:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:19:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:19:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:19:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:19:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:19:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:19:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:19:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:19:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:19:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:19:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:19:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:19:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:19:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:19:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:19:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:19:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:20:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:20:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:20:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:20:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:20:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:20:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:20:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:20:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:20:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:20:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:20:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:20:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:20:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:20:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:20:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:20:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:20:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:20:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:20:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:20:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:20:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:20:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:20:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:20:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:20:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:20:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:20:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:20:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:20:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:20:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:20:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:20:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:20:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:20:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:20:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:20:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:20:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:20:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:20:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:20:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:20:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:20:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:20:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:20:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:20:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:20:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:20:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:20:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:20:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:20:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:20:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:20:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:20:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:20:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:20:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:20:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:20:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:20:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:20:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:20:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:20:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:20:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:20:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:20:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:20:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:20:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:20:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:20:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:20:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:20:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:20:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:20:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:20:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:20:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:20:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:20:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:20:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:20:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:20:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:20:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:20:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:20:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:20:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:20:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:20:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:20:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:20:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:20:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:20:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:20:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:20:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:20:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:20:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:20:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:20:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:20:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:20:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:20:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:20:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:20:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:20:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:20:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:20:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:20:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:20:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:20:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:20:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:20:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:20:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:20:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:20:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:20:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:20:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:20:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:20:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:20:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:20:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:20:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:20:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:20:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:20:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:20:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:20:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:20:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:20:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:20:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:20:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:20:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:20:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:20:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:20:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:20:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:20:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:20:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:20:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:20:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:20:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:20:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:20:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:20:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:20:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:20:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:20:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:20:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:20:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:20:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:20:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:20:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:20:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:20:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:20:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:20:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:20:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:20:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:20:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:20:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:20:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:20:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:20:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:20:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:20:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:20:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:20:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:20:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:20:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:20:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:20:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:20:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:20:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:20:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:20:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:20:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:20:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:20:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:20:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:20:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:20:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:20:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:20:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:20:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:20:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:20:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:20:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:20:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:20:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:20:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:20:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:20:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:20:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:20:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:20:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:20:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:20:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:20:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:20:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:20:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:20:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:20:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:20:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:20:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:20:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:20:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:20:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:20:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:20:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:20:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:20:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:20:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:20:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:20:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:20:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:20:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:20:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:20:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:20:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:20:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:21:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:21:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:21:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:21:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:21:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:21:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:21:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:21:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:21:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:21:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:21:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:21:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:21:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:21:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:21:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:21:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:21:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:21:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:21:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:21:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:21:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:21:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:21:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:21:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:21:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:21:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:21:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:21:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:21:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:21:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:21:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:21:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:21:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:21:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:21:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:21:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:21:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:21:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:21:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:21:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:21:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:21:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:21:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:21:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:21:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:21:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:21:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:21:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:21:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:21:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:21:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:21:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:21:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:21:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:21:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:21:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:21:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:21:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:21:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:21:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:21:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:21:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:21:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:21:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:21:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:21:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:21:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:21:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:21:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:21:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:21:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:21:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:21:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:21:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:21:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:21:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:21:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:21:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:21:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:21:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:21:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:21:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:21:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:21:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:21:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:21:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:21:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:21:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:21:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:21:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:21:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:21:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:21:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:21:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:21:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:21:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:21:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:21:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:21:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:21:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:21:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:21:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:21:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:21:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:21:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:21:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:21:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:21:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:21:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:21:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:21:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:21:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:21:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:21:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:21:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:21:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:21:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:21:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:21:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:21:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:21:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:21:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:21:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:21:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:21:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:21:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:21:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:21:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:21:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:21:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:21:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:21:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:21:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:21:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:21:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:21:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:21:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:21:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:21:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:21:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:21:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:21:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:21:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:21:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:21:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:21:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:21:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:21:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:21:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:21:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:21:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:21:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:21:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:21:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:21:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:21:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:21:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:21:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:21:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:21:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:21:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:21:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:21:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:21:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:21:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:21:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:21:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:21:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:21:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:21:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:21:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:21:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:21:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:21:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:21:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:21:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:21:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:21:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:21:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:21:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:21:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:21:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:21:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:21:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:21:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:21:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:21:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:21:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:21:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:21:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:21:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:21:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:21:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:21:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:21:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:21:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:21:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:21:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:21:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:21:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:21:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:21:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:21:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:21:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:21:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:21:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:21:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:21:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:21:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:21:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:21:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:21:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:21:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:21:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:21:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:21:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:22:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:22:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:22:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:22:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:22:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:22:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:22:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:22:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:22:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:22:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:22:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:22:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:22:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:22:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:22:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:22:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:22:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:22:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:22:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:22:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:22:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:22:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:22:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:22:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:22:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:22:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:22:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:22:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:22:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:22:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:22:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:22:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:22:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:22:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:22:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:22:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:22:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:22:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:22:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:22:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:22:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:22:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:22:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:22:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:22:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:22:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:22:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:22:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:22:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:22:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:22:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:22:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:22:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:22:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:22:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:22:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:22:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:22:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:22:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:22:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:22:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:22:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:22:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:22:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:22:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:22:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:22:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:22:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:22:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:22:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:22:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:22:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:22:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:22:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:22:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:22:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:22:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:22:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:22:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:22:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:22:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:22:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:22:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:22:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:22:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:22:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:22:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:22:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:22:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:22:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:22:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:22:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:22:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:22:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:22:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:22:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:22:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:22:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:22:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:22:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:22:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:22:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:22:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:22:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:22:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:22:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:22:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:22:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:22:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:22:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:22:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:22:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:22:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:22:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:22:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:22:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:22:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:22:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:22:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:22:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:22:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:22:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:22:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:22:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:22:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:22:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:22:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:22:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:22:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:22:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:22:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:22:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:22:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:22:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:22:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:22:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:22:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:22:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:22:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:22:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:22:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:22:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:22:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:22:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:22:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:22:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:22:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:22:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:22:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:22:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:22:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:22:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:22:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:22:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:22:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:22:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:22:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:22:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:22:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:22:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:22:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:22:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:22:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:22:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:22:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:22:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:22:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:22:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:22:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:22:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:22:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:22:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:22:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:22:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:22:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:22:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:22:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:22:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:22:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:22:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:22:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:22:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:22:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:22:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:22:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:22:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:22:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:22:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:22:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:22:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:22:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:22:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:22:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:22:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:22:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:22:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:22:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:22:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:22:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:22:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:22:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:22:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:22:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:22:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:22:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:22:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:22:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:22:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:22:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:22:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:22:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:22:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:22:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:22:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:22:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:22:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:23:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:23:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:23:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:23:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:23:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:23:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:23:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:23:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:23:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:23:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:23:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:23:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:23:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:23:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:23:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:23:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:23:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:23:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:23:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:23:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:23:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:23:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:23:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:23:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:23:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:23:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:23:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:23:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:23:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:23:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:23:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:23:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:23:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:23:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:23:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:23:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:23:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:23:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:23:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:23:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:23:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:23:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:23:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:23:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:23:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:23:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:23:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:23:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:23:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:23:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:23:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:23:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:23:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:23:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:25:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:25:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:25:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:25:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:25:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:25:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:25:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:25:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:25:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:25:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:25:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:25:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:25:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:25:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:25:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:25:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:25:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:25:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:25:24 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 09:25:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:25:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:25:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:25:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:25:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:25:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:25:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:25:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:25:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:25:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:25:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:25:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:25:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:25:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:25:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:25:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:25:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:25:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:25:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:25:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:25:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:25:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:25:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:25:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:25:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:25:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:25:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:25:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:25:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:25:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:25:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:25:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:25:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:25:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:25:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:25:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:25:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:25:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:25:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:25:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:25:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:25:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:25:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:25:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:25:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:25:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:25:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:25:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:25:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:25:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:25:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:25:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:25:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:25:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:25:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:25:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:25:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:25:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:25:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:25:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:25:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:25:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:25:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:25:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:25:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:25:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:25:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:25:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:25:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:25:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:25:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:25:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:25:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:25:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:25:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:25:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:26:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:26:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:26:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:26:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:26:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:26:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:26:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:26:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:26:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:26:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:26:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:26:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:26:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:26:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:26:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:26:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:26:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:26:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:26:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:26:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:26:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:26:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:26:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:26:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:26:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:26:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:26:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:26:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:26:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:26:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:26:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:26:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:26:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:26:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:26:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:26:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:26:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:26:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:26:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:26:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:26:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:26:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:26:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:26:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:26:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:26:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:26:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:26:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:26:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:26:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:26:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:26:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:26:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:26:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:26:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:26:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:26:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:26:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:26:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:26:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:26:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:26:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:26:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:26:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:26:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:26:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:26:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:26:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:26:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:26:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:26:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:26:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:26:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:26:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:26:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:26:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:26:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:26:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:26:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:26:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:26:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:26:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:26:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:26:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:26:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:26:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:26:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:26:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:26:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:26:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:26:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:26:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:26:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:26:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:26:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:26:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:26:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:26:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:26:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:26:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:26:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:26:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:26:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:26:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:26:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:26:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:26:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:26:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:26:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:26:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:26:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:26:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:26:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:26:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:26:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:26:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:26:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:26:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:26:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:26:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:26:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:26:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:26:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:26:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:26:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:26:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:26:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:26:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:26:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:26:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:26:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:26:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:26:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:26:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:26:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:26:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:26:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:26:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:26:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:26:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:26:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:26:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:26:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:26:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:26:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:26:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:26:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:26:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:26:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:26:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:26:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:26:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:26:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:26:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:26:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:26:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:26:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:26:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:26:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:26:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:26:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:26:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:26:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:26:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:26:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:26:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:26:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:26:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:26:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:26:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:26:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:26:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:26:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:26:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:26:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:26:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:26:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:26:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:26:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:26:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:26:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:26:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:26:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:26:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:26:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:26:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:26:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:26:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:26:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:26:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:26:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:26:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:26:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:26:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:26:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:26:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:26:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:26:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:26:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:26:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:26:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:26:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:26:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:26:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:26:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:26:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:26:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:26:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:26:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:26:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:26:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:26:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:26:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:26:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:26:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:26:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:27:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:27:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:27:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:27:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:27:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:27:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:27:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:27:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:27:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:27:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:27:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:27:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:27:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:27:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:27:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:27:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:27:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:27:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:27:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:27:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:27:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:27:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:27:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:27:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:27:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:27:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:27:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:27:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:27:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:27:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:27:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:27:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:27:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:27:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:27:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:27:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:27:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:27:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:27:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:27:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:27:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:27:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:27:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:27:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:27:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:27:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:27:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:27:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:27:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:27:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:27:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:27:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:27:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:27:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:27:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:27:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:27:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:27:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:27:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:27:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:27:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:27:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:27:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:27:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:27:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:27:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:27:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:27:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:27:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:27:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:27:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:27:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:27:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:27:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:27:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:27:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:27:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:27:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:27:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:27:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:27:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:27:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:27:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:27:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:27:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:27:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:27:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:27:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:27:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:27:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:27:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:27:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:27:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:27:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:27:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:27:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:27:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:27:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:27:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:27:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:27:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:27:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:27:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:27:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:27:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:27:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:27:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:27:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:27:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:27:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:27:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:27:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:27:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:27:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:27:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:27:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:27:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:27:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:27:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:27:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:27:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:27:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:27:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:27:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:27:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:27:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:27:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:27:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:27:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:27:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:27:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:27:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:27:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:27:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:27:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:27:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:27:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:27:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:27:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:27:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:27:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:27:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:27:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:27:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:27:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:27:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:27:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:27:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:27:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:27:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:27:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:27:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:27:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:27:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:27:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:27:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:27:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:27:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:27:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:27:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:27:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:27:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:27:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:27:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:27:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:27:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:27:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:27:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:27:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:27:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:27:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:27:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:27:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:27:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:27:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:27:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:27:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:27:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:27:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:27:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:27:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:27:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:27:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:27:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:27:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:27:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:27:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:27:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:27:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:27:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:27:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:27:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:27:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:27:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:27:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:27:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:27:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:27:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:27:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:27:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:27:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:27:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:27:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:27:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:27:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:27:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:27:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:27:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:27:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:27:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:27:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:27:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:27:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:27:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:27:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:27:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:28:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:28:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:28:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:28:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:28:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:28:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:28:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:28:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:28:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:28:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:28:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:28:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:28:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:28:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:28:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:28:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:28:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:28:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:28:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:28:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:28:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:28:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:28:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:28:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:28:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:28:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:28:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:28:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:28:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:28:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:28:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:28:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:28:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:28:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:28:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:28:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:28:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:28:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:28:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:28:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:28:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:28:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:28:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:28:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:28:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:28:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:28:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:28:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:28:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:28:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:28:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:28:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:28:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:28:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:28:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:28:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:28:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:28:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:28:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:28:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:28:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:28:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:28:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:28:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:28:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:28:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:28:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:28:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:28:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:28:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:28:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:28:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:28:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:28:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:28:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:28:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:28:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:28:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:28:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:28:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:28:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:28:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:28:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:28:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:28:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:28:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:28:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:28:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:28:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:28:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:28:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:28:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:28:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:28:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:28:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:28:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:28:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:28:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:28:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:28:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:28:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:28:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:28:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:28:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:28:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:28:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:28:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:28:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:28:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:28:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:28:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:28:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:28:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:28:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:28:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:28:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:28:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:28:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:28:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:28:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:28:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:28:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:28:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:28:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:28:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:28:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:28:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:28:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:28:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:28:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:28:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:28:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:28:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:28:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:28:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:28:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:28:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:28:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:28:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:28:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:28:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:28:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:28:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:28:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:28:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:28:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:28:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:28:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:28:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:28:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:28:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:28:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:28:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:28:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:28:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:28:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:28:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:28:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:28:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:28:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:28:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:28:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:28:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:28:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:28:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:28:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:28:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:28:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:28:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:28:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:28:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:28:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:28:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:28:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:28:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:28:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:28:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:28:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:28:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:28:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:28:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:28:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:28:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:28:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:28:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:28:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:28:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:28:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:28:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:28:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:28:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:28:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:28:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:28:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:28:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:28:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:28:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:28:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:28:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:28:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:28:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:28:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:28:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:28:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:28:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:28:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:28:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:28:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:28:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:28:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:28:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:28:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:28:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:28:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:28:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:28:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:29:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:29:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:29:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:29:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:29:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:29:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:29:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:29:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:29:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:29:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:29:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:29:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:29:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:29:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:29:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:29:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:29:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:29:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:29:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:29:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:29:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:29:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:29:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:29:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:29:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:29:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:29:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:29:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:29:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:29:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:29:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:29:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:29:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:29:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:29:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:29:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:29:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:29:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:29:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:29:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:29:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:29:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:29:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:29:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:29:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:29:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:29:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:29:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:29:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:29:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:29:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:29:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:29:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:29:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:29:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:29:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:29:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:29:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:29:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:29:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:29:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:29:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:29:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:29:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:29:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:29:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:29:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:29:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:29:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:29:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:29:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:29:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:29:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:29:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:29:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:29:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:29:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:29:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:29:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:29:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:29:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:29:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:29:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:29:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:29:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:29:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:29:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:29:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:29:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:29:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:29:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:29:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:29:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:29:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:29:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:29:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:29:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:29:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:29:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:29:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:29:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:29:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:29:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:29:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:29:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:29:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:29:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:29:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:29:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:29:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:29:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:29:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:29:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:29:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:29:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:29:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:29:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:29:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:29:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:29:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:29:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:29:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:29:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:29:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:29:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:29:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:29:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:29:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:29:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:29:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:29:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:29:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:29:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:29:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:29:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:29:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:29:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:29:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:29:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:29:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:29:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:29:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:29:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:29:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:29:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:29:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:29:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:29:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:29:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:29:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:29:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:29:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:29:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:29:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:29:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:29:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:29:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:29:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:29:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:29:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:29:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:29:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:29:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:29:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:29:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:29:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:29:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:29:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:29:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:29:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:29:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:29:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:29:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:29:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:29:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:29:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:29:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:29:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:29:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:29:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:29:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:29:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:29:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:29:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:29:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:29:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:29:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:29:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:29:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:29:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:29:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:29:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:29:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:29:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:29:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:29:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:29:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:29:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:29:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:29:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:29:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:29:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:29:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:29:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:29:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:29:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:29:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:29:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:29:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:29:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:29:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:29:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:29:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:29:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:29:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:29:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:30:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:30:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:30:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:30:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:30:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:30:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:30:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:30:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:30:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:30:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:30:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:30:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:30:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:30:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:30:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:30:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:30:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:30:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:30:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:30:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:30:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:30:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:30:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:30:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:30:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:30:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:30:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:30:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:30:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:30:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:30:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:30:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:30:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:30:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:30:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:30:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:30:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:30:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:30:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:30:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:30:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:30:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:30:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:30:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:30:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:30:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:30:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:30:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:30:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:30:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:30:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:30:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:30:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:30:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:30:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:30:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:30:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:30:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:30:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:30:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:30:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:30:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:30:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:30:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:30:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:30:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:30:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:30:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:30:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:30:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:30:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:30:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:30:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:30:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:30:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:30:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:30:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:30:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:30:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:30:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:30:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:30:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:30:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:30:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:30:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:30:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:30:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:30:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:30:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:30:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:30:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:30:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:30:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:30:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:30:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:30:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:30:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:30:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:30:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:30:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:30:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:30:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:30:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:30:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:30:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:30:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:30:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:30:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:30:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:30:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:30:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:30:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:30:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:30:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:30:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:30:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:30:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:30:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:30:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:31:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:31:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:31:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:31:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:31:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:31:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:31:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:31:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:31:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:31:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:31:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:31:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:31:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:31:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:31:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:31:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:31:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:31:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:31:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:31:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:31:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:31:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:31:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:31:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:31:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:31:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:31:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:31:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:31:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:31:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:31:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:31:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:31:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:31:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:31:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:31:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:31:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:31:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:31:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:31:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:31:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:31:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:31:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:31:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:31:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:31:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:31:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:31:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:31:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:31:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:31:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:31:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:31:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:31:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:31:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:31:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:31:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:31:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:31:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:31:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:31:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:31:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:31:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:31:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:31:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:31:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:31:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:31:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:31:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:31:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:31:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:31:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:31:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:31:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:31:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:31:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:31:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:31:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:31:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:31:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:31:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:31:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:31:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:31:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:31:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:31:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:31:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:31:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:31:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:31:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:31:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:31:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:31:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:31:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:31:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:31:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:31:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:31:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:31:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:31:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:31:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:31:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:31:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:31:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:31:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:31:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:31:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:31:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:31:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:31:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:31:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:31:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:31:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:31:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:31:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:31:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:31:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:31:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:31:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:31:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:31:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:31:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:31:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:31:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:31:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:31:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:31:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:31:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:31:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:31:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:31:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:31:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:31:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:31:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:31:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:31:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:31:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:31:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:31:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:31:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:31:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:31:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:31:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:31:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:31:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:31:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:31:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:31:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:31:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:31:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:31:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:31:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:31:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:31:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:31:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:31:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:31:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:31:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:31:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:31:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:31:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:31:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:31:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:31:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:31:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:31:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:31:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:31:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:31:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:31:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:31:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:31:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:31:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:31:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:31:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:31:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:31:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:31:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:31:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:31:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:31:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:31:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:31:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:31:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:31:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:31:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:31:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:31:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:31:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:31:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:31:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:31:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:31:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:31:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:31:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:31:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:31:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:31:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:31:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:31:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:31:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:31:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:31:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:31:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:31:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:31:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:32:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:32:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:32:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:32:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:32:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:32:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:32:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:32:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:32:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:32:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:32:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:32:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:32:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:32:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:32:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:32:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:32:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:32:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:32:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:32:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:32:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:32:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:32:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:32:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:32:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:32:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:32:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:32:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:32:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:32:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:32:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:32:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:32:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:32:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:32:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:32:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:32:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:32:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:32:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:32:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:32:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:32:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:32:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:32:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:32:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:32:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:32:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:32:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:32:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:32:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:32:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:32:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:32:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:32:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:32:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:32:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:32:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:32:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:32:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:32:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:32:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:32:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:32:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:32:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:32:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:32:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:32:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:32:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:32:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:32:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:32:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:32:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:32:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:32:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:32:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:32:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:32:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:32:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:32:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:32:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:32:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:32:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:32:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:32:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:32:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:32:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:32:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:32:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:32:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:32:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:32:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:32:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:32:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:32:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:32:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:32:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:32:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:32:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:32:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:32:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:32:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:32:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:32:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:32:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:32:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:32:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:32:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:32:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:32:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:32:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:32:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:32:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:32:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:32:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:32:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:32:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:32:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:32:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:32:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:32:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:32:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:32:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:32:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:32:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:32:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:32:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:32:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:32:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:32:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:32:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:32:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:32:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:32:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:32:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:32:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:32:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:32:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:32:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:32:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:32:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:32:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:32:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:32:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:32:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:32:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:32:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:32:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:32:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:32:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:32:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:32:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:32:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:32:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:32:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:32:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:32:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:32:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:32:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:32:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:32:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:32:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:32:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:32:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:32:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:32:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:32:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:32:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:32:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:32:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:32:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:32:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:32:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:32:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:32:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:32:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:32:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:32:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:32:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:32:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:32:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:32:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:32:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:32:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:32:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:32:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:32:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:32:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:32:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:32:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:32:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:32:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:32:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:32:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:32:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:32:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:32:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:32:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:32:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:32:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:32:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:32:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:32:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:32:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:32:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:32:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:32:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:32:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:32:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:32:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:32:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:32:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:32:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:32:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:32:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:32:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:32:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:33:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:33:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:33:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:33:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:33:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:33:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:33:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:33:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:33:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:33:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:33:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:33:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:33:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:33:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:33:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:33:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:33:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:33:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:33:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:33:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:33:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:33:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:33:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:33:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:33:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:33:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:33:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:33:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:33:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:33:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:33:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:33:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:33:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:33:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:33:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:33:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:33:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:33:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:33:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:33:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:33:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:33:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:33:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:33:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:33:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:33:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:33:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:33:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:33:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:33:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:33:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:33:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:33:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:33:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:33:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:33:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:33:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:33:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:33:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:33:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:33:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:33:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:33:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:33:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:33:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:33:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:33:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:33:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:33:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:33:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:33:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:33:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:35:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:35:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:35:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:35:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:35:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:35:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:35:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:35:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:35:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:35:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:35:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:35:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:35:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:35:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:35:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:35:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:35:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:35:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:35:29 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 09:35:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:35:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:35:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:35:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:35:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:35:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:35:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:35:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:35:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:35:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:35:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:35:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:35:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:35:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:35:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:35:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:35:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:35:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:35:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:35:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:35:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:35:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:35:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:35:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:35:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:35:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:35:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:35:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:35:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:35:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:35:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:35:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:35:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:35:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:35:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:35:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:35:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:35:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:35:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:35:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:35:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:35:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:35:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:35:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:35:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:35:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:35:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:35:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:35:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:35:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:35:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:35:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:35:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:35:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:35:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:35:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:35:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:35:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:35:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:35:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:35:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:35:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:35:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:35:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:35:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:35:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:35:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:35:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:35:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:35:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:35:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:35:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:35:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:35:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:35:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:35:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:35:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:35:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:35:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:35:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:35:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:35:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:35:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:35:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:35:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:35:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:35:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:35:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:35:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:35:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:35:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:35:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:35:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:35:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:35:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:35:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:35:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:35:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:35:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:35:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:35:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:35:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:35:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:35:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:35:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:35:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:35:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:35:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:36:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:36:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:36:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:36:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:36:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:36:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:36:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:36:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:36:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:36:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:36:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:36:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:36:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:36:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:36:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:36:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:36:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:36:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:36:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:36:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:36:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:36:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:36:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:36:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:36:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:36:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:36:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:36:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:36:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:36:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:36:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:36:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:36:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:36:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:36:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:36:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:36:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:36:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:36:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:36:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:36:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:36:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:36:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:36:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:36:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:36:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:36:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:36:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:36:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:36:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:36:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:36:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:36:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:36:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:36:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:36:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:36:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:36:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:36:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:36:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:36:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:36:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:36:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:36:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:36:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:36:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:36:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:36:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:36:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:36:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:36:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:36:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:36:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:36:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:36:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:36:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:36:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:36:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:36:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:36:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:36:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:36:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:36:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:36:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:36:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:36:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:36:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:36:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:36:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:36:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:36:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:36:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:36:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:36:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:36:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:36:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:36:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:36:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:36:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:36:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:36:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:36:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:36:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:36:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:36:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:36:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:36:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:36:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:36:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:36:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:36:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:36:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:36:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:36:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:36:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:36:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:36:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:36:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:36:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:36:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:36:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:36:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:36:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:36:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:36:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:36:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:36:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:36:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:36:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:36:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:36:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:36:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:36:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:36:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:36:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:36:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:36:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:36:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:36:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:36:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:36:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:36:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:36:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:36:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:36:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:36:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:36:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:36:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:36:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:36:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:36:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:36:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:36:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:36:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:36:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:36:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:36:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:36:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:36:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:36:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:36:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:36:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:36:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:36:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:36:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:36:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:36:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:36:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:36:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:36:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:36:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:36:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:36:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:36:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:36:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:36:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:36:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:36:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:36:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:36:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:36:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:36:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:36:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:36:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:36:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:36:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:36:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:36:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:36:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:36:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:36:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:36:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:36:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:36:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:36:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:36:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:36:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:36:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:36:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:36:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:36:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:36:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:36:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:36:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:36:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:36:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:36:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:36:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:36:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:36:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:36:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:36:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:36:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:36:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:36:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:36:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:37:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:37:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:37:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:37:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:37:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:37:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:37:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:37:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:37:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:37:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:37:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:37:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:37:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:37:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:37:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:37:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:37:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:37:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:37:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:37:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:37:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:37:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:37:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:37:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:37:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:37:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:37:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:37:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:37:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:37:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:37:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:37:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:37:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:37:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:37:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:37:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:37:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:37:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:37:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:37:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:37:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:37:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:37:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:37:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:37:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:37:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:37:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:37:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:37:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:37:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:37:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:37:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:37:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:37:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:37:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:37:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:37:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:37:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:37:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:37:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:37:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:37:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:37:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:37:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:37:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:37:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:37:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:37:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:37:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:37:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:37:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:37:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:37:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:37:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:37:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:37:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:37:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:37:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:37:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:37:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:37:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:37:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:37:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:37:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:37:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:37:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:37:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:37:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:37:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:37:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:37:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:37:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:37:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:37:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:37:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:37:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:37:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:37:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:37:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:37:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:37:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:37:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:37:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:37:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:37:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:37:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:37:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:37:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:37:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:37:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:37:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:37:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:37:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:37:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:37:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:37:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:37:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:37:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:37:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:37:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:37:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:37:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:37:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:37:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:37:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:37:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:37:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:37:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:37:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:37:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:37:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:37:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:37:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:37:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:37:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:37:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:37:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:37:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:37:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:37:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:37:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:37:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:37:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:37:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:37:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:37:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:37:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:37:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:37:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:37:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:37:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:37:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:37:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:37:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:37:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:37:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:37:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:37:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:37:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:37:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:37:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:37:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:37:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:37:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:37:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:37:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:37:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:37:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:37:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:37:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:37:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:37:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:37:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:37:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:37:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:37:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:37:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:37:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:37:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:37:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:37:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:37:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:37:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:37:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:37:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:37:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:37:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:37:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:37:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:37:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:37:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:37:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:37:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:37:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:37:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:37:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:37:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:37:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:37:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:37:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:37:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:37:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:37:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:37:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:37:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:37:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:37:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:37:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:37:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:37:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:37:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:37:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:37:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:37:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:37:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:37:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:38:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:38:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:38:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:38:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:38:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:38:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:38:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:38:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:38:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:38:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:38:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:38:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:38:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:38:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:38:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:38:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:38:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:38:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:38:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:38:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:38:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:38:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:38:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:38:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:38:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:38:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:38:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:38:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:38:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:38:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:38:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:38:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:38:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:38:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:38:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:38:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:38:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:38:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:38:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:38:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:38:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:38:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:38:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:38:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:38:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:38:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:38:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:38:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:38:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:38:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:38:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:38:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:38:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:38:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:38:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:38:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:38:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:38:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:38:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:38:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:38:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:38:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:38:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:38:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:38:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:38:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:38:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:38:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:38:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:38:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:38:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:38:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:38:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:38:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:38:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:38:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:38:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:38:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:38:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:38:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:38:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:38:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:38:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:38:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:38:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:38:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:38:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:38:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:38:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:38:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:38:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:38:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:38:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:38:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:38:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:38:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:38:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:38:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:38:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:38:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:38:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:38:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:38:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:38:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:38:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:38:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:38:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:38:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:38:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:38:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:38:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:38:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:38:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:38:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:38:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:38:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:38:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:38:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:38:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:38:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:38:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:38:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:38:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:38:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:38:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:38:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:38:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:38:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:38:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:38:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:38:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:38:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:38:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:38:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:38:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:38:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:38:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:38:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:38:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:38:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:38:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:38:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:38:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:38:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:38:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:38:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:38:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:38:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:38:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:38:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:38:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:38:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:38:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:38:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:38:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:38:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:38:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:38:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:38:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:38:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:38:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:38:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:38:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:38:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:38:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:38:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:38:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:38:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:38:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:38:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:38:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:38:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:38:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:38:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:38:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:38:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:38:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:38:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:38:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:38:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:38:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:38:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:38:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:38:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:38:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:38:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:38:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:38:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:38:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:38:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:38:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:38:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:38:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:38:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:38:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:38:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:38:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:38:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:38:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:38:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:38:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:38:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:38:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:38:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:38:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:38:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:38:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:38:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:38:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:38:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:38:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:38:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:38:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:38:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:38:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:38:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:39:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:39:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:39:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:39:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:39:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:39:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:39:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:39:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:39:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:39:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:39:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:39:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:39:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:39:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:39:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:39:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:39:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:39:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:39:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:39:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:39:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:39:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:39:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:39:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:39:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:39:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:39:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:39:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:39:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:39:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:39:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:39:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:39:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:39:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:39:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:39:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:39:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:39:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:39:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:39:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:39:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:39:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:39:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:39:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:39:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:39:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:39:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:39:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:39:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:39:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:39:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:39:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:39:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:39:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:39:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:39:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:39:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:39:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:39:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:39:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:39:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:39:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:39:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:39:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:39:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:39:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:39:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:39:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:39:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:39:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:39:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:39:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:39:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:39:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:39:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:39:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:39:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:39:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:39:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:39:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:39:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:39:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:39:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:39:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:39:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:39:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:39:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:39:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:39:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:39:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:39:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:39:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:39:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:39:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:39:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:39:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:39:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:39:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:39:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:39:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:39:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:39:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:39:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:39:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:39:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:39:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:39:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:39:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:39:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:39:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:39:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:39:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:39:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:39:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:39:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:39:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:39:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:39:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:39:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:39:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:39:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:39:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:39:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:39:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:39:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:39:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:39:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:39:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:39:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:39:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:39:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:39:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:39:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:39:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:39:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:39:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:39:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:39:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:39:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:39:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:39:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:39:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:39:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:39:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:39:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:39:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:39:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:39:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:39:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:39:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:39:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:39:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:39:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:39:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:39:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:39:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:40:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:40:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:40:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:40:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:40:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:40:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:40:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:40:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:40:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:40:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:40:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:40:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:40:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:40:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:40:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:40:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:40:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:40:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:40:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:40:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:40:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:40:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:40:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:40:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:40:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:40:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:40:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:40:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:40:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:40:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:40:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:40:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:40:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:40:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:40:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:40:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:40:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:40:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:40:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:40:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:40:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:40:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:40:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:40:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:40:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:40:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:40:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:40:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:40:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:40:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:40:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:40:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:40:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:40:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:40:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:40:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:40:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:40:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:40:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:40:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:40:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:40:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:40:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:40:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:40:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:40:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:40:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:40:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:40:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:40:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:40:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:40:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:40:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:40:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:40:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:40:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:40:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:40:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:40:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:40:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:40:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:40:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:40:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:40:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:40:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:40:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:40:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:40:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:40:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:40:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:40:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:40:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:40:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:40:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:40:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:40:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:40:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:40:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:40:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:40:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:40:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:40:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:40:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:40:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:40:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:40:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:40:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:40:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:40:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:40:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:40:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:40:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:40:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:40:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:40:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:40:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:40:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:40:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:40:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:40:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:40:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:40:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:40:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:40:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:40:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:40:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:40:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:40:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:40:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:40:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:40:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:40:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:40:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:40:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:40:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:40:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:40:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:40:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:40:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:40:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:40:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:40:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:40:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:40:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:40:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:40:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:40:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:40:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:40:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:40:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:40:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:40:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:40:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:40:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:40:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:40:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:40:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:40:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:40:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:40:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:40:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:40:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:40:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:40:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:40:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:40:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:40:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:40:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:40:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:40:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:40:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:40:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:40:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:40:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:40:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:40:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:40:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:40:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:40:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:40:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:40:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:40:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:40:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:40:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:40:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:40:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:40:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:40:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:40:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:40:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:40:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:40:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:40:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:40:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:40:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:40:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:40:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:40:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:40:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:40:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:40:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:40:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:40:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:40:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:40:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:40:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:40:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:40:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:40:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:40:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:40:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:40:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:40:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:40:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:40:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:40:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:41:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:41:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:41:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:41:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:41:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:41:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:41:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:41:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:41:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:41:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:41:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:41:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:41:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:41:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:41:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:41:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:41:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:41:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:41:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:41:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:41:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:41:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:41:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:41:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:41:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:41:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:41:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:41:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:41:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:41:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:41:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:41:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:41:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:41:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:41:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:41:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:41:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:41:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:41:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:41:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:41:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:41:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:41:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:41:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:41:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:41:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:41:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:41:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:41:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:41:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:41:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:41:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:41:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:41:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:41:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:41:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:41:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:41:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:41:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:41:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:41:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:41:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:41:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:41:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:41:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:41:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:41:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:41:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:41:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:41:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:41:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:41:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:41:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:41:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:41:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:41:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:41:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:41:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:41:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:41:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:41:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:41:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:41:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:41:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:41:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:41:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:41:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:41:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:41:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:41:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:41:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:41:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:41:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:41:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:41:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:41:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:41:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:41:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:41:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:41:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:41:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:41:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:41:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:41:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:41:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:41:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:41:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:41:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:41:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:41:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:41:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:41:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:41:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:41:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:41:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:41:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:41:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:41:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:41:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:41:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:41:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:41:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:41:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:41:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:41:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:41:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:41:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:41:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:41:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:41:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:41:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:41:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:41:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:41:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:41:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:41:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:41:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:41:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:41:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:41:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:41:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:41:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:41:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:41:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:41:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:41:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:41:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:41:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:41:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:41:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:41:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:41:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:41:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:41:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:41:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:41:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:41:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:41:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:41:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:41:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:41:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:41:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:41:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:41:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:41:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:41:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:41:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:41:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:41:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:41:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:41:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:41:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:41:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:41:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:41:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:41:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:41:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:41:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:41:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:41:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:41:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:41:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:41:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:41:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:41:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:41:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:41:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:41:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:41:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:41:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:41:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:41:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:41:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:41:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:41:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:41:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:41:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:41:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:41:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:41:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:41:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:41:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:41:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:41:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:41:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:41:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:41:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:41:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:41:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:41:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:41:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:41:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:41:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:41:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:41:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:41:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:42:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:42:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:42:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:42:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:42:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:42:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:42:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:42:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:42:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:42:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:42:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:42:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:42:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:42:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:42:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:42:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:42:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:42:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:42:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:42:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:42:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:42:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:42:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:42:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:42:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:42:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:42:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:42:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:42:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:42:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:42:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:42:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:42:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:42:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:42:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:42:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:42:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:42:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:42:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:42:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:42:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:42:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:42:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:42:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:42:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:42:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:42:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:42:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:42:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:42:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:42:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:42:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:42:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:42:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:42:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:42:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:42:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:42:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:42:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:42:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:42:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:42:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:42:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:42:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:42:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:42:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:42:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:42:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:42:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:42:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:42:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:42:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:42:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:42:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:42:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:42:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:42:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:42:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:42:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:42:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:42:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:42:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:42:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:42:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:42:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:42:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:42:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:42:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:42:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:42:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:42:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:42:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:42:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:42:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:42:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:42:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:42:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:42:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:42:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:42:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:42:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:42:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:42:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:42:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:42:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:42:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:42:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:42:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:42:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:42:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:42:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:42:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:42:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:42:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:42:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:42:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:42:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:42:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:42:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:42:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:42:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:42:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:42:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:42:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:42:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:42:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:42:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:42:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:42:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:42:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:42:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:42:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:42:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:42:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:42:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:42:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:42:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:42:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:42:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:42:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:42:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:42:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:42:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:42:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:42:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:42:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:42:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:42:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:42:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:42:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:42:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:42:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:42:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:42:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:42:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:42:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:42:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:42:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:42:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:42:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:42:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:42:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:42:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:42:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:42:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:42:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:42:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:42:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:42:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:42:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:42:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:42:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:42:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:42:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:42:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:42:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:42:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:42:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:42:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:42:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:42:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:42:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:42:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:42:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:42:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:42:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:42:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:42:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:42:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:42:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:42:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:42:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:42:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:42:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:42:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:42:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:42:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:42:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:42:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:42:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:42:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:42:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:42:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:42:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:42:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:42:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:42:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:42:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:42:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:42:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:42:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:42:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:42:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:42:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:42:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:42:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:43:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:43:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:43:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:43:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:43:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:43:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:43:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:43:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:43:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:43:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:43:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:43:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:43:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:43:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:43:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:43:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:43:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:43:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:43:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:43:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:43:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:43:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:43:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:43:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:43:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:43:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:43:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:43:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:43:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:43:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:43:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:43:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:43:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:43:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:43:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:43:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:43:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:43:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:43:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:43:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:43:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:43:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:43:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:43:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:43:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:43:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:43:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:43:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:43:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:43:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:43:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:43:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:43:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:43:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:43:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:43:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:43:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:43:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:43:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:43:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:43:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:43:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:43:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:43:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:43:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:43:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:43:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:43:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:43:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:43:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:43:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:43:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:43:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:43:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:43:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:43:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:43:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:43:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:43:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:43:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:43:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:43:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:43:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:43:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:43:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:43:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:43:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:43:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:43:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:43:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:45:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:45:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:45:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:45:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:45:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:45:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:45:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:45:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:45:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:45:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:45:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:45:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:45:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:45:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:45:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:45:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:45:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:45:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:45:34 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 09:45:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:45:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:45:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:45:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:45:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:45:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:45:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:45:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:45:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:45:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:45:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:45:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:45:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:45:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:45:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:45:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:45:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:45:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:45:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:45:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:45:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:45:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:45:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:45:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:45:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:45:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:45:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:45:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:45:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:45:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:45:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:45:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:45:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:45:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:45:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:45:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:45:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:45:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:45:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:45:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:45:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:45:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:45:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:45:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:45:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:45:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:45:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:45:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:45:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:45:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:45:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:45:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:45:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:45:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:45:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:45:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:45:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:45:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:45:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:45:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:45:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:45:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:45:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:45:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:45:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:45:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:45:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:45:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:45:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:45:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:45:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:45:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:45:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:45:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:45:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:45:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:45:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:45:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:45:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:45:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:45:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:45:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:45:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:45:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:45:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:45:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:45:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:45:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:45:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:45:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:46:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:46:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:46:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:46:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:46:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:46:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:46:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:46:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:46:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:46:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:46:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:46:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:46:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:46:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:46:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:46:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:46:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:46:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:46:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:46:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:46:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:46:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:46:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:46:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:46:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:46:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:46:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:46:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:46:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:46:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:46:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:46:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:46:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:46:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:46:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:46:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:46:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:46:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:46:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:46:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:46:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:46:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:46:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:46:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:46:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:46:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:46:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:46:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:46:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:46:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:46:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:46:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:46:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:46:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:46:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:46:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:46:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:46:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:46:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:46:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:46:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:46:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:46:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:46:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:46:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:46:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:46:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:46:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:46:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:46:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:46:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:46:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:46:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:46:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:46:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:46:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:46:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:46:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:46:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:46:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:46:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:46:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:46:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:46:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:46:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:46:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:46:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:46:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:46:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:46:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:46:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:46:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:46:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:46:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:46:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:46:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:46:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:46:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:46:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:46:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:46:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:46:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:46:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:46:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:46:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:46:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:46:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:46:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:46:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:46:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:46:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:46:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:46:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:46:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:46:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:46:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:46:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:46:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:46:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:46:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:46:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:46:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:46:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:46:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:46:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:46:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:46:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:46:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:46:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:46:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:46:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:46:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:46:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:46:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:46:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:46:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:46:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:46:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:46:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:46:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:46:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:46:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:46:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:46:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:46:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:46:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:46:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:46:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:46:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:46:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:46:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:46:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:46:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:46:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:46:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:46:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:46:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:46:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:46:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:46:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:46:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:46:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:46:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:46:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:46:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:46:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:46:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:46:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:46:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:46:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:46:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:46:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:46:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:46:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:46:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:46:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:46:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:46:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:46:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:46:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:46:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:46:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:46:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:46:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:46:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:46:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:46:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:46:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:46:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:46:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:46:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:46:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:46:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:46:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:46:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:46:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:46:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:46:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:46:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:46:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:46:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:46:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:46:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:46:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:46:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:46:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:46:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:46:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:46:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:46:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:46:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:46:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:46:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:46:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:46:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:46:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:47:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:47:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:47:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:47:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:47:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:47:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:47:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:47:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:47:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:47:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:47:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:47:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:47:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:47:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:47:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:47:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:47:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:47:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:47:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:47:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:47:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:47:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:47:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:47:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:47:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:47:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:47:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:47:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:47:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:47:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:47:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:47:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:47:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:47:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:47:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:47:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:47:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:47:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:47:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:47:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:47:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:47:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:47:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:47:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:47:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:47:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:47:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:47:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:47:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:47:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:47:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:47:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:47:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:47:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:47:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:47:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:47:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:47:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:47:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:47:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:47:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:47:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:47:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:47:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:47:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:47:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:47:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:47:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:47:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:47:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:47:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:47:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:47:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:47:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:47:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:47:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:47:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:47:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:47:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:47:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:47:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:47:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:47:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:47:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:47:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:47:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:47:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:47:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:47:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:47:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:47:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:47:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:47:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:47:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:47:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:47:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:47:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:47:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:47:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:47:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:47:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:47:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:47:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:47:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:47:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:47:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:47:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:47:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:47:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:47:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:47:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:47:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:47:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:47:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:47:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:47:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:47:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:47:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:47:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:47:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:47:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:47:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:47:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:47:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:47:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:47:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:47:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:47:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:47:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:47:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:47:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:47:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:47:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:47:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:47:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:47:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:47:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:47:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:47:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:47:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:47:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:47:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:47:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:47:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:47:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:47:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:47:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:47:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:47:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:47:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:47:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:47:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:47:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:47:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:47:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:47:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:47:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:47:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:47:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:47:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:47:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:47:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:47:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:47:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:47:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:47:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:47:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:47:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:47:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:47:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:47:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:47:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:47:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:47:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:47:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:47:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:48:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:48:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:48:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:48:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:48:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:48:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:48:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:48:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:48:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:48:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:48:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:48:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:48:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:48:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:48:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:48:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:48:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:48:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:48:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:48:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:48:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:48:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:48:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:48:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:48:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:48:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:48:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:48:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:48:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:48:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:48:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:48:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:48:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:48:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:48:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:48:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:48:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:48:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:48:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:48:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:48:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:48:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:48:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:48:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:48:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:48:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:48:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:48:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:48:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:48:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:48:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:48:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:48:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:48:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:48:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:48:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:48:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:48:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:48:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:48:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:48:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:48:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:48:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:48:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:48:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:48:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:48:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:48:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:48:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:48:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:48:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:48:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:48:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:48:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:48:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:48:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:48:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:48:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:48:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:48:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:48:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:48:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:48:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:48:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:48:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:48:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:48:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:48:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:48:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:48:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:48:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:48:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:48:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:48:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:48:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:48:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:48:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:48:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:48:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:48:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:48:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:48:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:48:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:48:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:48:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:48:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:48:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:48:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:48:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:48:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:48:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:48:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:48:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:48:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:48:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:48:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:48:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:48:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:48:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:48:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:48:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:48:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:48:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:48:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:48:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:48:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:48:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:48:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:48:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:48:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:48:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:48:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:48:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:48:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:48:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:48:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:48:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:48:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:48:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:48:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:48:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:48:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:48:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:48:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:48:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:48:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:48:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:48:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:48:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:48:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:48:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:48:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:48:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:48:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:48:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:48:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:48:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:48:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:48:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:48:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:48:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:48:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:48:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:48:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:48:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:48:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:48:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:48:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:48:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:48:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:48:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:48:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:48:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:48:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:48:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:48:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:48:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:48:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:48:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:48:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:48:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:48:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:48:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:48:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:48:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:48:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:48:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:48:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:48:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:48:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:48:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:48:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:48:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:48:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:48:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:48:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:48:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:48:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:48:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:48:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:48:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:48:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:48:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:48:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:48:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:48:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:48:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:48:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:48:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:48:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:48:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:48:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:48:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:48:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:48:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:48:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:49:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:49:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:49:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:49:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:49:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:49:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:49:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:49:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:49:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:49:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:49:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:49:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:49:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:49:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:49:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:49:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:49:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:49:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:49:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:49:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:49:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:49:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:49:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:49:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:49:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:49:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:49:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:49:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:49:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:49:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:49:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:49:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:49:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:49:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:49:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:49:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:49:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:49:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:49:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:49:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:49:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:49:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:49:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:49:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:49:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:49:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:49:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:49:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:49:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:49:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:49:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:49:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:49:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:49:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:49:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:49:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:49:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:49:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:49:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:49:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:49:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:49:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:49:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:49:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:49:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:49:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:49:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:49:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:49:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:49:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:49:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:49:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:49:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:49:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:49:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:49:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:49:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:49:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:49:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:49:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:49:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:49:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:49:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:49:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:49:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:49:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:49:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:49:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:49:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:49:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:49:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:49:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:49:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:49:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:49:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:49:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:49:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:49:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:49:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:49:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:49:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:49:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:49:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:49:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:49:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:49:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:49:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:49:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:49:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:49:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:49:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:49:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:49:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:49:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:49:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:49:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:49:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:49:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:49:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:49:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:49:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:49:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:49:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:49:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:49:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:49:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:49:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:49:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:49:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:49:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:49:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:49:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:49:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:49:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:49:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:49:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:49:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:49:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:49:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:49:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:49:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:49:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:49:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:49:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:49:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:49:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:49:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:49:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:49:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:49:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:49:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:49:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:49:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:49:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:49:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:49:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:49:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:49:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:49:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:49:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:49:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:49:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:49:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:49:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:49:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:49:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:49:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:49:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:49:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:49:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:49:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:49:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:49:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:49:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:49:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:49:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:49:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:49:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:49:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:49:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:49:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:49:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:49:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:49:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:49:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:49:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:49:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:49:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:49:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:49:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:49:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:49:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:49:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:49:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:49:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:49:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:49:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:49:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:49:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:49:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:49:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:49:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:49:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:49:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:49:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:49:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:50:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:50:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:50:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:50:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:50:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:50:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:50:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:50:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:50:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:50:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:50:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:50:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:50:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:50:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:50:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:50:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:50:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:50:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:50:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:50:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:50:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:50:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:50:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:50:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:50:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:50:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:50:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:50:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:50:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:50:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:50:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:50:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:50:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:50:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:50:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:50:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:50:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:50:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:50:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:50:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:50:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:50:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:50:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:50:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:50:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:50:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:50:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:50:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:50:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:50:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:50:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:50:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:50:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:50:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:50:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:50:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:50:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:50:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:50:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:50:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:50:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:50:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:50:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:50:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:50:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:50:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:50:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:50:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:50:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:50:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:50:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:50:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:50:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:50:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:50:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:50:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:50:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:50:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:50:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:50:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:50:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:50:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:50:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:50:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:50:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:50:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:50:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:50:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:50:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:50:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:50:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:50:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:50:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:50:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:50:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:50:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:50:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:50:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:50:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:50:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:50:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:50:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:50:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:50:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:50:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:50:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:50:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:50:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:50:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:50:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:50:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:50:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:50:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:50:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:50:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:50:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:50:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:50:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:50:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:50:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:50:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:50:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:50:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:50:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:50:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:50:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:50:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:50:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:50:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:50:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:50:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:50:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:50:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:50:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:50:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:50:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:50:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:50:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:50:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:50:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:50:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:50:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:50:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:50:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:50:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:50:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:50:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:50:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:50:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:50:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:50:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:50:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:50:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:50:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:50:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:50:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:50:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:50:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:50:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:50:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:50:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:50:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:50:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:50:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:50:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:50:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:50:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:50:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:50:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:50:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:50:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:50:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:50:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:50:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:50:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:50:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:50:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:50:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:50:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:50:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:50:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:50:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:50:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:50:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:50:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:50:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:50:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:50:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:50:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:50:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:50:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:50:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:50:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:50:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:50:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:50:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:50:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:50:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:50:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:50:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:50:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:50:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:50:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:50:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:50:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:50:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:50:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:50:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:50:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:50:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:50:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:50:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:50:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:50:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:50:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:50:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:51:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:51:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:51:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:51:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:51:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:51:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:51:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:51:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:51:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:51:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:51:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:51:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:51:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:51:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:51:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:51:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:51:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:51:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:51:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:51:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:51:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:51:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:51:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:51:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:51:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:51:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:51:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:51:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:51:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:51:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:51:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:51:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:51:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:51:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:51:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:51:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:51:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:51:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:51:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:51:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:51:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:51:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:51:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:51:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:51:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:51:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:51:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:51:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:51:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:51:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:51:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:51:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:51:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:51:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:51:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:51:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:51:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:51:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:51:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:51:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:51:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:51:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:51:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:51:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:51:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:51:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:51:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:51:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:51:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:51:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:51:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:51:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:51:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:51:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:51:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:51:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:51:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:51:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:51:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:51:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:51:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:51:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:51:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:51:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:51:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:51:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:51:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:51:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:51:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:51:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:51:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:51:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:51:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:51:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:51:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:51:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:51:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:51:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:51:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:51:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:51:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:51:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:51:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:51:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:51:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:51:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:51:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:51:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:51:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:51:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:51:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:51:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:51:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:51:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:51:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:51:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:51:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:51:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:51:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:51:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:51:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:51:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:51:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:51:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:51:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:51:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:51:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:51:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:51:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:51:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:51:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:51:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:51:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:51:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:51:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:51:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:51:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:51:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:51:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:51:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:51:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:51:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:51:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:51:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:51:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:51:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:51:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:51:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:51:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:51:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:51:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:51:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:51:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:51:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:51:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:51:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:51:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:51:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:51:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:51:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:51:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:51:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:51:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:51:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:51:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:51:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:51:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:51:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:51:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:51:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:51:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:51:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:51:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:51:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:51:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:51:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:51:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:51:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:51:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:51:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:51:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:51:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:51:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:51:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:51:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:51:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:51:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:51:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:51:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:51:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:51:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:51:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:51:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:51:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:51:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:51:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:51:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:51:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:51:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:51:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:51:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:51:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:51:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:51:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:51:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:51:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:51:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:51:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:51:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:51:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:51:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:51:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:51:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:51:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:51:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:51:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:52:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:52:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:52:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:52:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:52:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:52:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:52:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:52:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:52:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:52:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:52:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:52:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:52:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:52:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:52:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:52:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:52:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:52:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:52:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:52:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:52:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:52:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:52:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:52:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:52:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:52:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:52:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:52:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:52:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:52:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:52:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:52:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:52:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:52:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:52:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:52:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:52:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:52:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:52:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:52:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:52:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:52:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:52:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:52:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:52:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:52:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:52:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:52:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:52:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:52:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:52:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:52:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:52:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:52:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:52:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:52:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:52:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:52:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:52:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:52:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:52:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:52:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:52:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:52:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:52:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:52:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:52:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:52:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:52:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:52:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:52:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:52:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:52:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:52:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:52:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:52:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:52:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:52:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:52:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:52:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:52:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:52:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:52:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:52:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:52:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:52:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:52:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:52:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:52:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:52:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:52:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:52:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:52:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:52:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:52:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:52:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:52:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:52:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:52:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:52:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:52:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:52:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:52:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:52:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:52:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:52:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:52:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:52:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:52:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:52:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:52:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:52:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:52:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:52:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:52:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:52:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:52:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:52:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:52:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:52:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:52:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:52:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:52:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:52:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:52:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:52:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:52:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:52:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:52:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:52:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:52:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:52:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:52:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:52:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:52:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:52:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:52:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:52:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:52:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:52:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:52:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:52:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:52:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:52:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:52:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:52:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:52:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:52:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:52:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:52:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:52:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:52:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:52:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:52:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:52:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:52:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:52:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:52:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:52:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:52:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:52:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:52:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:52:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:52:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:52:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:52:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:52:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:52:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:52:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:52:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:52:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:52:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:52:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:52:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:52:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:52:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:52:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:52:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:52:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:52:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:52:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:52:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:52:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:52:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:52:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:52:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:52:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:52:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:52:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:52:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:52:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:52:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:52:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:52:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:52:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:52:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:52:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:52:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:52:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:52:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:52:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:52:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:52:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:52:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:52:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:52:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:52:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:52:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:52:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:52:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:52:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:52:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:52:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:52:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:52:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:52:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:53:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:53:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:53:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:53:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:53:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:53:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:53:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:53:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:53:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:53:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:53:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:53:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:53:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:53:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:53:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:53:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:53:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:53:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:53:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:53:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:53:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:53:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:53:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:53:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:53:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:53:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:53:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:53:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:53:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:53:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:53:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:53:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:53:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:53:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:53:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:53:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:53:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:53:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:53:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:53:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:53:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:53:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:53:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:53:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:53:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:53:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:53:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:53:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:53:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:53:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:53:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:53:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:53:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:53:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:53:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:53:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:53:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:53:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:55:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:55:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:55:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:55:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:55:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:55:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:55:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:55:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:55:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:55:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:55:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:55:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:55:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:55:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:55:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:55:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:55:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:55:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:55:35 rdma06 rsyslogd[1041]: imjournal: 5082 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 09:55:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:55:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:55:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:55:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:55:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:55:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:55:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:55:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:55:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:55:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:55:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:55:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:55:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:55:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:55:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:55:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:55:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:55:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:55:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:55:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:55:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:55:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:55:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:55:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:55:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:55:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:55:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:55:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:55:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:55:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:55:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:55:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:55:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:55:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:55:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:55:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:55:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:55:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:55:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:55:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:55:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:55:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:55:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:55:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:55:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:55:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:55:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:55:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:55:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:55:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:55:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:55:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:55:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:55:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:55:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:55:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:55:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:55:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:55:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:55:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:55:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:55:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:55:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:55:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:55:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:55:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:55:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:55:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:55:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:55:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:55:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:55:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:55:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:55:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:55:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:55:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:55:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:55:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:55:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:55:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:55:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:55:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:55:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:55:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:55:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:55:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:55:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:55:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:55:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:55:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:56:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:56:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:56:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:56:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:56:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:56:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:56:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:56:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:56:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:56:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:56:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:56:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:56:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:56:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:56:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:56:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:56:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:56:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:56:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:56:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:56:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:56:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:56:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:56:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:56:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:56:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:56:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:56:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:56:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:56:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:56:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:56:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:56:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:56:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:56:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:56:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:56:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:56:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:56:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:56:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:56:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:56:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:56:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:56:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:56:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:56:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:56:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:56:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:56:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:56:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:56:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:56:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:56:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:56:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:56:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:56:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:56:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:56:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:56:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:56:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:56:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:56:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:56:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:56:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:56:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:56:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:56:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:56:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:56:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:56:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:56:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:56:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:56:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:56:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:56:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:56:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:56:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:56:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:56:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:56:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:56:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:56:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:56:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:56:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:56:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:56:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:56:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:56:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:56:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:56:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:56:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:56:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:56:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:56:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:56:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:56:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:56:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:56:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:56:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:56:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:56:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:56:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:56:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:56:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:56:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:56:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:56:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:56:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:56:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:56:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:56:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:56:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:56:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:56:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:56:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:56:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:56:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:56:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:56:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:56:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:56:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:56:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:56:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:56:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:56:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:56:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:56:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:56:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:56:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:56:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:56:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:56:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:56:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:56:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:56:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:56:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:56:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:56:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:56:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:56:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:56:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:56:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:56:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:56:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:56:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:56:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:56:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:56:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:56:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:56:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:56:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:56:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:56:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:56:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:56:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:56:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:56:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:56:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:56:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:56:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:56:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:56:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:56:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:56:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:56:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:56:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:56:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:56:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:56:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:56:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:56:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:56:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:56:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:56:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:56:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:56:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:56:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:56:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:56:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:56:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:56:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:56:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:56:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:56:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:56:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:56:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:56:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:56:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:56:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:56:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:56:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:56:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:56:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:56:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:56:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:56:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:56:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:56:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:56:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:56:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:56:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:56:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:56:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:56:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:56:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:56:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:56:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:56:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:56:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:56:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:56:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:56:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:56:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:56:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:56:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:56:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:57:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:57:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:57:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:57:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:57:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:57:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:57:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:57:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:57:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:57:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:57:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:57:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:57:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:57:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:57:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:57:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:57:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:57:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:57:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:57:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:57:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:57:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:57:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:57:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:57:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:57:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:57:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:57:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:57:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:57:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:57:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:57:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:57:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:57:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:57:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:57:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:57:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:57:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:57:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:57:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:57:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:57:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:57:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:57:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:57:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:57:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:57:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:57:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:57:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:57:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:57:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:57:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:57:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:57:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:57:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:57:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:57:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:57:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:57:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:57:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:57:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:57:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:57:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:57:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:57:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:57:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:57:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:57:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:57:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:57:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:57:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:57:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:57:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:57:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:57:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:57:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:57:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:57:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:57:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:57:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:57:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:57:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:57:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:57:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:57:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:57:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:57:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:57:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:57:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:57:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:57:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:57:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:57:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:57:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:57:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:57:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:57:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:57:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:57:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:57:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:57:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:57:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:57:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:57:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:57:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:57:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:57:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:57:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:57:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:57:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:57:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:57:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:57:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:57:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:57:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:57:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:57:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:57:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:57:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:57:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:57:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:57:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:57:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:57:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:57:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:57:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:57:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:57:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:57:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:57:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:57:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:57:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:57:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:57:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:57:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:57:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:57:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:57:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:57:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:57:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:57:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:57:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:57:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:57:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:57:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:57:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:57:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:57:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:57:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:57:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:57:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:57:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:57:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:57:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:57:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:57:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:57:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:57:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:57:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:57:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:57:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:57:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:57:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:57:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:57:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:57:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:57:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:57:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:57:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:57:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:57:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:57:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:57:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:57:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:57:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:57:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:57:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:57:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:57:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:57:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:57:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:57:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:57:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:57:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:57:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:57:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:57:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:57:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:57:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:57:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:57:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:57:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:57:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:57:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:57:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:57:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:57:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:57:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:57:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:57:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:57:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:57:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:57:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:57:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:57:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:57:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:57:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:57:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:57:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:57:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:57:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:57:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:57:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:57:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:57:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:57:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:58:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:58:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:58:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:58:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:58:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:58:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:58:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:58:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:58:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:58:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:58:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:58:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:58:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:58:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:58:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:58:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:58:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:58:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:58:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:58:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:58:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:58:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:58:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:58:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:58:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:58:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:58:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:58:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:58:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:58:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:58:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:58:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:58:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:58:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:58:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:58:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:58:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:58:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:58:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:58:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:58:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:58:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:58:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:58:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:58:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:58:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:58:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:58:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:58:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:58:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:58:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:58:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:58:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:58:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:58:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:58:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:58:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:58:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:58:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:58:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:58:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:58:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:58:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:58:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:58:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:58:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:58:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:58:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:58:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:58:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:58:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:58:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:58:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:58:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:58:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:58:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:58:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:58:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:58:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:58:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:58:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:58:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:58:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:58:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:58:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:58:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:58:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:58:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:58:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:58:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:58:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:58:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:58:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:58:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:58:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:58:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:58:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:58:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:58:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:58:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:58:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:58:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:58:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:58:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:58:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:58:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:58:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:58:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:58:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:58:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:58:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:58:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:58:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:58:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:58:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:58:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:58:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:58:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:58:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:58:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:58:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:58:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:58:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:58:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:58:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:58:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:58:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:58:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:58:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:58:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:58:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:58:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:58:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:58:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:58:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:58:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:58:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:58:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:58:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:58:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:58:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:58:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:58:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:58:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:58:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:58:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:58:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:58:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:58:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:58:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:58:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:58:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:58:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:58:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:58:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:58:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:58:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:58:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:58:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:58:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:58:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:58:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:58:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:58:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:58:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:58:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:58:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:58:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:58:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:58:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:58:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:58:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:58:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:58:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:58:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:58:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:58:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:58:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:58:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:58:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:58:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:58:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:58:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:58:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:58:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:58:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:58:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:58:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:58:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:58:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:58:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:58:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:58:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:58:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:58:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:58:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:58:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:58:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:58:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:58:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:58:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:58:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:58:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:58:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:58:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:58:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:58:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:58:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:58:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:58:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:58:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:58:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:58:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:58:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:58:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:58:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:59:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:59:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:59:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:59:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:59:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:59:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:59:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:59:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:59:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:59:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:59:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:59:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:59:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:59:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:59:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:59:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:59:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:59:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:59:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:59:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:59:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:59:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:59:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:59:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:59:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:59:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:59:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:59:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:59:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:59:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:59:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:59:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:59:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:59:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:59:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:59:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:59:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:59:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:59:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:59:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:59:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:59:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:59:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:59:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:59:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:59:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:59:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:59:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:59:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:59:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:59:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:59:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:59:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:59:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:59:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:59:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:59:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:59:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:59:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:59:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:59:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:59:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:59:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:59:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:59:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:59:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:59:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:59:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:59:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:59:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:59:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:59:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:59:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:59:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:59:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:59:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:59:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:59:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:59:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:59:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:59:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:59:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:59:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:59:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:59:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:59:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:59:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:59:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:59:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:59:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:59:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:59:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:59:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:59:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:59:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:59:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:59:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:59:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:59:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:59:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:59:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:59:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:59:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:59:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:59:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:59:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:59:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:59:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:59:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:59:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:59:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:59:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:59:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:59:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:59:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:59:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:59:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:59:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:59:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:59:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:59:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:59:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:59:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:59:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:59:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:59:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:59:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:59:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:59:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:59:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:59:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:59:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:59:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:59:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:59:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:59:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:59:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:59:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 09:59:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 09:59:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:59:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:59:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:59:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 09:59:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 09:59:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 09:59:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 09:59:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 09:59:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 09:59:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 09:59:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 09:59:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 09:59:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 09:59:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 09:59:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 09:59:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 09:59:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:00:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:00:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:00:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:00:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:00:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:00:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:00:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:00:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:00:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:00:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:00:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:00:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:00:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:00:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:00:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:00:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:00:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:00:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:00:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:00:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:00:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:00:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:00:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:00:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:00:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:00:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:00:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:00:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:00:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:00:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:00:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:00:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:00:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:00:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:00:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:00:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:00:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:00:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:00:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:00:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:00:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:00:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:00:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:00:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:00:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:00:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:00:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:00:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:00:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:00:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:00:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:00:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:00:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:00:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:00:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:00:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:00:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:00:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:00:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:00:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:00:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:00:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:00:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:00:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:00:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:00:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:00:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:00:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:00:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:00:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:00:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:00:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:00:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:00:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:00:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:00:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:00:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:00:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:00:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:00:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:00:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:00:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:00:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:00:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:00:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:00:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:00:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:00:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:00:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:00:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:00:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:00:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:00:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:00:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:00:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:00:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:00:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:00:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:00:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:00:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:00:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:00:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:00:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:00:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:00:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:00:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:00:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:00:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:00:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:00:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:00:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:00:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:00:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:00:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:00:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:00:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:00:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:00:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:00:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:00:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:00:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:00:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:00:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:00:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:00:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:00:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:00:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:00:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:00:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:00:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:00:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:00:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:00:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:00:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:00:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:00:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:00:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:00:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:00:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:00:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:00:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:00:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:00:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:00:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:00:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:00:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:00:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:00:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:00:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:00:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:00:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:00:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:00:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:00:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:00:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:00:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:00:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:00:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:00:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:00:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:00:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:00:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:00:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:00:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:00:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:00:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:00:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:00:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:00:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:00:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:00:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:01:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:01:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:01:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:01:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:01:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:01:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:01:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:01:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:01:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:01:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:01:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:01:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:01:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:01:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:01:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:01:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:01:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:01:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:01:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:01:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:01:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:01:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:01:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:01:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:01:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:01:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:01:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:01:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:01:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:01:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:01:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:01:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:01:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:01:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:01:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:01:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:01:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:01:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:01:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:01:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:01:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:01:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:01:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:01:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:01:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:01:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:01:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:01:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:01:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:01:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:01:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:01:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:01:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:01:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:01:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:01:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:01:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:01:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:01:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:01:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:01:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:01:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:01:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:01:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:01:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:01:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:01:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:01:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:01:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:01:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:01:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:01:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:01:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:01:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:01:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:01:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:01:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:01:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:01:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:01:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:01:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:01:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:01:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:01:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:01:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:01:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:01:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:01:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:01:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:01:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:01:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:01:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:01:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:01:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:01:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:01:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:01:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:01:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:01:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:01:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:01:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:01:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:01:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:01:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:01:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:01:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:01:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:01:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:01:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:01:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:01:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:01:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:01:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:01:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:01:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:01:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:01:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:01:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:01:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:01:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:01:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:01:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:01:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:01:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:01:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:01:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:01:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:01:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:01:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:01:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:01:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:01:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:01:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:01:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:01:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:01:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:01:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:01:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:01:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:01:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:01:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:01:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:01:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:01:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:01:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:01:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:01:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:01:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:01:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:01:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:01:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:01:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:01:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:01:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:01:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:01:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:02:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:02:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:02:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:02:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:02:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:02:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:02:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:02:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:02:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:02:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:02:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:02:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:02:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:02:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:02:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:02:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:02:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:02:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:02:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:02:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:02:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:02:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:02:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:02:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:02:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:02:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:02:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:02:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:02:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:02:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:02:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:02:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:02:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:02:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:02:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:02:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:02:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:02:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:02:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:02:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:02:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:02:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:02:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:02:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:02:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:02:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:02:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:02:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:02:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:02:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:02:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:02:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:02:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:02:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:02:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:02:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:02:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:02:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:02:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:02:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:02:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:02:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:02:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:02:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:02:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:02:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:02:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:02:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:02:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:02:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:02:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:02:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:02:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:02:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:02:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:02:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:02:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:02:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:02:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:02:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:02:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:02:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:02:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:02:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:02:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:02:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:02:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:02:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:02:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:02:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:02:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:02:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:02:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:02:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:02:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:02:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:02:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:02:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:02:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:02:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:02:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:02:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:02:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:02:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:02:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:02:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:02:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:02:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:02:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:02:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:02:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:02:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:02:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:02:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:02:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:02:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:02:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:02:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:02:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:02:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:02:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:02:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:02:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:02:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:02:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:02:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:02:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:02:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:02:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:02:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:02:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:02:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:02:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:02:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:02:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:02:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:02:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:02:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:02:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:02:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:02:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:02:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:02:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:02:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:02:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:02:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:02:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:02:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:02:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:02:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:02:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:02:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:02:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:02:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:02:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:02:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:02:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:02:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:02:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:02:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:02:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:02:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:02:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:02:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:02:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:02:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:02:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:02:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:02:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:02:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:02:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:02:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:02:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:02:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:02:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:02:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:02:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:02:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:02:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:02:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:02:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:02:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:02:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:02:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:02:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:02:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:02:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:02:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:02:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:02:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:02:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:02:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:02:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:02:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:02:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:02:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:02:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:02:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:02:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:02:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:02:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:02:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:02:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:02:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:02:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:02:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:02:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:02:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:02:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:02:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:02:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:02:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:02:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:02:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:02:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:02:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:03:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:03:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:03:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:03:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:03:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:03:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:03:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:03:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:03:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:03:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:03:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:03:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:03:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:03:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:03:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:03:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:03:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:03:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:03:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:03:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:03:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:03:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:03:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:03:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:03:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:03:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:03:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:03:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:03:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:03:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:03:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:03:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:03:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:03:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:03:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:03:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:03:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:03:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:03:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:03:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:03:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:03:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:03:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:03:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:03:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:03:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:03:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:03:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:03:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:03:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:03:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:03:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:03:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:03:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:03:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:03:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:03:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:03:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:03:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:03:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:03:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:03:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:03:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:03:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:03:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:03:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:03:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:03:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:03:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:03:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:03:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:03:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:03:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:03:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:03:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:03:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:03:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:03:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:03:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:03:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:03:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:03:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:03:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:03:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:03:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:03:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:03:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:03:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:03:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:03:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:03:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:03:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:03:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:03:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:03:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:03:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:03:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:03:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:03:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:03:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:03:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:03:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:03:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:03:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:03:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:03:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:03:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:03:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:03:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:03:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:03:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:03:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:03:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:03:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:03:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:03:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:03:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:03:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:03:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:03:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:03:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:03:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:03:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:03:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:03:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:03:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:05:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:05:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:05:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:05:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:05:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:05:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:05:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:05:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:05:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:05:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:05:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:05:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:05:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:05:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:05:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:05:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:05:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:05:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:05:40 rdma06 rsyslogd[1041]: imjournal: 5295 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 10:05:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:05:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:05:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:05:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:05:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:05:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:05:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:05:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:05:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:05:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:05:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:05:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:05:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:05:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:05:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:05:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:05:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:05:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:05:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:05:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:05:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:05:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:05:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:05:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:05:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:05:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:05:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:05:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:05:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:05:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:05:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:05:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:05:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:05:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:05:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:05:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:05:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:05:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:05:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:05:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:05:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:05:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:05:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:05:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:05:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:05:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:05:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:05:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:05:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:05:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:05:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:05:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:05:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:05:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:06:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:06:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:06:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:06:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:06:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:06:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:06:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:06:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:06:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:06:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:06:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:06:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:06:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:06:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:06:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:06:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:06:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:06:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:06:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:06:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:06:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:06:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:06:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:06:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:06:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:06:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:06:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:06:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:06:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:06:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:06:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:06:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:06:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:06:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:06:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:06:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:06:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:06:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:06:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:06:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:06:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:06:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:06:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:06:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:06:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:06:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:06:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:06:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:06:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:06:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:06:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:06:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:06:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:06:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:06:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:06:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:06:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:06:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:06:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:06:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:06:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:06:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:06:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:06:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:06:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:06:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:06:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:06:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:06:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:06:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:06:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:06:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:06:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:06:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:06:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:06:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:06:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:06:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:06:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:06:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:06:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:06:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:06:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:06:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:06:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:06:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:06:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:06:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:06:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:06:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:06:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:06:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:06:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:06:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:06:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:06:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:06:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:06:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:06:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:06:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:06:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:06:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:06:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:06:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:06:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:06:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:06:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:06:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:06:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:06:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:06:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:06:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:06:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:06:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:06:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:06:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:06:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:06:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:06:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:06:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:06:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:06:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:06:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:06:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:06:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:06:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:06:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:06:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:06:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:06:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:06:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:06:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:06:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:06:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:06:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:06:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:06:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:06:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:06:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:06:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:06:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:06:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:06:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:06:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:06:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:06:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:06:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:06:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:06:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:06:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:06:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:06:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:06:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:06:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:06:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:06:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:06:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:06:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:06:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:06:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:06:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:06:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:06:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:06:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:06:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:06:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:06:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:06:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:06:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:06:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:06:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:06:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:06:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:06:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:06:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:06:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:06:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:06:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:06:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:06:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:06:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:06:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:06:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:06:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:06:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:06:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:06:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:06:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:06:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:06:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:06:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:06:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:06:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:06:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:06:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:06:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:06:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:06:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:06:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:06:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:06:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:06:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:06:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:06:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:06:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:06:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:06:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:06:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:06:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:06:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:06:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:06:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:06:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:06:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:06:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:06:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:07:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:07:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:07:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:07:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:07:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:07:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:07:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:07:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:07:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:07:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:07:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:07:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:07:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:07:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:07:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:07:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:07:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:07:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:07:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:07:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:07:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:07:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:07:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:07:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:07:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:07:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:07:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:07:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:07:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:07:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:07:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:07:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:07:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:07:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:07:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:07:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:07:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:07:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:07:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:07:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:07:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:07:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:07:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:07:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:07:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:07:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:07:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:07:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:07:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:07:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:07:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:07:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:07:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:07:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:07:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:07:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:07:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:07:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:07:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:07:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:07:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:07:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:07:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:07:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:07:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:07:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:07:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:07:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:07:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:07:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:07:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:07:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:07:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:07:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:07:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:07:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:07:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:07:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:07:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:07:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:07:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:07:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:07:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:07:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:07:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:07:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:07:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:07:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:07:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:07:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:07:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:07:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:07:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:07:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:07:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:07:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:07:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:07:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:07:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:07:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:07:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:07:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:07:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:07:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:07:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:07:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:07:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:07:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:07:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:07:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:07:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:07:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:07:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:07:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:07:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:07:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:07:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:07:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:07:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:07:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:07:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:07:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:07:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:07:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:07:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:07:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:07:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:07:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:07:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:07:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:07:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:07:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:07:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:07:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:07:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:07:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:07:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:07:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:07:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:07:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:07:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:07:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:07:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:07:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:07:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:07:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:07:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:07:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:07:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:07:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:07:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:07:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:07:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:07:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:07:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:07:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:07:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:07:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:07:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:07:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:07:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:07:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:07:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:07:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:07:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:07:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:07:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:07:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:07:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:07:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:07:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:07:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:07:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:07:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:07:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:07:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:07:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:07:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:07:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:07:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:07:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:07:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:07:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:07:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:07:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:07:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:07:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:07:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:07:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:07:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:07:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:07:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:07:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:07:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:07:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:07:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:07:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:07:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:07:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:07:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:07:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:07:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:07:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:07:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:07:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:07:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:07:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:07:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:07:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:07:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:07:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:07:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:07:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:07:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:07:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:07:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:08:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:08:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:08:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:08:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:08:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:08:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:08:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:08:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:08:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:08:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:08:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:08:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:08:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:08:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:08:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:08:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:08:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:08:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:08:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:08:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:08:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:08:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:08:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:08:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:08:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:08:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:08:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:08:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:08:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:08:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:08:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:08:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:08:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:08:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:08:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:08:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:08:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:08:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:08:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:08:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:08:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:08:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:08:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:08:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:08:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:08:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:08:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:08:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:08:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:08:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:08:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:08:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:08:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:08:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:08:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:08:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:08:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:08:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:08:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:08:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:08:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:08:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:08:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:08:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:08:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:08:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:08:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:08:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:08:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:08:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:08:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:08:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:08:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:08:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:08:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:08:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:08:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:08:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:08:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:08:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:08:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:08:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:08:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:08:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:08:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:08:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:08:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:08:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:08:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:08:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:08:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:08:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:08:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:08:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:08:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:08:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:08:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:08:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:08:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:08:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:08:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:08:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:08:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:08:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:08:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:08:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:08:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:08:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:08:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:08:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:08:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:08:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:08:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:08:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:08:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:08:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:08:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:08:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:08:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:08:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:08:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:08:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:08:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:08:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:08:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:08:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:08:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:08:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:08:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:08:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:08:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:08:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:08:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:08:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:08:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:08:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:08:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:08:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:08:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:08:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:08:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:08:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:08:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:08:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:08:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:08:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:08:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:08:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:08:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:08:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:08:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:08:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:08:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:08:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:08:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:08:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:08:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:08:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:08:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:08:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:08:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:08:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:08:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:08:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:08:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:08:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:08:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:08:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:08:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:08:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:08:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:08:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:08:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:08:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:08:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:08:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:08:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:08:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:08:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:08:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:08:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:08:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:08:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:08:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:08:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:08:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:08:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:08:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:08:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:08:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:08:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:08:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:08:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:08:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:08:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:08:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:08:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:08:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:08:50 rdma06 systemd-journald[790]: Data hash table of /run/log/journal/95379d2f723847b68d9e976dee8967a9/system.journal has a fill level at 75.0 (163750 of 218332 items, 75497472 file size, 461 bytes per hash table item), suggesting rotation.
Apr 17 10:08:50 rdma06 systemd-journald[790]: /run/log/journal/95379d2f723847b68d9e976dee8967a9/system.journal: Journal header limits reached or header out-of-date, rotating.
Apr 17 10:08:50 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 17 10:08:50 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 17 10:08:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:08:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:08:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:08:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:08:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:08:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:08:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:08:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:08:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:08:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:08:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:08:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:08:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:08:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:08:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:08:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:08:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:08:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:09:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:09:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:09:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:09:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:09:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:09:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:09:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:09:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:09:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:09:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:09:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:09:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:09:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:09:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:09:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:09:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:09:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:09:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:09:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:09:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:09:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:09:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:09:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:09:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:09:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:09:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:09:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:09:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:09:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:09:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:09:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:09:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:09:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:09:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:09:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:09:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:09:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:09:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:09:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:09:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:09:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:09:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:09:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:09:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:09:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:09:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:09:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:09:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:09:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:09:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:09:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:09:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:09:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:09:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:09:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:09:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:09:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:09:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:09:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:09:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:09:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:09:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:09:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:09:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:09:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:09:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:09:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:09:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:09:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:09:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:09:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:09:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:09:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:09:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:09:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:09:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:09:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:09:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:09:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:09:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:09:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:09:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:09:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:09:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:09:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:09:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:09:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:09:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:09:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:09:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:09:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:09:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:09:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:09:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:09:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:09:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:09:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:09:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:09:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:09:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:09:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:09:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:09:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:09:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:09:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:09:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:09:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:09:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:09:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:09:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:09:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:09:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:09:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:09:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:09:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:09:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:09:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:09:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:09:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:09:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:09:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:09:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:09:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:09:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:09:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:09:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:09:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:09:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:09:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:09:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:09:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:09:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:09:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:09:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:09:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:09:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:09:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:09:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:09:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:09:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:09:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:09:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:09:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:09:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:09:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:09:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:09:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:09:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:09:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:09:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:09:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:09:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:09:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:09:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:09:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:09:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:09:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:09:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:09:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:09:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:09:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:09:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:09:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:09:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:09:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:09:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:09:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:09:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:09:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:09:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:09:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:09:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:09:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:09:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:09:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:09:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:09:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:09:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:09:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:09:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:09:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:09:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:09:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:09:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:09:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:09:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:09:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:09:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:09:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:09:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:09:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:09:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:09:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:09:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:09:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:09:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:09:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:09:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:09:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:09:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:09:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:09:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:09:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:09:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:09:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:09:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:09:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:09:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:09:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:09:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:09:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:09:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:09:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:09:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:09:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:09:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:10:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:10:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:10:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:10:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:10:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:10:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:10:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:10:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:10:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:10:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:10:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:10:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:10:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:10:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:10:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:10:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:10:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:10:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:10:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:10:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:10:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:10:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:10:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:10:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:10:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:10:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:10:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:10:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:10:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:10:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:10:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:10:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:10:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:10:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:10:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:10:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:10:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:10:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:10:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:10:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:10:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:10:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:10:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:10:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:10:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:10:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:10:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:10:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:10:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:10:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:10:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:10:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:10:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:10:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:10:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:10:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:10:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:10:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:10:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:10:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:10:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:10:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:10:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:10:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:10:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:10:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:10:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:10:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:10:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:10:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:10:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:10:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:10:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:10:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:10:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:10:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:10:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:10:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:10:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:10:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:10:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:10:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:10:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:10:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:10:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:10:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:10:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:10:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:10:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:10:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:10:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:10:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:10:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:10:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:10:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:10:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:10:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:10:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:10:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:10:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:10:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:10:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:10:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:10:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:10:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:10:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:10:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:10:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:10:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:10:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:10:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:10:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:10:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:10:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:10:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:10:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:10:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:10:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:10:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:10:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:10:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:10:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:10:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:10:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:10:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:10:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:10:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:10:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:10:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:10:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:10:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:10:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:10:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:10:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:10:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:10:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:10:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:10:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:10:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:10:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:10:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:10:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:10:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:10:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:10:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:10:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:10:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:10:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:10:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:10:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:10:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:10:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:10:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:10:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:10:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:10:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:10:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:10:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:10:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:10:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:10:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:10:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:10:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:10:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:10:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:10:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:10:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:10:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:10:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:10:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:10:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:10:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:10:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:10:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:10:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:10:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:10:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:10:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:10:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:10:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:10:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:10:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:10:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:10:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:10:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:10:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:10:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:10:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:10:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:10:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:10:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:10:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:10:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:10:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:10:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:10:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:10:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:10:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:10:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:10:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:10:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:10:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:10:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:10:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:10:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:10:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:10:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:10:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:10:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:10:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:10:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:10:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:10:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:10:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:10:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:10:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:11:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:11:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:11:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:11:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:11:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:11:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:11:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:11:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:11:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:11:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:11:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:11:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:11:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:11:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:11:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:11:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:11:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:11:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:11:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:11:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:11:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:11:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:11:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:11:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:11:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:11:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:11:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:11:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:11:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:11:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:11:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:11:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:11:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:11:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:11:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:11:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:11:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:11:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:11:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:11:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:11:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:11:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:11:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:11:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:11:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:11:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:11:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:11:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:11:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:11:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:11:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:11:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:11:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:11:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:11:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:11:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:11:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:11:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:11:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:11:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:11:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:11:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:11:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:11:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:11:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:11:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:11:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:11:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:11:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:11:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:11:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:11:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:11:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:11:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:11:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:11:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:11:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:11:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:11:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:11:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:11:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:11:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:11:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:11:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:11:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:11:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:11:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:11:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:11:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:11:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:11:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:11:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:11:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:11:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:11:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:11:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:11:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:11:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:11:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:11:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:11:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:11:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:11:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:11:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:11:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:11:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:11:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:11:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:11:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:11:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:11:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:11:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:11:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:11:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:11:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:11:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:11:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:11:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:11:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:11:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:11:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:11:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:11:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:11:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:11:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:11:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:11:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:11:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:11:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:11:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:11:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:11:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:11:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:11:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:11:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:11:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:11:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:11:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:11:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:11:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:11:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:11:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:11:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:11:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:11:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:11:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:11:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:11:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:11:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:11:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:11:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:11:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:11:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:11:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:11:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:11:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:11:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:11:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:11:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:11:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:11:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:11:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:11:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:11:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:11:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:11:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:11:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:11:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:11:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:11:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:11:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:11:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:11:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:11:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:11:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:11:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:11:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:11:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:11:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:11:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:11:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:11:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:11:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:11:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:11:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:11:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:12:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:12:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:12:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:12:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:12:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:12:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:12:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:12:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:12:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:12:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:12:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:12:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:12:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:12:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:12:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:12:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:12:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:12:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:12:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:12:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:12:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:12:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:12:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:12:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:12:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:12:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:12:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:12:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:12:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:12:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:12:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:12:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:12:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:12:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:12:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:12:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:12:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:12:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:12:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:12:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:12:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:12:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:12:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:12:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:12:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:12:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:12:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:12:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:12:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:12:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:12:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:12:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:12:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:12:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:12:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:12:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:12:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:12:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:12:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:12:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:12:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:12:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:12:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:12:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:12:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:12:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:12:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:12:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:12:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:12:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:12:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:12:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:12:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:12:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:12:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:12:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:12:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:12:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:12:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:12:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:12:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:12:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:12:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:12:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:12:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:12:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:12:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:12:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:12:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:12:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:12:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:12:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:12:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:12:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:12:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:12:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:12:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:12:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:12:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:12:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:12:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:12:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:12:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:12:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:12:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:12:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:12:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:12:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:12:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:12:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:12:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:12:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:12:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:12:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:12:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:12:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:12:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:12:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:12:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:12:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:12:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:12:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:12:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:12:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:12:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:12:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:12:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:12:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:12:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:12:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:12:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:12:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:12:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:12:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:12:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:12:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:12:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:12:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:12:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:12:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:12:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:12:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:12:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:12:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:12:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:12:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:12:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:12:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:12:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:12:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:12:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:12:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:12:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:12:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:12:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:12:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:12:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:12:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:12:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:12:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:12:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:12:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:12:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:12:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:12:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:12:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:12:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:12:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:12:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:12:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:12:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:12:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:12:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:12:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:12:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:12:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:12:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:12:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:12:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:12:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:12:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:12:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:12:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:12:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:12:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:12:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:12:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:12:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:12:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:12:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:12:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:12:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:12:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:12:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:12:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:12:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:12:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:12:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:12:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:12:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:12:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:12:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:12:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:12:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:12:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:12:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:12:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:12:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:12:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:12:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:12:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:12:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:12:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:12:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:12:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:12:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:13:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:13:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:13:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:13:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:13:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:13:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:13:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:13:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:13:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:13:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:13:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:13:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:13:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:13:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:13:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:13:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:13:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:13:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:13:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:13:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:13:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:13:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:13:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:13:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:13:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:13:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:13:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:13:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:13:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:13:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:13:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:13:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:13:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:13:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:13:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:13:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:13:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:13:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:13:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:13:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:13:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:13:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:13:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:13:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:13:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:13:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:13:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:13:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:13:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:13:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:13:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:13:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:13:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:13:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:13:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:13:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:13:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:13:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:13:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:13:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:13:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:13:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:13:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:13:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:13:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:13:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:13:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:13:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:13:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:13:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:13:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:13:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:13:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:13:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:13:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:13:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:13:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:13:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:13:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:13:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:13:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:13:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:13:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:13:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:13:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:13:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:13:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:13:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:13:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:13:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:13:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:13:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:13:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:13:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:13:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:13:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:13:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:13:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:13:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:13:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:13:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:13:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:13:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:13:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:13:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:13:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:13:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:13:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:13:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:13:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:13:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:13:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:13:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:13:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:13:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:13:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:13:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:13:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:13:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:13:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:13:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:13:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:13:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:13:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:13:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:13:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:13:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:13:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:13:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:13:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:13:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:13:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:13:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:13:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:13:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:13:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:13:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:13:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:13:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:13:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:13:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:13:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:13:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:13:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:15:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:15:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:15:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:15:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:15:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:15:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:15:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:15:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:15:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:15:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:15:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:15:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:15:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:15:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:15:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:15:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:15:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:15:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:15:45 rdma06 rsyslogd[1041]: imjournal: 5295 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 10:15:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:15:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:15:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:15:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:15:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:15:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:15:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:15:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:15:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:15:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:15:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:15:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:15:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:15:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:15:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:15:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:15:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:15:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:15:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:15:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:15:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:15:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:15:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:15:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:15:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:15:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:15:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:15:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:15:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:15:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:15:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:15:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:15:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:15:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:15:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:15:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:16:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:16:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:16:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:16:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:16:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:16:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:16:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:16:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:16:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:16:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:16:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:16:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:16:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:16:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:16:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:16:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:16:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:16:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:16:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:16:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:16:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:16:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:16:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:16:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:16:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:16:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:16:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:16:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:16:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:16:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:16:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:16:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:16:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:16:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:16:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:16:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:16:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:16:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:16:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:16:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:16:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:16:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:16:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:16:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:16:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:16:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:16:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:16:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:16:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:16:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:16:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:16:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:16:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:16:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:16:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:16:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:16:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:16:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:16:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:16:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:16:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:16:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:16:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:16:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:16:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:16:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:16:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:16:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:16:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:16:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:16:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:16:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:16:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:16:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:16:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:16:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:16:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:16:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:16:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:16:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:16:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:16:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:16:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:16:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:16:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:16:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:16:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:16:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:16:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:16:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:16:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:16:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:16:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:16:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:16:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:16:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:16:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:16:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:16:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:16:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:16:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:16:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:16:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:16:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:16:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:16:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:16:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:16:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:16:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:16:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:16:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:16:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:16:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:16:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:16:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:16:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:16:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:16:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:16:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:16:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:16:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:16:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:16:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:16:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:16:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:16:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:16:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:16:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:16:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:16:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:16:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:16:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:16:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:16:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:16:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:16:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:16:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:16:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:16:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:16:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:16:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:16:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:16:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:16:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:16:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:16:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:16:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:16:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:16:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:16:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:16:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:16:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:16:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:16:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:16:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:16:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:16:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:16:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:16:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:16:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:16:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:16:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:16:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:16:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:16:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:16:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:16:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:16:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:16:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:16:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:16:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:16:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:16:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:16:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:16:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:16:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:16:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:16:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:16:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:16:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:16:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:16:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:16:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:16:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:16:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:16:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:16:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:16:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:16:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:16:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:16:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:16:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:16:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:16:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:16:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:16:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:16:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:16:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:16:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:16:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:16:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:16:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:16:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:16:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:16:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:16:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:16:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:16:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:16:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:16:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:16:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:16:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:16:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:16:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:16:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:16:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:17:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:17:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:17:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:17:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:17:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:17:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:17:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:17:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:17:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:17:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:17:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:17:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:17:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:17:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:17:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:17:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:17:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:17:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:17:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:17:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:17:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:17:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:17:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:17:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:17:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:17:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:17:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:17:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:17:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:17:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:17:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:17:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:17:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:17:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:17:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:17:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:17:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:17:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:17:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:17:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:17:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:17:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:17:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:17:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:17:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:17:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:17:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:17:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:17:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:17:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:17:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:17:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:17:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:17:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:17:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:17:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:17:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:17:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:17:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:17:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:17:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:17:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:17:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:17:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:17:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:17:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:17:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:17:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:17:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:17:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:17:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:17:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:17:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:17:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:17:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:17:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:17:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:17:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:17:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:17:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:17:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:17:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:17:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:17:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:17:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:17:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:17:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:17:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:17:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:17:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:17:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:17:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:17:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:17:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:17:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:17:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:17:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:17:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:17:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:17:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:17:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:17:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:17:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:17:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:17:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:17:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:17:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:17:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:17:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:17:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:17:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:17:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:17:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:17:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:17:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:17:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:17:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:17:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:17:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:17:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:17:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:17:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:17:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:17:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:17:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:17:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:17:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:17:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:17:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:17:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:17:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:17:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:17:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:17:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:17:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:17:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:17:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:17:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:17:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:17:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:17:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:17:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:17:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:17:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:17:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:17:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:17:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:17:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:17:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:17:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:17:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:17:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:17:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:17:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:17:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:17:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:17:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:17:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:17:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:17:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:17:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:17:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:17:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:17:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:17:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:17:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:17:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:17:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:17:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:17:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:17:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:17:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:17:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:17:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:17:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:17:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:17:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:17:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:17:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:17:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:17:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:17:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:17:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:17:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:17:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:17:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:17:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:17:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:17:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:17:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:17:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:17:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:17:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:17:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:17:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:17:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:17:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:17:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:17:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:17:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:17:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:17:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:17:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:17:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:17:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:17:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:17:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:17:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:17:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:17:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:17:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:17:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:17:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:17:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:17:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:17:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:18:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:18:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:18:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:18:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:18:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:18:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:18:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:18:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:18:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:18:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:18:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:18:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:18:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:18:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:18:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:18:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:18:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:18:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:18:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:18:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:18:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:18:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:18:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:18:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:18:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:18:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:18:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:18:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:18:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:18:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:18:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:18:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:18:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:18:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:18:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:18:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:18:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:18:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:18:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:18:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:18:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:18:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:18:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:18:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:18:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:18:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:18:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:18:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:18:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:18:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:18:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:18:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:18:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:18:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:18:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:18:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:18:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:18:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:18:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:18:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:18:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:18:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:18:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:18:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:18:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:18:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:18:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:18:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:18:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:18:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:18:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:18:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:18:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:18:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:18:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:18:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:18:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:18:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:18:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:18:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:18:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:18:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:18:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:18:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:18:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:18:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:18:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:18:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:18:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:18:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:18:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:18:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:18:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:18:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:18:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:18:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:18:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:18:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:18:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:18:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:18:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:18:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:18:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:18:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:18:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:18:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:18:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:18:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:18:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:18:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:18:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:18:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:18:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:18:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:18:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:18:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:18:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:18:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:18:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:18:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:18:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:18:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:18:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:18:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:18:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:18:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:18:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:18:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:18:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:18:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:18:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:18:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:18:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:18:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:18:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:18:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:18:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:18:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:18:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:18:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:18:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:18:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:18:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:18:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:18:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:18:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:18:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:18:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:18:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:18:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:18:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:18:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:18:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:18:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:18:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:18:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:18:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:18:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:18:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:18:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:18:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:18:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:18:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:18:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:18:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:18:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:18:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:18:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:18:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:18:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:18:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:18:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:18:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:18:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:18:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:18:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:18:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:18:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:18:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:18:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:18:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:18:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:18:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:18:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:18:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:18:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:18:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:18:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:18:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:18:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:18:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:18:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:18:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:18:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:18:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:18:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:18:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:18:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:18:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:18:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:18:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:18:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:18:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:18:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:18:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:18:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:18:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:18:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:18:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:18:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:18:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:18:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:18:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:18:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:18:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:18:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:19:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:19:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:19:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:19:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:19:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:19:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:19:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:19:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:19:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:19:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:19:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:19:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:19:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:19:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:19:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:19:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:19:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:19:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:19:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:19:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:19:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:19:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:19:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:19:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:19:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:19:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:19:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:19:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:19:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:19:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:19:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:19:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:19:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:19:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:19:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:19:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:19:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:19:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:19:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:19:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:19:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:19:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:19:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:19:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:19:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:19:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:19:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:19:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:19:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:19:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:19:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:19:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:19:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:19:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:19:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:19:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:19:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:19:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:19:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:19:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:19:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:19:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:19:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:19:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:19:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:19:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:19:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:19:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:19:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:19:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:19:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:19:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:19:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:19:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:19:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:19:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:19:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:19:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:19:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:19:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:19:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:19:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:19:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:19:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:19:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:19:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:19:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:19:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:19:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:19:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:19:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:19:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:19:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:19:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:19:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:19:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:19:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:19:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:19:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:19:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:19:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:19:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:19:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:19:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:19:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:19:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:19:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:19:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:19:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:19:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:19:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:19:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:19:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:19:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:19:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:19:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:19:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:19:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:19:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:19:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:19:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:19:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:19:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:19:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:19:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:19:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:19:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:19:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:19:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:19:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:19:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:19:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:19:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:19:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:19:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:19:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:19:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:19:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:19:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:19:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:19:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:19:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:19:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:19:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:19:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:19:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:19:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:19:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:19:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:19:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:19:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:19:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:19:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:19:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:19:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:19:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:19:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:19:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:19:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:19:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:19:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:19:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:19:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:19:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:19:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:19:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:19:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:19:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:19:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:19:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:19:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:19:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:19:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:19:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:19:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:19:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:19:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:19:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:19:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:19:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:19:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:19:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:19:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:19:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:19:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:19:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:19:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:19:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:19:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:19:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:19:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:19:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:19:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:19:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:19:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:19:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:19:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:19:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:19:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:19:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:19:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:19:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:19:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:19:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:19:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:19:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:19:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:19:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:19:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:19:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:19:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:19:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:19:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:19:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:19:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:19:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:20:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:20:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:20:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:20:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:20:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:20:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:20:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:20:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:20:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:20:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:20:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:20:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:20:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:20:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:20:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:20:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:20:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:20:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:20:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:20:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:20:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:20:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:20:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:20:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:20:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:20:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:20:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:20:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:20:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:20:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:20:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:20:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:20:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:20:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:20:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:20:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:20:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:20:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:20:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:20:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:20:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:20:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:20:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:20:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:20:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:20:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:20:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:20:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:20:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:20:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:20:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:20:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:20:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:20:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:20:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:20:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:20:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:20:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:20:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:20:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:20:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:20:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:20:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:20:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:20:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:20:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:20:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:20:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:20:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:20:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:20:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:20:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:20:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:20:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:20:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:20:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:20:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:20:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:20:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:20:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:20:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:20:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:20:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:20:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:20:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:20:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:20:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:20:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:20:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:20:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:20:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:20:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:20:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:20:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:20:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:20:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:20:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:20:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:20:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:20:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:20:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:20:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:20:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:20:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:20:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:20:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:20:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:20:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:20:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:20:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:20:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:20:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:20:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:20:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:20:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:20:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:20:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:20:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:20:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:20:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:20:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:20:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:20:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:20:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:20:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:20:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:20:32 rdma06 NetworkManager[1025]: <info>  [1681741232.9304] dhcp4 (eno1): state changed new lease, address=10.73.131.45
Apr 17 10:20:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:20:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:20:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:20:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:20:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:20:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:20:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:20:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:20:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:20:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:20:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:20:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:20:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:20:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:20:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:20:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:20:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:20:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:20:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:20:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:20:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:20:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:20:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:20:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:20:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:20:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:20:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:20:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:20:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:20:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:20:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:20:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:20:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:20:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:20:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:20:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:20:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:20:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:20:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:20:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:20:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:20:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:20:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:20:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:20:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:20:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:20:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:20:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:20:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:20:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:20:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:20:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:20:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:20:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:20:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:20:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:20:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:20:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:20:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:20:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:20:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:20:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:20:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:20:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:20:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:20:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:20:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:20:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:20:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:20:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:20:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:20:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:20:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:20:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:20:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:20:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:20:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:20:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:20:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:20:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:20:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:20:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:20:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:20:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:20:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:20:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:20:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:20:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:20:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:20:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:21:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:21:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:21:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:21:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:21:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:21:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:21:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:21:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:21:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:21:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:21:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:21:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:21:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:21:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:21:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:21:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:21:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:21:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:21:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:21:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:21:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:21:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:21:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:21:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:21:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:21:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:21:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:21:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:21:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:21:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:21:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:21:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:21:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:21:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:21:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:21:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:21:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:21:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:21:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:21:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:21:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:21:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:21:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:21:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:21:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:21:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:21:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:21:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:21:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:21:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:21:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:21:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:21:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:21:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:21:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:21:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:21:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:21:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:21:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:21:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:21:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:21:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:21:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:21:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:21:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:21:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:21:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:21:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:21:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:21:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:21:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:21:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:21:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:21:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:21:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:21:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:21:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:21:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:21:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:21:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:21:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:21:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:21:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:21:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:21:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:21:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:21:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:21:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:21:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:21:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:21:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:21:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:21:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:21:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:21:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:21:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:21:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:21:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:21:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:21:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:21:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:21:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:21:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:21:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:21:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:21:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:21:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:21:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:21:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:21:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:21:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:21:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:21:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:21:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:21:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:21:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:21:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:21:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:21:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:21:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:21:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:21:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:21:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:21:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:21:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:21:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:21:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:21:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:21:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:21:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:21:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:21:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:21:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:21:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:21:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:21:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:21:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:21:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:21:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:21:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:21:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:21:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:21:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:21:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:21:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:21:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:21:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:21:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:21:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:21:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:21:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:21:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:21:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:21:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:21:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:21:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:21:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:21:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:21:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:21:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:21:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:21:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:21:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:21:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:21:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:21:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:21:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:21:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:21:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:21:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:21:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:21:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:21:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:21:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:21:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:21:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:21:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:21:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:21:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:21:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:21:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:21:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:21:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:21:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:21:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:21:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:21:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:21:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:21:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:21:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:21:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:21:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:21:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:21:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:21:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:21:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:21:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:21:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:21:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:21:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:21:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:21:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:21:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:21:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:21:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:21:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:21:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:21:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:21:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:21:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:21:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:21:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:21:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:21:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:21:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:21:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:22:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:22:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:22:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:22:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:22:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:22:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:22:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:22:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:22:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:22:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:22:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:22:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:22:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:22:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:22:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:22:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:22:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:22:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:22:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:22:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:22:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:22:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:22:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:22:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:22:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:22:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:22:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:22:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:22:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:22:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:22:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:22:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:22:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:22:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:22:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:22:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:22:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:22:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:22:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:22:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:22:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:22:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:22:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:22:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:22:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:22:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:22:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:22:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:22:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:22:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:22:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:22:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:22:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:22:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:22:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:22:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:22:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:22:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:22:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:22:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:22:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:22:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:22:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:22:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:22:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:22:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:22:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:22:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:22:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:22:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:22:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:22:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:22:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:22:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:22:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:22:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:22:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:22:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:22:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:22:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:22:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:22:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:22:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:22:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:22:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:22:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:22:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:22:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:22:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:22:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:22:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:22:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:22:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:22:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:22:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:22:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:22:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:22:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:22:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:22:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:22:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:22:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:22:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:22:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:22:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:22:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:22:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:22:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:22:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:22:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:22:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:22:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:22:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:22:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:22:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:22:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:22:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:22:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:22:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:22:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:22:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:22:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:22:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:22:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:22:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:22:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:22:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:22:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:22:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:22:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:22:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:22:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:22:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:22:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:22:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:22:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:22:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:22:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:22:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:22:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:22:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:22:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:22:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:22:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:22:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:22:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:22:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:22:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:22:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:22:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:22:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:22:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:22:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:22:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:22:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:22:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:22:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:22:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:22:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:22:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:22:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:22:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:22:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:22:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:22:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:22:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:22:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:22:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:22:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:22:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:22:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:22:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:22:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:22:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:22:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:22:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:22:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:22:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:22:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:22:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:22:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:22:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:22:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:22:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:22:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:22:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:22:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:22:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:22:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:22:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:22:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:22:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:22:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:22:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:22:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:22:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:22:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:22:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:22:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:22:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:22:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:22:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:22:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:22:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:22:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:22:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:22:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:22:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:22:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:22:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:22:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:22:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:22:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:22:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:22:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:22:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:23:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:23:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:23:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:23:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:23:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:23:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:23:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:23:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:23:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:23:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:23:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:23:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:23:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:23:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:23:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:23:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:23:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:23:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:23:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:23:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:23:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:23:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:23:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:23:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:23:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:23:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:23:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:23:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:23:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:23:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:23:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:23:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:23:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:23:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:23:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:23:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:23:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:23:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:23:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:23:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:23:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:23:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:23:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:23:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:23:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:23:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:23:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:23:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:23:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:23:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:23:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:23:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:23:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:23:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:23:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:23:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:23:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:23:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:23:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:23:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:23:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:23:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:23:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:23:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:23:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:23:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:23:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:23:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:23:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:23:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:23:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:23:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:23:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:23:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:23:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:23:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:23:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:23:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:23:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:23:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:23:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:23:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:23:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:23:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:23:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:23:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:23:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:23:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:23:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:23:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:23:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:23:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:23:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:23:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:23:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:23:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:23:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:23:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:23:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:23:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:23:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:23:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:23:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:23:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:23:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:23:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:23:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:23:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:23:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:23:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:23:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:23:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:23:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:23:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:23:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:23:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:23:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:23:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:23:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:23:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:23:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:23:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:23:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:23:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:23:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:23:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:23:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:23:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:23:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:23:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:23:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:23:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:23:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:23:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:23:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:23:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:23:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:23:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:23:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:23:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:23:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:23:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:23:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:23:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:23:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:23:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:23:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:23:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:23:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:23:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:23:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:23:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:23:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:23:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:23:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:23:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:23:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:23:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:23:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:23:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:23:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:23:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:25:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:25:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:25:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:25:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:25:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:25:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:25:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:25:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:25:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:25:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:25:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:25:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:25:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:25:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:25:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:25:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:25:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:25:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:25:46 rdma06 rsyslogd[1041]: imjournal: 5083 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 10:25:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:25:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:25:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:25:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:25:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:25:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:25:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:25:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:25:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:25:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:25:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:25:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:25:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:25:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:25:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:25:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:25:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:25:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:25:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:25:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:25:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:25:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:25:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:25:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:25:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:25:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:25:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:25:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:25:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:25:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:25:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:25:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:25:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:25:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:25:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:25:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:26:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:26:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:26:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:26:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:26:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:26:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:26:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:26:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:26:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:26:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:26:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:26:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:26:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:26:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:26:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:26:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:26:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:26:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:26:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:26:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:26:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:26:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:26:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:26:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:26:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:26:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:26:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:26:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:26:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:26:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:26:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:26:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:26:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:26:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:26:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:26:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:26:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:26:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:26:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:26:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:26:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:26:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:26:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:26:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:26:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:26:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:26:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:26:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:26:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:26:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:26:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:26:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:26:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:26:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:26:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:26:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:26:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:26:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:26:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:26:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:26:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:26:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:26:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:26:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:26:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:26:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:26:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:26:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:26:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:26:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:26:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:26:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:26:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:26:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:26:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:26:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:26:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:26:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:26:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:26:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:26:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:26:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:26:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:26:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:26:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:26:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:26:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:26:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:26:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:26:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:26:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:26:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:26:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:26:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:26:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:26:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:26:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:26:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:26:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:26:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:26:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:26:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:26:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:26:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:26:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:26:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:26:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:26:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:26:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:26:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:26:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:26:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:26:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:26:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:26:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:26:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:26:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:26:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:26:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:26:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:26:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:26:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:26:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:26:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:26:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:26:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:26:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:26:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:26:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:26:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:26:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:26:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:26:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:26:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:26:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:26:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:26:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:26:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:26:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:26:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:26:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:26:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:26:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:26:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:26:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:26:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:26:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:26:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:26:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:26:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:26:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:26:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:26:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:26:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:26:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:26:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:26:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:26:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:26:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:26:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:26:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:26:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:26:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:26:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:26:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:26:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:26:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:26:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:26:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:26:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:26:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:26:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:26:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:26:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:26:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:26:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:26:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:26:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:26:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:26:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:26:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:26:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:26:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:26:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:26:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:26:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:26:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:26:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:26:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:26:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:26:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:26:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:26:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:26:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:26:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:26:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:26:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:26:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:26:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:26:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:26:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:26:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:26:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:26:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:26:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:26:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:26:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:26:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:26:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:26:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:26:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:26:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:26:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:26:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:26:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:26:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:27:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:27:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:27:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:27:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:27:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:27:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:27:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:27:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:27:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:27:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:27:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:27:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:27:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:27:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:27:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:27:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:27:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:27:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:27:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:27:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:27:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:27:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:27:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:27:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:27:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:27:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:27:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:27:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:27:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:27:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:27:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:27:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:27:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:27:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:27:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:27:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:27:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:27:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:27:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:27:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:27:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:27:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:27:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:27:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:27:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:27:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:27:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:27:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:27:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:27:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:27:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:27:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:27:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:27:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:27:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:27:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:27:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:27:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:27:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:27:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:27:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:27:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:27:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:27:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:27:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:27:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:27:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:27:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:27:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:27:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:27:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:27:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:27:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:27:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:27:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:27:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:27:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:27:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:27:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:27:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:27:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:27:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:27:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:27:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:27:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:27:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:27:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:27:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:27:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:27:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:27:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:27:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:27:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:27:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:27:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:27:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:27:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:27:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:27:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:27:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:27:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:27:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:27:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:27:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:27:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:27:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:27:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:27:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:27:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:27:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:27:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:27:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:27:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:27:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:27:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:27:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:27:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:27:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:27:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:27:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:27:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:27:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:27:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:27:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:27:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:27:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:27:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:27:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:27:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:27:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:27:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:27:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:27:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:27:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:27:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:27:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:27:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:27:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:27:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:27:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:27:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:27:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:27:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:27:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:27:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:27:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:27:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:27:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:27:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:27:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:27:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:27:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:27:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:27:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:27:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:27:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:27:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:27:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:27:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:27:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:27:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:27:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:27:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:27:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:27:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:27:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:27:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:27:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:27:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:27:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:27:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:27:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:27:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:27:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:27:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:27:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:27:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:27:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:27:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:27:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:27:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:27:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:27:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:27:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:27:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:27:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:27:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:27:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:27:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:27:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:27:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:27:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:27:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:27:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:27:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:27:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:27:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:27:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:27:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:27:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:27:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:27:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:27:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:27:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:27:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:27:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:27:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:27:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:27:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:27:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:27:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:27:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:27:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:27:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:27:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:27:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:28:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:28:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:28:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:28:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:28:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:28:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:28:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:28:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:28:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:28:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:28:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:28:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:28:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:28:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:28:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:28:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:28:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:28:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:28:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:28:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:28:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:28:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:28:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:28:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:28:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:28:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:28:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:28:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:28:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:28:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:28:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:28:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:28:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:28:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:28:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:28:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:28:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:28:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:28:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:28:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:28:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:28:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:28:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:28:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:28:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:28:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:28:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:28:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:28:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:28:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:28:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:28:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:28:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:28:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:28:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:28:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:28:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:28:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:28:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:28:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:28:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:28:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:28:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:28:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:28:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:28:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:28:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:28:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:28:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:28:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:28:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:28:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:28:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:28:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:28:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:28:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:28:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:28:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:28:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:28:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:28:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:28:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:28:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:28:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:28:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:28:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:28:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:28:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:28:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:28:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:28:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:28:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:28:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:28:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:28:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:28:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:28:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:28:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:28:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:28:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:28:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:28:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:28:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:28:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:28:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:28:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:28:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:28:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:28:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:28:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:28:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:28:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:28:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:28:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:28:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:28:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:28:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:28:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:28:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:28:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:28:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:28:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:28:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:28:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:28:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:28:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:28:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:28:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:28:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:28:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:28:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:28:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:28:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:28:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:28:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:28:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:28:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:28:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:28:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:28:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:28:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:28:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:28:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:28:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:28:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:28:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:28:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:28:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:28:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:28:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:28:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:28:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:28:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:28:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:28:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:28:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:28:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:28:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:28:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:28:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:28:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:28:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:28:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:28:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:28:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:28:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:28:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:28:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:28:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:28:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:28:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:28:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:28:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:28:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:28:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:28:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:28:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:28:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:28:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:28:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:28:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:28:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:28:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:28:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:28:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:28:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:28:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:28:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:28:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:28:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:28:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:28:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:28:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:28:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:28:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:28:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:28:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:28:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:28:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:28:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:28:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:28:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:28:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:28:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:28:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:28:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:28:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:28:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:28:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:28:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:28:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:28:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:28:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:28:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:28:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:28:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:29:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:29:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:29:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:29:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:29:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:29:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:29:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:29:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:29:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:29:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:29:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:29:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:29:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:29:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:29:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:29:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:29:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:29:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:29:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:29:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:29:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:29:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:29:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:29:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:29:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:29:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:29:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:29:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:29:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:29:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:29:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:29:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:29:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:29:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:29:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:29:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:29:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:29:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:29:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:29:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:29:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:29:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:29:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:29:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:29:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:29:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:29:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:29:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:29:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:29:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:29:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:29:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:29:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:29:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:29:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:29:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:29:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:29:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:29:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:29:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:29:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:29:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:29:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:29:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:29:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:29:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:29:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:29:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:29:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:29:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:29:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:29:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:29:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:29:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:29:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:29:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:29:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:29:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:29:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:29:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:29:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:29:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:29:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:29:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:29:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:29:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:29:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:29:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:29:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:29:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:29:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:29:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:29:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:29:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:29:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:29:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:29:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:29:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:29:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:29:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:29:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:29:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:29:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:29:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:29:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:29:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:29:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:29:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:29:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:29:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:29:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:29:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:29:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:29:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:29:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:29:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:29:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:29:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:29:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:29:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:29:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:29:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:29:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:29:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:29:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:29:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:29:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:29:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:29:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:29:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:29:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:29:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:29:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:29:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:29:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:29:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:29:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:29:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:29:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:29:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:29:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:29:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:29:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:29:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:29:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:29:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:29:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:29:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:29:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:29:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:29:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:29:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:29:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:29:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:29:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:29:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:29:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:29:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:29:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:29:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:29:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:29:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:29:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:29:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:29:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:29:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:29:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:29:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:29:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:29:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:29:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:29:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:29:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:29:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:29:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:29:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:29:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:29:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:29:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:29:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:29:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:29:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:29:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:29:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:29:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:29:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:29:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:29:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:29:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:29:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:29:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:29:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:29:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:29:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:29:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:29:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:29:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:29:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:29:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:29:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:29:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:29:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:29:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:29:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:29:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:29:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:29:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:29:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:29:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:29:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:29:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:29:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:29:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:29:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:29:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:29:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:30:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:30:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:30:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:30:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:30:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:30:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:30:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:30:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:30:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:30:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:30:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:30:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:30:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:30:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:30:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:30:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:30:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:30:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:30:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:30:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:30:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:30:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:30:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:30:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:30:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:30:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:30:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:30:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:30:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:30:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:30:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:30:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:30:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:30:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:30:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:30:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:30:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:30:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:30:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:30:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:30:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:30:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:30:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:30:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:30:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:30:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:30:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:30:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:30:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:30:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:30:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:30:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:30:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:30:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:30:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:30:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:30:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:30:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:30:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:30:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:30:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:30:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:30:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:30:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:30:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:30:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:30:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:30:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:30:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:30:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:30:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:30:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:30:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:30:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:30:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:30:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:30:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:30:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:30:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:30:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:30:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:30:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:30:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:30:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:30:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:30:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:30:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:30:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:30:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:30:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:30:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:30:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:30:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:30:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:30:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:30:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:30:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:30:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:30:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:30:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:30:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:30:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:30:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:30:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:30:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:30:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:30:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:30:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:30:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:30:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:30:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:30:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:30:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:30:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:30:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:30:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:30:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:30:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:30:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:30:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:30:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:30:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:30:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:30:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:30:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:30:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:30:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:30:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:30:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:30:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:30:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:30:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:30:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:30:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:30:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:30:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:30:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:30:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:30:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:30:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:30:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:30:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:30:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:30:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:30:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:30:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:30:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:30:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:30:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:30:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:30:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:30:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:30:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:30:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:30:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:30:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:30:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:30:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:30:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:30:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:30:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:30:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:30:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:30:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:30:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:30:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:30:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:30:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:30:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:30:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:30:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:30:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:30:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:30:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:30:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:30:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:30:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:30:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:30:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:30:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:30:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:30:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:30:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:30:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:30:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:30:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:30:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:30:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:30:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:30:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:30:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:30:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:30:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:30:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:30:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:30:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:30:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:30:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:30:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:30:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:30:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:30:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:30:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:30:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:30:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:30:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:30:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:30:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:30:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:30:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:30:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:30:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:30:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:30:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:30:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:30:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:31:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:31:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:31:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:31:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:31:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:31:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:31:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:31:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:31:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:31:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:31:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:31:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:31:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:31:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:31:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:31:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:31:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:31:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:31:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:31:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:31:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:31:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:31:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:31:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:31:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:31:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:31:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:31:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:31:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:31:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:31:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:31:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:31:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:31:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:31:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:31:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:31:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:31:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:31:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:31:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:31:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:31:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:31:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:31:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:31:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:31:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:31:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:31:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:31:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:31:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:31:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:31:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:31:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:31:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:31:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:31:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:31:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:31:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:31:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:31:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:31:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:31:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:31:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:31:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:31:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:31:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:31:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:31:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:31:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:31:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:31:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:31:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:31:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:31:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:31:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:31:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:31:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:31:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:31:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:31:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:31:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:31:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:31:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:31:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:31:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:31:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:31:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:31:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:31:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:31:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:31:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:31:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:31:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:31:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:31:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:31:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:31:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:31:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:31:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:31:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:31:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:31:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:31:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:31:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:31:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:31:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:31:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:31:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:31:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:31:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:31:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:31:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:31:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:31:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:31:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:31:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:31:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:31:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:31:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:31:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:31:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:31:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:31:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:31:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:31:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:31:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:31:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:31:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:31:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:31:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:31:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:31:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:31:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:31:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:31:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:31:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:31:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:31:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:31:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:31:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:31:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:31:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:31:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:31:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:31:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:31:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:31:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:31:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:31:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:31:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:31:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:31:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:31:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:31:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:31:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:31:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:31:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:31:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:31:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:31:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:31:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:31:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:31:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:31:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:31:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:31:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:32:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:32:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:32:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:32:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:32:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:32:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:32:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:32:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:32:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:32:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:32:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:32:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:32:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:32:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:32:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:32:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:32:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:32:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:32:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:32:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:32:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:32:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:32:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:32:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:32:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:32:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:32:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:32:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:32:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:32:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:32:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:32:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:32:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:32:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:32:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:32:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:32:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:32:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:32:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:32:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:32:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:32:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:32:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:32:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:32:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:32:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:32:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:32:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:32:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:32:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:32:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:32:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:32:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:32:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:32:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:32:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:32:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:32:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:32:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:32:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:32:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:32:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:32:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:32:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:32:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:32:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:32:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:32:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:32:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:32:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:32:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:32:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:32:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:32:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:32:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:32:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:32:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:32:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:32:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:32:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:32:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:32:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:32:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:32:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:32:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:32:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:32:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:32:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:32:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:32:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:32:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:32:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:32:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:32:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:32:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:32:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:32:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:32:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:32:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:32:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:32:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:32:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:32:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:32:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:32:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:32:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:32:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:32:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:32:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:32:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:32:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:32:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:32:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:32:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:32:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:32:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:32:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:32:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:32:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:32:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:32:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:32:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:32:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:32:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:32:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:32:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:32:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:32:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:32:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:32:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:32:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:32:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:32:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:32:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:32:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:32:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:32:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:32:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:32:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:32:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:32:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:32:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:32:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:32:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:32:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:32:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:32:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:32:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:32:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:32:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:32:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:32:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:32:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:32:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:32:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:32:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:32:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:32:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:32:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:32:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:32:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:32:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:32:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:32:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:32:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:32:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:32:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:32:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:32:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:32:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:32:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:32:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:32:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:32:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:32:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:32:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:32:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:32:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:32:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:32:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:32:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:32:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:32:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:32:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:32:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:32:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:32:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:32:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:32:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:32:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:32:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:32:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:32:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:32:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:32:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:32:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:32:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:32:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:32:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:32:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:32:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:32:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:32:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:32:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:32:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:32:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:32:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:32:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:32:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:32:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:32:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:32:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:32:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:32:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:32:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:32:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:33:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:33:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:33:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:33:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:33:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:33:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:33:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:33:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:33:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:33:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:33:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:33:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:33:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:33:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:33:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:33:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:33:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:33:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:33:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:33:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:33:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:33:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:33:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:33:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:33:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:33:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:33:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:33:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:33:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:33:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:33:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:33:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:33:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:33:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:33:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:33:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:33:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:33:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:33:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:33:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:33:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:33:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:33:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:33:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:33:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:33:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:33:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:33:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:33:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:33:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:33:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:33:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:33:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:33:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:33:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:33:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:33:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:33:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:33:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:33:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:33:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:33:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:33:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:33:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:33:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:33:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:33:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:33:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:33:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:33:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:33:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:33:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:33:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:33:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:33:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:33:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:33:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:33:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:33:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:33:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:33:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:33:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:33:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:33:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:33:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:33:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:33:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:33:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:33:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:33:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:33:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:33:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:33:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:33:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:33:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:33:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:33:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:33:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:33:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:33:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:33:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:33:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:33:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:33:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:33:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:33:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:33:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:33:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:33:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:33:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:33:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:33:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:33:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:33:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:33:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:33:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:33:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:33:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:33:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:33:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:33:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:33:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:33:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:33:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:33:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:33:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:33:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:33:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:33:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:33:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:33:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:33:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:33:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:33:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:33:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:33:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:33:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:33:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:33:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:33:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:33:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:33:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:33:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:33:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:33:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:33:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:33:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:33:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:33:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:33:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:33:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:33:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:33:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:33:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:33:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:33:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:33:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:33:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:33:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:33:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:33:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:33:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:35:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:35:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:35:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:35:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:35:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:35:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:35:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:35:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:35:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:35:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:35:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:35:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:35:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:35:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:35:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:35:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:35:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:35:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:35:51 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 10:35:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:35:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:35:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:35:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:35:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:35:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:35:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:35:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:35:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:35:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:35:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:35:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:35:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:35:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:35:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:35:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:35:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:35:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:36:00 rdma06 systemd[1]: Starting dnf makecache...
Apr 17 10:36:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:36:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:36:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:36:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:36:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:36:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:36:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:36:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:36:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:36:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:36:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:36:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:36:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:36:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:36:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:36:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:36:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:36:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:36:01 rdma06 dnf[1152362]: Updating Subscription Management repositories.
Apr 17 10:36:01 rdma06 dnf[1152362]: Unable to read consumer identity
Apr 17 10:36:01 rdma06 dnf[1152362]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 17 10:36:02 rdma06 dnf[1152362]: Copr repo for fmf owned by psss                 2.4 kB/s | 3.3 kB     00:01
Apr 17 10:36:02 rdma06 dnf[1152362]: beaker-AppStream                                 45 kB/s | 168  B     00:00
Apr 17 10:36:02 rdma06 dnf[1152362]: Errors during downloading metadata for repository 'beaker-AppStream':
Apr 17 10:36:02 rdma06 dnf[1152362]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 10:36:02 rdma06 dnf[1152362]: Error: Failed to download metadata for repo 'beaker-AppStream': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 10:36:02 rdma06 dnf[1152362]: beaker-AppStream-debuginfo                       56 kB/s | 168  B     00:00
Apr 17 10:36:02 rdma06 dnf[1152362]: Errors during downloading metadata for repository 'beaker-AppStream-debuginfo':
Apr 17 10:36:02 rdma06 dnf[1152362]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 10:36:02 rdma06 dnf[1152362]: Error: Failed to download metadata for repo 'beaker-AppStream-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 10:36:02 rdma06 dnf[1152362]: beaker-BaseOS                                    63 kB/s | 168  B     00:00
Apr 17 10:36:02 rdma06 dnf[1152362]: Errors during downloading metadata for repository 'beaker-BaseOS':
Apr 17 10:36:02 rdma06 dnf[1152362]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 10:36:02 rdma06 dnf[1152362]: Error: Failed to download metadata for repo 'beaker-BaseOS': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 10:36:02 rdma06 dnf[1152362]: beaker-BaseOS-debuginfo                          63 kB/s | 168  B     00:00
Apr 17 10:36:02 rdma06 dnf[1152362]: Errors during downloading metadata for repository 'beaker-BaseOS-debuginfo':
Apr 17 10:36:02 rdma06 dnf[1152362]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 10:36:02 rdma06 dnf[1152362]: Error: Failed to download metadata for repo 'beaker-BaseOS-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 10:36:02 rdma06 dnf[1152362]: beaker-CRB                                       61 kB/s | 168  B     00:00
Apr 17 10:36:02 rdma06 dnf[1152362]: Errors during downloading metadata for repository 'beaker-CRB':
Apr 17 10:36:02 rdma06 dnf[1152362]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 10:36:02 rdma06 dnf[1152362]: Error: Failed to download metadata for repo 'beaker-CRB': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 10:36:02 rdma06 dnf[1152362]: beaker-CRB-debuginfo                             65 kB/s | 168  B     00:00
Apr 17 10:36:02 rdma06 dnf[1152362]: Errors during downloading metadata for repository 'beaker-CRB-debuginfo':
Apr 17 10:36:02 rdma06 dnf[1152362]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 10:36:02 rdma06 dnf[1152362]: Error: Failed to download metadata for repo 'beaker-CRB-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 10:36:02 rdma06 dnf[1152362]: beaker-HighAvailability                          63 kB/s | 168  B     00:00
Apr 17 10:36:02 rdma06 dnf[1152362]: Errors during downloading metadata for repository 'beaker-HighAvailability':
Apr 17 10:36:02 rdma06 dnf[1152362]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 10:36:02 rdma06 dnf[1152362]: Error: Failed to download metadata for repo 'beaker-HighAvailability': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 10:36:02 rdma06 dnf[1152362]: beaker-HighAvailability-debuginfo                63 kB/s | 168  B     00:00
Apr 17 10:36:02 rdma06 dnf[1152362]: Errors during downloading metadata for repository 'beaker-HighAvailability-debuginfo':
Apr 17 10:36:02 rdma06 dnf[1152362]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 10:36:02 rdma06 dnf[1152362]: Error: Failed to download metadata for repo 'beaker-HighAvailability-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 10:36:02 rdma06 dnf[1152362]: beaker-NFV                                       62 kB/s | 168  B     00:00
Apr 17 10:36:02 rdma06 dnf[1152362]: Errors during downloading metadata for repository 'beaker-NFV':
Apr 17 10:36:02 rdma06 dnf[1152362]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 10:36:02 rdma06 dnf[1152362]: Error: Failed to download metadata for repo 'beaker-NFV': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 10:36:02 rdma06 dnf[1152362]: beaker-NFV-debuginfo                             62 kB/s | 168  B     00:00
Apr 17 10:36:02 rdma06 dnf[1152362]: Errors during downloading metadata for repository 'beaker-NFV-debuginfo':
Apr 17 10:36:02 rdma06 dnf[1152362]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 10:36:02 rdma06 dnf[1152362]: Error: Failed to download metadata for repo 'beaker-NFV-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 10:36:02 rdma06 dnf[1152362]: beaker-RT                                        60 kB/s | 168  B     00:00
Apr 17 10:36:02 rdma06 dnf[1152362]: Errors during downloading metadata for repository 'beaker-RT':
Apr 17 10:36:02 rdma06 dnf[1152362]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 10:36:02 rdma06 dnf[1152362]: Error: Failed to download metadata for repo 'beaker-RT': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 10:36:02 rdma06 dnf[1152362]: beaker-RT-debuginfo                              59 kB/s | 168  B     00:00
Apr 17 10:36:02 rdma06 dnf[1152362]: Errors during downloading metadata for repository 'beaker-RT-debuginfo':
Apr 17 10:36:02 rdma06 dnf[1152362]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 10:36:02 rdma06 dnf[1152362]: Error: Failed to download metadata for repo 'beaker-RT-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 10:36:02 rdma06 dnf[1152362]: beaker-ResilientStorage                          64 kB/s | 168  B     00:00
Apr 17 10:36:02 rdma06 dnf[1152362]: Errors during downloading metadata for repository 'beaker-ResilientStorage':
Apr 17 10:36:02 rdma06 dnf[1152362]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 10:36:02 rdma06 dnf[1152362]: Error: Failed to download metadata for repo 'beaker-ResilientStorage': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 10:36:02 rdma06 dnf[1152362]: beaker-ResilientStorage-debuginfo                68 kB/s | 168  B     00:00
Apr 17 10:36:02 rdma06 dnf[1152362]: Errors during downloading metadata for repository 'beaker-ResilientStorage-debuginfo':
Apr 17 10:36:02 rdma06 dnf[1152362]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 10:36:02 rdma06 dnf[1152362]: Error: Failed to download metadata for repo 'beaker-ResilientStorage-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 10:36:02 rdma06 dnf[1152362]: beaker-SAP                                       66 kB/s | 168  B     00:00
Apr 17 10:36:02 rdma06 dnf[1152362]: Errors during downloading metadata for repository 'beaker-SAP':
Apr 17 10:36:02 rdma06 dnf[1152362]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 10:36:02 rdma06 dnf[1152362]: Error: Failed to download metadata for repo 'beaker-SAP': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 10:36:02 rdma06 dnf[1152362]: beaker-SAP-debuginfo                             56 kB/s | 168  B     00:00
Apr 17 10:36:02 rdma06 dnf[1152362]: Errors during downloading metadata for repository 'beaker-SAP-debuginfo':
Apr 17 10:36:02 rdma06 dnf[1152362]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 10:36:02 rdma06 dnf[1152362]: Error: Failed to download metadata for repo 'beaker-SAP-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 10:36:02 rdma06 dnf[1152362]: beaker-SAPHANA                                   56 kB/s | 168  B     00:00
Apr 17 10:36:02 rdma06 dnf[1152362]: Errors during downloading metadata for repository 'beaker-SAPHANA':
Apr 17 10:36:02 rdma06 dnf[1152362]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 10:36:02 rdma06 dnf[1152362]: Error: Failed to download metadata for repo 'beaker-SAPHANA': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 10:36:02 rdma06 dnf[1152362]: beaker-SAPHANA-debuginfo                         59 kB/s | 168  B     00:00
Apr 17 10:36:02 rdma06 dnf[1152362]: Errors during downloading metadata for repository 'beaker-SAPHANA-debuginfo':
Apr 17 10:36:02 rdma06 dnf[1152362]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 10:36:02 rdma06 dnf[1152362]: Error: Failed to download metadata for repo 'beaker-SAPHANA-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 10:36:03 rdma06 dnf[1152362]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr 17 10:36:04 rdma06 dnf[1152362]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr 17 10:36:04 rdma06 dnf[1152362]: Ignoring repositories: beaker-AppStream, beaker-AppStream-debuginfo, beaker-BaseOS, beaker-BaseOS-debuginfo, beaker-CRB, beaker-CRB-debuginfo, beaker-HighAvailability, beaker-HighAvailability-debuginfo, beaker-NFV, beaker-NFV-debuginfo, beaker-RT, beaker-RT-debuginfo, beaker-ResilientStorage, beaker-ResilientStorage-debuginfo, beaker-SAP, beaker-SAP-debuginfo, beaker-SAPHANA, beaker-SAPHANA-debuginfo
Apr 17 10:36:04 rdma06 dnf[1152362]: Metadata cache created.
Apr 17 10:36:04 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 17 10:36:04 rdma06 systemd[1]: Finished dnf makecache.
Apr 17 10:36:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:36:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:36:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:36:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:36:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:36:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:36:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:36:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:36:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:36:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:36:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:36:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:36:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:36:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:36:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:36:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:36:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:36:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:36:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:36:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:36:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:36:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:36:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:36:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:36:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:36:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:36:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:36:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:36:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:36:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:36:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:36:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:36:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:36:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:36:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:36:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:36:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:36:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:36:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:36:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:36:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:36:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:36:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:36:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:36:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:36:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:36:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:36:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:36:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:36:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:36:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:36:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:36:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:36:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:36:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:36:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:36:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:36:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:36:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:36:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:36:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:36:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:36:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:36:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:36:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:36:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:36:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:36:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:36:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:36:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:36:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:36:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:36:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:36:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:36:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:36:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:36:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:36:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:36:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:36:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:36:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:36:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:36:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:36:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:36:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:36:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:36:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:36:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:36:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:36:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:36:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:36:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:36:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:36:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:36:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:36:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:36:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:36:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:36:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:36:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:36:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:36:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:36:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:36:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:36:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:36:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:36:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:36:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:36:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:36:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:36:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:36:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:36:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:36:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:36:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:36:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:36:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:36:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:36:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:36:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:36:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:36:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:36:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:36:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:36:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:36:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:36:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:36:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:36:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:36:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:36:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:36:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:36:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:36:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:36:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:36:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:36:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:36:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:36:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:36:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:36:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:36:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:36:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:36:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:36:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:36:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:36:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:36:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:36:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:36:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:36:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:36:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:36:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:36:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:36:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:36:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:36:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:36:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:36:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:36:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:36:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:36:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:36:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:36:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:36:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:36:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:36:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:36:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:36:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:36:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:36:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:36:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:36:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:36:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:36:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:36:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:36:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:36:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:36:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:36:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:36:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:36:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:36:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:36:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:36:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:36:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:36:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:36:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:36:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:36:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:36:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:36:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:36:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:36:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:36:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:36:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:36:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:36:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:37:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:37:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:37:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:37:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:37:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:37:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:37:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:37:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:37:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:37:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:37:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:37:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:37:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:37:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:37:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:37:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:37:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:37:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:37:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:37:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:37:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:37:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:37:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:37:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:37:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:37:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:37:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:37:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:37:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:37:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:37:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:37:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:37:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:37:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:37:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:37:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:37:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:37:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:37:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:37:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:37:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:37:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:37:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:37:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:37:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:37:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:37:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:37:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:37:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:37:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:37:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:37:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:37:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:37:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:37:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:37:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:37:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:37:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:37:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:37:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:37:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:37:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:37:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:37:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:37:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:37:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:37:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:37:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:37:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:37:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:37:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:37:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:37:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:37:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:37:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:37:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:37:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:37:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:37:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:37:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:37:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:37:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:37:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:37:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:37:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:37:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:37:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:37:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:37:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:37:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:37:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:37:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:37:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:37:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:37:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:37:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:37:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:37:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:37:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:37:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:37:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:37:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:37:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:37:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:37:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:37:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:37:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:37:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:37:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:37:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:37:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:37:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:37:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:37:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:37:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:37:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:37:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:37:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:37:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:37:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:37:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:37:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:37:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:37:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:37:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:37:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:37:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:37:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:37:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:37:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:37:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:37:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:37:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:37:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:37:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:37:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:37:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:37:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:37:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:37:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:37:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:37:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:37:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:37:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:37:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:37:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:37:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:37:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:37:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:37:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:37:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:37:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:37:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:37:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:37:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:37:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:37:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:37:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:37:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:37:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:37:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:37:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:37:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:37:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:37:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:37:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:37:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:37:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:37:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:37:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:37:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:37:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:37:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:37:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:37:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:37:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:37:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:37:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:37:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:37:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:37:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:37:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:37:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:37:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:37:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:37:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:37:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:37:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:37:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:37:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:37:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:37:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:37:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:37:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:37:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:37:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:37:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:37:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:37:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:37:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:37:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:37:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:37:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:37:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:37:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:37:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:37:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:37:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:37:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:37:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:37:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:37:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:37:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:37:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:37:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:37:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:38:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:38:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:38:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:38:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:38:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:38:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:38:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:38:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:38:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:38:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:38:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:38:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:38:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:38:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:38:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:38:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:38:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:38:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:38:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:38:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:38:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:38:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:38:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:38:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:38:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:38:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:38:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:38:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:38:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:38:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:38:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:38:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:38:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:38:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:38:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:38:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:38:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:38:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:38:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:38:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:38:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:38:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:38:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:38:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:38:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:38:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:38:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:38:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:38:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:38:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:38:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:38:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:38:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:38:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:38:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:38:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:38:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:38:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:38:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:38:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:38:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:38:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:38:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:38:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:38:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:38:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:38:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:38:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:38:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:38:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:38:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:38:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:38:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:38:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:38:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:38:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:38:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:38:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:38:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:38:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:38:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:38:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:38:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:38:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:38:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:38:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:38:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:38:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:38:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:38:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:38:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:38:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:38:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:38:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:38:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:38:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:38:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:38:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:38:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:38:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:38:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:38:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:38:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:38:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:38:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:38:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:38:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:38:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:38:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:38:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:38:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:38:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:38:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:38:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:38:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:38:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:38:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:38:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:38:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:38:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:38:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:38:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:38:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:38:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:38:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:38:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:38:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:38:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:38:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:38:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:38:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:38:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:38:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:38:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:38:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:38:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:38:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:38:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:38:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:38:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:38:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:38:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:38:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:38:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:38:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:38:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:38:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:38:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:38:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:38:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:38:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:38:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:38:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:38:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:38:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:38:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:38:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:38:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:38:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:38:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:38:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:38:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:38:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:38:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:38:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:38:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:38:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:38:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:38:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:38:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:38:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:38:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:38:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:38:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:38:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:38:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:38:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:38:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:38:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:38:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:38:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:38:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:38:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:38:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:38:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:38:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:38:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:38:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:38:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:38:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:38:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:38:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:38:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:38:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:38:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:38:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:38:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:38:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:38:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:38:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:38:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:38:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:38:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:38:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:38:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:38:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:38:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:38:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:38:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:38:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:38:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:38:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:38:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:38:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:38:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:38:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:39:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:39:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:39:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:39:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:39:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:39:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:39:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:39:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:39:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:39:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:39:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:39:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:39:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:39:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:39:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:39:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:39:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:39:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:39:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:39:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:39:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:39:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:39:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:39:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:39:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:39:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:39:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:39:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:39:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:39:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:39:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:39:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:39:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:39:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:39:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:39:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:39:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:39:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:39:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:39:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:39:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:39:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:39:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:39:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:39:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:39:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:39:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:39:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:39:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:39:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:39:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:39:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:39:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:39:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:39:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:39:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:39:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:39:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:39:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:39:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:39:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:39:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:39:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:39:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:39:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:39:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:39:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:39:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:39:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:39:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:39:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:39:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:39:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:39:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:39:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:39:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:39:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:39:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:39:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:39:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:39:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:39:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:39:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:39:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:39:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:39:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:39:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:39:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:39:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:39:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:39:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:39:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:39:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:39:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:39:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:39:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:39:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:39:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:39:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:39:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:39:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:39:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:39:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:39:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:39:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:39:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:39:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:39:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:39:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:39:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:39:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:39:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:39:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:39:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:39:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:39:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:39:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:39:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:39:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:39:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:39:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:39:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:39:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:39:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:39:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:39:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:39:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:39:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:39:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:39:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:39:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:39:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:39:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:39:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:39:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:39:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:39:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:39:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:39:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:39:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:39:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:39:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:39:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:39:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:39:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:39:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:39:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:39:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:39:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:39:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:39:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:39:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:39:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:39:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:39:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:39:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:39:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:39:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:39:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:39:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:39:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:39:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:39:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:39:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:39:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:39:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:39:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:39:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:39:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:39:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:39:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:39:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:39:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:39:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:39:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:39:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:39:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:39:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:39:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:39:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:39:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:39:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:39:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:39:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:39:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:39:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:39:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:39:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:39:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:39:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:39:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:39:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:39:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:39:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:39:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:39:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:39:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:39:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:39:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:39:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:39:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:39:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:39:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:39:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:39:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:39:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:39:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:39:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:39:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:39:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:39:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:39:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:39:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:39:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:39:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:39:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:40:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:40:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:40:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:40:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:40:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:40:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:40:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:40:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:40:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:40:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:40:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:40:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:40:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:40:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:40:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:40:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:40:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:40:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:40:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:40:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:40:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:40:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:40:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:40:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:40:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:40:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:40:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:40:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:40:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:40:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:40:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:40:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:40:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:40:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:40:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:40:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:40:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:40:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:40:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:40:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:40:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:40:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:40:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:40:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:40:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:40:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:40:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:40:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:40:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:40:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:40:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:40:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:40:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:40:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:40:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:40:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:40:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:40:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:40:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:40:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:40:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:40:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:40:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:40:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:40:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:40:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:40:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:40:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:40:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:40:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:40:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:40:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:40:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:40:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:40:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:40:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:40:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:40:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:40:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:40:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:40:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:40:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:40:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:40:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:40:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:40:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:40:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:40:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:40:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:40:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:40:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:40:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:40:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:40:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:40:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:40:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:40:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:40:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:40:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:40:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:40:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:40:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:40:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:40:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:40:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:40:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:40:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:40:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:40:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:40:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:40:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:40:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:40:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:40:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:40:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:40:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:40:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:40:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:40:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:40:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:40:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:40:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:40:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:40:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:40:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:40:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:40:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:40:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:40:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:40:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:40:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:40:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:40:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:40:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:40:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:40:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:40:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:40:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:40:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:40:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:40:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:40:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:40:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:40:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:40:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:40:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:40:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:40:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:40:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:40:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:40:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:40:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:40:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:40:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:40:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:40:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:40:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:40:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:40:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:40:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:40:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:40:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:40:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:40:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:40:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:40:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:40:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:40:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:40:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:40:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:40:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:40:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:40:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:40:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:40:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:40:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:40:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:40:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:40:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:40:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:40:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:40:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:40:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:40:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:40:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:40:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:40:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:40:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:40:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:40:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:40:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:40:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:40:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:40:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:40:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:40:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:40:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:40:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:40:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:40:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:40:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:40:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:40:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:40:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:40:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:40:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:40:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:40:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:40:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:40:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:40:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:40:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:40:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:40:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:40:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:40:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:41:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:41:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:41:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:41:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:41:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:41:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:41:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:41:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:41:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:41:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:41:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:41:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:41:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:41:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:41:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:41:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:41:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:41:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:41:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:41:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:41:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:41:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:41:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:41:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:41:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:41:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:41:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:41:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:41:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:41:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:41:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:41:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:41:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:41:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:41:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:41:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:41:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:41:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:41:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:41:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:41:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:41:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:41:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:41:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:41:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:41:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:41:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:41:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:41:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:41:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:41:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:41:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:41:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:41:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:41:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:41:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:41:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:41:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:41:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:41:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:41:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:41:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:41:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:41:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:41:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:41:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:41:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:41:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:41:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:41:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:41:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:41:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:41:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:41:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:41:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:41:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:41:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:41:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:41:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:41:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:41:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:41:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:41:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:41:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:41:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:41:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:41:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:41:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:41:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:41:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:41:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:41:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:41:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:41:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:41:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:41:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:41:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:41:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:41:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:41:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:41:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:41:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:41:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:41:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:41:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:41:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:41:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:41:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:41:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:41:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:41:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:41:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:41:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:41:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:41:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:41:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:41:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:41:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:41:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:41:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:41:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:41:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:41:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:41:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:41:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:41:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:41:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:41:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:41:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:41:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:41:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:41:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:41:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:41:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:41:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:41:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:41:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:41:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:41:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:41:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:41:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:41:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:41:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:41:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:41:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:41:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:41:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:41:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:41:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:41:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:41:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:41:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:41:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:41:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:41:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:41:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:41:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:41:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:41:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:41:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:41:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:41:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:41:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:41:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:41:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:41:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:41:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:41:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:41:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:41:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:41:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:41:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:41:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:41:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:41:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:41:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:41:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:41:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:41:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:41:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:41:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:41:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:41:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:41:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:41:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:41:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:41:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:41:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:41:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:41:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:41:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:41:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:41:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:41:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:41:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:41:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:41:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:41:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:41:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:41:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:41:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:41:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:41:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:41:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:41:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:41:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:41:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:41:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:41:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:41:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:41:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:41:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:41:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:41:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:41:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:41:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:42:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:42:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:42:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:42:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:42:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:42:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:42:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:42:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:42:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:42:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:42:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:42:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:42:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:42:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:42:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:42:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:42:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:42:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:42:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:42:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:42:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:42:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:42:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:42:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:42:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:42:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:42:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:42:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:42:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:42:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:42:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:42:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:42:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:42:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:42:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:42:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:42:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:42:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:42:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:42:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:42:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:42:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:42:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:42:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:42:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:42:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:42:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:42:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:42:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:42:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:42:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:42:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:42:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:42:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:42:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:42:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:42:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:42:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:42:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:42:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:42:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:42:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:42:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:42:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:42:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:42:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:42:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:42:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:42:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:42:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:42:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:42:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:42:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:42:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:42:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:42:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:42:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:42:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:42:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:42:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:42:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:42:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:42:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:42:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:42:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:42:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:42:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:42:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:42:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:42:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:42:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:42:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:42:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:42:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:42:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:42:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:42:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:42:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:42:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:42:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:42:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:42:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:42:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:42:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:42:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:42:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:42:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:42:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:42:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:42:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:42:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:42:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:42:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:42:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:42:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:42:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:42:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:42:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:42:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:42:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:42:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:42:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:42:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:42:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:42:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:42:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:42:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:42:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:42:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:42:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:42:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:42:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:42:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:42:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:42:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:42:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:42:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:42:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:42:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:42:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:42:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:42:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:42:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:42:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:42:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:42:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:42:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:42:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:42:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:42:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:42:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:42:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:42:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:42:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:42:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:42:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:42:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:42:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:42:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:42:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:42:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:42:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:42:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:42:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:42:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:42:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:42:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:42:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:42:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:42:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:42:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:42:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:42:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:42:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:42:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:42:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:42:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:42:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:42:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:42:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:42:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:42:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:42:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:42:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:42:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:42:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:42:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:42:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:42:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:42:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:42:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:42:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:42:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:42:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:42:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:42:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:42:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:42:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:42:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:42:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:42:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:42:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:42:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:42:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:42:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:42:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:42:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:42:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:42:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:42:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:42:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:42:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:42:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:42:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:42:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:42:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:43:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:43:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:43:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:43:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:43:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:43:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:43:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:43:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:43:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:43:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:43:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:43:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:43:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:43:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:43:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:43:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:43:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:43:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:43:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:43:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:43:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:43:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:43:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:43:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:43:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:43:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:43:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:43:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:43:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:43:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:43:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:43:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:43:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:43:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:43:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:43:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:43:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:43:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:43:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:43:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:43:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:43:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:43:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:43:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:43:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:43:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:43:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:43:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:43:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:43:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:43:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:43:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:43:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:43:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:43:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:43:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:43:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:43:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:43:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:43:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:43:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:43:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:43:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:43:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:43:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:43:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:43:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:43:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:43:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:43:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:43:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:43:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:43:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:43:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:43:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:43:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:43:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:43:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:43:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:43:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:43:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:43:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:43:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:43:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:43:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:43:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:43:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:43:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:43:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:43:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:43:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:43:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:43:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:43:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:43:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:43:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:43:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:43:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:43:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:43:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:43:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:43:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:43:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:43:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:43:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:43:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:43:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:43:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:43:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:43:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:43:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:43:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:43:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:43:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:43:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:43:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:43:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:43:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:43:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:43:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:43:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:43:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:43:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:43:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:43:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:43:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:43:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:43:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:43:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:43:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:43:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:43:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:43:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:43:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:43:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:43:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:43:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:43:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:43:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:43:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:43:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:43:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:43:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:43:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:43:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:43:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:43:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:43:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:43:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:43:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:43:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:43:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:43:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:43:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:43:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:43:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:43:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:43:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:43:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:43:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:43:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:43:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:43:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:43:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:43:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:43:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:43:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:43:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:43:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:43:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:43:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:43:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:43:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:43:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:43:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:43:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:43:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:45:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:45:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:45:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:45:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:45:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:45:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:45:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:45:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:45:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:45:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:45:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:45:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:45:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:45:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:45:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:45:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:45:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:45:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:45:56 rdma06 rsyslogd[1041]: imjournal: 5374 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 10:46:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:46:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:46:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:46:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:46:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:46:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:46:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:46:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:46:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:46:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:46:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:46:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:46:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:46:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:46:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:46:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:46:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:46:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:46:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:46:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:46:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:46:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:46:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:46:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:46:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:46:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:46:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:46:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:46:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:46:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:46:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:46:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:46:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:46:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:46:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:46:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:46:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:46:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:46:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:46:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:46:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:46:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:46:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:46:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:46:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:46:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:46:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:46:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:46:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:46:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:46:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:46:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:46:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:46:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:46:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:46:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:46:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:46:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:46:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:46:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:46:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:46:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:46:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:46:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:46:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:46:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:46:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:46:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:46:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:46:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:46:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:46:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:46:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:46:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:46:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:46:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:46:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:46:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:46:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:46:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:46:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:46:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:46:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:46:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:46:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:46:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:46:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:46:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:46:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:46:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:46:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:46:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:46:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:46:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:46:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:46:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:46:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:46:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:46:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:46:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:46:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:46:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:46:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:46:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:46:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:46:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:46:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:46:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:46:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:46:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:46:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:46:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:46:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:46:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:46:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:46:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:46:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:46:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:46:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:46:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:46:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:46:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:46:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:46:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:46:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:46:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:46:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:46:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:46:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:46:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:46:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:46:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:46:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:46:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:46:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:46:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:46:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:46:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:46:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:46:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:46:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:46:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:46:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:46:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:46:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:46:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:46:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:46:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:46:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:46:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:46:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:46:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:46:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:46:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:46:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:46:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:46:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:46:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:46:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:46:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:46:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:46:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:46:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:46:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:46:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:46:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:46:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:46:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:46:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:46:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:46:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:46:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:46:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:46:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:46:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:46:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:46:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:46:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:46:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:46:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:46:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:46:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:46:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:46:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:46:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:46:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:46:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:46:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:46:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:46:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:46:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:46:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:46:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:46:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:46:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:46:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:46:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:46:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:46:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:46:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:46:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:46:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:46:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:46:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:46:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:46:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:46:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:46:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:46:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:46:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:46:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:46:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:46:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:46:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:46:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:46:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:47:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:47:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:47:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:47:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:47:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:47:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:47:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:47:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:47:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:47:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:47:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:47:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:47:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:47:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:47:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:47:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:47:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:47:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:47:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:47:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:47:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:47:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:47:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:47:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:47:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:47:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:47:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:47:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:47:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:47:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:47:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:47:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:47:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:47:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:47:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:47:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:47:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:47:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:47:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:47:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:47:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:47:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:47:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:47:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:47:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:47:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:47:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:47:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:47:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:47:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:47:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:47:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:47:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:47:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:47:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:47:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:47:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:47:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:47:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:47:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:47:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:47:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:47:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:47:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:47:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:47:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:47:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:47:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:47:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:47:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:47:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:47:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:47:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:47:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:47:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:47:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:47:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:47:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:47:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:47:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:47:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:47:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:47:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:47:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:47:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:47:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:47:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:47:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:47:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:47:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:47:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:47:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:47:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:47:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:47:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:47:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:47:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:47:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:47:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:47:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:47:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:47:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:47:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:47:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:47:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:47:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:47:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:47:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:47:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:47:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:47:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:47:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:47:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:47:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:47:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:47:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:47:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:47:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:47:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:47:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:47:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:47:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:47:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:47:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:47:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:47:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:47:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:47:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:47:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:47:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:47:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:47:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:47:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:47:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:47:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:47:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:47:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:47:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:47:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:47:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:47:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:47:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:47:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:47:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:47:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:47:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:47:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:47:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:47:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:47:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:47:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:47:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:47:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:47:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:47:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:47:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:47:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:47:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:47:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:47:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:47:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:47:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:47:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:47:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:47:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:47:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:47:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:47:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:47:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:47:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:47:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:47:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:47:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:47:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:47:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:47:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:47:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:47:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:47:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:47:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:47:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:47:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:47:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:47:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:47:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:47:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:47:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:47:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:47:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:47:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:47:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:47:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:47:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:47:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:47:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:47:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:47:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:47:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:47:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:47:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:47:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:47:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:47:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:47:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:47:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:47:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:47:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:47:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:47:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:47:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:47:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:47:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:47:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:47:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:47:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:47:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:48:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:48:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:48:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:48:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:48:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:48:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:48:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:48:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:48:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:48:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:48:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:48:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:48:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:48:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:48:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:48:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:48:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:48:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:48:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:48:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:48:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:48:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:48:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:48:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:48:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:48:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:48:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:48:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:48:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:48:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:48:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:48:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:48:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:48:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:48:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:48:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:48:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:48:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:48:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:48:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:48:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:48:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:48:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:48:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:48:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:48:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:48:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:48:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:48:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:48:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:48:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:48:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:48:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:48:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:48:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:48:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:48:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:48:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:48:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:48:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:48:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:48:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:48:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:48:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:48:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:48:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:48:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:48:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:48:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:48:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:48:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:48:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:48:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:48:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:48:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:48:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:48:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:48:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:48:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:48:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:48:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:48:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:48:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:48:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:48:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:48:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:48:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:48:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:48:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:48:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:48:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:48:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:48:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:48:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:48:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:48:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:48:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:48:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:48:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:48:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:48:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:48:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:48:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:48:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:48:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:48:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:48:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:48:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:48:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:48:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:48:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:48:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:48:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:48:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:48:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:48:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:48:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:48:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:48:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:48:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:48:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:48:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:48:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:48:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:48:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:48:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:48:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:48:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:48:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:48:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:48:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:48:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:48:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:48:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:48:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:48:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:48:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:48:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:48:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:48:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:48:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:48:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:48:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:48:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:48:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:48:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:48:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:48:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:48:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:48:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:48:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:48:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:48:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:48:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:48:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:48:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:48:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:48:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:48:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:48:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:48:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:48:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:48:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:48:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:48:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:48:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:48:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:48:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:48:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:48:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:48:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:48:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:48:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:48:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:48:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:48:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:48:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:48:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:48:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:48:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:48:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:48:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:48:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:48:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:48:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:48:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:48:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:48:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:48:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:48:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:48:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:48:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:48:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:48:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:48:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:48:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:48:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:48:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:48:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:48:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:48:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:48:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:48:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:48:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:48:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:48:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:48:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:48:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:48:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:48:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:48:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:48:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:48:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:48:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:48:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:48:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:49:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:49:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:49:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:49:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:49:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:49:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:49:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:49:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:49:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:49:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:49:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:49:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:49:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:49:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:49:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:49:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:49:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:49:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:49:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:49:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:49:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:49:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:49:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:49:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:49:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:49:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:49:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:49:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:49:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:49:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:49:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:49:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:49:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:49:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:49:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:49:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:49:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:49:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:49:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:49:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:49:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:49:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:49:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:49:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:49:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:49:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:49:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:49:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:49:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:49:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:49:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:49:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:49:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:49:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:49:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:49:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:49:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:49:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:49:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:49:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:49:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:49:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:49:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:49:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:49:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:49:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:49:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:49:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:49:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:49:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:49:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:49:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:49:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:49:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:49:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:49:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:49:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:49:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:49:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:49:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:49:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:49:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:49:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:49:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:49:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:49:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:49:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:49:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:49:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:49:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:49:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:49:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:49:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:49:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:49:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:49:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:49:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:49:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:49:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:49:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:49:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:49:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:49:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:49:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:49:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:49:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:49:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:49:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:49:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:49:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:49:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:49:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:49:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:49:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:49:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:49:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:49:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:49:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:49:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:49:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:49:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:49:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:49:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:49:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:49:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:49:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:49:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:49:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:49:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:49:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:49:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:49:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:49:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:49:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:49:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:49:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:49:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:49:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:49:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:49:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:49:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:49:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:49:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:49:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:49:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:49:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:49:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:49:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:49:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:49:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:49:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:49:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:49:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:49:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:49:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:49:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:49:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:49:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:49:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:49:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:49:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:49:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:49:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:49:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:49:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:49:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:49:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:49:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:49:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:49:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:49:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:49:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:49:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:49:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:49:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:49:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:49:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:49:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:49:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:49:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:49:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:49:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:49:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:49:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:49:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:49:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:49:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:49:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:49:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:49:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:49:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:49:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:49:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:49:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:49:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:49:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:49:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:49:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:49:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:49:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:49:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:49:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:49:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:49:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:49:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:49:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:49:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:49:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:49:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:49:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:49:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:49:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:49:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:49:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:49:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:49:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:50:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:50:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:50:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:50:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:50:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:50:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:50:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:50:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:50:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:50:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:50:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:50:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:50:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:50:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:50:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:50:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:50:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:50:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:50:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:50:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:50:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:50:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:50:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:50:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:50:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:50:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:50:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:50:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:50:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:50:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:50:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:50:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:50:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:50:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:50:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:50:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:50:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:50:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:50:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:50:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:50:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:50:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:50:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:50:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:50:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:50:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:50:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:50:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:50:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:50:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:50:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:50:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:50:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:50:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:50:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:50:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:50:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:50:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:50:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:50:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:50:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:50:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:50:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:50:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:50:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:50:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:50:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:50:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:50:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:50:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:50:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:50:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:50:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:50:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:50:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:50:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:50:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:50:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:50:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:50:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:50:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:50:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:50:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:50:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:50:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:50:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:50:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:50:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:50:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:50:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:50:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:50:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:50:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:50:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:50:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:50:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:50:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:50:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:50:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:50:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:50:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:50:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:50:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:50:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:50:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:50:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:50:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:50:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:50:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:50:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:50:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:50:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:50:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:50:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:50:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:50:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:50:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:50:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:50:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:50:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:50:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:50:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:50:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:50:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:50:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:50:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:50:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:50:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:50:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:50:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:50:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:50:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:50:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:50:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:50:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:50:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:50:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:50:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:50:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:50:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:50:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:50:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:50:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:50:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:50:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:50:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:50:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:50:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:50:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:50:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:50:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:50:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:50:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:50:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:50:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:50:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:50:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:50:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:50:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:50:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:50:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:50:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:50:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:50:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:50:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:50:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:50:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:50:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:50:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:50:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:50:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:50:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:50:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:50:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:50:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:50:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:50:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:50:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:50:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:50:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:50:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:50:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:50:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:50:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:50:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:50:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:50:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:50:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:50:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:50:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:50:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:50:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:50:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:50:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:50:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:50:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:50:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:50:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:50:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:50:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:50:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:50:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:50:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:50:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:50:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:50:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:50:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:50:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:50:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:50:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:50:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:50:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:50:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:50:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:50:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:50:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:51:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:51:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:51:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:51:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:51:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:51:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:51:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:51:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:51:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:51:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:51:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:51:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:51:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:51:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:51:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:51:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:51:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:51:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:51:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:51:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:51:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:51:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:51:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:51:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:51:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:51:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:51:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:51:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:51:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:51:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:51:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:51:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:51:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:51:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:51:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:51:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:51:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:51:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:51:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:51:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:51:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:51:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:51:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:51:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:51:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:51:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:51:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:51:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:51:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:51:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:51:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:51:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:51:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:51:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:51:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:51:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:51:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:51:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:51:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:51:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:51:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:51:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:51:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:51:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:51:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:51:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:51:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:51:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:51:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:51:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:51:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:51:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:51:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:51:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:51:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:51:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:51:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:51:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:51:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:51:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:51:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:51:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:51:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:51:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:51:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:51:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:51:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:51:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:51:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:51:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:51:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:51:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:51:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:51:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:51:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:51:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:51:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:51:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:51:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:51:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:51:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:51:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:51:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:51:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:51:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:51:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:51:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:51:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:51:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:51:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:51:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:51:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:51:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:51:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:51:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:51:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:51:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:51:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:51:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:51:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:51:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:51:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:51:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:51:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:51:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:51:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:51:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:51:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:51:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:51:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:51:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:51:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:51:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:51:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:51:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:51:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:51:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:51:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:51:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:51:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:51:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:51:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:51:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:51:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:51:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:51:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:51:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:51:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:51:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:51:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:51:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:51:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:51:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:51:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:51:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:51:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:51:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:51:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:51:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:51:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:51:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:51:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:51:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:51:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:51:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:51:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:51:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:51:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:51:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:51:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:51:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:51:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:51:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:51:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:51:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:51:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:51:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:51:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:51:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:51:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:51:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:51:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:51:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:51:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:51:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:51:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:51:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:51:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:51:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:51:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:51:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:51:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:51:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:51:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:51:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:51:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:51:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:51:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:51:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:51:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:51:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:51:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:51:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:51:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:51:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:51:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:51:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:51:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:51:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:51:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:51:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:51:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:51:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:51:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:51:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:51:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:52:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:52:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:52:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:52:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:52:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:52:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:52:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:52:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:52:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:52:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:52:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:52:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:52:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:52:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:52:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:52:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:52:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:52:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:52:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:52:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:52:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:52:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:52:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:52:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:52:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:52:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:52:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:52:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:52:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:52:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:52:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:52:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:52:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:52:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:52:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:52:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:52:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:52:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:52:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:52:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:52:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:52:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:52:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:52:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:52:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:52:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:52:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:52:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:52:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:52:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:52:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:52:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:52:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:52:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:52:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:52:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:52:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:52:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:52:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:52:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:52:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:52:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:52:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:52:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:52:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:52:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:52:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:52:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:52:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:52:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:52:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:52:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:52:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:52:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:52:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:52:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:52:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:52:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:52:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:52:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:52:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:52:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:52:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:52:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:52:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:52:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:52:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:52:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:52:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:52:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:52:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:52:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:52:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:52:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:52:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:52:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:52:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:52:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:52:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:52:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:52:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:52:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:52:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:52:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:52:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:52:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:52:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:52:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:52:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:52:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:52:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:52:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:52:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:52:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:52:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:52:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:52:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:52:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:52:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:52:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:52:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:52:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:52:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:52:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:52:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:52:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:52:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:52:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:52:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:52:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:52:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:52:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:52:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:52:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:52:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:52:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:52:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:52:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:52:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:52:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:52:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:52:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:52:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:52:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:52:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:52:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:52:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:52:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:52:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:52:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:52:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:52:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:52:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:52:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:52:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:52:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:52:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:52:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:52:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:52:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:52:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:52:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:52:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:52:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:52:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:52:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:52:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:52:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:52:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:52:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:52:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:52:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:52:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:52:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:52:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:52:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:52:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:52:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:52:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:52:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:52:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:52:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:52:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:52:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:52:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:52:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:52:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:52:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:52:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:52:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:52:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:52:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:52:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:52:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:52:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:52:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:52:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:52:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:52:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:52:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:52:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:52:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:52:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:52:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:52:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:52:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:52:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:52:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:52:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:52:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:52:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:52:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:52:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:52:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:52:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:52:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:53:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:53:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:53:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:53:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:53:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:53:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:53:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:53:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:53:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:53:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:53:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:53:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:53:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:53:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:53:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:53:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:53:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:53:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:53:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:53:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:53:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:53:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:53:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:53:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:53:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:53:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:53:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:53:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:53:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:53:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:53:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:53:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:53:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:53:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:53:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:53:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:53:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:53:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:53:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:53:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:53:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:53:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:53:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:53:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:53:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:53:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:53:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:53:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:53:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:53:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:53:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:53:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:53:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:53:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:53:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:53:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:53:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:53:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:53:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:53:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:53:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:53:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:53:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:53:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:53:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:53:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:53:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:53:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:53:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:53:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:53:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:53:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:53:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:53:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:53:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:53:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:53:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:53:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:53:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:53:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:53:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:53:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:53:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:53:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:53:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:53:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:53:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:53:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:53:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:53:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:53:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:53:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:53:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:53:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:53:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:53:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:53:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:53:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:53:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:53:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:53:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:53:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:53:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:53:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:53:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:53:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:53:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:53:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:53:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:53:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:53:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:53:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:53:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:53:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:53:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:53:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:53:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:53:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:53:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:53:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:53:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:53:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:53:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:53:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:53:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:53:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:53:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:53:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:53:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:53:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:53:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:53:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:53:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:53:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:53:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:53:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:53:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:53:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:53:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:53:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:53:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:53:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:53:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:53:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:53:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:53:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:53:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:53:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:53:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:53:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:53:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:53:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:53:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:53:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:53:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:53:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:53:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:53:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:53:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:53:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:53:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:53:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:53:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:53:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:53:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:53:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:53:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:53:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:53:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:53:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:53:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:53:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:53:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:53:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:53:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:53:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:53:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:53:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:53:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:53:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:53:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:53:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:53:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:53:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:53:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:53:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:53:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:53:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:53:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:53:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:53:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:53:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:53:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:53:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:53:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:53:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:53:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:53:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:55:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:55:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:55:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:55:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:55:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:55:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:55:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:55:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:55:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:55:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:55:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:55:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:55:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:55:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:55:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:55:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:55:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:55:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:55:57 rdma06 rsyslogd[1041]: imjournal: 5082 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 10:56:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:56:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:56:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:56:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:56:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:56:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:56:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:56:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:56:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:56:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:56:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:56:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:56:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:56:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:56:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:56:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:56:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:56:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:56:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:56:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:56:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:56:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:56:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:56:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:56:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:56:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:56:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:56:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:56:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:56:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:56:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:56:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:56:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:56:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:56:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:56:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:56:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:56:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:56:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:56:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:56:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:56:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:56:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:56:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:56:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:56:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:56:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:56:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:56:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:56:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:56:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:56:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:56:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:56:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:56:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:56:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:56:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:56:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:56:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:56:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:56:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:56:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:56:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:56:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:56:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:56:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:56:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:56:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:56:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:56:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:56:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:56:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:56:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:56:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:56:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:56:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:56:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:56:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:56:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:56:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:56:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:56:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:56:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:56:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:56:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:56:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:56:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:56:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:56:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:56:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:56:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:56:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:56:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:56:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:56:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:56:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:56:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:56:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:56:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:56:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:56:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:56:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:56:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:56:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:56:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:56:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:56:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:56:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:56:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:56:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:56:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:56:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:56:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:56:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:56:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:56:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:56:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:56:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:56:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:56:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:56:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:56:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:56:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:56:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:56:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:56:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:56:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:56:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:56:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:56:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:56:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:56:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:56:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:56:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:56:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:56:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:56:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:56:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:56:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:56:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:56:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:56:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:56:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:56:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:56:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:56:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:56:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:56:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:56:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:56:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:56:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:56:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:56:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:56:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:56:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:56:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:56:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:56:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:56:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:56:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:56:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:56:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:56:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:56:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:56:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:56:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:56:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:56:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:56:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:56:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:56:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:56:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:56:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:56:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:56:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:56:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:56:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:56:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:56:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:56:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:56:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:56:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:56:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:56:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:56:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:56:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:56:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:56:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:56:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:56:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:56:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:56:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:56:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:56:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:56:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:56:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:56:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:56:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:56:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:56:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:56:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:56:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:56:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:56:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:56:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:56:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:56:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:56:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:56:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:56:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:56:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:56:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:56:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:56:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:56:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:56:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:57:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:57:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:57:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:57:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:57:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:57:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:57:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:57:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:57:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:57:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:57:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:57:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:57:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:57:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:57:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:57:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:57:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:57:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:57:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:57:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:57:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:57:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:57:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:57:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:57:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:57:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:57:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:57:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:57:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:57:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:57:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:57:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:57:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:57:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:57:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:57:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:57:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:57:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:57:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:57:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:57:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:57:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:57:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:57:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:57:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:57:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:57:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:57:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:57:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:57:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:57:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:57:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:57:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:57:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:57:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:57:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:57:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:57:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:57:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:57:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:57:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:57:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:57:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:57:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:57:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:57:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:57:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:57:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:57:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:57:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:57:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:57:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:57:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:57:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:57:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:57:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:57:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:57:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:57:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:57:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:57:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:57:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:57:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:57:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:57:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:57:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:57:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:57:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:57:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:57:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:57:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:57:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:57:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:57:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:57:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:57:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:57:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:57:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:57:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:57:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:57:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:57:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:57:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:57:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:57:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:57:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:57:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:57:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:57:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:57:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:57:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:57:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:57:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:57:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:57:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:57:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:57:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:57:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:57:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:57:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:57:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:57:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:57:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:57:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:57:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:57:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:57:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:57:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:57:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:57:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:57:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:57:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:57:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:57:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:57:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:57:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:57:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:57:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:57:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:57:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:57:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:57:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:57:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:57:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:57:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:57:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:57:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:57:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:57:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:57:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:57:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:57:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:57:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:57:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:57:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:57:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:57:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:57:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:57:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:57:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:57:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:57:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:57:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:57:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:57:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:57:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:57:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:57:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:57:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:57:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:57:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:57:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:57:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:57:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:57:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:57:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:57:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:57:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:57:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:57:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:57:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:57:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:57:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:57:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:57:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:57:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:57:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:57:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:57:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:57:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:57:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:57:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:57:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:57:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:57:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:57:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:57:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:57:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:57:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:57:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:57:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:57:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:57:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:57:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:57:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:57:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:57:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:57:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:57:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:57:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:57:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:57:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:57:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:57:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:57:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:57:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:58:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:58:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:58:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:58:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:58:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:58:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:58:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:58:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:58:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:58:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:58:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:58:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:58:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:58:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:58:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:58:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:58:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:58:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:58:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:58:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:58:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:58:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:58:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:58:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:58:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:58:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:58:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:58:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:58:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:58:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:58:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:58:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:58:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:58:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:58:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:58:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:58:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:58:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:58:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:58:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:58:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:58:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:58:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:58:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:58:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:58:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:58:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:58:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:58:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:58:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:58:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:58:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:58:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:58:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:58:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:58:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:58:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:58:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:58:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:58:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:58:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:58:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:58:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:58:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:58:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:58:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:58:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:58:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:58:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:58:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:58:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:58:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:58:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:58:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:58:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:58:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:58:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:58:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:58:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:58:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:58:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:58:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:58:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:58:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:58:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:58:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:58:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:58:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:58:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:58:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:58:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:58:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:58:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:58:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:58:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:58:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:58:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:58:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:58:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:58:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:58:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:58:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:58:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:58:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:58:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:58:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:58:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:58:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:58:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:58:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:58:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:58:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:58:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:58:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:58:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:58:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:58:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:58:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:58:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:58:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:58:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:58:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:58:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:58:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:58:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:58:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:58:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:58:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:58:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:58:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:58:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:58:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:58:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:58:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:58:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:58:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:58:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:58:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:58:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:58:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:58:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:58:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:58:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:58:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:58:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:58:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:58:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:58:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:58:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:58:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:58:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:58:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:58:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:58:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:58:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:58:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:58:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:58:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:58:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:58:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:58:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:58:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:58:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:58:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:58:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:58:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:58:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:58:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:58:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:58:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:58:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:58:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:58:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:58:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:58:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:58:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:58:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:58:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:58:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:58:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:58:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:58:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:58:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:58:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:58:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:58:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:58:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:58:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:58:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:58:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:58:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:58:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:58:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:58:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:58:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:58:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:58:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:58:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:58:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:58:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:58:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:58:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:58:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:58:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:58:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:58:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:58:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:58:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:58:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:58:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:58:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:58:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:58:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:58:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:58:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:58:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:59:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:59:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:59:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:59:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:59:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:59:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:59:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:59:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:59:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:59:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:59:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:59:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:59:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:59:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:59:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:59:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:59:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:59:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:59:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:59:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:59:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:59:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:59:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:59:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:59:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:59:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:59:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:59:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:59:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:59:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:59:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:59:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:59:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:59:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:59:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:59:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:59:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:59:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:59:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:59:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:59:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:59:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:59:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:59:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:59:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:59:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:59:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:59:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:59:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:59:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:59:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:59:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:59:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:59:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:59:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:59:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:59:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:59:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:59:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:59:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:59:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:59:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:59:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:59:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:59:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:59:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:59:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:59:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:59:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:59:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:59:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:59:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:59:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:59:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:59:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:59:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:59:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:59:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:59:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:59:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:59:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:59:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:59:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:59:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:59:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:59:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:59:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:59:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:59:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:59:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:59:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:59:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:59:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:59:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:59:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:59:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:59:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:59:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:59:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:59:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:59:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:59:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:59:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:59:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:59:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:59:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:59:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:59:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:59:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:59:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:59:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:59:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:59:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:59:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:59:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:59:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:59:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:59:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:59:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:59:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:59:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:59:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:59:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:59:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:59:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:59:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:59:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:59:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:59:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:59:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:59:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:59:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:59:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:59:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:59:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:59:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:59:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:59:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:59:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:59:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:59:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:59:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:59:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:59:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:59:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:59:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:59:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:59:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:59:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:59:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:59:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:59:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:59:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:59:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:59:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:59:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:59:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:59:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:59:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:59:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:59:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:59:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:59:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:59:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:59:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:59:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:59:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:59:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:59:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:59:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:59:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:59:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:59:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:59:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:59:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:59:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:59:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:59:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:59:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:59:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:59:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:59:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:59:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:59:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:59:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:59:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:59:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:59:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:59:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:59:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:59:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:59:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:59:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:59:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:59:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:59:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:59:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:59:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 10:59:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 10:59:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:59:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:59:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:59:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 10:59:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 10:59:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 10:59:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 10:59:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 10:59:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 10:59:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 10:59:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 10:59:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 10:59:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 10:59:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 10:59:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 10:59:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 10:59:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:00:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:00:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:00:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:00:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:00:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:00:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:00:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:00:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:00:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:00:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:00:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:00:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:00:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:00:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:00:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:00:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:00:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:00:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:00:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:00:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:00:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:00:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:00:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:00:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:00:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:00:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:00:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:00:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:00:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:00:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:00:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:00:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:00:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:00:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:00:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:00:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:00:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:00:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:00:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:00:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:00:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:00:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:00:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:00:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:00:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:00:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:00:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:00:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:00:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:00:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:00:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:00:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:00:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:00:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:00:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:00:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:00:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:00:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:00:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:00:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:00:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:00:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:00:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:00:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:00:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:00:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:00:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:00:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:00:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:00:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:00:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:00:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:00:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:00:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:00:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:00:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:00:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:00:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:00:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:00:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:00:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:00:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:00:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:00:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:00:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:00:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:00:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:00:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:00:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:00:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:00:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:00:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:00:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:00:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:00:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:00:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:00:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:00:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:00:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:00:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:00:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:00:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:00:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:00:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:00:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:00:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:00:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:00:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:00:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:00:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:00:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:00:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:00:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:00:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:00:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:00:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:00:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:00:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:00:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:00:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:00:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:00:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:00:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:00:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:00:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:00:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:00:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:00:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:00:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:00:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:00:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:00:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:00:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:00:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:00:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:00:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:00:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:00:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:00:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:00:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:00:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:00:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:00:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:00:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:00:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:00:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:00:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:00:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:00:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:00:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:00:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:00:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:00:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:00:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:00:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:00:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:00:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:00:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:00:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:00:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:00:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:00:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:00:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:00:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:00:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:00:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:00:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:00:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:00:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:00:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:00:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:00:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:00:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:00:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:00:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:00:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:00:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:00:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:00:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:00:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:00:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:00:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:00:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:00:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:00:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:00:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:00:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:00:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:00:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:00:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:00:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:00:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:00:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:00:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:00:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:00:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:00:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:00:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:00:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:00:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:00:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:00:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:00:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:00:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:00:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:00:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:00:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:00:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:00:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:00:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:00:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:00:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:00:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:00:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:00:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:00:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:01:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:01:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:01:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:01:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:01:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:01:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:01:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:01:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:01:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:01:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:01:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:01:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:01:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:01:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:01:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:01:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:01:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:01:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:01:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:01:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:01:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:01:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:01:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:01:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:01:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:01:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:01:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:01:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:01:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:01:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:01:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:01:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:01:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:01:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:01:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:01:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:01:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:01:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:01:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:01:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:01:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:01:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:01:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:01:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:01:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:01:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:01:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:01:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:01:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:01:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:01:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:01:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:01:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:01:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:01:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:01:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:01:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:01:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:01:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:01:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:01:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:01:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:01:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:01:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:01:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:01:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:01:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:01:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:01:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:01:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:01:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:01:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:01:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:01:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:01:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:01:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:01:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:01:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:01:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:01:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:01:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:01:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:01:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:01:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:01:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:01:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:01:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:01:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:01:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:01:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:01:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:01:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:01:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:01:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:01:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:01:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:01:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:01:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:01:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:01:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:01:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:01:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:01:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:01:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:01:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:01:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:01:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:01:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:01:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:01:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:01:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:01:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:01:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:01:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:01:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:01:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:01:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:01:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:01:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:01:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:01:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:01:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:01:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:01:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:01:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:01:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:01:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:01:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:01:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:01:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:01:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:01:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:01:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:01:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:01:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:01:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:01:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:01:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:01:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:01:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:01:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:01:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:01:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:01:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:01:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:01:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:01:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:01:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:01:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:01:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:01:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:01:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:01:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:01:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:01:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:01:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:01:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:01:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:01:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:01:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:01:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:01:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:01:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:01:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:01:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:01:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:01:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:01:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:01:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:01:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:01:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:01:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:01:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:01:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:01:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:01:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:01:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:01:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:01:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:01:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:01:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:01:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:01:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:01:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:01:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:01:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:01:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:01:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:01:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:01:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:01:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:01:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:01:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:01:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:01:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:01:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:01:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:01:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:01:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:01:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:01:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:01:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:01:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:01:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:01:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:01:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:01:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:01:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:01:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:01:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:01:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:01:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:01:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:01:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:01:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:01:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:02:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:02:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:02:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:02:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:02:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:02:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:02:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:02:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:02:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:02:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:02:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:02:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:02:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:02:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:02:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:02:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:02:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:02:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:02:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:02:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:02:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:02:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:02:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:02:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:02:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:02:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:02:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:02:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:02:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:02:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:02:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:02:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:02:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:02:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:02:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:02:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:02:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:02:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:02:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:02:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:02:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:02:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:02:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:02:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:02:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:02:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:02:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:02:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:02:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:02:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:02:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:02:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:02:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:02:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:02:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:02:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:02:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:02:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:02:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:02:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:02:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:02:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:02:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:02:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:02:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:02:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:02:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:02:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:02:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:02:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:02:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:02:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:02:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:02:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:02:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:02:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:02:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:02:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:02:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:02:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:02:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:02:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:02:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:02:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:02:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:02:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:02:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:02:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:02:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:02:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:02:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:02:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:02:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:02:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:02:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:02:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:02:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:02:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:02:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:02:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:02:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:02:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:02:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:02:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:02:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:02:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:02:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:02:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:02:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:02:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:02:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:02:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:02:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:02:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:02:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:02:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:02:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:02:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:02:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:02:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:02:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:02:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:02:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:02:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:02:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:02:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:02:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:02:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:02:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:02:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:02:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:02:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:02:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:02:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:02:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:02:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:02:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:02:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:02:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:02:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:02:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:02:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:02:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:02:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:02:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:02:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:02:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:02:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:02:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:02:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:02:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:02:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:02:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:02:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:02:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:02:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:02:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:02:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:02:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:02:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:02:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:02:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:02:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:02:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:02:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:02:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:02:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:02:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:02:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:02:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:02:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:02:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:02:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:02:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:02:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:02:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:02:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:02:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:02:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:02:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:02:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:02:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:02:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:02:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:02:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:02:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:02:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:02:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:02:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:02:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:02:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:02:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:02:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:02:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:02:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:02:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:02:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:02:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:02:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:02:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:02:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:02:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:02:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:02:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:02:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:02:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:02:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:02:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:02:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:02:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:02:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:02:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:02:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:02:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:02:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:02:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:03:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:03:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:03:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:03:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:03:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:03:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:03:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:03:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:03:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:03:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:03:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:03:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:03:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:03:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:03:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:03:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:03:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:03:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:03:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:03:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:03:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:03:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:03:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:03:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:03:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:03:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:03:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:03:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:03:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:03:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:03:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:03:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:03:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:03:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:03:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:03:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:03:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:03:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:03:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:03:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:03:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:03:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:03:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:03:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:03:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:03:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:03:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:03:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:03:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:03:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:03:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:03:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:03:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:03:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:03:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:03:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:03:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:03:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:03:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:03:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:03:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:03:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:03:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:03:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:03:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:03:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:03:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:03:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:03:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:03:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:03:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:03:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:03:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:03:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:03:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:03:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:03:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:03:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:03:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:03:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:03:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:03:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:03:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:03:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:03:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:03:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:03:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:03:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:03:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:03:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:03:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:03:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:03:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:03:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:03:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:03:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:03:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:03:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:03:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:03:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:03:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:03:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:03:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:03:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:03:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:03:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:03:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:03:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:03:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:03:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:03:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:03:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:03:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:03:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:03:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:03:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:03:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:03:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:03:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:03:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:03:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:03:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:03:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:03:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:03:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:03:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:03:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:03:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:03:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:03:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:03:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:03:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:03:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:03:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:03:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:03:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:03:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:03:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:03:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:03:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:03:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:03:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:03:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:03:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:03:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:03:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:03:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:03:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:03:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:03:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:03:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:03:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:03:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:03:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:03:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:03:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:03:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:03:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:03:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:03:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:03:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:03:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:03:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:03:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:03:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:03:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:03:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:03:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:03:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:03:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:03:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:03:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:03:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:03:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:03:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:03:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:03:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:03:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:03:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:03:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:03:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:03:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:03:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:03:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:03:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:03:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:03:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:03:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:03:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:03:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:03:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:03:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:03:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:03:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:03:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:03:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:03:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:03:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:06:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:06:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:06:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:06:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:06:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:06:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:06:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:06:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:06:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:06:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:06:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:06:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:06:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:06:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:06:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:06:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:06:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:06:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:06:02 rdma06 rsyslogd[1041]: imjournal: 5295 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 11:06:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:06:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:06:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:06:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:06:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:06:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:06:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:06:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:06:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:06:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:06:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:06:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:06:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:06:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:06:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:06:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:06:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:06:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:06:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:06:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:06:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:06:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:06:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:06:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:06:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:06:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:06:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:06:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:06:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:06:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:06:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:06:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:06:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:06:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:06:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:06:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:06:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:06:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:06:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:06:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:06:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:06:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:06:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:06:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:06:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:06:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:06:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:06:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:06:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:06:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:06:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:06:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:06:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:06:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:06:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:06:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:06:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:06:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:06:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:06:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:06:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:06:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:06:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:06:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:06:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:06:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:06:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:06:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:06:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:06:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:06:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:06:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:06:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:06:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:06:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:06:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:06:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:06:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:06:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:06:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:06:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:06:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:06:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:06:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:06:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:06:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:06:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:06:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:06:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:06:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:06:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:06:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:06:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:06:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:06:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:06:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:06:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:06:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:06:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:06:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:06:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:06:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:06:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:06:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:06:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:06:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:06:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:06:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:06:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:06:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:06:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:06:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:06:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:06:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:06:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:06:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:06:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:06:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:06:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:06:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:06:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:06:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:06:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:06:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:06:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:06:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:06:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:06:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:06:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:06:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:06:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:06:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:06:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:06:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:06:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:06:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:06:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:06:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:06:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:06:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:06:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:06:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:06:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:06:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:06:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:06:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:06:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:06:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:06:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:06:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:06:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:06:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:06:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:06:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:06:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:06:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:06:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:06:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:06:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:06:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:06:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:06:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:06:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:06:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:06:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:06:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:06:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:06:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:06:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:06:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:06:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:06:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:06:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:06:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:06:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:06:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:06:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:06:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:06:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:06:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:06:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:06:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:06:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:06:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:06:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:06:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:06:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:06:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:06:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:06:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:06:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:06:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:06:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:06:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:06:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:06:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:06:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:06:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:07:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:07:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:07:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:07:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:07:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:07:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:07:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:07:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:07:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:07:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:07:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:07:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:07:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:07:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:07:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:07:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:07:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:07:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:07:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:07:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:07:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:07:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:07:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:07:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:07:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:07:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:07:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:07:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:07:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:07:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:07:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:07:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:07:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:07:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:07:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:07:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:07:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:07:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:07:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:07:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:07:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:07:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:07:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:07:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:07:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:07:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:07:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:07:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:07:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:07:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:07:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:07:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:07:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:07:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:07:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:07:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:07:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:07:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:07:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:07:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:07:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:07:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:07:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:07:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:07:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:07:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:07:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:07:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:07:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:07:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:07:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:07:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:07:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:07:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:07:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:07:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:07:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:07:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:07:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:07:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:07:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:07:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:07:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:07:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:07:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:07:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:07:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:07:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:07:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:07:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:07:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:07:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:07:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:07:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:07:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:07:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:07:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:07:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:07:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:07:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:07:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:07:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:07:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:07:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:07:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:07:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:07:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:07:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:07:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:07:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:07:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:07:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:07:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:07:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:07:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:07:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:07:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:07:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:07:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:07:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:07:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:07:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:07:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:07:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:07:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:07:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:07:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:07:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:07:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:07:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:07:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:07:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:07:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:07:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:07:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:07:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:07:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:07:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:07:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:07:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:07:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:07:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:07:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:07:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:07:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:07:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:07:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:07:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:07:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:07:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:07:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:07:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:07:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:07:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:07:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:07:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:07:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:07:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:07:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:07:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:07:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:07:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:07:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:07:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:07:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:07:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:07:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:07:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:07:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:07:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:07:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:07:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:07:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:07:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:07:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:07:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:07:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:07:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:07:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:07:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:07:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:07:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:07:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:07:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:07:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:07:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:07:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:07:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:07:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:07:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:07:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:07:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:07:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:07:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:07:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:07:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:07:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:07:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:07:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:07:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:07:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:07:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:07:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:07:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:07:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:07:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:07:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:07:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:07:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:07:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:07:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:07:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:07:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:07:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:07:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:07:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:08:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:08:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:08:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:08:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:08:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:08:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:08:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:08:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:08:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:08:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:08:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:08:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:08:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:08:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:08:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:08:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:08:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:08:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:08:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:08:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:08:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:08:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:08:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:08:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:08:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:08:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:08:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:08:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:08:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:08:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:08:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:08:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:08:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:08:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:08:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:08:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:08:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:08:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:08:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:08:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:08:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:08:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:08:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:08:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:08:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:08:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:08:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:08:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:08:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:08:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:08:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:08:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:08:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:08:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:08:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:08:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:08:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:08:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:08:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:08:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:08:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:08:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:08:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:08:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:08:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:08:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:08:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:08:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:08:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:08:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:08:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:08:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:08:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:08:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:08:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:08:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:08:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:08:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:08:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:08:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:08:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:08:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:08:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:08:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:08:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:08:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:08:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:08:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:08:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:08:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:08:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:08:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:08:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:08:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:08:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:08:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:08:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:08:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:08:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:08:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:08:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:08:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:08:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:08:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:08:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:08:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:08:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:08:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:08:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:08:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:08:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:08:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:08:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:08:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:08:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:08:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:08:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:08:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:08:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:08:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:08:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:08:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:08:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:08:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:08:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:08:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:08:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:08:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:08:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:08:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:08:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:08:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:08:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:08:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:08:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:08:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:08:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:08:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:08:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:08:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:08:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:08:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:08:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:08:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:08:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:08:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:08:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:08:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:08:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:08:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:08:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:08:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:08:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:08:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:08:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:08:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:08:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:08:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:08:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:08:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:08:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:08:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:08:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:08:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:08:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:08:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:08:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:08:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:08:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:08:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:08:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:08:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:08:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:08:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:08:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:08:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:08:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:08:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:08:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:08:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:08:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:08:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:08:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:08:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:08:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:08:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:08:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:08:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:08:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:08:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:08:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:08:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:08:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:08:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:08:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:08:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:08:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:08:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:08:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:08:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:08:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:08:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:08:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:08:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:08:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:08:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:08:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:08:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:08:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:08:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:08:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:08:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:08:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:08:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:08:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:08:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:09:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:09:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:09:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:09:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:09:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:09:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:09:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:09:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:09:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:09:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:09:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:09:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:09:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:09:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:09:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:09:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:09:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:09:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:09:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:09:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:09:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:09:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:09:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:09:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:09:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:09:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:09:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:09:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:09:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:09:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:09:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:09:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:09:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:09:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:09:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:09:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:09:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:09:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:09:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:09:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:09:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:09:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:09:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:09:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:09:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:09:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:09:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:09:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:09:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:09:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:09:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:09:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:09:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:09:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:09:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:09:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:09:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:09:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:09:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:09:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:09:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:09:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:09:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:09:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:09:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:09:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:09:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:09:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:09:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:09:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:09:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:09:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:09:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:09:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:09:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:09:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:09:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:09:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:09:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:09:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:09:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:09:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:09:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:09:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:09:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:09:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:09:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:09:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:09:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:09:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:09:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:09:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:09:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:09:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:09:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:09:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:09:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:09:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:09:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:09:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:09:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:09:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:09:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:09:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:09:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:09:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:09:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:09:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:09:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:09:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:09:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:09:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:09:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:09:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:09:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:09:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:09:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:09:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:09:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:09:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:09:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:09:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:09:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:09:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:09:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:09:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:09:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:09:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:09:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:09:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:09:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:09:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:09:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:09:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:09:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:09:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:09:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:09:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:09:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:09:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:09:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:09:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:09:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:09:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:09:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:09:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:09:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:09:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:09:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:09:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:09:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:09:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:09:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:09:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:09:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:09:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:09:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:09:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:09:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:09:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:09:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:09:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:09:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:09:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:09:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:09:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:09:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:09:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:09:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:09:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:09:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:09:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:09:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:09:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:09:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:09:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:09:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:09:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:09:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:09:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:09:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:09:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:09:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:09:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:09:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:09:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:09:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:09:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:09:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:09:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:09:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:09:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:09:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:09:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:09:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:09:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:09:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:09:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:09:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:09:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:09:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:09:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:09:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:09:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:09:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:09:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:09:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:09:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:09:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:09:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:09:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:09:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:09:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:09:57 rdma06 systemd-journald[790]: Data hash table of /run/log/journal/95379d2f723847b68d9e976dee8967a9/system.journal has a fill level at 75.0 (163750 of 218332 items, 75497472 file size, 461 bytes per hash table item), suggesting rotation.
Apr 17 11:09:57 rdma06 systemd-journald[790]: /run/log/journal/95379d2f723847b68d9e976dee8967a9/system.journal: Journal header limits reached or header out-of-date, rotating.
Apr 17 11:09:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:09:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:09:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:09:57 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 17 11:09:57 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 17 11:09:57 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 17 11:10:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:10:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:10:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:10:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:10:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:10:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:10:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:10:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:10:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:10:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:10:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:10:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:10:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:10:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:10:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:10:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:10:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:10:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:10:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:10:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:10:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:10:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:10:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:10:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:10:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:10:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:10:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:10:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:10:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:10:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:10:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:10:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:10:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:10:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:10:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:10:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:10:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:10:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:10:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:10:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:10:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:10:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:10:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:10:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:10:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:10:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:10:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:10:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:10:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:10:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:10:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:10:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:10:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:10:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:10:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:10:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:10:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:10:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:10:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:10:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:10:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:10:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:10:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:10:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:10:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:10:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:10:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:10:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:10:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:10:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:10:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:10:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:10:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:10:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:10:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:10:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:10:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:10:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:10:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:10:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:10:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:10:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:10:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:10:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:10:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:10:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:10:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:10:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:10:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:10:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:10:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:10:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:10:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:10:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:10:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:10:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:10:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:10:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:10:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:10:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:10:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:10:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:10:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:10:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:10:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:10:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:10:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:10:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:10:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:10:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:10:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:10:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:10:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:10:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:10:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:10:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:10:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:10:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:10:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:10:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:10:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:10:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:10:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:10:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:10:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:10:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:10:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:10:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:10:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:10:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:10:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:10:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:10:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:10:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:10:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:10:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:10:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:10:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:10:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:10:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:10:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:10:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:10:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:10:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:10:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:10:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:10:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:10:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:10:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:10:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:10:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:10:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:10:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:10:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:10:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:10:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:10:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:10:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:10:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:10:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:10:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:10:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:10:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:10:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:10:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:10:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:10:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:10:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:10:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:10:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:10:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:10:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:10:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:10:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:10:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:10:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:10:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:10:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:10:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:10:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:10:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:10:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:10:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:10:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:10:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:10:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:10:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:10:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:10:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:10:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:10:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:10:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:10:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:10:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:10:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:10:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:10:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:10:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:10:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:10:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:10:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:10:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:10:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:10:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:10:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:10:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:10:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:10:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:10:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:10:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:10:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:10:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:10:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:10:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:10:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:10:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:11:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:11:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:11:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:11:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:11:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:11:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:11:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:11:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:11:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:11:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:11:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:11:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:11:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:11:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:11:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:11:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:11:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:11:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:11:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:11:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:11:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:11:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:11:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:11:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:11:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:11:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:11:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:11:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:11:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:11:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:11:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:11:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:11:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:11:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:11:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:11:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:11:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:11:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:11:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:11:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:11:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:11:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:11:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:11:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:11:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:11:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:11:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:11:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:11:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:11:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:11:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:11:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:11:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:11:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:11:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:11:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:11:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:11:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:11:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:11:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:11:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:11:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:11:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:11:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:11:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:11:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:11:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:11:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:11:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:11:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:11:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:11:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:11:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:11:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:11:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:11:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:11:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:11:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:11:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:11:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:11:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:11:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:11:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:11:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:11:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:11:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:11:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:11:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:11:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:11:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:11:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:11:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:11:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:11:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:11:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:11:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:11:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:11:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:11:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:11:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:11:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:11:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:11:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:11:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:11:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:11:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:11:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:11:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:11:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:11:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:11:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:11:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:11:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:11:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:11:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:11:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:11:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:11:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:11:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:11:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:11:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:11:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:11:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:11:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:11:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:11:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:11:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:11:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:11:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:11:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:11:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:11:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:11:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:11:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:11:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:11:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:11:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:11:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:11:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:11:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:11:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:11:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:11:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:11:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:11:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:11:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:11:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:11:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:11:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:11:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:11:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:11:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:11:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:11:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:11:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:11:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:11:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:11:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:11:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:11:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:11:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:11:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:11:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:11:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:11:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:11:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:11:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:11:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:11:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:11:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:11:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:11:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:11:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:11:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:11:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:11:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:11:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:11:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:11:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:11:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:11:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:11:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:11:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:11:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:11:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:11:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:11:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:11:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:11:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:11:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:11:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:11:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:11:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:11:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:11:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:11:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:11:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:11:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:11:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:11:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:11:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:11:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:11:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:11:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:11:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:11:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:11:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:11:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:11:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:11:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:11:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:11:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:11:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:11:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:11:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:11:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:12:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:12:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:12:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:12:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:12:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:12:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:12:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:12:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:12:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:12:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:12:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:12:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:12:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:12:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:12:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:12:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:12:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:12:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:12:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:12:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:12:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:12:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:12:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:12:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:12:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:12:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:12:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:12:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:12:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:12:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:12:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:12:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:12:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:12:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:12:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:12:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:12:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:12:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:12:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:12:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:12:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:12:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:12:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:12:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:12:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:12:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:12:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:12:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:12:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:12:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:12:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:12:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:12:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:12:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:12:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:12:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:12:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:12:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:12:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:12:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:12:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:12:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:12:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:12:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:12:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:12:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:12:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:12:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:12:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:12:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:12:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:12:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:12:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:12:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:12:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:12:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:12:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:12:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:12:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:12:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:12:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:12:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:12:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:12:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:12:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:12:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:12:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:12:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:12:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:12:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:12:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:12:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:12:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:12:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:12:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:12:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:12:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:12:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:12:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:12:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:12:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:12:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:12:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:12:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:12:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:12:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:12:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:12:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:12:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:12:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:12:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:12:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:12:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:12:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:12:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:12:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:12:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:12:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:12:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:12:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:12:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:12:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:12:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:12:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:12:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:12:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:12:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:12:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:12:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:12:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:12:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:12:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:12:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:12:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:12:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:12:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:12:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:12:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:12:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:12:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:12:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:12:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:12:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:12:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:12:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:12:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:12:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:12:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:12:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:12:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:12:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:12:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:12:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:12:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:12:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:12:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:12:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:12:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:12:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:12:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:12:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:12:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:12:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:12:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:12:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:12:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:12:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:12:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:12:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:12:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:12:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:12:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:12:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:12:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:12:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:12:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:12:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:12:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:12:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:12:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:12:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:12:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:12:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:12:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:12:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:12:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:12:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:12:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:12:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:12:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:12:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:12:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:12:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:12:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:12:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:12:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:12:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:12:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:12:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:12:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:12:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:12:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:12:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:12:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:12:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:12:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:12:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:12:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:12:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:12:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:12:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:12:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:12:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:12:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:12:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:12:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:13:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:13:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:13:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:13:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:13:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:13:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:13:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:13:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:13:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:13:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:13:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:13:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:13:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:13:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:13:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:13:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:13:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:13:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:13:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:13:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:13:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:13:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:13:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:13:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:13:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:13:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:13:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:13:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:13:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:13:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:13:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:13:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:13:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:13:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:13:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:13:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:13:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:13:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:13:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:13:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:13:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:13:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:13:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:13:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:13:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:13:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:13:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:13:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:13:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:13:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:13:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:13:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:13:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:13:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:13:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:13:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:13:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:13:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:13:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:13:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:13:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:13:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:13:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:13:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:13:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:13:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:13:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:13:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:13:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:13:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:13:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:13:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:13:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:13:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:13:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:13:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:13:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:13:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:13:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:13:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:13:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:13:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:13:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:13:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:13:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:13:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:13:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:13:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:13:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:13:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:13:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:13:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:13:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:13:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:13:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:13:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:13:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:13:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:13:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:13:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:13:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:13:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:13:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:13:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:13:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:13:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:13:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:13:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:13:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:13:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:13:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:13:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:13:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:13:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:13:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:13:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:13:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:13:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:13:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:13:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:13:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:13:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:13:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:13:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:13:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:13:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:13:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:13:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:13:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:13:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:13:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:13:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:13:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:13:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:13:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:13:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:13:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:13:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:13:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:13:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:13:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:13:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:13:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:13:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:13:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:13:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:13:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:13:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:13:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:13:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:13:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:13:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:13:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:13:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:13:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:13:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:13:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:13:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:13:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:13:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:13:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:13:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:13:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:13:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:13:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:13:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:13:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:13:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:13:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:13:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:13:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:13:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:13:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:13:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:13:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:13:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:13:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:13:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:13:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:13:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:13:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:13:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:13:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:13:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:13:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:13:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:13:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:13:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:13:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:13:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:13:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:13:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:13:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:13:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:13:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:13:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:13:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:13:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:13:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:13:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:13:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:13:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:13:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:13:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:13:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:13:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:13:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:13:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:13:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:13:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:13:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:13:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:13:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:13:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:13:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:13:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:16:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:16:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:16:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:16:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:16:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:16:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:16:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:16:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:16:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:16:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:16:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:16:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:16:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:16:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:16:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:16:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:16:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:16:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:16:03 rdma06 rsyslogd[1041]: imjournal: 5087 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 11:16:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:16:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:16:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:16:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:16:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:16:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:16:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:16:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:16:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:16:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:16:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:16:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:16:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:16:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:16:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:16:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:16:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:16:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:16:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:16:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:16:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:16:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:16:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:16:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:16:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:16:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:16:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:16:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:16:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:16:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:16:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:16:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:16:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:16:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:16:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:16:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:16:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:16:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:16:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:16:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:16:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:16:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:16:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:16:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:16:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:16:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:16:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:16:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:16:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:16:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:16:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:16:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:16:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:16:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:16:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:16:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:16:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:16:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:16:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:16:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:16:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:16:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:16:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:16:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:16:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:16:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:16:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:16:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:16:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:16:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:16:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:16:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:16:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:16:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:16:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:16:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:16:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:16:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:16:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:16:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:16:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:16:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:16:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:16:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:16:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:16:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:16:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:16:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:16:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:16:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:16:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:16:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:16:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:16:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:16:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:16:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:16:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:16:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:16:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:16:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:16:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:16:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:16:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:16:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:16:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:16:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:16:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:16:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:16:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:16:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:16:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:16:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:16:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:16:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:16:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:16:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:16:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:16:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:16:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:16:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:16:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:16:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:16:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:16:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:16:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:16:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:16:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:16:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:16:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:16:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:16:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:16:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:16:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:16:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:16:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:16:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:16:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:16:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:16:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:16:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:16:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:16:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:16:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:16:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:16:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:16:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:16:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:16:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:16:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:16:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:16:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:16:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:16:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:16:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:16:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:16:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:16:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:16:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:16:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:16:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:16:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:16:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:16:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:16:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:16:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:16:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:16:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:16:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:16:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:16:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:16:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:16:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:16:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:16:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:16:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:16:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:16:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:16:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:16:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:16:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:16:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:16:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:16:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:16:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:16:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:16:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:16:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:16:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:16:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:16:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:16:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:16:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:16:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:16:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:16:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:16:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:16:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:16:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:17:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:17:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:17:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:17:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:17:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:17:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:17:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:17:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:17:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:17:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:17:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:17:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:17:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:17:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:17:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:17:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:17:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:17:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:17:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:17:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:17:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:17:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:17:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:17:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:17:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:17:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:17:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:17:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:17:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:17:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:17:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:17:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:17:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:17:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:17:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:17:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:17:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:17:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:17:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:17:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:17:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:17:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:17:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:17:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:17:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:17:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:17:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:17:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:17:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:17:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:17:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:17:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:17:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:17:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:17:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:17:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:17:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:17:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:17:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:17:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:17:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:17:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:17:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:17:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:17:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:17:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:17:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:17:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:17:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:17:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:17:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:17:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:17:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:17:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:17:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:17:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:17:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:17:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:17:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:17:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:17:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:17:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:17:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:17:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:17:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:17:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:17:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:17:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:17:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:17:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:17:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:17:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:17:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:17:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:17:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:17:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:17:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:17:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:17:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:17:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:17:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:17:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:17:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:17:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:17:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:17:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:17:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:17:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:17:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:17:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:17:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:17:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:17:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:17:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:17:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:17:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:17:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:17:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:17:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:17:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:17:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:17:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:17:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:17:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:17:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:17:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:17:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:17:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:17:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:17:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:17:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:17:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:17:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:17:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:17:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:17:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:17:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:17:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:17:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:17:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:17:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:17:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:17:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:17:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:17:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:17:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:17:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:17:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:17:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:17:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:17:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:17:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:17:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:17:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:17:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:17:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:17:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:17:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:17:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:17:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:17:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:17:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:17:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:17:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:17:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:17:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:17:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:17:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:17:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:17:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:17:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:17:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:17:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:17:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:17:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:17:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:17:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:17:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:17:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:17:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:17:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:17:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:17:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:17:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:17:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:17:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:17:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:17:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:17:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:17:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:17:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:17:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:17:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:17:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:17:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:17:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:17:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:17:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:17:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:17:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:17:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:17:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:17:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:17:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:17:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:17:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:17:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:17:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:17:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:17:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:17:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:17:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:17:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:17:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:17:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:17:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:18:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:18:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:18:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:18:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:18:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:18:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:18:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:18:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:18:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:18:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:18:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:18:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:18:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:18:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:18:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:18:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:18:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:18:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:18:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:18:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:18:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:18:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:18:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:18:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:18:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:18:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:18:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:18:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:18:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:18:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:18:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:18:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:18:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:18:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:18:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:18:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:18:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:18:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:18:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:18:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:18:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:18:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:18:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:18:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:18:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:18:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:18:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:18:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:18:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:18:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:18:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:18:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:18:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:18:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:18:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:18:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:18:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:18:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:18:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:18:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:18:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:18:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:18:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:18:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:18:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:18:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:18:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:18:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:18:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:18:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:18:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:18:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:18:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:18:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:18:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:18:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:18:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:18:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:18:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:18:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:18:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:18:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:18:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:18:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:18:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:18:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:18:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:18:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:18:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:18:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:18:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:18:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:18:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:18:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:18:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:18:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:18:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:18:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:18:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:18:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:18:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:18:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:18:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:18:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:18:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:18:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:18:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:18:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:18:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:18:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:18:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:18:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:18:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:18:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:18:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:18:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:18:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:18:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:18:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:18:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:18:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:18:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:18:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:18:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:18:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:18:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:18:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:18:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:18:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:18:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:18:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:18:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:18:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:18:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:18:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:18:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:18:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:18:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:18:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:18:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:18:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:18:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:18:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:18:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:18:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:18:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:18:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:18:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:18:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:18:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:18:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:18:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:18:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:18:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:18:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:18:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:18:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:18:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:18:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:18:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:18:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:18:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:18:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:18:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:18:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:18:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:18:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:18:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:18:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:18:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:18:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:18:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:18:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:18:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:18:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:18:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:18:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:18:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:18:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:18:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:18:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:18:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:18:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:18:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:18:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:18:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:18:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:18:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:18:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:18:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:18:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:18:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:18:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:18:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:18:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:18:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:18:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:18:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:18:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:18:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:18:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:18:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:18:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:18:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:18:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:18:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:18:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:18:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:18:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:18:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:18:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:18:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:18:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:18:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:18:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:18:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:19:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:19:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:19:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:19:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:19:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:19:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:19:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:19:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:19:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:19:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:19:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:19:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:19:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:19:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:19:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:19:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:19:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:19:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:19:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:19:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:19:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:19:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:19:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:19:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:19:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:19:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:19:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:19:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:19:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:19:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:19:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:19:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:19:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:19:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:19:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:19:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:19:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:19:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:19:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:19:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:19:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:19:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:19:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:19:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:19:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:19:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:19:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:19:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:19:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:19:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:19:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:19:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:19:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:19:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:19:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:19:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:19:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:19:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:19:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:19:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:19:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:19:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:19:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:19:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:19:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:19:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:19:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:19:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:19:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:19:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:19:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:19:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:19:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:19:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:19:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:19:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:19:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:19:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:19:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:19:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:19:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:19:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:19:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:19:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:19:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:19:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:19:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:19:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:19:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:19:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:19:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:19:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:19:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:19:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:19:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:19:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:19:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:19:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:19:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:19:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:19:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:19:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:19:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:19:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:19:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:19:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:19:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:19:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:19:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:19:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:19:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:19:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:19:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:19:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:19:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:19:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:19:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:19:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:19:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:19:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:19:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:19:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:19:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:19:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:19:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:19:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:19:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:19:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:19:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:19:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:19:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:19:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:19:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:19:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:19:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:19:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:19:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:19:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:19:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:19:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:19:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:19:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:19:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:19:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:19:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:19:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:19:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:19:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:19:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:19:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:19:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:19:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:19:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:19:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:19:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:19:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:19:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:19:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:19:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:19:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:19:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:19:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:19:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:19:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:19:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:19:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:19:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:19:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:19:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:19:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:19:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:19:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:19:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:19:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:19:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:19:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:19:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:19:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:19:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:19:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:19:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:19:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:19:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:19:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:19:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:19:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:19:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:19:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:19:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:19:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:19:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:19:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:19:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:19:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:19:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:19:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:19:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:19:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:19:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:19:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:19:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:19:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:19:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:19:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:19:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:19:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:19:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:19:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:19:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:19:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:19:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:19:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:19:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:19:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:19:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:19:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:20:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:20:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:20:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:20:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:20:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:20:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:20:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:20:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:20:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:20:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:20:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:20:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:20:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:20:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:20:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:20:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:20:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:20:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:20:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:20:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:20:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:20:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:20:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:20:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:20:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:20:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:20:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:20:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:20:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:20:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:20:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:20:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:20:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:20:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:20:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:20:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:20:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:20:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:20:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:20:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:20:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:20:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:20:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:20:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:20:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:20:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:20:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:20:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:20:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:20:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:20:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:20:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:20:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:20:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:20:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:20:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:20:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:20:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:20:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:20:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:20:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:20:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:20:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:20:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:20:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:20:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:20:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:20:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:20:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:20:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:20:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:20:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:20:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:20:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:20:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:20:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:20:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:20:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:20:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:20:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:20:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:20:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:20:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:20:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:20:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:20:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:20:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:20:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:20:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:20:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:20:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:20:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:20:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:20:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:20:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:20:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:20:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:20:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:20:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:20:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:20:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:20:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:20:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:20:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:20:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:20:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:20:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:20:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:20:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:20:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:20:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:20:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:20:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:20:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:20:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:20:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:20:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:20:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:20:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:20:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:20:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:20:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:20:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:20:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:20:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:20:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:20:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:20:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:20:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:20:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:20:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:20:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:20:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:20:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:20:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:20:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:20:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:20:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:20:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:20:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:20:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:20:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:20:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:20:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:20:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:20:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:20:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:20:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:20:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:20:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:20:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:20:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:20:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:20:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:20:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:20:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:20:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:20:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:20:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:20:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:20:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:20:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:20:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:20:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:20:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:20:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:20:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:20:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:20:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:20:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:20:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:20:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:20:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:20:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:20:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:20:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:20:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:20:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:20:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:20:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:20:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:20:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:20:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:20:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:20:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:20:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:20:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:20:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:20:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:20:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:20:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:20:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:20:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:20:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:20:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:20:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:20:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:20:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:20:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:20:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:20:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:20:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:20:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:20:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:20:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:20:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:20:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:20:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:20:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:20:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:20:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:20:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:20:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:20:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:20:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:20:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:21:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:21:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:21:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:21:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:21:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:21:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:21:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:21:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:21:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:21:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:21:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:21:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:21:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:21:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:21:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:21:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:21:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:21:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:21:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:21:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:21:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:21:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:21:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:21:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:21:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:21:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:21:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:21:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:21:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:21:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:21:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:21:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:21:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:21:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:21:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:21:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:21:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:21:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:21:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:21:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:21:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:21:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:21:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:21:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:21:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:21:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:21:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:21:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:21:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:21:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:21:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:21:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:21:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:21:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:21:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:21:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:21:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:21:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:21:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:21:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:21:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:21:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:21:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:21:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:21:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:21:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:21:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:21:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:21:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:21:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:21:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:21:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:21:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:21:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:21:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:21:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:21:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:21:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:21:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:21:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:21:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:21:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:21:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:21:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:21:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:21:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:21:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:21:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:21:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:21:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:21:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:21:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:21:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:21:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:21:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:21:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:21:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:21:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:21:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:21:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:21:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:21:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:21:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:21:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:21:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:21:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:21:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:21:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:21:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:21:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:21:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:21:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:21:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:21:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:21:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:21:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:21:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:21:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:21:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:21:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:21:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:21:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:21:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:21:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:21:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:21:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:21:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:21:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:21:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:21:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:21:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:21:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:21:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:21:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:21:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:21:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:21:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:21:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:21:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:21:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:21:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:21:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:21:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:21:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:21:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:21:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:21:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:21:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:21:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:21:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:21:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:21:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:21:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:21:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:21:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:21:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:21:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:21:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:21:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:21:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:21:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:21:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:21:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:21:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:21:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:21:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:21:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:21:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:21:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:21:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:21:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:21:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:21:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:21:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:21:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:21:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:21:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:21:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:21:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:21:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:21:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:21:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:21:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:21:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:21:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:21:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:21:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:21:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:21:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:21:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:21:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:21:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:21:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:21:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:21:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:21:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:21:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:21:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:21:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:21:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:21:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:21:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:21:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:21:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:21:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:21:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:22:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:22:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:22:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:22:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:22:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:22:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:22:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:22:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:22:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:22:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:22:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:22:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:22:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:22:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:22:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:22:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:22:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:22:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:22:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:22:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:22:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:22:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:22:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:22:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:22:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:22:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:22:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:22:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:22:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:22:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:22:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:22:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:22:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:22:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:22:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:22:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:22:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:22:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:22:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:22:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:22:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:22:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:22:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:22:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:22:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:22:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:22:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:22:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:22:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:22:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:22:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:22:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:22:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:22:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:22:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:22:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:22:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:22:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:22:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:22:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:22:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:22:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:22:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:22:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:22:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:22:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:22:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:22:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:22:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:22:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:22:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:22:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:22:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:22:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:22:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:22:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:22:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:22:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:22:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:22:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:22:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:22:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:22:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:22:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:22:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:22:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:22:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:22:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:22:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:22:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:22:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:22:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:22:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:22:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:22:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:22:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:22:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:22:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:22:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:22:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:22:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:22:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:22:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:22:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:22:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:22:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:22:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:22:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:22:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:22:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:22:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:22:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:22:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:22:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:22:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:22:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:22:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:22:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:22:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:22:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:22:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:22:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:22:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:22:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:22:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:22:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:22:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:22:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:22:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:22:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:22:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:22:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:22:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:22:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:22:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:22:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:22:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:22:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:22:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:22:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:22:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:22:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:22:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:22:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:22:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:22:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:22:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:22:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:22:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:22:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:22:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:22:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:22:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:22:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:22:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:22:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:22:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:22:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:22:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:22:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:22:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:22:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:22:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:22:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:22:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:22:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:22:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:22:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:22:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:22:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:22:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:22:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:22:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:22:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:22:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:22:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:22:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:22:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:22:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:22:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:22:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:22:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:22:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:22:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:22:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:22:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:22:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:22:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:22:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:22:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:22:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:22:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:22:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:22:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:22:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:22:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:23:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:23:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:23:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:23:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:23:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:23:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:23:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:23:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:23:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:23:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:23:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:23:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:23:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:23:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:23:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:23:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:23:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:23:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:23:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:23:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:23:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:23:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:23:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:23:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:23:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:23:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:23:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:23:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:23:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:23:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:23:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:23:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:23:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:23:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:23:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:23:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:23:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:23:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:23:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:23:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:23:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:23:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:23:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:23:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:23:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:23:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:23:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:23:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:23:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:23:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:23:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:23:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:23:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:23:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:23:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:23:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:23:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:23:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:23:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:23:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:23:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:23:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:23:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:23:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:23:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:23:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:23:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:23:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:23:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:23:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:23:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:23:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:23:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:23:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:23:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:23:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:23:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:23:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:23:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:23:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:23:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:23:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:23:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:23:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:23:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:23:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:23:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:23:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:23:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:23:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:23:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:23:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:23:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:23:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:23:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:23:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:23:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:23:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:23:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:23:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:23:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:23:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:23:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:23:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:23:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:23:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:23:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:23:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:23:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:23:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:23:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:23:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:23:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:23:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:23:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:23:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:23:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:23:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:23:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:23:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:23:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:23:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:23:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:23:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:23:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:23:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:23:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:23:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:23:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:23:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:23:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:23:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:23:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:23:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:23:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:23:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:23:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:23:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:23:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:23:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:23:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:23:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:23:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:23:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:23:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:23:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:23:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:23:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:23:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:23:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:23:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:23:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:23:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:23:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:23:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:23:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:23:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:23:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:23:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:23:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:23:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:23:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:23:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:23:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:23:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:23:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:23:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:23:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:23:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:23:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:23:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:23:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:23:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:23:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:23:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:23:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:23:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:23:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:23:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:23:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:23:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:23:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:23:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:23:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:23:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:23:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:23:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:23:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:23:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:23:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:23:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:23:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:23:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:23:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:23:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:23:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:23:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:23:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:23:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:23:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:23:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:23:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:23:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:23:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:23:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:23:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:23:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:23:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:23:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:23:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:23:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:23:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:23:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:23:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:23:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:23:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:26:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:26:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:26:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:26:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:26:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:26:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:26:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:26:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:26:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:26:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:26:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:26:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:26:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:26:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:26:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:26:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:26:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:26:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:26:08 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 11:26:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:26:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:26:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:26:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:26:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:26:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:26:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:26:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:26:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:26:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:26:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:26:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:26:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:26:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:26:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:26:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:26:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:26:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:26:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:26:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:26:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:26:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:26:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:26:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:26:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:26:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:26:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:26:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:26:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:26:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:26:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:26:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:26:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:26:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:26:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:26:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:26:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:26:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:26:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:26:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:26:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:26:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:26:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:26:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:26:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:26:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:26:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:26:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:26:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:26:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:26:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:26:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:26:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:26:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:26:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:26:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:26:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:26:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:26:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:26:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:26:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:26:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:26:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:26:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:26:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:26:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:26:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:26:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:26:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:26:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:26:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:26:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:26:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:26:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:26:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:26:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:26:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:26:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:26:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:26:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:26:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:26:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:26:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:26:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:26:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:26:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:26:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:26:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:26:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:26:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:26:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:26:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:26:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:26:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:26:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:26:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:26:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:26:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:26:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:26:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:26:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:26:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:26:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:26:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:26:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:26:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:26:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:26:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:26:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:26:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:26:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:26:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:26:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:26:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:26:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:26:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:26:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:26:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:26:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:26:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:26:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:26:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:26:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:26:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:26:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:26:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:26:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:26:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:26:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:26:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:26:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:26:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:26:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:26:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:26:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:26:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:26:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:26:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:26:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:26:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:26:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:26:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:26:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:26:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:26:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:26:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:26:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:26:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:26:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:26:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:26:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:26:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:26:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:26:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:26:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:26:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:26:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:26:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:26:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:26:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:26:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:26:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:26:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:26:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:26:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:26:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:26:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:26:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:26:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:26:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:26:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:26:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:26:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:26:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:26:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:26:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:26:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:26:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:26:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:26:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:27:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:27:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:27:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:27:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:27:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:27:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:27:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:27:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:27:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:27:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:27:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:27:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:27:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:27:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:27:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:27:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:27:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:27:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:27:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:27:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:27:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:27:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:27:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:27:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:27:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:27:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:27:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:27:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:27:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:27:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:27:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:27:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:27:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:27:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:27:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:27:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:27:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:27:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:27:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:27:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:27:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:27:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:27:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:27:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:27:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:27:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:27:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:27:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:27:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:27:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:27:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:27:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:27:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:27:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:27:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:27:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:27:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:27:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:27:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:27:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:27:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:27:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:27:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:27:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:27:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:27:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:27:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:27:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:27:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:27:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:27:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:27:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:27:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:27:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:27:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:27:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:27:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:27:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:27:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:27:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:27:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:27:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:27:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:27:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:27:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:27:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:27:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:27:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:27:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:27:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:27:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:27:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:27:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:27:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:27:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:27:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:27:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:27:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:27:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:27:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:27:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:27:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:27:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:27:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:27:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:27:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:27:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:27:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:27:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:27:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:27:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:27:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:27:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:27:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:27:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:27:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:27:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:27:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:27:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:27:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:27:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:27:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:27:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:27:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:27:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:27:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:27:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:27:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:27:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:27:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:27:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:27:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:27:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:27:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:27:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:27:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:27:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:27:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:27:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:27:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:27:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:27:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:27:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:27:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:27:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:27:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:27:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:27:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:27:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:27:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:27:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:27:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:27:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:27:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:27:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:27:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:27:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:27:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:27:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:27:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:27:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:27:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:27:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:27:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:27:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:27:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:27:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:27:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:27:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:27:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:27:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:27:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:27:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:27:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:27:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:27:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:27:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:27:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:27:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:27:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:27:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:27:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:27:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:27:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:27:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:27:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:27:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:27:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:27:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:27:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:27:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:27:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:27:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:27:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:27:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:27:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:27:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:27:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:27:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:27:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:27:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:27:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:27:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:27:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:27:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:27:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:27:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:27:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:27:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:27:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:27:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:27:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:27:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:27:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:27:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:27:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:28:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:28:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:28:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:28:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:28:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:28:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:28:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:28:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:28:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:28:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:28:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:28:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:28:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:28:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:28:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:28:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:28:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:28:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:28:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:28:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:28:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:28:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:28:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:28:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:28:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:28:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:28:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:28:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:28:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:28:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:28:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:28:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:28:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:28:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:28:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:28:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:28:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:28:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:28:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:28:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:28:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:28:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:28:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:28:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:28:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:28:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:28:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:28:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:28:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:28:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:28:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:28:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:28:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:28:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:28:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:28:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:28:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:28:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:28:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:28:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:28:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:28:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:28:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:28:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:28:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:28:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:28:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:28:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:28:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:28:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:28:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:28:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:28:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:28:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:28:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:28:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:28:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:28:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:28:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:28:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:28:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:28:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:28:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:28:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:28:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:28:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:28:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:28:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:28:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:28:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:28:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:28:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:28:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:28:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:28:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:28:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:28:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:28:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:28:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:28:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:28:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:28:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:28:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:28:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:28:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:28:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:28:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:28:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:28:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:28:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:28:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:28:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:28:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:28:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:28:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:28:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:28:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:28:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:28:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:28:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:28:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:28:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:28:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:28:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:28:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:28:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:28:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:28:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:28:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:28:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:28:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:28:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:28:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:28:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:28:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:28:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:28:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:28:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:28:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:28:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:28:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:28:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:28:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:28:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:28:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:28:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:28:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:28:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:28:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:28:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:28:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:28:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:28:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:28:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:28:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:28:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:28:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:28:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:28:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:28:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:28:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:28:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:28:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:28:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:28:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:28:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:28:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:28:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:28:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:28:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:28:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:28:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:28:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:28:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:28:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:28:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:28:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:28:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:28:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:28:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:28:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:28:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:28:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:28:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:28:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:28:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:28:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:28:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:28:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:28:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:28:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:28:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:28:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:28:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:28:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:28:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:28:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:28:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:28:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:28:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:28:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:28:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:28:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:28:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:28:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:28:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:28:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:28:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:28:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:28:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:28:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:28:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:28:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:28:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:28:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:28:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:29:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:29:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:29:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:29:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:29:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:29:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:29:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:29:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:29:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:29:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:29:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:29:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:29:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:29:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:29:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:29:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:29:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:29:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:29:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:29:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:29:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:29:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:29:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:29:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:29:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:29:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:29:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:29:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:29:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:29:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:29:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:29:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:29:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:29:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:29:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:29:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:29:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:29:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:29:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:29:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:29:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:29:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:29:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:29:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:29:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:29:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:29:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:29:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:29:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:29:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:29:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:29:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:29:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:29:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:29:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:29:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:29:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:29:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:29:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:29:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:29:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:29:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:29:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:29:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:29:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:29:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:29:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:29:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:29:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:29:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:29:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:29:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:29:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:29:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:29:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:29:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:29:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:29:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:29:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:29:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:29:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:29:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:29:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:29:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:29:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:29:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:29:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:29:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:29:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:29:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:29:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:29:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:29:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:29:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:29:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:29:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:29:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:29:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:29:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:29:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:29:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:29:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:29:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:29:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:29:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:29:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:29:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:29:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:29:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:29:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:29:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:29:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:29:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:29:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:29:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:29:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:29:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:29:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:29:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:29:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:29:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:29:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:29:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:29:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:29:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:29:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:29:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:29:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:29:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:29:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:29:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:29:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:29:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:29:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:29:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:29:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:29:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:29:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:29:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:29:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:29:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:29:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:29:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:29:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:29:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:29:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:29:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:29:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:29:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:29:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:29:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:29:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:29:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:29:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:29:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:29:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:29:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:29:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:29:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:29:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:29:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:29:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:29:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:29:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:29:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:29:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:29:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:29:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:29:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:29:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:29:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:29:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:29:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:29:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:29:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:29:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:29:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:29:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:29:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:29:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:29:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:29:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:29:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:29:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:29:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:29:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:29:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:29:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:29:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:29:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:29:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:29:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:29:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:29:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:29:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:29:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:29:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:29:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:29:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:29:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:29:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:29:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:29:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:29:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:29:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:29:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:29:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:29:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:29:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:29:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:29:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:29:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:29:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:29:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:29:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:29:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:30:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:30:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:30:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:30:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:30:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:30:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:30:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:30:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:30:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:30:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:30:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:30:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:30:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:30:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:30:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:30:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:30:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:30:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:30:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:30:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:30:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:30:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:30:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:30:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:30:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:30:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:30:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:30:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:30:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:30:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:30:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:30:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:30:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:30:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:30:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:30:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:30:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:30:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:30:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:30:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:30:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:30:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:30:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:30:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:30:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:30:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:30:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:30:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:30:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:30:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:30:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:30:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:30:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:30:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:30:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:30:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:30:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:30:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:30:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:30:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:30:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:30:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:30:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:30:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:30:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:30:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:30:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:30:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:30:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:30:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:30:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:30:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:30:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:30:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:30:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:30:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:30:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:30:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:30:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:30:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:30:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:30:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:30:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:30:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:30:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:30:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:30:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:30:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:30:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:30:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:30:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:30:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:30:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:30:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:30:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:30:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:30:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:30:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:30:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:30:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:30:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:30:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:30:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:30:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:30:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:30:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:30:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:30:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:30:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:30:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:30:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:30:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:30:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:30:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:30:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:30:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:31:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:31:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:31:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:31:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:31:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:31:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:31:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:31:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:31:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:31:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:31:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:31:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:31:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:31:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:31:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:31:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:31:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:31:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:31:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:31:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:31:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:31:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:31:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:31:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:31:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:31:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:31:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:31:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:31:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:31:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:31:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:31:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:31:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:31:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:31:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:31:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:31:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:31:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:31:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:31:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:31:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:31:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:31:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:31:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:31:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:31:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:31:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:31:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:31:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:31:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:31:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:31:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:31:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:31:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:31:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:31:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:31:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:31:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:31:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:31:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:31:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:31:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:31:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:31:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:31:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:31:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:31:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:31:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:31:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:31:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:31:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:31:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:31:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:31:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:31:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:31:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:31:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:31:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:31:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:31:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:31:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:31:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:31:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:31:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:31:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:31:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:31:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:31:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:31:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:31:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:31:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:31:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:31:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:31:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:31:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:31:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:31:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:31:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:31:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:31:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:31:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:31:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:31:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:31:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:31:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:31:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:31:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:31:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:31:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:31:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:31:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:31:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:31:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:31:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:31:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:31:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:31:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:31:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:31:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:31:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:31:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:31:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:31:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:31:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:31:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:31:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:31:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:31:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:31:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:31:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:31:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:31:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:31:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:31:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:31:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:31:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:31:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:31:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:31:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:31:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:31:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:31:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:31:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:31:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:31:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:31:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:31:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:31:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:31:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:31:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:31:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:31:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:31:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:31:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:31:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:31:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:31:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:31:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:31:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:31:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:31:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:31:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:31:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:31:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:31:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:31:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:31:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:31:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:31:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:31:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:31:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:31:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:31:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:31:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:31:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:31:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:31:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:31:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:31:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:31:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:31:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:31:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:31:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:31:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:31:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:31:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:31:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:31:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:31:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:31:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:31:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:31:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:31:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:31:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:31:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:31:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:31:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:31:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:31:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:31:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:31:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:31:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:31:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:31:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:31:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:31:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:31:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:31:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:31:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:31:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:31:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:31:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:31:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:31:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:31:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:31:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:32:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:32:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:32:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:32:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:32:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:32:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:32:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:32:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:32:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:32:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:32:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:32:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:32:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:32:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:32:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:32:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:32:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:32:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:32:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:32:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:32:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:32:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:32:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:32:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:32:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:32:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:32:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:32:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:32:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:32:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:32:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:32:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:32:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:32:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:32:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:32:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:32:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:32:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:32:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:32:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:32:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:32:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:32:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:32:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:32:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:32:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:32:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:32:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:32:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:32:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:32:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:32:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:32:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:32:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:32:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:32:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:32:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:32:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:32:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:32:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:32:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:32:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:32:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:32:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:32:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:32:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:32:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:32:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:32:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:32:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:32:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:32:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:32:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:32:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:32:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:32:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:32:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:32:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:32:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:32:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:32:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:32:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:32:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:32:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:32:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:32:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:32:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:32:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:32:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:32:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:32:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:32:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:32:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:32:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:32:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:32:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:32:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:32:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:32:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:32:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:32:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:32:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:32:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:32:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:32:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:32:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:32:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:32:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:32:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:32:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:32:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:32:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:32:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:32:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:32:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:32:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:32:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:32:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:32:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:32:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:32:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:32:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:32:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:32:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:32:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:32:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:32:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:32:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:32:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:32:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:32:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:32:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:32:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:32:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:32:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:32:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:32:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:32:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:32:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:32:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:32:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:32:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:32:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:32:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:32:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:32:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:32:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:32:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:32:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:32:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:32:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:32:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:32:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:32:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:32:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:32:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:32:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:32:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:32:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:32:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:32:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:32:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:32:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:32:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:32:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:32:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:32:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:32:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:32:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:32:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:32:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:32:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:32:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:32:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:32:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:32:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:32:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:32:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:32:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:32:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:32:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:32:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:32:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:32:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:32:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:32:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:32:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:32:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:32:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:32:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:32:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:32:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:32:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:32:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:32:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:32:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:32:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:32:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:32:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:32:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:32:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:32:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:32:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:32:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:32:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:32:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:32:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:32:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:32:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:32:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:32:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:32:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:32:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:32:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:32:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:32:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:33:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:33:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:33:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:33:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:33:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:33:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:33:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:33:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:33:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:33:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:33:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:33:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:33:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:33:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:33:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:33:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:33:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:33:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:33:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:33:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:33:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:33:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:33:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:33:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:33:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:33:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:33:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:33:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:33:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:33:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:33:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:33:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:33:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:33:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:33:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:33:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:33:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:33:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:33:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:33:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:33:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:33:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:33:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:33:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:33:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:33:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:33:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:33:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:33:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:33:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:33:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:33:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:33:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:33:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:33:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:33:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:33:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:33:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:33:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:33:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:33:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:33:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:33:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:33:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:33:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:33:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:33:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:33:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:33:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:33:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:33:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:33:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:33:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:33:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:33:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:33:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:33:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:33:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:33:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:33:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:33:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:33:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:33:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:33:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:33:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:33:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:33:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:33:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:33:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:33:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:33:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:33:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:33:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:33:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:33:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:33:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:33:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:33:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:33:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:33:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:33:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:33:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:33:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:33:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:33:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:33:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:33:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:33:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:33:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:33:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:33:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:33:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:33:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:33:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:33:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:33:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:33:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:33:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:33:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:33:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:33:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:33:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:33:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:33:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:33:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:33:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:33:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:33:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:33:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:33:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:33:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:33:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:33:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:33:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:33:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:33:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:33:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:33:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:33:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:33:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:33:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:33:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:33:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:33:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:33:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:33:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:33:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:33:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:33:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:33:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:33:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:33:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:33:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:33:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:33:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:33:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:33:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:33:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:33:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:33:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:33:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:33:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:33:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:33:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:33:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:33:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:33:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:33:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:33:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:33:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:33:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:33:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:33:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:33:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:33:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:33:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:33:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:33:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:33:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:33:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:33:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:33:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:33:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:33:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:33:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:33:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:33:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:33:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:33:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:33:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:33:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:33:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:33:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:33:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:33:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:33:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:33:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:33:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:33:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:33:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:33:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:33:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:33:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:33:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:33:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:33:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:33:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:33:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:33:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:33:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:33:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:33:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:33:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:33:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:33:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:33:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:34:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:34:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:34:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:34:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:34:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:34:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:34:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:34:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:34:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:34:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:34:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:34:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:34:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:34:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:34:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:34:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:34:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:34:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:36:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:36:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:36:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:36:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:36:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:36:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:36:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:36:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:36:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:36:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:36:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:36:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:36:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:36:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:36:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:36:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:36:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:36:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:36:13 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 11:36:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:36:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:36:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:36:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:36:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:36:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:36:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:36:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:36:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:36:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:36:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:36:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:36:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:36:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:36:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:36:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:36:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:36:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:36:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:36:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:36:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:36:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:36:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:36:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:36:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:36:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:36:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:36:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:36:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:36:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:36:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:36:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:36:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:36:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:36:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:36:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:36:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:36:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:36:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:36:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:36:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:36:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:36:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:36:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:36:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:36:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:36:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:36:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:36:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:36:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:36:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:36:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:36:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:36:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:36:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:36:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:36:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:36:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:36:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:36:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:36:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:36:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:36:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:36:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:36:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:36:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:36:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:36:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:36:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:36:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:36:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:36:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:36:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:36:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:36:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:36:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:36:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:36:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:36:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:36:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:36:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:36:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:36:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:36:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:36:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:36:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:36:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:36:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:36:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:36:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:36:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:36:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:36:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:36:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:36:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:36:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:36:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:36:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:36:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:36:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:36:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:36:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:36:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:36:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:36:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:36:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:36:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:36:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:36:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:36:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:36:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:36:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:36:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:36:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:36:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:36:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:36:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:36:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:36:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:36:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:36:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:36:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:36:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:36:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:36:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:36:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:36:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:36:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:36:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:36:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:36:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:36:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:36:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:36:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:36:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:36:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:36:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:36:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:36:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:36:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:36:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:36:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:36:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:36:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:36:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:36:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:36:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:36:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:36:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:36:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:36:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:36:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:36:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:36:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:36:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:36:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:36:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:36:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:36:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:36:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:36:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:36:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:37:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:37:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:37:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:37:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:37:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:37:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:37:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:37:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:37:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:37:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:37:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:37:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:37:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:37:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:37:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:37:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:37:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:37:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:37:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:37:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:37:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:37:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:37:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:37:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:37:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:37:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:37:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:37:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:37:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:37:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:37:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:37:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:37:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:37:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:37:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:37:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:37:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:37:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:37:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:37:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:37:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:37:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:37:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:37:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:37:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:37:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:37:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:37:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:37:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:37:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:37:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:37:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:37:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:37:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:37:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:37:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:37:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:37:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:37:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:37:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:37:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:37:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:37:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:37:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:37:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:37:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:37:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:37:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:37:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:37:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:37:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:37:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:37:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:37:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:37:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:37:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:37:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:37:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:37:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:37:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:37:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:37:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:37:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:37:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:37:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:37:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:37:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:37:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:37:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:37:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:37:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:37:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:37:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:37:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:37:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:37:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:37:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:37:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:37:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:37:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:37:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:37:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:37:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:37:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:37:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:37:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:37:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:37:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:37:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:37:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:37:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:37:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:37:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:37:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:37:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:37:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:37:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:37:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:37:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:37:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:37:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:37:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:37:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:37:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:37:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:37:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:37:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:37:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:37:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:37:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:37:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:37:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:37:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:37:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:37:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:37:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:37:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:37:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:37:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:37:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:37:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:37:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:37:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:37:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:37:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:37:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:37:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:37:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:37:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:37:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:37:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:37:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:37:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:37:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:37:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:37:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:37:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:37:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:37:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:37:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:37:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:37:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:37:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:37:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:37:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:37:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:37:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:37:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:37:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:37:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:37:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:37:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:37:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:37:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:37:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:37:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:37:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:37:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:37:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:37:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:37:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:37:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:37:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:37:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:37:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:37:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:37:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:37:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:37:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:37:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:37:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:37:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:37:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:37:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:37:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:37:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:37:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:37:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:37:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:37:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:37:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:37:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:37:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:37:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:37:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:37:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:37:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:37:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:37:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:37:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:37:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:37:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:37:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:37:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:37:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:37:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:38:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:38:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:38:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:38:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:38:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:38:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:38:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:38:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:38:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:38:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:38:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:38:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:38:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:38:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:38:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:38:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:38:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:38:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:38:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:38:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:38:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:38:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:38:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:38:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:38:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:38:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:38:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:38:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:38:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:38:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:38:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:38:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:38:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:38:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:38:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:38:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:38:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:38:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:38:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:38:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:38:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:38:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:38:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:38:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:38:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:38:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:38:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:38:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:38:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:38:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:38:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:38:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:38:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:38:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:38:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:38:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:38:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:38:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:38:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:38:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:38:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:38:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:38:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:38:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:38:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:38:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:38:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:38:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:38:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:38:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:38:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:38:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:38:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:38:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:38:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:38:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:38:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:38:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:38:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:38:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:38:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:38:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:38:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:38:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:38:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:38:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:38:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:38:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:38:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:38:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:38:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:38:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:38:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:38:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:38:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:38:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:38:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:38:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:38:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:38:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:38:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:38:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:38:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:38:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:38:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:38:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:38:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:38:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:38:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:38:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:38:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:38:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:38:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:38:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:38:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:38:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:38:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:38:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:38:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:38:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:38:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:38:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:38:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:38:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:38:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:38:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:38:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:38:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:38:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:38:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:38:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:38:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:38:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:38:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:38:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:38:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:38:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:38:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:38:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:38:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:38:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:38:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:38:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:38:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:38:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:38:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:38:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:38:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:38:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:38:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:38:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:38:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:38:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:38:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:38:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:38:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:38:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:38:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:38:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:38:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:38:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:38:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:38:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:38:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:38:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:38:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:38:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:38:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:38:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:38:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:38:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:38:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:38:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:38:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:38:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:38:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:38:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:38:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:38:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:38:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:38:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:38:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:38:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:38:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:38:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:38:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:38:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:38:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:38:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:38:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:38:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:38:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:38:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:38:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:38:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:38:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:38:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:38:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:38:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:38:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:38:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:38:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:38:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:38:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:38:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:38:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:38:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:38:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:38:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:38:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:38:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:38:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:38:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:38:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:38:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:38:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:39:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:39:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:39:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:39:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:39:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:39:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:39:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:39:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:39:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:39:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:39:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:39:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:39:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:39:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:39:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:39:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:39:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:39:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:39:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:39:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:39:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:39:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:39:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:39:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:39:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:39:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:39:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:39:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:39:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:39:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:39:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:39:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:39:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:39:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:39:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:39:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:39:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:39:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:39:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:39:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:39:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:39:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:39:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:39:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:39:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:39:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:39:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:39:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:39:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:39:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:39:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:39:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:39:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:39:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:39:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:39:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:39:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:39:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:39:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:39:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:39:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:39:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:39:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:39:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:39:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:39:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:39:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:39:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:39:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:39:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:39:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:39:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:39:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:39:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:39:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:39:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:39:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:39:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:39:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:39:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:39:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:39:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:39:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:39:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:39:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:39:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:39:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:39:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:39:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:39:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:39:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:39:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:39:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:39:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:39:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:39:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:39:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:39:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:39:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:39:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:39:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:39:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:39:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:39:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:39:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:39:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:39:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:39:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:39:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:39:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:39:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:39:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:39:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:39:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:39:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:39:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:39:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:39:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:39:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:39:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:39:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:39:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:39:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:39:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:39:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:39:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:39:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:39:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:39:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:39:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:39:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:39:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:39:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:39:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:39:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:39:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:39:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:39:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:39:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:39:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:39:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:39:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:39:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:39:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:39:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:39:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:39:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:39:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:39:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:39:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:39:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:39:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:39:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:39:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:39:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:39:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:39:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:39:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:39:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:39:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:39:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:39:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:39:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:39:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:39:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:39:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:39:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:39:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:39:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:39:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:39:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:39:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:39:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:39:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:39:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:39:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:39:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:39:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:39:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:39:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:39:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:39:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:39:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:39:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:39:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:39:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:39:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:39:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:39:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:39:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:39:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:39:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:39:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:39:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:39:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:39:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:39:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:39:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:39:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:39:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:39:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:39:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:39:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:39:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:39:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:39:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:39:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:39:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:39:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:39:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:39:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:39:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:39:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:39:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:39:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:39:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:40:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:40:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:40:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:40:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:40:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:40:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:40:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:40:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:40:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:40:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:40:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:40:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:40:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:40:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:40:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:40:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:40:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:40:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:40:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:40:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:40:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:40:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:40:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:40:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:40:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:40:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:40:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:40:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:40:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:40:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:40:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:40:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:40:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:40:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:40:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:40:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:40:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:40:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:40:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:40:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:40:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:40:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:40:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:40:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:40:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:40:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:40:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:40:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:40:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:40:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:40:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:40:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:40:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:40:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:40:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:40:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:40:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:40:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:40:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:40:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:40:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:40:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:40:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:40:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:40:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:40:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:40:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:40:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:40:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:40:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:40:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:40:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:40:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:40:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:40:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:40:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:40:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:40:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:40:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:40:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:40:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:40:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:40:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:40:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:40:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:40:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:40:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:40:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:40:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:40:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:40:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:40:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:40:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:40:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:40:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:40:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:40:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:40:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:40:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:40:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:40:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:40:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:40:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:40:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:40:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:40:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:40:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:40:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:40:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:40:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:40:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:40:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:40:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:40:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:40:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:40:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:40:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:40:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:40:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:40:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:40:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:40:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:40:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:40:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:40:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:40:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:40:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:40:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:40:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:40:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:40:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:40:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:40:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:40:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:40:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:40:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:40:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:40:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:40:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:40:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:40:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:40:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:40:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:40:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:40:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:40:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:40:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:40:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:40:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:40:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:40:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:40:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:40:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:40:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:40:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:40:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:40:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:40:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:40:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:40:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:40:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:40:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:40:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:40:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:40:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:40:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:40:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:40:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:40:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:40:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:40:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:40:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:40:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:40:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:40:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:40:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:40:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:40:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:40:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:40:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:40:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:40:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:40:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:40:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:40:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:40:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:40:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:40:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:40:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:40:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:40:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:40:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:40:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:40:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:40:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:40:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:40:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:40:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:40:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:40:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:40:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:40:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:40:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:40:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:40:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:40:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:40:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:40:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:40:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:40:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:40:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:40:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:40:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:40:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:40:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:40:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:41:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:41:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:41:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:41:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:41:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:41:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:41:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:41:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:41:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:41:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:41:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:41:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:41:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:41:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:41:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:41:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:41:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:41:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:41:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:41:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:41:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:41:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:41:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:41:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:41:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:41:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:41:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:41:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:41:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:41:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:41:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:41:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:41:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:41:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:41:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:41:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:41:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:41:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:41:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:41:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:41:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:41:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:41:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:41:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:41:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:41:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:41:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:41:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:41:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:41:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:41:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:41:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:41:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:41:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:41:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:41:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:41:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:41:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:41:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:41:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:41:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:41:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:41:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:41:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:41:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:41:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:41:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:41:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:41:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:41:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:41:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:41:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:41:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:41:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:41:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:41:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:41:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:41:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:41:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:41:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:41:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:41:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:41:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:41:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:41:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:41:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:41:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:41:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:41:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:41:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:41:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:41:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:41:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:41:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:41:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:41:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:41:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:41:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:41:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:41:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:41:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:41:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:41:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:41:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:41:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:41:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:41:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:41:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:41:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:41:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:41:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:41:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:41:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:41:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:41:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:41:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:41:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:41:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:41:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:41:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:41:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:41:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:41:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:41:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:41:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:41:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:41:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:41:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:41:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:41:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:41:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:41:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:41:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:41:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:41:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:41:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:41:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:41:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:41:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:41:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:41:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:41:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:41:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:41:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:41:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:41:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:41:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:41:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:41:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:41:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:41:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:41:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:41:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:41:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:41:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:41:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:41:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:41:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:41:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:41:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:41:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:41:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:41:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:41:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:41:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:41:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:41:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:41:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:41:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:41:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:41:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:41:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:41:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:41:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:41:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:41:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:41:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:41:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:41:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:41:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:41:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:41:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:41:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:41:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:41:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:41:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:41:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:41:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:41:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:41:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:41:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:41:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:41:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:41:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:41:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:41:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:41:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:41:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:41:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:41:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:41:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:41:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:41:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:41:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:41:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:41:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:41:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:41:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:41:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:41:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:41:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:41:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:41:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:41:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:41:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:41:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:42:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:42:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:42:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:42:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:42:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:42:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:42:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:42:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:42:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:42:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:42:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:42:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:42:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:42:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:42:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:42:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:42:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:42:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:42:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:42:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:42:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:42:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:42:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:42:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:42:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:42:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:42:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:42:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:42:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:42:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:42:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:42:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:42:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:42:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:42:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:42:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:42:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:42:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:42:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:42:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:42:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:42:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:42:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:42:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:42:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:42:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:42:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:42:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:42:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:42:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:42:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:42:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:42:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:42:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:42:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:42:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:42:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:42:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:42:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:42:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:42:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:42:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:42:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:42:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:42:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:42:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:42:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:42:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:42:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:42:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:42:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:42:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:42:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:42:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:42:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:42:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:42:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:42:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:42:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:42:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:42:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:42:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:42:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:42:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:42:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:42:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:42:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:42:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:42:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:42:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:42:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:42:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:42:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:42:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:42:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:42:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:42:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:42:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:42:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:42:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:42:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:42:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:42:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:42:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:42:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:42:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:42:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:42:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:42:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:42:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:42:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:42:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:42:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:42:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:42:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:42:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:42:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:42:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:42:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:42:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:42:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:42:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:42:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:42:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:42:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:42:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:42:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:42:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:42:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:42:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:42:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:42:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:42:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:42:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:42:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:42:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:42:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:42:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:42:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:42:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:42:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:42:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:42:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:42:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:42:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:42:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:42:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:42:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:42:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:42:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:42:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:42:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:42:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:42:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:42:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:42:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:42:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:42:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:42:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:42:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:42:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:42:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:42:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:42:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:42:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:42:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:42:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:42:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:42:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:42:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:42:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:42:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:42:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:42:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:42:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:42:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:42:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:42:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:42:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:42:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:42:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:42:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:42:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:42:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:42:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:42:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:42:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:42:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:42:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:42:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:42:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:42:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:42:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:42:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:42:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:42:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:42:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:42:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:42:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:42:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:42:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:42:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:42:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:42:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:42:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:42:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:42:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:42:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:42:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:42:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:42:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:42:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:42:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:42:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:42:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:42:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:43:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:43:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:43:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:43:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:43:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:43:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:43:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:43:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:43:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:43:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:43:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:43:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:43:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:43:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:43:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:43:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:43:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:43:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:43:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:43:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:43:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:43:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:43:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:43:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:43:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:43:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:43:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:43:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:43:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:43:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:43:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:43:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:43:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:43:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:43:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:43:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:43:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:43:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:43:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:43:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:43:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:43:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:43:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:43:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:43:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:43:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:43:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:43:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:43:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:43:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:43:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:43:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:43:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:43:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:43:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:43:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:43:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:43:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:43:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:43:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:43:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:43:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:43:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:43:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:43:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:43:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:43:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:43:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:43:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:43:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:43:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:43:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:43:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:43:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:43:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:43:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:43:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:43:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:43:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:43:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:43:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:43:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:43:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:43:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:43:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:43:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:43:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:43:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:43:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:43:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:43:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:43:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:43:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:43:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:43:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:43:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:43:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:43:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:43:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:43:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:43:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:43:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:43:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:43:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:43:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:43:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:43:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:43:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:43:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:43:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:43:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:43:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:43:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:43:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:43:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:43:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:43:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:43:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:43:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:43:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:43:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:43:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:43:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:43:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:43:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:43:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:43:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:43:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:43:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:43:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:43:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:43:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:43:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:43:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:43:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:43:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:43:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:43:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:43:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:43:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:43:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:43:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:43:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:43:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:43:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:43:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:43:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:43:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:43:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:43:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:43:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:43:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:43:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:43:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:43:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:43:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:43:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:43:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:43:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:43:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:43:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:43:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:43:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:43:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:43:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:43:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:43:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:43:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:43:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:43:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:43:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:43:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:43:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:43:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:43:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:43:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:43:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:43:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:43:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:43:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:43:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:43:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:43:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:43:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:43:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:43:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:43:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:43:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:43:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:43:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:43:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:43:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:43:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:43:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:43:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:43:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:43:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:43:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:43:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:43:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:43:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:43:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:43:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:43:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:43:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:43:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:43:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:43:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:43:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:43:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:43:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:43:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:43:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:43:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:43:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:43:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:44:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:44:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:44:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:44:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:44:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:44:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:44:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:44:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:44:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:44:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:44:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:44:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:44:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:44:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:44:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:44:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:44:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:44:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:44:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:44:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:44:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:44:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:44:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:44:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:44:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:44:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:44:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:44:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:44:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:44:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:44:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:44:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:44:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:44:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:44:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:44:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:46:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:46:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:46:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:46:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:46:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:46:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:46:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:46:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:46:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:46:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:46:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:46:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:46:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:46:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:46:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:46:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:46:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:46:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:46:14 rdma06 rsyslogd[1041]: imjournal: 5082 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 11:46:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:46:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:46:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:46:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:46:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:46:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:46:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:46:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:46:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:46:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:46:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:46:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:46:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:46:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:46:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:46:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:46:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:46:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:46:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:46:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:46:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:46:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:46:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:46:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:46:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:46:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:46:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:46:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:46:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:46:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:46:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:46:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:46:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:46:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:46:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:46:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:46:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:46:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:46:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:46:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:46:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:46:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:46:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:46:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:46:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:46:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:46:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:46:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:46:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:46:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:46:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:46:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:46:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:46:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:46:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:46:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:46:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:46:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:46:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:46:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:46:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:46:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:46:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:46:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:46:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:46:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:46:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:46:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:46:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:46:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:46:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:46:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:46:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:46:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:46:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:46:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:46:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:46:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:46:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:46:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:46:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:46:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:46:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:46:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:46:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:46:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:46:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:46:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:46:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:46:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:46:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:46:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:46:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:46:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:46:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:46:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:46:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:46:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:46:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:46:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:46:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:46:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:46:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:46:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:46:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:46:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:46:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:46:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:46:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:46:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:46:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:46:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:46:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:46:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:46:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:46:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:46:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:46:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:46:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:46:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:46:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:46:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:46:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:46:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:46:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:46:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:46:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:46:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:46:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:46:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:46:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:46:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:46:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:46:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:46:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:46:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:46:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:46:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:46:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:46:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:46:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:46:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:46:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:46:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:46:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:46:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:46:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:46:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:46:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:46:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:46:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:46:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:46:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:46:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:46:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:46:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:46:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:46:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:46:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:46:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:46:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:46:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:47:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:47:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:47:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:47:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:47:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:47:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:47:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:47:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:47:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:47:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:47:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:47:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:47:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:47:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:47:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:47:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:47:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:47:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:47:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:47:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:47:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:47:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:47:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:47:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:47:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:47:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:47:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:47:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:47:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:47:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:47:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:47:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:47:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:47:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:47:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:47:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:47:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:47:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:47:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:47:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:47:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:47:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:47:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:47:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:47:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:47:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:47:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:47:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:47:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:47:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:47:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:47:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:47:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:47:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:47:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:47:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:47:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:47:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:47:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:47:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:47:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:47:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:47:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:47:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:47:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:47:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:47:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:47:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:47:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:47:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:47:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:47:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:47:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:47:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:47:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:47:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:47:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:47:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:47:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:47:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:47:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:47:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:47:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:47:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:47:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:47:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:47:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:47:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:47:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:47:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:47:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:47:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:47:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:47:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:47:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:47:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:47:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:47:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:47:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:47:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:47:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:47:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:47:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:47:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:47:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:47:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:47:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:47:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:47:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:47:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:47:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:47:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:47:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:47:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:47:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:47:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:47:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:47:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:47:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:47:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:47:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:47:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:47:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:47:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:47:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:47:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:47:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:47:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:47:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:47:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:47:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:47:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:47:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:47:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:47:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:47:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:47:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:47:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:47:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:47:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:47:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:47:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:47:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:47:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:47:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:47:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:47:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:47:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:47:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:47:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:47:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:47:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:47:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:47:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:47:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:47:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:47:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:47:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:47:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:47:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:47:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:47:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:47:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:47:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:47:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:47:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:47:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:47:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:47:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:47:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:47:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:47:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:47:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:47:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:47:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:47:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:47:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:47:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:47:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:47:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:47:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:47:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:47:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:47:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:47:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:47:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:47:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:47:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:47:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:47:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:47:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:47:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:47:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:47:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:47:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:47:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:47:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:47:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:47:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:47:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:47:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:47:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:47:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:47:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:47:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:47:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:47:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:47:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:47:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:47:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:47:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:47:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:47:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:47:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:47:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:47:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:48:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:48:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:48:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:48:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:48:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:48:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:48:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:48:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:48:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:48:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:48:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:48:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:48:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:48:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:48:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:48:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:48:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:48:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:48:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:48:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:48:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:48:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:48:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:48:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:48:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:48:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:48:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:48:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:48:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:48:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:48:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:48:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:48:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:48:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:48:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:48:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:48:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:48:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:48:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:48:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:48:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:48:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:48:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:48:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:48:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:48:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:48:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:48:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:48:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:48:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:48:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:48:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:48:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:48:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:48:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:48:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:48:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:48:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:48:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:48:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:48:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:48:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:48:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:48:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:48:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:48:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:48:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:48:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:48:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:48:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:48:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:48:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:48:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:48:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:48:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:48:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:48:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:48:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:48:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:48:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:48:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:48:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:48:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:48:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:48:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:48:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:48:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:48:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:48:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:48:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:48:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:48:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:48:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:48:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:48:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:48:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:48:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:48:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:48:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:48:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:48:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:48:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:48:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:48:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:48:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:48:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:48:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:48:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:48:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:48:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:48:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:48:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:48:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:48:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:48:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:48:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:48:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:48:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:48:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:48:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:48:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:48:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:48:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:48:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:48:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:48:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:48:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:48:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:48:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:48:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:48:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:48:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:48:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:48:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:48:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:48:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:48:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:48:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:48:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:48:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:48:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:48:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:48:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:48:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:48:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:48:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:48:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:48:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:48:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:48:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:48:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:48:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:48:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:48:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:48:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:48:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:48:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:48:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:48:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:48:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:48:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:48:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:48:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:48:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:48:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:48:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:48:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:48:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:48:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:48:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:48:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:48:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:48:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:48:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:48:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:48:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:48:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:48:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:48:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:48:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:48:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:48:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:48:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:48:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:48:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:48:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:48:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:48:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:48:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:48:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:48:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:48:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:48:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:48:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:48:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:48:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:48:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:48:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:48:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:48:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:48:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:48:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:48:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:48:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:48:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:48:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:48:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:48:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:48:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:48:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:48:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:48:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:48:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:48:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:48:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:48:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:49:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:49:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:49:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:49:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:49:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:49:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:49:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:49:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:49:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:49:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:49:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:49:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:49:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:49:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:49:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:49:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:49:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:49:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:49:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:49:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:49:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:49:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:49:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:49:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:49:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:49:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:49:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:49:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:49:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:49:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:49:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:49:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:49:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:49:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:49:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:49:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:49:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:49:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:49:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:49:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:49:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:49:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:49:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:49:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:49:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:49:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:49:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:49:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:49:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:49:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:49:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:49:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:49:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:49:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:49:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:49:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:49:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:49:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:49:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:49:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:49:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:49:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:49:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:49:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:49:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:49:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:49:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:49:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:49:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:49:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:49:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:49:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:49:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:49:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:49:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:49:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:49:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:49:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:49:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:49:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:49:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:49:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:49:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:49:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:49:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:49:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:49:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:49:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:49:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:49:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:49:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:49:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:49:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:49:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:49:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:49:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:49:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:49:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:49:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:49:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:49:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:49:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:49:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:49:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:49:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:49:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:49:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:49:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:49:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:49:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:49:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:49:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:49:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:49:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:49:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:49:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:49:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:49:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:49:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:49:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:49:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:49:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:49:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:49:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:49:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:49:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:49:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:49:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:49:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:49:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:49:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:49:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:49:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:49:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:49:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:49:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:49:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:49:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:49:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:49:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:49:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:49:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:49:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:49:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:49:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:49:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:49:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:49:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:49:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:49:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:49:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:49:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:49:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:49:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:49:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:49:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:49:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:49:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:49:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:49:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:49:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:49:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:49:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:49:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:49:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:49:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:49:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:49:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:49:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:49:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:49:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:49:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:49:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:49:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:49:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:49:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:49:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:49:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:49:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:49:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:49:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:49:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:49:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:49:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:49:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:49:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:49:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:49:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:49:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:49:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:49:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:49:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:49:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:49:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:49:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:49:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:49:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:49:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:49:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:49:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:49:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:49:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:49:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:49:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:49:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:49:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:49:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:49:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:49:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:49:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:49:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:49:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:49:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:49:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:49:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:49:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:50:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:50:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:50:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:50:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:50:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:50:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:50:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:50:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:50:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:50:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:50:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:50:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:50:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:50:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:50:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:50:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:50:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:50:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:50:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:50:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:50:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:50:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:50:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:50:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:50:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:50:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:50:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:50:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:50:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:50:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:50:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:50:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:50:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:50:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:50:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:50:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:50:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:50:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:50:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:50:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:50:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:50:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:50:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:50:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:50:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:50:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:50:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:50:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:50:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:50:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:50:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:50:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:50:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:50:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:50:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:50:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:50:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:50:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:50:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:50:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:50:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:50:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:50:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:50:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:50:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:50:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:50:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:50:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:50:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:50:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:50:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:50:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:50:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:50:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:50:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:50:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:50:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:50:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:50:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:50:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:50:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:50:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:50:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:50:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:50:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:50:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:50:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:50:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:50:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:50:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:50:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:50:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:50:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:50:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:50:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:50:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:50:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:50:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:50:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:50:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:50:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:50:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:50:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:50:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:50:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:50:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:50:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:50:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:50:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:50:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:50:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:50:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:50:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:50:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:50:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:50:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:50:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:50:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:50:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:50:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:50:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:50:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:50:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:50:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:50:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:50:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:50:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:50:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:50:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:50:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:50:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:50:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:50:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:50:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:50:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:50:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:50:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:50:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:50:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:50:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:50:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:50:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:50:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:50:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:50:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:50:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:50:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:50:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:50:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:50:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:50:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:50:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:50:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:50:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:50:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:50:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:50:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:50:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:50:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:50:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:50:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:50:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:50:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:50:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:50:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:50:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:50:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:50:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:50:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:50:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:50:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:50:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:50:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:50:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:50:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:50:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:50:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:50:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:50:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:50:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:50:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:50:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:50:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:50:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:50:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:50:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:50:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:50:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:50:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:50:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:50:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:50:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:50:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:50:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:50:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:50:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:50:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:50:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:50:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:50:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:50:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:50:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:50:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:50:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:50:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:50:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:50:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:50:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:50:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:50:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:50:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:50:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:50:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:50:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:50:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:50:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:51:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:51:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:51:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:51:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:51:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:51:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:51:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:51:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:51:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:51:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:51:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:51:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:51:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:51:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:51:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:51:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:51:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:51:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:51:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:51:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:51:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:51:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:51:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:51:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:51:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:51:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:51:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:51:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:51:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:51:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:51:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:51:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:51:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:51:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:51:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:51:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:51:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:51:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:51:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:51:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:51:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:51:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:51:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:51:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:51:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:51:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:51:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:51:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:51:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:51:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:51:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:51:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:51:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:51:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:51:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:51:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:51:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:51:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:51:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:51:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:51:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:51:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:51:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:51:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:51:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:51:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:51:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:51:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:51:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:51:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:51:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:51:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:51:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:51:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:51:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:51:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:51:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:51:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:51:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:51:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:51:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:51:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:51:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:51:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:51:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:51:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:51:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:51:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:51:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:51:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:51:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:51:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:51:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:51:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:51:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:51:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:51:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:51:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:51:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:51:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:51:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:51:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:51:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:51:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:51:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:51:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:51:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:51:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:51:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:51:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:51:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:51:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:51:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:51:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:51:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:51:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:51:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:51:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:51:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:51:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:51:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:51:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:51:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:51:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:51:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:51:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:51:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:51:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:51:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:51:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:51:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:51:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:51:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:51:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:51:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:51:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:51:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:51:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:51:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:51:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:51:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:51:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:51:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:51:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:51:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:51:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:52:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:52:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:52:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:52:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:52:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:52:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:52:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:52:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:52:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:52:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:52:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:52:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:52:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:52:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:52:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:52:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:52:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:52:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:52:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:52:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:52:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:52:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:52:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:52:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:52:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:52:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:52:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:52:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:52:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:52:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:52:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:52:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:52:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:52:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:52:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:52:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:52:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:52:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:52:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:52:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:52:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:52:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:52:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:52:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:52:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:52:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:52:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:52:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:52:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:52:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:52:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:52:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:52:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:52:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:52:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:52:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:52:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:52:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:52:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:52:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:52:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:52:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:52:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:52:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:52:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:52:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:52:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:52:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:52:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:52:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:52:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:52:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:52:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:52:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:52:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:52:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:52:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:52:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:52:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:52:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:52:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:52:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:52:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:52:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:52:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:52:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:52:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:52:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:52:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:52:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:52:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:52:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:52:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:52:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:52:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:52:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:52:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:52:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:52:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:52:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:52:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:52:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:52:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:52:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:52:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:52:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:52:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:52:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:52:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:52:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:52:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:52:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:52:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:52:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:52:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:52:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:52:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:52:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:52:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:52:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:52:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:52:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:52:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:52:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:52:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:52:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:52:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:52:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:52:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:52:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:52:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:52:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:52:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:52:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:52:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:52:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:52:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:52:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:52:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:52:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:52:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:52:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:52:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:52:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:52:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:52:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:52:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:52:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:52:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:52:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:52:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:52:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:52:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:52:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:52:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:52:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:52:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:52:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:52:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:52:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:52:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:52:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:52:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:52:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:52:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:52:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:52:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:52:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:52:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:52:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:52:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:52:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:52:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:52:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:52:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:52:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:53:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:53:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:53:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:53:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:53:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:53:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:53:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:53:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:53:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:53:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:53:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:53:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:53:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:53:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:53:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:53:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:53:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:53:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:53:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:53:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:53:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:53:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:53:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:53:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:53:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:53:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:53:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:53:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:53:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:53:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:53:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:53:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:53:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:53:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:53:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:53:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:53:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:53:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:53:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:53:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:53:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:53:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:53:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:53:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:53:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:53:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:53:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:53:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:53:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:53:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:53:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:53:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:53:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:53:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:53:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:53:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:53:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:53:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:53:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:53:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:53:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:53:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:53:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:53:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:53:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:53:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:53:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:53:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:53:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:53:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:53:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:53:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:53:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:53:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:53:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:53:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:53:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:53:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:53:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:53:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:53:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:53:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:53:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:53:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:53:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:53:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:53:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:53:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:53:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:53:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:53:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:53:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:53:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:53:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:53:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:53:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:53:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:53:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:53:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:53:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:53:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:53:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:53:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:53:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:53:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:53:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:53:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:53:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:53:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:53:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:53:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:53:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:53:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:53:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:53:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:53:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:53:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:53:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:53:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:53:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:53:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:53:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:53:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:53:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:53:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:53:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:53:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:53:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:53:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:53:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:53:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:53:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:53:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:53:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:53:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:53:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:53:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:53:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:53:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:53:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:53:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:53:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:53:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:53:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:53:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:53:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:53:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:53:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:53:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:53:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:53:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:53:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:53:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:53:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:53:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:53:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:53:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:53:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:53:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:53:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:53:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:53:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:53:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:53:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:53:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:53:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:53:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:53:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:53:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:53:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:53:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:53:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:53:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:53:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:53:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:53:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:53:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:53:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:53:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:53:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:53:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:53:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:53:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:53:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:53:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:53:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:53:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:53:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:53:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:53:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:53:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:53:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:53:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:53:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:53:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:53:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:53:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:53:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:53:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:53:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:53:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:53:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:53:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:53:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:53:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:53:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:53:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:53:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:53:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:53:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:53:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:53:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:53:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:53:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:53:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:53:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:54:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:54:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:54:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:54:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:54:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:54:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:54:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:54:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:54:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:54:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:54:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:54:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:54:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:54:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:54:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:54:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:54:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:54:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:54:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:54:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:54:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:54:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:54:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:54:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:54:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:54:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:54:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:54:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:54:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:54:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:54:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:54:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:54:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:54:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:54:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:54:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:56:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:56:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:56:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:56:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:56:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:56:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:56:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:56:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:56:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:56:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:56:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:56:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:56:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:56:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:56:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:56:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:56:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:56:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:56:19 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 11:56:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:56:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:56:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:56:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:56:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:56:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:56:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:56:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:56:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:56:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:56:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:56:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:56:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:56:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:56:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:56:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:56:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:56:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:56:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:56:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:56:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:56:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:56:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:56:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:56:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:56:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:56:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:56:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:56:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:56:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:56:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:56:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:56:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:56:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:56:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:56:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:56:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:56:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:56:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:56:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:56:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:56:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:56:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:56:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:56:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:56:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:56:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:56:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:56:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:56:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:56:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:56:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:56:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:56:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:56:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:56:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:56:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:56:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:56:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:56:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:56:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:56:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:56:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:56:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:56:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:56:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:56:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:56:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:56:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:56:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:56:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:56:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:56:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:56:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:56:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:56:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:56:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:56:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:56:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:56:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:56:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:56:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:56:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:56:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:56:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:56:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:56:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:56:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:56:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:56:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:56:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:56:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:56:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:56:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:56:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:56:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:56:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:56:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:56:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:56:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:56:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:56:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:56:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:56:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:56:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:56:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:56:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:56:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:56:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:56:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:56:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:56:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:56:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:56:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:56:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:56:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:56:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:56:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:56:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:56:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:56:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:56:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:56:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:56:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:56:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:56:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:56:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:56:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:56:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:56:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:56:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:56:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:56:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:56:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:56:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:56:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:56:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:56:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:56:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:56:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:56:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:56:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:56:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:56:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:57:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:57:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:57:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:57:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:57:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:57:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:57:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:57:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:57:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:57:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:57:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:57:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:57:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:57:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:57:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:57:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:57:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:57:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:57:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:57:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:57:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:57:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:57:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:57:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:57:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:57:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:57:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:57:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:57:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:57:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:57:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:57:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:57:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:57:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:57:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:57:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:57:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:57:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:57:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:57:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:57:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:57:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:57:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:57:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:57:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:57:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:57:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:57:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:57:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:57:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:57:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:57:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:57:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:57:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:57:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:57:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:57:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:57:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:57:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:57:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:57:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:57:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:57:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:57:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:57:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:57:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:57:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:57:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:57:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:57:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:57:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:57:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:57:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:57:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:57:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:57:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:57:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:57:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:57:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:57:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:57:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:57:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:57:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:57:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:57:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:57:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:57:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:57:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:57:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:57:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:57:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:57:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:57:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:57:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:57:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:57:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:57:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:57:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:57:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:57:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:57:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:57:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:57:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:57:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:57:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:57:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:57:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:57:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:57:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:57:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:57:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:57:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:57:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:57:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:57:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:57:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:57:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:57:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:57:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:57:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:57:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:57:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:57:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:57:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:57:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:57:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:57:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:57:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:57:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:57:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:57:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:57:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:57:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:57:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:57:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:57:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:57:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:57:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:57:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:57:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:57:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:57:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:57:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:57:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:57:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:57:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:57:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:57:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:57:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:57:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:57:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:57:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:57:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:57:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:57:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:57:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:57:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:57:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:57:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:57:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:57:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:57:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:57:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:57:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:57:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:57:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:57:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:57:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:57:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:57:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:57:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:57:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:57:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:57:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:57:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:57:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:57:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:57:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:57:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:57:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:57:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:57:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:57:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:57:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:57:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:57:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:57:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:57:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:57:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:57:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:57:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:57:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:57:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:57:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:57:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:57:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:57:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:57:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:57:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:57:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:57:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:57:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:57:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:57:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:57:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:57:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:57:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:57:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:57:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:57:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:57:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:57:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:57:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:57:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:57:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:57:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:58:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:58:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:58:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:58:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:58:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:58:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:58:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:58:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:58:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:58:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:58:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:58:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:58:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:58:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:58:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:58:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:58:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:58:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:58:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:58:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:58:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:58:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:58:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:58:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:58:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:58:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:58:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:58:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:58:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:58:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:58:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:58:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:58:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:58:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:58:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:58:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:58:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:58:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:58:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:58:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:58:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:58:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:58:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:58:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:58:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:58:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:58:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:58:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:58:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:58:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:58:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:58:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:58:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:58:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:58:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:58:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:58:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:58:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:58:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:58:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:58:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:58:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:58:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:58:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:58:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:58:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:58:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:58:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:58:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:58:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:58:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:58:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:58:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:58:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:58:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:58:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:58:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:58:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:58:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:58:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:58:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:58:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:58:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:58:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:58:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:58:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:58:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:58:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:58:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:58:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:58:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:58:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:58:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:58:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:58:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:58:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:58:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:58:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:58:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:58:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:58:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:58:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:58:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:58:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:58:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:58:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:58:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:58:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:58:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:58:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:58:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:58:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:58:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:58:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:58:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:58:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:58:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:58:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:58:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:58:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:58:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:58:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:58:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:58:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:58:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:58:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:58:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:58:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:58:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:58:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:58:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:58:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:58:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:58:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:58:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:58:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:58:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:58:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:58:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:58:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:58:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:58:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:58:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:58:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:58:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:58:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:58:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:58:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:58:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:58:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:58:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:58:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:58:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:58:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:58:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:58:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:58:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:58:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:58:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:58:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:58:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:58:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:58:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:58:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:58:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:58:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:58:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:58:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:58:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:58:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:58:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:58:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:58:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:58:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:58:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:58:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:58:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:58:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:58:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:58:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:58:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:58:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:58:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:58:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:58:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:58:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:58:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:58:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:58:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:58:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:58:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:58:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:58:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:58:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:58:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:58:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:58:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:58:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:58:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:58:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:58:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:58:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:58:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:58:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:58:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:58:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:58:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:58:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:58:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:58:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:58:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:58:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:58:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:58:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:58:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:58:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:59:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:59:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:59:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:59:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:59:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:59:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:59:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:59:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:59:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:59:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:59:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:59:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:59:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:59:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:59:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:59:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:59:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:59:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:59:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:59:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:59:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:59:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:59:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:59:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:59:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:59:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:59:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:59:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:59:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:59:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:59:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:59:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:59:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:59:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:59:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:59:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:59:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:59:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:59:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:59:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:59:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:59:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:59:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:59:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:59:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:59:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:59:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:59:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:59:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:59:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:59:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:59:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:59:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:59:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:59:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:59:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:59:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:59:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:59:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:59:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:59:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:59:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:59:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:59:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:59:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:59:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:59:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:59:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:59:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:59:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:59:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:59:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:59:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:59:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:59:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:59:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:59:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:59:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:59:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:59:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:59:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:59:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:59:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:59:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:59:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:59:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:59:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:59:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:59:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:59:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:59:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:59:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:59:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:59:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:59:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:59:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:59:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:59:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:59:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:59:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:59:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:59:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:59:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:59:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:59:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:59:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:59:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:59:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:59:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:59:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:59:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:59:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:59:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:59:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:59:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:59:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:59:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:59:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:59:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:59:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:59:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:59:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:59:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:59:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:59:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:59:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:59:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:59:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:59:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:59:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:59:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:59:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:59:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:59:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:59:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:59:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:59:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:59:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 11:59:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 11:59:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:59:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:59:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:59:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 11:59:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 11:59:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 11:59:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 11:59:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 11:59:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 11:59:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 11:59:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 11:59:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 11:59:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 11:59:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 11:59:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 11:59:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 11:59:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:00:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:00:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:00:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:00:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:00:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:00:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:00:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:00:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:00:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:00:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:00:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:00:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:00:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:00:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:00:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:00:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:00:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:00:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:00:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:00:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:00:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:00:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:00:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:00:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:00:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:00:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:00:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:00:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:00:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:00:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:00:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:00:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:00:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:00:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:00:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:00:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:00:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:00:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:00:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:00:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:00:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:00:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:00:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:00:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:00:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:00:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:00:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:00:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:00:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:00:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:00:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:00:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:00:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:00:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:00:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:00:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:00:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:00:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:00:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:00:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:00:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:00:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:00:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:00:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:00:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:00:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:00:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:00:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:00:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:00:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:00:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:00:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:00:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:00:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:00:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:00:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:00:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:00:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:00:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:00:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:00:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:00:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:00:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:00:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:00:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:00:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:00:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:00:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:00:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:00:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:00:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:00:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:00:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:00:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:00:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:00:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:00:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:00:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:00:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:00:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:00:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:00:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:00:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:00:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:00:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:00:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:00:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:00:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:00:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:00:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:00:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:00:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:00:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:00:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:00:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:00:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:00:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:00:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:00:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:00:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:00:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:00:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:00:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:00:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:00:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:00:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:00:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:00:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:00:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:00:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:00:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:00:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:00:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:00:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:00:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:00:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:00:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:00:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:00:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:00:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:00:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:00:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:00:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:00:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:00:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:00:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:00:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:00:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:00:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:00:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:00:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:00:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:00:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:00:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:00:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:00:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:00:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:00:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:00:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:00:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:00:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:00:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:00:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:00:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:00:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:00:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:00:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:00:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:00:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:00:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:00:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:00:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:00:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:00:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:00:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:00:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:00:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:00:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:00:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:00:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:00:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:00:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:00:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:00:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:00:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:00:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:01:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:01:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:01:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:01:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:01:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:01:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:01:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:01:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:01:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:01:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:01:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:01:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:01:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:01:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:01:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:01:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:01:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:01:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:01:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:01:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:01:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:01:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:01:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:01:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:01:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:01:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:01:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:01:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:01:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:01:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:01:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:01:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:01:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:01:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:01:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:01:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:01:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:01:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:01:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:01:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:01:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:01:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:01:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:01:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:01:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:01:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:01:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:01:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:01:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:01:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:01:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:01:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:01:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:01:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:01:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:01:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:01:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:01:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:01:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:01:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:01:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:01:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:01:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:01:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:01:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:01:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:01:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:01:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:01:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:01:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:01:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:01:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:01:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:01:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:01:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:01:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:01:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:01:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:01:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:01:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:01:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:01:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:01:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:01:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:01:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:01:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:01:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:01:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:01:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:01:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:01:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:01:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:01:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:01:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:01:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:01:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:01:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:01:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:01:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:01:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:01:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:01:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:01:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:01:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:01:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:01:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:01:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:01:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:01:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:01:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:01:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:01:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:01:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:01:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:01:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:01:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:01:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:01:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:01:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:01:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:01:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:01:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:01:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:01:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:01:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:01:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:01:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:01:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:01:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:01:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:01:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:01:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:01:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:01:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:01:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:01:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:01:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:01:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:01:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:01:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:01:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:01:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:01:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:01:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:01:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:01:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:01:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:01:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:01:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:01:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:01:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:01:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:01:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:01:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:01:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:01:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:01:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:01:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:01:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:01:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:01:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:01:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:01:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:01:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:01:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:01:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:01:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:01:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:01:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:01:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:01:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:01:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:01:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:01:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:01:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:01:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:01:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:01:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:01:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:01:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:01:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:01:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:01:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:01:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:01:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:01:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:01:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:01:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:01:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:01:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:01:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:01:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:01:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:01:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:01:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:01:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:01:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:01:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:01:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:01:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:01:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:01:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:01:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:01:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:01:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:01:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:01:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:01:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:01:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:01:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:01:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:01:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:01:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:01:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:01:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:01:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:02:00 rdma06 systemd[1]: Starting dnf makecache...
Apr 17 12:02:01 rdma06 dnf[1154493]: Updating Subscription Management repositories.
Apr 17 12:02:01 rdma06 dnf[1154493]: Unable to read consumer identity
Apr 17 12:02:01 rdma06 dnf[1154493]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 17 12:02:01 rdma06 dnf[1154493]: Metadata cache refreshed recently.
Apr 17 12:02:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 17 12:02:01 rdma06 systemd[1]: Finished dnf makecache.
Apr 17 12:02:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:02:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:02:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:02:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:02:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:02:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:02:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:02:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:02:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:02:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:02:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:02:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:02:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:02:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:02:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:02:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:02:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:02:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:02:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:02:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:02:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:02:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:02:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:02:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:02:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:02:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:02:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:02:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:02:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:02:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:02:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:02:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:02:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:02:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:02:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:02:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:02:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:02:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:02:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:02:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:02:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:02:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:02:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:02:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:02:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:02:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:02:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:02:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:02:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:02:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:02:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:02:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:02:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:02:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:02:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:02:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:02:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:02:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:02:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:02:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:02:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:02:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:02:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:02:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:02:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:02:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:02:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:02:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:02:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:02:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:02:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:02:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:02:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:02:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:02:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:02:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:02:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:02:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:02:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:02:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:02:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:02:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:02:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:02:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:02:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:02:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:02:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:02:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:02:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:02:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:02:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:02:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:02:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:02:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:02:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:02:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:02:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:02:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:02:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:02:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:02:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:02:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:02:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:02:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:02:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:02:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:02:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:02:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:02:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:02:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:02:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:02:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:02:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:02:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:02:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:02:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:02:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:02:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:02:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:02:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:02:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:02:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:02:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:02:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:02:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:02:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:02:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:02:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:02:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:02:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:02:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:02:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:02:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:02:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:02:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:02:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:02:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:02:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:02:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:02:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:02:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:02:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:02:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:02:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:02:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:02:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:02:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:02:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:02:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:02:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:02:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:02:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:02:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:02:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:02:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:02:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:02:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:02:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:02:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:02:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:02:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:02:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:02:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:02:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:02:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:02:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:02:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:02:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:02:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:02:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:02:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:02:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:02:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:02:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:02:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:02:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:02:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:02:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:02:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:02:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:02:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:02:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:02:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:02:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:02:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:02:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:02:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:02:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:02:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:02:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:02:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:02:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:02:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:02:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:02:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:02:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:02:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:02:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:02:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:02:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:02:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:02:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:02:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:02:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:02:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:02:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:02:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:02:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:02:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:02:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:02:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:02:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:02:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:02:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:02:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:02:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:03:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:03:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:03:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:03:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:03:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:03:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:03:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:03:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:03:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:03:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:03:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:03:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:03:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:03:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:03:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:03:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:03:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:03:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:03:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:03:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:03:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:03:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:03:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:03:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:03:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:03:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:03:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:03:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:03:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:03:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:03:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:03:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:03:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:03:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:03:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:03:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:03:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:03:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:03:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:03:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:03:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:03:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:03:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:03:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:03:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:03:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:03:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:03:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:03:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:03:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:03:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:03:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:03:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:03:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:03:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:03:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:03:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:03:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:03:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:03:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:03:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:03:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:03:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:03:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:03:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:03:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:03:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:03:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:03:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:03:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:03:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:03:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:03:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:03:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:03:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:03:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:03:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:03:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:03:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:03:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:03:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:03:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:03:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:03:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:03:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:03:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:03:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:03:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:03:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:03:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:03:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:03:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:03:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:03:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:03:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:03:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:03:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:03:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:03:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:03:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:03:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:03:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:03:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:03:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:03:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:03:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:03:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:03:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:03:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:03:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:03:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:03:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:03:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:03:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:03:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:03:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:03:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:03:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:03:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:03:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:03:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:03:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:03:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:03:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:03:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:03:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:03:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:03:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:03:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:03:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:03:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:03:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:03:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:03:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:03:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:03:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:03:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:03:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:03:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:03:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:03:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:03:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:03:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:03:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:03:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:03:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:03:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:03:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:03:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:03:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:03:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:03:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:03:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:03:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:03:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:03:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:03:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:03:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:03:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:03:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:03:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:03:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:03:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:03:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:03:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:03:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:03:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:03:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:03:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:03:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:03:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:03:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:03:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:03:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:03:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:03:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:03:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:03:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:03:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:03:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:03:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:03:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:03:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:03:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:03:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:03:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:03:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:03:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:03:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:03:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:03:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:03:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:03:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:03:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:03:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:03:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:03:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:03:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:03:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:03:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:03:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:03:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:03:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:03:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:03:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:03:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:03:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:03:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:03:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:03:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:03:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:03:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:03:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:03:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:03:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:03:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:04:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:04:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:04:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:04:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:04:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:04:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:04:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:04:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:04:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:04:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:04:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:04:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:04:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:04:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:04:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:04:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:04:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:04:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:04:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:04:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:04:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:04:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:04:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:04:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:04:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:04:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:04:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:04:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:04:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:04:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:04:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:04:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:04:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:04:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:04:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:04:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:04:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:04:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:04:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:04:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:04:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:04:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:04:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:04:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:04:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:04:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:04:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:04:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:04:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:04:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:04:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:04:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:04:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:04:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:06:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:06:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:06:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:06:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:06:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:06:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:06:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:06:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:06:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:06:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:06:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:06:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:06:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:06:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:06:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:06:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:06:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:06:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:06:24 rdma06 rsyslogd[1041]: imjournal: 5302 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 12:06:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:06:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:06:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:06:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:06:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:06:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:06:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:06:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:06:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:06:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:06:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:06:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:06:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:06:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:06:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:06:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:06:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:06:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:06:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:06:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:06:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:06:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:06:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:06:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:06:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:06:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:06:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:06:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:06:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:06:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:06:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:06:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:06:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:06:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:06:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:06:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:06:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:06:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:06:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:06:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:06:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:06:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:06:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:06:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:06:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:06:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:06:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:06:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:06:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:06:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:06:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:06:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:06:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:06:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:06:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:06:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:06:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:06:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:06:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:06:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:06:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:06:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:06:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:06:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:06:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:06:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:06:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:06:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:06:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:06:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:06:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:06:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:06:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:06:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:06:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:06:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:06:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:06:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:06:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:06:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:06:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:06:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:06:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:06:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:06:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:06:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:06:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:06:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:06:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:06:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:06:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:06:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:06:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:06:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:06:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:06:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:06:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:06:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:06:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:06:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:06:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:06:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:06:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:06:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:06:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:06:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:06:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:06:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:06:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:06:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:06:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:06:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:06:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:06:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:06:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:06:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:06:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:06:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:06:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:06:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:06:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:06:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:06:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:06:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:06:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:06:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:07:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:07:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:07:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:07:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:07:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:07:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:07:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:07:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:07:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:07:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:07:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:07:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:07:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:07:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:07:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:07:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:07:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:07:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:07:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:07:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:07:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:07:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:07:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:07:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:07:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:07:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:07:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:07:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:07:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:07:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:07:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:07:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:07:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:07:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:07:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:07:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:07:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:07:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:07:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:07:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:07:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:07:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:07:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:07:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:07:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:07:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:07:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:07:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:07:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:07:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:07:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:07:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:07:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:07:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:07:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:07:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:07:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:07:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:07:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:07:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:07:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:07:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:07:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:07:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:07:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:07:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:07:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:07:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:07:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:07:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:07:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:07:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:07:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:07:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:07:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:07:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:07:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:07:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:07:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:07:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:07:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:07:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:07:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:07:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:07:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:07:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:07:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:07:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:07:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:07:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:07:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:07:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:07:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:07:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:07:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:07:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:07:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:07:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:07:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:07:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:07:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:07:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:07:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:07:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:07:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:07:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:07:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:07:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:07:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:07:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:07:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:07:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:07:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:07:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:07:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:07:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:07:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:07:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:07:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:07:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:07:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:07:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:07:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:07:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:07:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:07:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:07:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:07:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:07:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:07:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:07:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:07:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:07:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:07:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:07:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:07:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:07:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:07:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:07:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:07:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:07:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:07:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:07:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:07:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:07:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:07:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:07:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:07:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:07:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:07:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:07:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:07:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:07:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:07:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:07:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:07:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:07:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:07:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:07:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:07:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:07:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:07:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:07:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:07:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:07:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:07:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:07:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:07:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:07:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:07:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:07:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:07:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:07:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:07:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:07:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:07:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:07:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:07:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:07:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:07:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:07:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:07:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:07:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:07:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:07:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:07:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:07:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:07:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:07:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:07:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:07:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:07:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:07:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:07:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:07:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:07:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:07:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:07:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:07:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:07:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:07:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:07:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:07:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:07:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:07:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:07:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:07:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:07:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:07:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:07:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:07:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:07:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:07:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:07:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:07:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:07:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:08:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:08:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:08:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:08:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:08:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:08:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:08:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:08:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:08:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:08:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:08:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:08:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:08:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:08:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:08:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:08:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:08:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:08:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:08:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:08:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:08:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:08:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:08:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:08:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:08:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:08:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:08:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:08:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:08:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:08:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:08:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:08:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:08:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:08:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:08:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:08:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:08:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:08:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:08:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:08:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:08:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:08:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:08:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:08:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:08:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:08:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:08:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:08:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:08:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:08:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:08:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:08:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:08:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:08:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:08:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:08:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:08:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:08:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:08:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:08:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:08:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:08:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:08:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:08:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:08:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:08:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:08:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:08:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:08:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:08:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:08:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:08:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:08:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:08:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:08:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:08:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:08:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:08:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:08:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:08:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:08:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:08:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:08:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:08:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:08:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:08:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:08:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:08:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:08:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:08:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:08:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:08:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:08:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:08:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:08:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:08:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:08:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:08:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:08:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:08:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:08:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:08:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:08:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:08:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:08:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:08:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:08:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:08:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:08:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:08:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:08:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:08:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:08:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:08:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:08:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:08:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:08:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:08:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:08:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:08:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:08:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:08:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:08:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:08:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:08:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:08:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:08:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:08:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:08:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:08:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:08:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:08:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:08:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:08:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:08:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:08:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:08:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:08:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:08:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:08:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:08:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:08:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:08:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:08:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:08:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:08:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:08:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:08:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:08:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:08:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:08:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:08:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:08:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:08:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:08:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:08:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:08:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:08:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:08:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:08:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:08:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:08:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:08:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:08:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:08:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:08:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:08:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:08:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:08:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:08:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:08:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:08:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:08:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:08:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:08:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:08:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:08:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:08:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:08:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:08:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:08:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:08:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:08:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:08:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:08:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:08:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:08:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:08:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:08:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:08:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:08:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:08:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:08:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:08:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:08:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:08:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:08:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:08:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:08:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:08:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:08:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:08:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:08:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:08:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:08:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:08:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:08:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:08:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:08:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:08:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:08:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:08:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:08:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:08:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:08:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:08:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:09:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:09:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:09:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:09:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:09:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:09:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:09:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:09:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:09:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:09:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:09:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:09:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:09:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:09:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:09:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:09:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:09:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:09:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:09:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:09:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:09:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:09:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:09:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:09:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:09:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:09:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:09:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:09:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:09:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:09:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:09:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:09:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:09:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:09:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:09:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:09:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:09:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:09:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:09:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:09:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:09:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:09:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:09:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:09:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:09:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:09:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:09:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:09:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:09:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:09:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:09:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:09:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:09:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:09:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:09:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:09:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:09:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:09:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:09:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:09:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:09:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:09:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:09:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:09:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:09:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:09:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:09:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:09:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:09:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:09:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:09:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:09:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:09:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:09:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:09:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:09:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:09:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:09:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:09:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:09:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:09:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:09:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:09:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:09:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:09:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:09:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:09:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:09:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:09:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:09:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:09:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:09:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:09:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:09:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:09:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:09:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:09:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:09:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:09:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:09:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:09:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:09:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:09:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:09:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:09:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:09:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:09:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:09:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:09:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:09:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:09:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:09:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:09:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:09:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:09:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:09:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:09:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:09:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:09:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:09:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:09:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:09:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:09:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:09:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:09:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:09:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:09:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:09:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:09:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:09:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:09:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:09:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:09:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:09:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:09:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:09:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:09:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:09:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:09:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:09:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:09:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:09:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:09:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:09:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:09:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:09:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:09:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:09:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:09:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:09:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:09:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:09:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:09:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:09:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:09:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:09:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:09:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:09:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:09:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:09:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:09:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:09:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:09:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:09:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:09:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:09:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:09:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:09:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:09:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:09:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:09:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:09:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:09:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:09:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:09:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:09:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:09:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:09:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:09:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:09:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:09:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:09:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:09:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:09:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:09:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:09:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:09:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:09:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:09:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:09:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:09:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:09:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:09:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:09:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:09:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:09:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:09:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:09:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:09:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:09:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:09:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:09:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:09:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:09:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:09:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:09:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:09:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:09:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:09:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:09:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:09:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:09:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:09:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:09:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:09:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:09:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:10:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:10:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:10:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:10:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:10:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:10:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:10:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:10:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:10:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:10:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:10:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:10:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:10:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:10:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:10:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:10:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:10:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:10:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:10:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:10:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:10:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:10:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:10:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:10:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:10:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:10:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:10:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:10:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:10:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:10:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:10:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:10:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:10:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:10:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:10:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:10:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:10:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:10:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:10:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:10:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:10:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:10:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:10:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:10:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:10:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:10:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:10:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:10:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:10:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:10:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:10:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:10:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:10:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:10:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:10:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:10:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:10:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:10:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:10:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:10:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:10:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:10:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:10:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:10:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:10:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:10:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:10:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:10:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:10:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:10:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:10:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:10:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:10:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:10:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:10:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:10:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:10:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:10:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:10:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:10:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:10:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:10:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:10:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:10:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:10:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:10:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:10:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:10:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:10:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:10:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:10:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:10:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:10:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:10:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:10:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:10:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:10:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:10:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:10:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:10:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:10:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:10:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:10:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:10:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:10:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:10:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:10:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:10:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:10:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:10:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:10:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:10:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:10:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:10:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:10:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:10:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:10:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:10:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:10:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:10:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:10:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:10:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:10:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:10:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:10:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:10:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:10:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:10:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:10:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:10:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:10:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:10:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:10:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:10:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:10:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:10:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:10:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:10:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:10:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:10:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:10:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:10:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:10:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:10:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:10:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:10:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:10:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:10:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:10:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:10:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:10:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:10:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:10:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:10:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:10:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:10:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:10:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:10:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:10:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:10:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:10:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:10:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:10:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:10:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:10:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:10:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:10:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:10:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:10:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:10:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:10:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:10:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:10:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:10:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:10:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:10:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:10:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:10:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:10:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:10:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:10:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:10:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:10:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:10:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:10:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:10:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:10:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:10:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:10:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:10:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:10:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:10:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:10:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:10:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:10:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:10:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:10:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:10:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:10:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:10:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:10:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:10:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:10:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:10:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:10:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:10:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:10:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:10:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:10:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:10:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:10:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:10:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:10:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:10:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:10:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:10:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:11:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:11:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:11:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:11:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:11:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:11:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:11:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:11:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:11:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:11:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:11:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:11:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:11:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:11:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:11:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:11:04 rdma06 systemd-journald[790]: Data hash table of /run/log/journal/95379d2f723847b68d9e976dee8967a9/system.journal has a fill level at 75.0 (163750 of 218332 items, 75497472 file size, 461 bytes per hash table item), suggesting rotation.
Apr 17 12:11:04 rdma06 systemd-journald[790]: /run/log/journal/95379d2f723847b68d9e976dee8967a9/system.journal: Journal header limits reached or header out-of-date, rotating.
Apr 17 12:11:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:11:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:11:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:11:04 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 17 12:11:04 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 17 12:11:04 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 17 12:11:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:11:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:11:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:11:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:11:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:11:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:11:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:11:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:11:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:11:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:11:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:11:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:11:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:11:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:11:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:11:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:11:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:11:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:11:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:11:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:11:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:11:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:11:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:11:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:11:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:11:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:11:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:11:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:11:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:11:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:11:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:11:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:11:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:11:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:11:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:11:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:11:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:11:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:11:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:11:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:11:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:11:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:11:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:11:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:11:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:11:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:11:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:11:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:11:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:11:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:11:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:11:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:11:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:11:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:11:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:11:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:11:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:11:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:11:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:11:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:11:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:11:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:11:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:11:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:11:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:11:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:11:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:11:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:11:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:11:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:11:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:11:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:11:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:11:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:11:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:11:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:11:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:11:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:11:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:11:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:11:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:11:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:11:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:11:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:11:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:11:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:11:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:11:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:11:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:11:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:11:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:11:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:11:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:11:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:11:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:11:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:11:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:11:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:11:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:11:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:11:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:11:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:11:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:11:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:11:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:11:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:11:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:11:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:11:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:11:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:11:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:11:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:11:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:11:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:11:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:11:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:11:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:11:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:11:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:11:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:11:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:11:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:11:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:11:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:11:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:11:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:11:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:11:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:11:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:11:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:11:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:11:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:11:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:11:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:11:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:11:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:11:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:11:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:11:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:11:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:11:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:11:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:11:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:11:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:11:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:11:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:11:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:11:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:11:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:11:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:11:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:11:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:11:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:11:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:11:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:11:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:11:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:11:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:11:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:11:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:11:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:11:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:11:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:11:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:11:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:11:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:11:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:11:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:11:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:11:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:11:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:11:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:11:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:11:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:11:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:11:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:11:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:11:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:11:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:11:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:11:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:11:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:11:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:11:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:11:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:11:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:11:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:11:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:11:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:11:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:11:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:11:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:11:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:11:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:11:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:11:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:11:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:11:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:12:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:12:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:12:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:12:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:12:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:12:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:12:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:12:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:12:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:12:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:12:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:12:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:12:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:12:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:12:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:12:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:12:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:12:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:12:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:12:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:12:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:12:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:12:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:12:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:12:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:12:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:12:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:12:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:12:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:12:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:12:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:12:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:12:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:12:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:12:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:12:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:12:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:12:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:12:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:12:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:12:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:12:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:12:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:12:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:12:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:12:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:12:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:12:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:12:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:12:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:12:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:12:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:12:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:12:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:12:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:12:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:12:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:12:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:12:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:12:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:12:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:12:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:12:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:12:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:12:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:12:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:12:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:12:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:12:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:12:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:12:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:12:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:12:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:12:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:12:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:12:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:12:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:12:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:12:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:12:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:12:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:12:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:12:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:12:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:12:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:12:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:12:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:12:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:12:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:12:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:12:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:12:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:12:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:12:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:12:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:12:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:12:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:12:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:12:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:12:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:12:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:12:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:12:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:12:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:12:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:12:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:12:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:12:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:12:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:12:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:12:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:12:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:12:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:12:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:12:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:12:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:12:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:12:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:12:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:12:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:12:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:12:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:12:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:12:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:12:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:12:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:12:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:12:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:12:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:12:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:12:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:12:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:12:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:12:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:12:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:12:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:12:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:12:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:12:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:12:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:12:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:12:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:12:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:12:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:12:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:12:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:12:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:12:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:12:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:12:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:12:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:12:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:12:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:12:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:12:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:12:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:12:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:12:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:12:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:12:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:12:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:12:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:12:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:12:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:12:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:12:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:12:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:12:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:12:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:12:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:12:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:12:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:12:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:12:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:12:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:12:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:12:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:12:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:12:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:12:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:12:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:12:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:12:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:12:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:12:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:12:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:12:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:12:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:12:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:12:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:12:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:12:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:12:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:12:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:12:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:12:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:12:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:12:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:12:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:12:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:12:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:12:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:12:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:12:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:12:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:12:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:12:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:12:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:12:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:12:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:12:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:12:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:12:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:12:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:12:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:12:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:13:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:13:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:13:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:13:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:13:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:13:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:13:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:13:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:13:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:13:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:13:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:13:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:13:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:13:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:13:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:13:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:13:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:13:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:13:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:13:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:13:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:13:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:13:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:13:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:13:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:13:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:13:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:13:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:13:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:13:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:13:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:13:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:13:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:13:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:13:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:13:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:13:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:13:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:13:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:13:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:13:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:13:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:13:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:13:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:13:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:13:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:13:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:13:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:13:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:13:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:13:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:13:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:13:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:13:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:13:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:13:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:13:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:13:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:13:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:13:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:13:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:13:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:13:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:13:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:13:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:13:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:13:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:13:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:13:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:13:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:13:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:13:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:13:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:13:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:13:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:13:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:13:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:13:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:13:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:13:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:13:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:13:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:13:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:13:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:13:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:13:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:13:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:13:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:13:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:13:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:13:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:13:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:13:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:13:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:13:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:13:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:13:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:13:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:13:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:13:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:13:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:13:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:13:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:13:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:13:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:13:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:13:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:13:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:13:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:13:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:13:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:13:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:13:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:13:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:13:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:13:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:13:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:13:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:13:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:13:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:13:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:13:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:13:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:13:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:13:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:13:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:13:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:13:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:13:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:13:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:13:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:13:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:13:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:13:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:13:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:13:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:13:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:13:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:13:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:13:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:13:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:13:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:13:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:13:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:13:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:13:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:13:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:13:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:13:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:13:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:13:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:13:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:13:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:13:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:13:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:13:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:13:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:13:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:13:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:13:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:13:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:13:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:13:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:13:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:13:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:13:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:13:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:13:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:13:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:13:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:13:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:13:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:13:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:13:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:13:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:13:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:13:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:13:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:13:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:13:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:13:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:13:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:13:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:13:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:13:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:13:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:13:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:13:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:13:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:13:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:13:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:13:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:13:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:13:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:13:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:13:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:13:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:13:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:13:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:13:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:13:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:13:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:13:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:13:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:13:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:13:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:13:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:13:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:13:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:13:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:13:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:13:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:13:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:13:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:13:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:13:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:14:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:14:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:14:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:14:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:14:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:14:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:14:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:14:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:14:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:14:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:14:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:14:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:14:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:14:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:14:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:14:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:14:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:14:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:14:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:14:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:14:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:14:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:14:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:14:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:14:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:14:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:14:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:14:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:14:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:14:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:14:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:14:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:14:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:14:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:14:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:14:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:14:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:14:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:14:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:14:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:14:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:14:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:14:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:14:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:14:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:14:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:14:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:14:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:14:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:14:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:14:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:14:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:14:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:14:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:14:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:14:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:14:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:14:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:14:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:14:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:14:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:14:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:14:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:14:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:14:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:14:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:14:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:14:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:14:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:14:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:14:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:14:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:16:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:16:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:16:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:16:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:16:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:16:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:16:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:16:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:16:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:16:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:16:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:16:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:16:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:16:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:16:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:16:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:16:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:16:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:16:25 rdma06 rsyslogd[1041]: imjournal: 5087 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 12:16:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:16:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:16:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:16:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:16:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:16:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:16:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:16:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:16:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:16:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:16:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:16:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:16:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:16:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:16:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:16:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:16:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:16:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:16:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:16:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:16:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:16:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:16:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:16:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:16:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:16:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:16:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:16:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:16:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:16:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:16:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:16:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:16:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:16:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:16:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:16:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:16:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:16:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:16:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:16:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:16:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:16:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:16:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:16:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:16:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:16:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:16:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:16:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:16:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:16:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:16:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:16:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:16:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:16:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:16:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:16:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:16:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:16:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:16:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:16:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:16:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:16:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:16:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:16:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:16:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:16:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:16:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:16:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:16:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:16:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:16:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:16:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:16:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:16:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:16:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:16:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:16:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:16:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:16:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:16:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:16:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:16:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:16:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:16:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:16:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:16:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:16:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:16:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:16:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:16:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:16:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:16:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:16:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:16:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:16:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:16:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:16:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:16:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:16:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:16:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:16:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:16:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:16:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:16:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:16:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:16:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:16:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:16:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:16:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:16:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:16:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:16:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:16:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:16:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:16:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:16:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:16:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:16:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:16:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:16:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:16:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:16:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:16:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:16:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:16:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:16:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:17:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:17:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:17:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:17:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:17:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:17:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:17:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:17:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:17:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:17:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:17:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:17:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:17:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:17:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:17:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:17:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:17:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:17:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:17:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:17:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:17:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:17:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:17:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:17:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:17:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:17:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:17:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:17:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:17:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:17:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:17:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:17:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:17:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:17:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:17:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:17:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:17:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:17:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:17:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:17:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:17:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:17:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:17:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:17:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:17:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:17:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:17:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:17:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:17:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:17:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:17:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:17:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:17:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:17:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:17:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:17:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:17:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:17:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:17:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:17:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:17:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:17:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:17:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:17:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:17:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:17:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:17:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:17:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:17:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:17:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:17:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:17:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:17:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:17:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:17:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:17:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:17:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:17:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:17:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:17:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:17:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:17:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:17:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:17:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:17:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:17:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:17:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:17:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:17:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:17:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:17:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:17:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:17:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:17:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:17:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:17:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:17:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:17:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:17:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:17:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:17:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:17:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:17:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:17:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:17:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:17:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:17:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:17:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:17:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:17:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:17:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:17:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:17:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:17:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:17:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:17:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:17:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:17:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:17:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:17:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:17:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:17:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:17:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:17:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:17:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:17:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:17:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:17:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:17:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:17:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:17:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:17:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:17:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:17:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:17:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:17:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:17:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:17:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:17:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:17:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:17:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:17:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:17:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:17:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:17:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:17:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:17:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:17:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:17:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:17:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:17:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:17:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:17:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:17:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:17:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:17:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:17:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:17:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:17:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:17:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:17:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:17:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:17:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:17:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:17:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:17:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:17:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:17:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:17:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:17:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:17:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:17:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:17:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:17:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:17:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:17:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:17:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:17:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:17:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:17:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:17:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:17:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:17:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:17:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:17:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:17:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:17:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:17:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:17:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:17:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:17:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:17:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:17:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:17:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:17:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:17:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:17:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:17:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:17:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:17:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:17:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:17:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:17:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:17:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:17:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:17:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:17:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:17:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:17:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:17:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:17:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:17:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:17:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:17:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:17:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:17:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:18:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:18:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:18:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:18:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:18:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:18:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:18:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:18:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:18:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:18:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:18:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:18:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:18:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:18:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:18:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:18:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:18:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:18:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:18:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:18:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:18:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:18:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:18:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:18:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:18:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:18:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:18:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:18:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:18:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:18:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:18:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:18:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:18:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:18:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:18:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:18:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:18:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:18:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:18:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:18:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:18:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:18:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:18:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:18:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:18:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:18:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:18:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:18:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:18:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:18:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:18:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:18:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:18:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:18:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:18:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:18:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:18:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:18:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:18:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:18:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:18:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:18:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:18:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:18:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:18:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:18:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:18:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:18:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:18:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:18:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:18:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:18:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:18:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:18:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:18:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:18:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:18:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:18:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:18:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:18:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:18:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:18:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:18:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:18:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:18:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:18:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:18:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:18:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:18:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:18:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:18:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:18:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:18:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:18:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:18:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:18:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:18:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:18:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:18:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:18:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:18:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:18:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:18:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:18:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:18:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:18:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:18:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:18:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:18:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:18:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:18:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:18:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:18:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:18:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:18:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:18:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:18:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:18:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:18:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:18:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:18:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:18:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:18:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:18:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:18:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:18:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:18:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:18:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:18:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:18:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:18:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:18:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:18:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:18:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:18:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:18:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:18:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:18:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:18:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:18:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:18:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:18:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:18:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:18:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:18:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:18:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:18:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:18:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:18:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:18:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:18:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:18:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:18:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:18:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:18:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:18:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:18:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:18:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:18:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:18:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:18:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:18:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:18:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:18:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:18:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:18:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:18:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:18:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:18:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:18:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:18:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:18:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:18:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:18:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:18:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:18:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:18:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:18:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:18:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:18:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:18:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:18:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:18:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:18:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:18:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:18:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:18:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:18:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:18:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:18:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:18:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:18:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:18:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:18:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:18:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:18:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:18:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:18:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:18:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:18:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:18:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:18:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:18:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:18:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:18:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:18:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:18:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:18:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:18:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:18:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:18:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:18:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:18:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:18:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:18:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:18:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:19:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:19:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:19:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:19:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:19:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:19:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:19:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:19:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:19:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:19:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:19:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:19:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:19:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:19:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:19:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:19:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:19:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:19:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:19:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:19:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:19:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:19:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:19:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:19:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:19:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:19:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:19:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:19:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:19:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:19:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:19:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:19:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:19:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:19:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:19:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:19:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:19:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:19:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:19:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:19:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:19:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:19:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:19:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:19:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:19:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:19:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:19:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:19:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:19:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:19:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:19:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:19:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:19:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:19:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:19:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:19:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:19:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:19:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:19:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:19:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:19:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:19:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:19:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:19:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:19:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:19:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:19:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:19:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:19:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:19:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:19:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:19:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:19:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:19:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:19:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:19:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:19:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:19:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:19:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:19:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:19:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:19:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:19:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:19:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:19:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:19:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:19:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:19:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:19:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:19:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:19:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:19:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:19:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:19:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:19:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:19:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:19:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:19:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:19:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:19:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:19:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:19:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:19:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:19:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:19:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:19:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:19:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:19:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:19:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:19:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:19:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:19:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:19:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:19:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:19:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:19:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:19:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:19:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:19:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:19:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:19:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:19:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:19:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:19:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:19:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:19:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:19:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:19:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:19:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:19:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:19:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:19:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:19:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:19:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:19:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:19:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:19:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:19:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:19:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:19:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:19:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:19:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:19:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:19:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:19:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:19:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:19:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:19:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:19:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:19:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:19:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:19:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:19:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:19:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:19:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:19:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:19:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:19:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:19:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:19:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:19:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:19:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:19:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:19:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:19:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:19:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:19:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:19:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:19:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:19:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:19:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:19:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:19:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:19:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:19:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:19:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:19:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:19:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:19:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:19:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:19:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:19:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:19:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:19:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:19:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:19:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:19:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:19:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:19:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:19:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:19:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:19:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:19:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:19:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:19:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:19:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:19:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:19:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:19:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:19:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:19:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:19:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:19:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:19:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:19:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:19:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:19:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:19:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:19:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:19:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:19:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:19:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:19:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:19:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:19:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:19:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:20:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:20:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:20:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:20:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:20:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:20:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:20:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:20:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:20:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:20:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:20:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:20:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:20:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:20:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:20:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:20:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:20:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:20:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:20:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:20:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:20:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:20:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:20:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:20:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:20:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:20:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:20:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:20:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:20:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:20:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:20:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:20:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:20:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:20:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:20:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:20:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:20:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:20:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:20:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:20:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:20:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:20:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:20:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:20:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:20:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:20:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:20:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:20:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:20:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:20:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:20:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:20:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:20:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:20:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:20:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:20:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:20:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:20:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:20:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:20:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:20:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:20:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:20:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:20:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:20:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:20:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:20:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:20:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:20:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:20:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:20:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:20:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:20:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:20:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:20:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:20:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:20:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:20:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:20:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:20:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:20:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:20:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:20:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:20:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:20:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:20:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:20:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:20:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:20:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:20:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:20:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:20:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:20:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:20:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:20:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:20:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:20:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:20:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:20:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:20:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:20:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:20:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:20:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:20:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:20:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:20:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:20:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:20:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:20:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:20:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:20:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:20:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:20:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:20:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:20:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:20:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:20:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:20:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:20:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:20:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:20:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:20:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:20:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:20:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:20:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:20:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:20:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:20:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:20:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:20:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:20:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:20:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:20:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:20:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:20:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:20:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:20:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:20:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:20:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:20:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:20:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:20:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:20:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:20:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:20:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:20:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:20:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:20:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:20:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:20:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:20:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:20:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:20:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:20:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:20:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:20:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:20:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:20:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:20:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:20:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:20:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:20:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:20:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:20:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:20:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:20:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:20:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:20:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:20:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:20:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:20:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:20:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:20:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:20:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:20:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:20:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:20:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:20:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:20:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:20:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:20:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:20:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:20:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:20:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:20:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:20:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:20:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:20:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:20:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:20:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:20:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:20:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:20:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:20:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:20:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:20:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:20:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:20:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:20:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:20:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:20:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:20:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:20:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:20:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:20:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:20:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:20:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:20:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:20:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:20:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:20:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:20:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:20:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:20:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:20:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:20:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:21:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:21:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:21:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:21:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:21:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:21:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:21:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:21:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:21:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:21:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:21:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:21:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:21:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:21:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:21:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:21:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:21:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:21:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:21:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:21:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:21:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:21:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:21:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:21:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:21:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:21:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:21:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:21:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:21:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:21:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:21:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:21:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:21:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:21:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:21:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:21:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:21:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:21:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:21:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:21:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:21:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:21:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:21:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:21:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:21:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:21:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:21:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:21:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:21:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:21:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:21:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:21:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:21:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:21:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:21:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:21:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:21:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:21:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:21:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:21:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:21:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:21:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:21:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:21:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:21:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:21:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:21:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:21:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:21:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:21:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:21:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:21:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:21:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:21:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:21:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:21:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:21:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:21:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:21:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:21:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:21:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:21:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:21:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:21:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:21:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:21:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:21:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:21:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:21:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:21:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:21:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:21:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:21:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:21:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:21:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:21:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:21:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:21:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:21:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:21:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:21:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:21:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:21:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:21:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:21:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:21:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:21:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:21:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:21:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:21:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:21:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:21:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:21:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:21:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:21:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:21:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:22:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:22:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:22:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:22:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:22:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:22:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:22:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:22:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:22:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:22:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:22:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:22:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:22:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:22:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:22:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:22:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:22:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:22:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:22:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:22:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:22:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:22:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:22:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:22:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:22:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:22:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:22:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:22:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:22:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:22:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:22:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:22:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:22:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:22:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:22:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:22:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:22:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:22:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:22:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:22:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:22:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:22:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:22:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:22:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:22:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:22:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:22:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:22:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:22:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:22:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:22:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:22:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:22:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:22:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:22:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:22:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:22:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:22:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:22:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:22:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:22:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:22:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:22:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:22:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:22:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:22:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:22:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:22:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:22:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:22:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:22:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:22:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:22:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:22:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:22:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:22:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:22:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:22:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:22:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:22:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:22:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:22:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:22:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:22:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:22:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:22:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:22:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:22:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:22:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:22:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:22:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:22:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:22:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:22:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:22:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:22:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:22:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:22:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:22:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:22:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:22:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:22:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:22:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:22:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:22:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:22:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:22:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:22:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:22:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:22:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:22:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:22:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:22:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:22:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:22:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:22:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:22:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:22:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:22:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:22:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:22:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:22:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:22:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:22:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:22:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:22:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:22:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:22:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:22:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:22:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:22:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:22:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:22:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:22:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:22:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:22:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:22:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:22:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:22:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:22:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:22:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:22:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:22:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:22:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:22:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:22:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:22:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:22:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:22:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:22:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:22:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:22:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:22:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:22:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:22:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:22:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:22:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:22:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:22:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:22:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:22:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:22:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:22:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:22:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:22:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:22:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:22:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:22:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:22:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:22:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:22:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:22:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:22:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:22:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:22:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:22:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:22:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:22:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:22:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:22:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:22:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:22:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:22:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:22:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:22:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:22:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:22:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:22:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:22:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:22:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:22:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:22:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:22:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:22:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:22:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:22:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:22:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:22:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:22:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:22:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:22:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:22:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:22:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:22:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:22:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:22:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:22:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:22:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:22:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:22:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:22:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:22:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:22:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:22:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:22:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:22:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:23:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:23:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:23:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:23:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:23:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:23:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:23:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:23:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:23:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:23:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:23:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:23:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:23:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:23:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:23:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:23:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:23:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:23:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:23:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:23:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:23:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:23:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:23:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:23:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:23:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:23:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:23:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:23:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:23:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:23:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:23:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:23:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:23:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:23:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:23:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:23:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:23:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:23:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:23:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:23:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:23:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:23:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:23:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:23:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:23:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:23:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:23:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:23:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:23:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:23:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:23:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:23:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:23:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:23:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:23:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:23:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:23:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:23:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:23:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:23:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:23:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:23:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:23:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:23:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:23:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:23:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:23:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:23:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:23:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:23:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:23:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:23:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:23:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:23:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:23:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:23:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:23:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:23:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:23:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:23:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:23:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:23:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:23:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:23:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:23:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:23:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:23:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:23:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:23:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:23:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:23:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:23:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:23:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:23:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:23:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:23:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:23:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:23:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:23:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:23:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:23:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:23:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:23:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:23:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:23:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:23:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:23:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:23:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:23:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:23:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:23:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:23:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:23:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:23:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:23:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:23:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:23:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:23:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:23:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:23:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:23:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:23:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:23:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:23:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:23:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:23:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:23:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:23:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:23:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:23:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:23:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:23:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:23:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:23:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:23:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:23:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:23:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:23:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:23:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:23:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:23:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:23:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:23:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:23:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:23:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:23:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:23:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:23:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:23:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:23:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:23:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:23:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:23:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:23:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:23:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:23:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:23:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:23:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:23:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:23:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:23:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:23:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:23:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:23:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:23:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:23:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:23:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:23:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:23:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:23:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:23:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:23:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:23:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:23:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:23:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:23:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:23:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:23:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:23:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:23:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:23:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:23:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:23:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:23:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:23:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:23:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:23:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:23:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:23:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:23:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:23:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:23:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:23:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:23:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:23:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:23:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:23:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:23:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:23:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:23:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:23:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:23:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:23:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:23:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:23:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:23:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:23:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:23:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:23:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:23:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:23:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:23:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:23:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:23:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:23:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:23:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:24:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:24:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:24:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:24:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:24:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:24:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:24:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:24:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:24:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:24:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:24:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:24:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:24:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:24:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:24:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:24:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:24:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:24:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:24:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:24:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:24:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:24:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:24:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:24:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:24:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:24:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:24:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:24:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:24:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:24:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:24:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:24:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:24:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:24:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:24:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:24:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:24:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:24:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:24:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:24:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:24:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:24:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:24:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:24:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:24:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:24:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:24:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:24:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:24:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:24:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:24:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:24:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:24:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:24:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:24:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:24:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:24:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:24:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:24:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:24:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:24:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:24:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:24:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:24:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:24:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:24:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:24:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:24:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:24:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:24:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:24:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:24:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:26:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:26:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:26:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:26:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:26:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:26:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:26:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:26:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:26:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:26:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:26:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:26:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:26:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:26:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:26:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:26:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:26:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:26:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:26:30 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 12:26:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:26:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:26:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:26:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:26:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:26:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:26:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:26:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:26:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:26:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:26:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:26:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:26:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:26:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:26:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:26:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:26:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:26:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:26:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:26:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:26:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:26:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:26:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:26:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:26:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:26:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:26:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:26:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:26:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:26:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:26:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:26:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:26:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:26:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:26:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:26:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:26:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:26:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:26:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:26:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:26:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:26:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:26:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:26:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:26:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:26:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:26:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:26:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:26:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:26:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:26:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:26:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:26:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:26:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:26:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:26:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:26:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:26:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:26:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:26:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:26:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:26:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:26:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:26:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:26:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:26:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:26:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:26:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:26:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:26:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:26:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:26:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:26:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:26:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:26:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:26:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:26:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:26:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:26:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:26:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:26:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:26:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:26:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:26:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:26:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:26:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:26:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:26:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:26:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:26:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:26:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:26:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:26:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:26:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:26:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:26:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:26:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:26:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:26:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:26:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:26:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:26:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:26:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:26:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:26:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:26:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:26:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:26:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:27:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:27:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:27:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:27:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:27:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:27:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:27:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:27:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:27:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:27:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:27:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:27:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:27:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:27:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:27:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:27:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:27:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:27:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:27:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:27:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:27:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:27:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:27:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:27:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:27:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:27:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:27:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:27:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:27:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:27:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:27:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:27:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:27:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:27:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:27:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:27:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:27:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:27:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:27:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:27:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:27:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:27:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:27:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:27:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:27:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:27:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:27:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:27:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:27:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:27:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:27:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:27:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:27:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:27:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:27:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:27:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:27:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:27:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:27:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:27:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:27:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:27:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:27:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:27:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:27:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:27:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:27:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:27:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:27:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:27:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:27:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:27:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:27:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:27:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:27:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:27:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:27:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:27:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:27:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:27:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:27:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:27:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:27:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:27:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:27:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:27:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:27:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:27:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:27:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:27:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:27:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:27:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:27:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:27:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:27:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:27:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:27:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:27:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:27:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:27:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:27:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:27:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:27:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:27:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:27:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:27:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:27:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:27:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:27:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:27:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:27:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:27:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:27:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:27:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:27:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:27:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:27:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:27:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:27:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:27:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:27:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:27:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:27:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:27:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:27:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:27:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:27:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:27:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:27:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:27:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:27:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:27:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:27:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:27:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:27:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:27:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:27:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:27:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:27:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:27:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:27:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:27:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:27:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:27:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:27:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:27:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:27:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:27:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:27:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:27:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:27:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:27:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:27:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:27:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:27:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:27:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:27:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:27:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:27:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:27:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:27:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:27:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:27:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:27:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:27:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:27:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:27:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:27:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:27:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:27:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:27:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:27:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:27:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:27:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:27:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:27:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:27:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:27:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:27:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:27:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:27:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:27:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:27:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:27:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:27:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:27:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:27:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:27:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:27:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:27:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:27:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:27:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:27:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:27:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:27:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:27:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:27:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:27:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:28:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:28:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:28:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:28:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:28:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:28:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:28:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:28:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:28:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:28:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:28:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:28:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:28:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:28:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:28:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:28:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:28:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:28:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:28:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:28:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:28:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:28:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:28:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:28:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:28:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:28:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:28:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:28:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:28:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:28:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:28:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:28:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:28:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:28:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:28:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:28:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:28:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:28:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:28:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:28:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:28:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:28:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:28:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:28:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:28:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:28:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:28:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:28:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:28:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:28:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:28:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:28:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:28:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:28:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:28:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:28:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:28:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:28:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:28:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:28:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:28:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:28:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:28:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:28:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:28:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:28:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:28:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:28:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:28:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:28:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:28:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:28:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:28:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:28:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:28:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:28:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:28:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:28:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:28:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:28:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:28:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:28:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:28:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:28:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:28:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:28:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:28:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:28:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:28:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:28:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:28:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:28:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:28:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:28:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:28:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:28:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:28:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:28:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:28:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:28:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:28:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:28:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:28:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:28:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:28:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:28:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:28:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:28:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:28:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:28:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:28:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:28:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:28:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:28:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:28:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:28:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:28:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:28:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:28:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:28:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:28:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:28:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:28:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:28:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:28:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:28:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:28:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:28:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:28:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:28:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:28:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:28:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:28:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:28:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:28:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:28:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:28:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:28:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:28:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:28:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:28:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:28:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:28:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:28:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:28:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:28:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:28:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:28:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:28:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:28:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:28:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:28:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:28:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:28:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:28:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:28:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:28:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:28:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:28:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:28:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:28:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:28:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:28:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:28:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:28:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:28:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:28:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:28:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:28:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:28:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:28:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:28:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:28:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:28:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:28:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:28:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:28:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:28:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:28:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:28:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:28:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:28:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:28:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:28:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:28:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:28:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:28:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:28:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:28:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:28:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:28:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:28:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:28:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:28:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:28:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:28:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:28:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:28:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:28:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:28:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:28:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:28:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:28:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:28:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:28:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:28:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:28:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:28:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:28:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:28:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:28:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:28:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:28:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:28:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:28:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:28:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:29:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:29:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:29:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:29:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:29:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:29:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:29:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:29:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:29:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:29:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:29:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:29:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:29:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:29:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:29:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:29:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:29:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:29:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:29:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:29:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:29:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:29:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:29:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:29:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:29:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:29:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:29:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:29:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:29:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:29:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:29:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:29:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:29:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:29:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:29:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:29:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:29:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:29:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:29:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:29:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:29:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:29:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:29:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:29:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:29:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:29:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:29:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:29:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:29:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:29:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:29:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:29:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:29:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:29:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:29:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:29:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:29:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:29:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:29:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:29:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:29:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:29:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:29:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:29:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:29:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:29:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:29:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:29:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:29:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:29:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:29:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:29:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:29:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:29:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:29:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:29:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:29:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:29:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:29:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:29:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:29:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:29:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:29:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:29:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:29:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:29:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:29:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:29:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:29:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:29:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:29:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:29:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:29:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:29:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:29:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:29:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:29:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:29:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:29:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:29:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:29:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:29:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:29:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:29:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:29:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:29:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:29:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:29:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:29:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:29:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:29:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:29:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:29:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:29:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:29:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:29:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:29:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:29:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:29:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:29:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:29:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:29:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:29:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:29:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:29:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:29:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:29:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:29:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:29:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:29:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:29:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:29:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:29:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:29:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:29:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:29:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:29:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:29:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:29:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:29:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:29:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:29:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:29:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:29:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:29:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:29:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:29:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:29:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:29:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:29:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:29:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:29:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:29:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:29:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:29:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:29:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:29:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:29:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:29:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:29:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:29:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:29:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:29:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:29:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:29:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:29:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:29:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:29:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:29:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:29:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:29:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:29:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:29:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:29:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:29:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:29:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:29:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:29:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:29:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:29:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:29:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:29:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:29:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:29:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:29:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:29:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:29:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:29:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:29:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:29:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:29:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:29:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:29:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:29:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:29:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:29:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:29:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:29:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:29:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:29:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:29:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:29:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:29:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:29:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:29:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:29:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:29:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:29:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:29:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:29:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:29:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:29:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:29:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:29:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:29:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:29:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:30:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:30:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:30:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:30:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:30:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:30:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:30:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:30:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:30:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:30:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:30:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:30:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:30:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:30:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:30:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:30:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:30:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:30:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:30:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:30:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:30:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:30:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:30:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:30:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:30:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:30:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:30:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:30:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:30:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:30:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:30:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:30:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:30:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:30:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:30:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:30:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:30:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:30:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:30:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:30:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:30:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:30:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:30:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:30:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:30:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:30:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:30:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:30:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:30:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:30:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:30:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:30:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:30:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:30:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:30:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:30:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:30:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:30:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:30:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:30:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:30:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:30:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:30:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:30:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:30:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:30:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:30:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:30:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:30:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:30:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:30:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:30:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:30:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:30:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:30:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:30:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:30:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:30:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:30:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:30:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:30:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:30:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:30:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:30:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:30:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:30:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:30:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:30:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:30:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:30:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:30:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:30:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:30:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:30:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:30:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:30:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:30:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:30:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:30:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:30:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:30:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:30:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:30:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:30:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:30:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:30:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:30:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:30:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:30:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:30:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:30:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:30:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:30:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:30:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:30:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:30:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:30:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:30:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:30:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:30:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:30:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:30:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:30:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:30:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:30:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:30:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:30:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:30:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:30:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:30:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:30:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:30:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:30:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:30:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:30:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:30:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:30:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:30:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:30:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:30:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:30:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:30:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:30:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:30:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:30:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:30:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:30:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:30:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:30:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:30:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:30:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:30:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:30:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:30:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:30:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:30:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:30:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:30:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:30:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:30:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:30:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:30:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:30:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:30:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:30:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:30:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:30:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:30:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:30:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:30:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:30:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:30:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:30:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:30:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:30:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:30:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:30:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:30:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:30:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:30:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:30:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:30:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:30:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:30:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:30:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:30:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:30:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:30:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:30:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:30:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:30:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:30:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:30:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:30:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:30:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:30:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:30:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:30:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:30:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:30:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:30:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:30:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:30:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:30:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:30:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:30:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:30:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:30:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:30:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:30:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:30:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:30:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:30:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:30:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:30:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:30:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:31:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:31:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:31:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:31:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:31:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:31:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:31:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:31:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:31:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:31:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:31:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:31:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:31:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:31:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:31:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:31:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:31:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:31:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:31:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:31:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:31:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:31:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:31:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:31:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:31:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:31:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:31:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:31:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:31:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:31:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:31:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:31:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:31:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:31:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:31:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:31:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:31:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:31:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:31:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:31:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:31:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:31:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:31:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:31:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:31:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:31:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:31:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:31:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:31:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:31:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:31:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:31:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:31:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:31:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:31:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:31:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:31:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:31:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:31:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:31:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:31:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:31:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:31:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:31:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:31:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:31:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:31:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:31:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:31:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:31:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:31:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:31:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:31:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:31:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:31:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:31:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:31:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:31:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:31:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:31:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:31:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:31:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:31:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:31:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:31:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:31:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:31:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:31:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:31:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:31:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:31:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:31:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:31:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:31:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:31:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:31:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:31:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:31:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:31:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:31:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:31:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:31:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:31:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:31:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:31:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:31:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:31:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:31:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:31:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:31:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:31:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:31:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:31:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:31:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:31:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:31:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:31:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:31:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:31:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:31:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:31:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:31:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:31:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:31:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:31:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:31:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:31:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:31:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:31:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:31:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:31:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:31:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:31:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:31:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:31:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:31:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:31:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:31:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:31:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:31:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:31:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:31:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:31:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:31:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:31:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:31:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:31:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:31:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:31:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:31:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:31:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:31:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:31:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:31:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:31:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:31:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:31:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:31:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:31:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:31:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:31:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:31:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:31:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:31:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:31:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:31:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:31:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:31:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:31:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:31:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:31:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:31:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:31:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:31:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:31:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:31:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:31:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:31:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:31:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:31:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:31:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:31:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:31:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:31:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:31:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:31:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:31:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:31:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:31:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:31:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:31:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:31:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:31:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:31:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:31:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:31:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:31:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:31:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:31:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:31:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:31:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:31:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:31:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:31:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:31:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:31:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:31:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:31:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:31:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:31:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:31:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:31:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:31:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:31:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:31:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:31:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:32:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:32:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:32:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:32:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:32:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:32:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:32:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:32:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:32:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:32:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:32:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:32:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:32:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:32:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:32:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:32:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:32:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:32:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:32:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:32:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:32:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:32:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:32:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:32:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:32:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:32:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:32:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:32:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:32:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:32:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:32:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:32:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:32:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:32:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:32:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:32:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:32:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:32:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:32:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:32:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:32:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:32:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:32:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:32:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:32:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:32:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:32:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:32:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:32:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:32:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:32:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:32:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:32:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:32:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:32:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:32:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:32:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:32:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:32:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:32:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:32:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:32:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:32:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:32:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:32:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:32:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:32:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:32:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:32:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:32:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:32:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:32:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:32:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:32:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:32:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:32:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:32:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:32:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:32:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:32:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:32:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:32:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:32:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:32:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:32:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:32:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:32:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:32:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:32:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:32:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:32:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:32:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:32:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:32:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:32:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:32:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:32:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:32:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:32:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:32:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:32:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:32:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:32:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:32:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:32:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:32:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:32:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:32:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:32:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:32:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:32:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:32:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:32:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:32:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:32:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:32:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:32:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:32:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:32:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:32:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:32:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:32:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:32:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:32:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:32:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:32:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:32:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:32:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:32:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:32:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:32:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:32:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:32:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:32:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:32:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:32:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:32:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:32:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:32:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:32:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:32:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:32:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:32:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:32:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:32:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:32:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:32:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:32:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:32:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:32:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:32:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:32:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:32:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:32:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:32:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:32:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:32:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:32:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:32:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:32:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:32:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:32:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:32:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:32:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:32:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:32:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:32:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:32:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:32:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:32:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:32:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:32:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:32:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:32:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:32:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:32:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:32:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:32:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:32:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:32:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:32:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:32:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:32:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:32:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:32:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:32:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:32:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:32:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:32:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:32:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:32:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:32:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:32:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:32:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:32:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:32:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:32:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:32:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:32:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:32:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:32:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:32:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:32:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:32:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:32:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:32:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:32:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:32:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:32:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:32:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:32:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:32:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:32:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:32:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:32:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:32:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:33:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:33:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:33:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:33:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:33:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:33:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:33:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:33:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:33:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:33:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:33:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:33:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:33:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:33:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:33:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:33:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:33:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:33:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:33:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:33:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:33:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:33:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:33:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:33:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:33:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:33:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:33:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:33:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:33:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:33:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:33:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:33:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:33:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:33:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:33:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:33:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:33:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:33:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:33:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:33:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:33:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:33:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:33:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:33:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:33:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:33:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:33:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:33:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:33:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:33:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:33:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:33:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:33:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:33:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:33:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:33:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:33:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:33:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:33:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:33:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:33:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:33:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:33:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:33:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:33:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:33:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:33:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:33:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:33:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:33:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:33:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:33:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:33:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:33:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:33:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:33:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:33:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:33:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:33:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:33:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:33:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:33:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:33:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:33:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:33:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:33:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:33:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:33:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:33:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:33:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:33:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:33:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:33:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:33:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:33:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:33:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:33:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:33:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:33:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:33:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:33:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:33:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:33:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:33:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:33:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:33:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:33:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:33:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:33:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:33:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:33:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:33:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:33:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:33:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:33:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:33:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:33:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:33:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:33:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:33:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:33:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:33:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:33:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:33:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:33:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:33:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:33:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:33:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:33:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:33:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:33:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:33:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:33:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:33:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:33:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:33:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:33:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:33:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:33:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:33:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:33:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:33:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:33:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:33:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:33:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:33:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:33:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:33:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:33:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:33:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:33:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:33:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:33:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:33:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:33:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:33:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:33:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:33:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:33:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:33:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:33:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:33:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:33:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:33:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:33:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:33:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:33:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:33:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:33:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:33:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:33:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:33:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:33:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:33:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:33:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:33:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:33:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:33:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:33:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:33:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:33:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:33:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:33:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:33:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:33:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:33:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:33:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:33:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:33:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:33:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:33:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:33:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:33:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:33:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:33:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:33:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:33:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:33:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:33:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:33:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:33:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:33:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:33:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:33:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:33:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:33:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:33:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:33:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:33:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:33:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:33:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:33:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:33:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:33:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:33:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:33:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:34:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:34:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:34:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:34:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:34:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:34:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:34:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:34:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:34:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:34:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:34:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:34:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:34:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:34:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:34:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:34:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:34:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:34:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:34:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:34:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:34:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:34:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:34:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:34:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:34:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:34:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:34:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:34:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:34:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:34:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:34:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:34:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:34:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:34:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:34:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:34:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:34:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:34:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:34:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:34:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:34:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:34:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:34:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:34:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:34:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:34:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:34:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:34:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:34:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:34:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:34:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:34:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:34:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:34:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:34:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:34:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:34:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:34:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:34:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:34:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:34:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:34:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:34:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:34:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:34:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:34:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:34:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:34:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:34:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:34:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:34:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:34:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:34:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:34:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:34:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:34:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:34:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:34:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:34:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:34:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:34:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:34:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:34:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:34:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:34:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:34:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:34:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:34:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:34:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:34:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:34:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:34:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:34:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:34:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:34:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:34:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:34:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:34:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:34:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:34:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:34:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:34:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:34:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:34:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:34:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:34:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:34:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:34:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:36:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:36:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:36:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:36:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:36:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:36:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:36:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:36:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:36:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:36:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:36:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:36:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:36:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:36:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:36:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:36:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:36:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:36:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:36:35 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 12:36:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:36:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:36:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:36:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:36:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:36:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:36:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:36:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:36:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:36:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:36:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:36:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:36:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:36:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:36:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:36:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:36:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:36:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:36:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:36:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:36:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:36:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:36:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:36:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:36:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:36:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:36:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:36:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:36:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:36:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:36:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:36:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:36:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:36:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:36:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:36:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:36:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:36:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:36:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:36:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:36:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:36:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:36:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:36:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:36:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:36:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:36:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:36:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:36:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:36:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:36:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:36:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:36:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:36:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:36:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:36:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:36:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:36:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:36:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:36:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:36:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:36:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:36:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:36:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:36:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:36:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:36:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:36:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:36:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:36:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:36:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:36:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:37:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:37:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:37:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:37:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:37:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:37:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:37:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:37:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:37:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:37:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:37:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:37:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:37:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:37:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:37:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:37:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:37:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:37:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:37:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:37:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:37:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:37:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:37:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:37:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:37:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:37:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:37:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:37:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:37:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:37:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:37:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:37:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:37:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:37:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:37:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:37:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:37:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:37:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:37:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:37:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:37:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:37:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:37:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:37:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:37:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:37:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:37:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:37:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:37:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:37:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:37:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:37:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:37:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:37:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:37:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:37:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:37:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:37:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:37:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:37:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:37:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:37:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:37:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:37:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:37:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:37:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:37:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:37:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:37:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:37:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:37:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:37:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:37:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:37:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:37:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:37:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:37:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:37:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:37:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:37:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:37:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:37:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:37:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:37:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:37:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:37:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:37:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:37:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:37:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:37:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:37:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:37:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:37:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:37:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:37:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:37:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:37:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:37:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:37:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:37:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:37:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:37:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:37:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:37:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:37:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:37:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:37:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:37:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:37:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:37:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:37:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:37:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:37:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:37:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:37:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:37:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:37:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:37:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:37:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:37:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:37:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:37:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:37:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:37:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:37:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:37:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:37:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:37:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:37:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:37:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:37:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:37:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:37:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:37:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:37:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:37:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:37:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:37:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:37:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:37:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:37:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:37:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:37:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:37:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:37:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:37:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:37:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:37:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:37:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:37:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:37:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:37:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:37:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:37:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:37:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:37:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:37:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:37:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:37:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:37:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:37:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:37:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:37:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:37:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:37:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:37:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:37:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:37:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:37:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:37:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:37:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:37:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:37:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:37:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:37:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:37:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:37:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:37:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:37:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:37:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:37:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:37:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:37:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:37:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:37:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:37:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:37:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:37:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:37:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:37:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:37:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:37:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:37:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:37:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:37:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:37:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:37:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:37:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:37:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:37:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:37:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:37:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:37:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:37:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:37:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:37:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:37:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:37:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:37:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:37:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:37:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:37:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:37:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:37:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:37:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:37:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:38:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:38:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:38:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:38:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:38:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:38:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:38:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:38:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:38:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:38:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:38:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:38:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:38:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:38:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:38:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:38:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:38:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:38:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:38:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:38:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:38:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:38:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:38:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:38:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:38:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:38:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:38:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:38:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:38:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:38:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:38:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:38:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:38:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:38:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:38:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:38:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:38:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:38:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:38:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:38:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:38:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:38:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:38:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:38:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:38:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:38:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:38:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:38:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:38:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:38:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:38:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:38:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:38:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:38:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:38:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:38:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:38:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:38:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:38:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:38:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:38:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:38:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:38:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:38:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:38:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:38:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:38:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:38:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:38:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:38:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:38:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:38:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:38:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:38:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:38:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:38:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:38:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:38:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:38:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:38:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:38:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:38:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:38:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:38:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:38:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:38:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:38:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:38:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:38:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:38:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:38:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:38:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:38:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:38:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:38:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:38:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:38:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:38:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:38:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:38:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:38:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:38:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:38:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:38:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:38:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:38:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:38:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:38:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:38:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:38:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:38:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:38:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:38:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:38:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:38:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:38:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:38:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:38:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:38:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:38:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:38:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:38:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:38:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:38:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:38:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:38:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:38:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:38:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:38:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:38:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:38:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:38:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:38:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:38:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:38:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:38:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:38:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:38:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:38:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:38:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:38:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:38:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:38:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:38:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:38:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:38:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:38:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:38:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:38:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:38:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:38:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:38:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:38:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:38:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:38:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:38:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:38:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:38:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:38:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:38:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:38:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:38:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:38:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:38:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:38:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:38:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:38:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:38:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:38:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:38:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:38:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:38:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:38:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:38:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:38:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:38:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:38:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:38:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:38:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:38:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:38:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:38:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:38:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:38:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:38:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:38:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:38:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:38:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:38:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:38:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:38:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:38:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:38:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:38:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:38:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:38:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:38:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:38:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:38:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:38:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:38:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:38:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:38:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:38:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:38:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:38:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:38:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:38:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:38:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:38:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:38:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:38:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:38:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:38:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:38:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:38:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:39:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:39:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:39:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:39:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:39:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:39:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:39:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:39:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:39:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:39:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:39:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:39:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:39:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:39:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:39:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:39:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:39:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:39:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:39:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:39:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:39:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:39:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:39:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:39:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:39:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:39:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:39:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:39:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:39:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:39:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:39:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:39:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:39:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:39:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:39:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:39:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:39:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:39:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:39:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:39:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:39:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:39:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:39:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:39:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:39:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:39:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:39:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:39:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:39:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:39:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:39:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:39:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:39:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:39:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:39:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:39:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:39:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:39:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:39:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:39:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:39:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:39:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:39:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:39:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:39:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:39:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:39:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:39:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:39:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:39:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:39:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:39:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:39:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:39:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:39:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:39:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:39:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:39:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:39:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:39:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:39:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:39:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:39:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:39:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:39:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:39:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:39:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:39:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:39:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:39:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:39:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:39:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:39:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:39:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:39:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:39:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:39:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:39:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:39:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:39:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:39:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:39:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:39:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:39:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:39:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:39:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:39:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:39:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:39:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:39:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:39:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:39:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:39:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:39:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:39:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:39:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:39:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:39:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:39:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:39:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:39:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:39:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:39:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:39:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:39:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:39:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:39:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:39:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:39:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:39:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:39:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:39:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:39:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:39:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:39:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:39:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:39:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:39:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:39:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:39:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:39:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:39:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:39:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:39:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:39:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:39:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:39:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:39:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:39:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:39:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:39:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:39:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:39:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:39:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:39:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:39:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:39:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:39:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:39:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:39:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:39:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:39:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:39:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:39:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:39:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:39:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:39:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:39:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:39:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:39:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:39:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:39:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:39:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:39:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:39:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:39:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:39:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:39:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:39:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:39:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:39:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:39:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:39:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:39:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:39:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:39:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:39:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:39:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:39:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:39:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:39:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:39:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:39:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:39:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:39:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:39:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:39:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:39:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:39:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:39:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:39:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:39:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:39:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:39:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:39:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:39:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:39:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:39:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:39:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:39:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:39:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:39:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:39:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:39:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:39:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:39:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:40:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:40:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:40:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:40:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:40:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:40:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:40:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:40:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:40:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:40:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:40:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:40:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:40:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:40:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:40:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:40:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:40:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:40:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:40:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:40:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:40:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:40:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:40:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:40:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:40:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:40:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:40:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:40:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:40:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:40:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:40:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:40:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:40:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:40:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:40:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:40:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:40:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:40:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:40:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:40:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:40:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:40:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:40:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:40:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:40:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:40:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:40:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:40:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:40:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:40:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:40:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:40:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:40:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:40:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:40:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:40:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:40:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:40:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:40:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:40:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:40:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:40:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:40:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:40:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:40:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:40:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:40:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:40:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:40:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:40:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:40:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:40:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:40:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:40:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:40:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:40:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:40:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:40:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:40:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:40:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:40:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:40:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:40:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:40:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:40:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:40:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:40:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:40:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:40:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:40:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:40:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:40:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:40:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:40:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:40:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:40:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:40:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:40:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:40:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:40:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:40:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:40:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:40:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:40:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:40:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:40:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:40:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:40:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:40:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:40:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:40:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:40:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:40:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:40:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:40:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:40:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:40:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:40:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:40:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:40:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:40:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:40:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:40:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:40:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:40:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:40:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:40:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:40:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:40:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:40:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:40:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:40:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:40:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:40:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:40:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:40:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:40:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:40:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:40:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:40:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:40:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:40:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:40:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:40:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:40:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:40:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:40:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:40:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:40:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:40:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:40:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:40:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:40:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:40:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:40:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:40:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:40:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:40:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:40:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:40:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:40:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:40:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:40:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:40:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:40:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:40:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:40:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:40:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:40:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:40:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:40:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:40:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:40:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:40:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:40:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:40:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:40:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:40:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:40:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:40:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:40:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:40:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:40:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:40:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:40:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:40:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:40:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:40:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:40:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:40:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:40:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:40:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:40:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:40:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:40:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:40:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:40:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:40:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:40:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:40:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:40:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:40:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:40:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:40:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:40:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:40:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:40:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:40:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:40:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:40:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:40:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:40:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:40:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:40:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:40:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:40:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:41:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:41:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:41:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:41:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:41:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:41:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:41:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:41:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:41:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:41:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:41:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:41:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:41:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:41:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:41:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:41:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:41:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:41:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:41:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:41:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:41:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:41:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:41:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:41:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:41:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:41:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:41:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:41:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:41:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:41:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:41:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:41:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:41:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:41:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:41:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:41:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:41:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:41:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:41:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:41:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:41:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:41:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:41:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:41:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:41:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:41:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:41:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:41:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:41:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:41:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:41:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:41:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:41:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:41:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:41:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:41:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:41:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:41:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:41:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:41:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:41:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:41:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:41:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:41:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:41:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:41:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:41:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:41:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:41:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:41:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:41:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:41:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:41:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:41:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:41:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:41:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:41:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:41:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:41:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:41:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:41:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:41:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:41:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:41:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:41:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:41:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:41:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:41:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:41:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:41:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:41:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:41:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:41:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:41:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:41:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:41:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:41:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:41:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:41:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:41:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:41:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:41:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:41:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:41:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:41:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:41:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:41:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:41:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:41:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:41:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:41:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:41:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:41:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:41:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:41:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:41:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:41:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:41:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:41:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:41:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:41:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:41:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:41:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:41:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:41:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:41:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:41:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:41:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:41:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:41:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:41:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:41:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:41:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:41:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:41:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:41:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:41:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:41:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:41:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:41:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:41:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:41:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:41:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:41:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:41:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:41:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:41:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:41:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:41:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:41:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:41:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:41:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:41:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:41:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:41:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:41:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:41:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:41:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:41:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:41:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:41:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:41:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:41:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:41:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:41:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:41:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:41:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:41:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:41:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:41:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:41:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:41:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:41:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:41:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:41:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:41:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:41:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:41:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:41:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:41:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:41:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:41:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:41:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:41:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:41:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:41:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:41:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:41:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:41:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:41:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:41:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:41:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:41:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:41:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:41:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:41:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:41:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:41:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:41:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:41:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:41:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:41:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:41:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:41:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:41:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:41:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:41:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:41:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:41:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:41:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:41:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:41:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:41:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:41:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:41:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:41:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:42:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:42:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:42:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:42:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:42:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:42:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:42:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:42:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:42:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:42:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:42:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:42:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:42:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:42:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:42:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:42:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:42:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:42:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:42:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:42:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:42:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:42:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:42:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:42:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:42:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:42:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:42:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:42:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:42:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:42:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:42:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:42:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:42:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:42:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:42:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:42:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:42:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:42:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:42:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:42:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:42:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:42:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:42:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:42:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:42:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:42:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:42:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:42:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:42:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:42:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:42:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:42:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:42:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:42:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:42:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:42:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:42:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:42:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:42:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:42:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:42:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:42:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:42:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:42:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:42:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:42:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:42:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:42:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:42:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:42:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:42:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:42:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:42:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:42:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:42:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:42:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:42:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:42:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:42:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:42:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:42:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:42:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:42:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:42:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:42:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:42:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:42:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:42:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:42:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:42:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:42:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:42:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:42:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:42:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:42:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:42:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:42:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:42:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:42:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:42:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:42:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:42:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:42:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:42:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:42:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:42:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:42:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:42:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:42:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:42:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:42:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:42:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:42:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:42:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:42:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:42:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:42:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:42:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:42:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:42:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:42:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:42:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:42:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:42:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:42:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:42:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:42:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:42:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:42:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:42:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:42:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:42:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:42:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:42:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:42:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:42:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:42:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:42:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:42:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:42:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:42:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:42:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:42:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:42:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:42:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:42:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:42:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:42:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:42:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:42:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:42:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:42:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:42:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:42:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:42:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:42:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:42:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:42:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:42:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:42:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:42:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:42:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:42:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:42:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:42:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:42:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:42:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:42:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:42:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:42:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:42:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:42:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:42:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:42:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:42:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:42:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:42:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:42:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:42:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:42:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:42:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:42:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:42:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:42:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:42:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:42:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:42:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:42:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:42:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:42:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:42:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:42:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:42:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:42:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:42:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:42:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:42:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:42:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:42:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:42:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:42:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:42:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:42:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:42:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:42:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:42:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:42:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:42:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:42:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:42:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:42:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:42:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:42:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:42:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:42:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:42:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:43:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:43:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:43:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:43:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:43:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:43:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:43:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:43:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:43:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:43:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:43:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:43:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:43:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:43:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:43:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:43:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:43:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:43:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:43:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:43:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:43:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:43:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:43:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:43:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:43:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:43:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:43:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:43:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:43:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:43:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:43:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:43:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:43:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:43:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:43:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:43:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:43:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:43:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:43:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:43:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:43:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:43:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:43:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:43:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:43:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:43:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:43:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:43:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:43:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:43:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:43:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:43:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:43:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:43:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:43:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:43:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:43:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:43:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:43:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:43:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:43:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:43:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:43:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:43:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:43:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:43:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:43:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:43:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:43:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:43:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:43:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:43:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:43:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:43:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:43:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:43:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:43:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:43:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:43:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:43:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:43:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:43:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:43:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:43:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:43:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:43:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:43:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:43:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:43:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:43:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:43:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:43:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:43:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:43:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:43:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:43:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:43:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:43:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:43:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:43:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:43:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:43:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:43:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:43:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:43:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:43:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:43:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:43:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:43:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:43:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:43:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:43:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:43:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:43:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:43:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:43:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:43:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:43:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:43:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:43:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:43:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:43:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:43:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:43:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:43:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:43:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:43:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:43:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:43:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:43:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:43:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:43:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:43:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:43:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:43:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:43:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:43:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:43:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:43:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:43:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:43:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:43:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:43:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:43:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:43:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:43:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:43:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:43:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:43:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:43:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:43:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:43:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:43:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:43:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:43:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:43:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:43:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:43:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:43:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:43:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:43:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:43:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:43:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:43:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:43:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:43:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:43:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:43:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:43:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:43:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:43:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:43:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:43:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:43:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:43:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:43:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:43:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:43:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:43:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:43:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:43:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:43:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:43:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:43:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:43:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:43:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:43:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:43:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:43:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:43:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:43:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:43:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:43:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:43:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:43:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:43:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:43:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:43:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:43:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:43:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:43:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:43:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:43:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:43:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:43:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:43:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:43:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:43:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:43:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:43:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:43:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:43:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:43:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:43:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:43:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:43:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:44:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:44:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:44:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:44:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:44:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:44:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:44:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:44:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:44:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:44:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:44:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:44:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:44:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:44:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:44:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:44:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:44:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:44:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:44:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:44:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:44:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:44:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:44:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:44:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:44:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:44:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:44:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:44:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:44:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:44:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:44:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:44:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:44:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:44:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:44:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:44:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:44:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:44:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:44:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:44:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:44:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:44:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:44:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:44:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:44:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:44:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:44:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:44:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:44:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:44:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:44:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:44:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:44:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:44:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:44:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:44:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:44:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:44:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:44:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:44:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:44:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:44:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:44:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:44:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:44:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:44:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:44:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:44:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:44:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:44:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:44:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:44:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:44:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:44:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:44:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:44:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:44:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:44:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:44:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:44:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:44:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:44:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:44:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:44:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:44:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:44:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:44:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:44:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:44:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:44:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:44:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:44:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:44:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:44:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:44:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:44:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:44:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:44:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:44:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:44:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:44:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:44:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:44:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:44:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:44:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:44:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:44:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:44:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:44:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:44:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:44:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:44:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:44:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:44:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:44:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:44:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:44:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:44:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:44:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:44:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:44:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:44:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:44:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:44:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:44:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:44:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:46:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:46:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:46:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:46:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:46:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:46:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:46:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:46:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:46:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:46:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:46:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:46:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:46:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:46:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:46:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:46:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:46:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:46:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:46:36 rdma06 rsyslogd[1041]: imjournal: 5082 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 12:46:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:46:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:46:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:46:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:46:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:46:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:46:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:46:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:46:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:46:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:46:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:46:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:46:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:46:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:46:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:46:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:46:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:46:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:46:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:46:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:46:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:46:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:46:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:46:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:46:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:46:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:46:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:46:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:46:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:46:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:46:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:46:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:46:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:46:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:46:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:46:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:46:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:46:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:46:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:46:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:46:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:46:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:46:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:46:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:46:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:46:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:46:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:46:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:46:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:46:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:46:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:46:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:46:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:46:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:46:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:46:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:46:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:46:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:46:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:46:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:46:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:46:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:46:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:46:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:46:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:46:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:46:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:46:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:46:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:46:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:46:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:46:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:47:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:47:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:47:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:47:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:47:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:47:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:47:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:47:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:47:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:47:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:47:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:47:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:47:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:47:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:47:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:47:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:47:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:47:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:47:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:47:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:47:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:47:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:47:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:47:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:47:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:47:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:47:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:47:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:47:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:47:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:47:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:47:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:47:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:47:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:47:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:47:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:47:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:47:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:47:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:47:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:47:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:47:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:47:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:47:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:47:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:47:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:47:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:47:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:47:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:47:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:47:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:47:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:47:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:47:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:47:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:47:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:47:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:47:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:47:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:47:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:47:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:47:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:47:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:47:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:47:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:47:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:47:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:47:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:47:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:47:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:47:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:47:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:47:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:47:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:47:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:47:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:47:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:47:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:47:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:47:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:47:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:47:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:47:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:47:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:47:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:47:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:47:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:47:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:47:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:47:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:47:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:47:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:47:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:47:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:47:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:47:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:47:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:47:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:47:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:47:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:47:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:47:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:47:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:47:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:47:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:47:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:47:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:47:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:47:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:47:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:47:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:47:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:47:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:47:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:47:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:47:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:47:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:47:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:47:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:47:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:47:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:47:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:47:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:47:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:47:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:47:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:47:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:47:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:47:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:47:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:47:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:47:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:47:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:47:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:47:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:47:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:47:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:47:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:47:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:47:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:47:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:47:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:47:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:47:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:47:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:47:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:47:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:47:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:47:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:47:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:47:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:47:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:47:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:47:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:47:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:47:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:47:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:47:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:47:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:47:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:47:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:47:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:47:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:47:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:47:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:47:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:47:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:47:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:47:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:47:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:47:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:47:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:47:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:47:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:47:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:47:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:47:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:47:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:47:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:47:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:47:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:47:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:47:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:47:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:47:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:47:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:47:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:47:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:47:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:47:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:47:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:47:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:47:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:47:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:47:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:47:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:47:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:47:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:47:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:47:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:47:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:47:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:47:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:47:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:47:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:47:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:47:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:47:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:47:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:47:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:47:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:47:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:47:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:47:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:47:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:47:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:48:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:48:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:48:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:48:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:48:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:48:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:48:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:48:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:48:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:48:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:48:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:48:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:48:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:48:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:48:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:48:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:48:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:48:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:48:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:48:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:48:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:48:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:48:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:48:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:48:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:48:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:48:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:48:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:48:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:48:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:48:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:48:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:48:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:48:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:48:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:48:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:48:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:48:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:48:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:48:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:48:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:48:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:48:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:48:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:48:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:48:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:48:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:48:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:48:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:48:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:48:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:48:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:48:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:48:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:48:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:48:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:48:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:48:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:48:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:48:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:48:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:48:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:48:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:48:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:48:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:48:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:48:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:48:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:48:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:48:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:48:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:48:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:48:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:48:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:48:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:48:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:48:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:48:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:48:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:48:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:48:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:48:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:48:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:48:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:48:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:48:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:48:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:48:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:48:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:48:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:48:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:48:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:48:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:48:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:48:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:48:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:48:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:48:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:48:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:48:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:48:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:48:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:48:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:48:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:48:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:48:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:48:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:48:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:48:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:48:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:48:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:48:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:48:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:48:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:48:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:48:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:48:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:48:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:48:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:48:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:48:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:48:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:48:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:48:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:48:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:48:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:48:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:48:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:48:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:48:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:48:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:48:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:48:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:48:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:48:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:48:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:48:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:48:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:48:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:48:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:48:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:48:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:48:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:48:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:48:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:48:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:48:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:48:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:48:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:48:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:48:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:48:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:48:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:48:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:48:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:48:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:48:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:48:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:48:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:48:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:48:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:48:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:48:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:48:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:48:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:48:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:48:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:48:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:48:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:48:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:48:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:48:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:48:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:48:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:48:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:48:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:48:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:48:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:48:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:48:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:48:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:48:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:48:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:48:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:48:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:48:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:48:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:48:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:48:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:48:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:48:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:48:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:48:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:48:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:48:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:48:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:48:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:48:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:48:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:48:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:48:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:48:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:48:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:48:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:48:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:48:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:48:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:48:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:48:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:48:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:48:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:48:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:48:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:48:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:48:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:48:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:49:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:49:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:49:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:49:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:49:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:49:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:49:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:49:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:49:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:49:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:49:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:49:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:49:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:49:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:49:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:49:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:49:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:49:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:49:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:49:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:49:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:49:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:49:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:49:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:49:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:49:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:49:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:49:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:49:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:49:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:49:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:49:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:49:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:49:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:49:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:49:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:49:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:49:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:49:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:49:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:49:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:49:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:49:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:49:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:49:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:49:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:49:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:49:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:49:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:49:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:49:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:49:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:49:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:49:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:49:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:49:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:49:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:49:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:49:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:49:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:49:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:49:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:49:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:49:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:49:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:49:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:49:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:49:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:49:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:49:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:49:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:49:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:49:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:49:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:49:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:49:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:49:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:49:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:49:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:49:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:49:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:49:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:49:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:49:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:49:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:49:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:49:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:49:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:49:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:49:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:49:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:49:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:49:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:49:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:49:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:49:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:49:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:49:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:49:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:49:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:49:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:49:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:49:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:49:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:49:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:49:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:49:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:49:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:49:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:49:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:49:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:49:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:49:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:49:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:49:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:49:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:49:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:49:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:49:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:49:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:49:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:49:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:49:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:49:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:49:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:49:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:49:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:49:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:49:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:49:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:49:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:49:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:49:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:49:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:49:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:49:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:49:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:49:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:49:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:49:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:49:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:49:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:49:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:49:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:49:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:49:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:49:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:49:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:49:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:49:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:49:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:49:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:49:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:49:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:49:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:49:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:49:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:49:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:49:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:49:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:49:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:49:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:49:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:49:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:49:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:49:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:49:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:49:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:49:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:49:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:49:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:49:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:49:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:49:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:49:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:49:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:49:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:49:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:49:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:49:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:49:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:49:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:49:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:49:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:49:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:49:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:49:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:49:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:49:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:49:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:49:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:49:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:49:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:49:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:49:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:49:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:49:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:49:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:49:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:49:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:49:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:49:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:49:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:49:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:49:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:49:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:49:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:49:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:49:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:49:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:49:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:49:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:49:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:49:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:49:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:49:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:50:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:50:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:50:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:50:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:50:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:50:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:50:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:50:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:50:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:50:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:50:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:50:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:50:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:50:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:50:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:50:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:50:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:50:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:50:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:50:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:50:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:50:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:50:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:50:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:50:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:50:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:50:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:50:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:50:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:50:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:50:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:50:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:50:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:50:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:50:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:50:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:50:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:50:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:50:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:50:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:50:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:50:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:50:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:50:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:50:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:50:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:50:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:50:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:50:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:50:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:50:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:50:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:50:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:50:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:50:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:50:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:50:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:50:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:50:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:50:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:50:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:50:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:50:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:50:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:50:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:50:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:50:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:50:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:50:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:50:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:50:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:50:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:50:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:50:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:50:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:50:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:50:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:50:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:50:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:50:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:50:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:50:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:50:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:50:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:50:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:50:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:50:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:50:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:50:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:50:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:50:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:50:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:50:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:50:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:50:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:50:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:50:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:50:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:50:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:50:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:50:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:50:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:50:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:50:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:50:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:50:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:50:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:50:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:50:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:50:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:50:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:50:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:50:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:50:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:50:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:50:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:50:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:50:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:50:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:50:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:50:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:50:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:50:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:50:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:50:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:50:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:50:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:50:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:50:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:50:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:50:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:50:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:50:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:50:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:50:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:50:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:50:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:50:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:50:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:50:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:50:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:50:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:50:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:50:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:50:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:50:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:50:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:50:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:50:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:50:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:50:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:50:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:50:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:50:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:50:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:50:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:50:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:50:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:50:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:50:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:50:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:50:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:50:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:50:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:50:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:50:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:50:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:50:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:50:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:50:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:50:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:50:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:50:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:50:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:50:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:50:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:50:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:50:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:50:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:50:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:50:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:50:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:50:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:50:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:50:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:50:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:50:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:50:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:50:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:50:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:50:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:50:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:50:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:50:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:50:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:50:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:50:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:50:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:50:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:50:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:50:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:50:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:50:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:50:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:50:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:50:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:50:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:50:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:50:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:50:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:50:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:50:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:50:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:50:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:50:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:50:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:51:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:51:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:51:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:51:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:51:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:51:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:51:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:51:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:51:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:51:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:51:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:51:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:51:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:51:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:51:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:51:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:51:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:51:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:51:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:51:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:51:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:51:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:51:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:51:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:51:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:51:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:51:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:51:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:51:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:51:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:51:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:51:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:51:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:51:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:51:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:51:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:51:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:51:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:51:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:51:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:51:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:51:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:51:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:51:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:51:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:51:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:51:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:51:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:51:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:51:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:51:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:51:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:51:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:51:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:51:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:51:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:51:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:51:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:51:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:51:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:51:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:51:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:51:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:51:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:51:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:51:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:51:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:51:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:51:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:51:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:51:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:51:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:51:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:51:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:51:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:51:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:51:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:51:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:51:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:51:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:51:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:51:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:51:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:51:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:51:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:51:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:51:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:51:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:51:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:51:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:51:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:51:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:51:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:51:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:51:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:51:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:51:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:51:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:51:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:51:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:51:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:51:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:51:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:51:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:51:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:51:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:51:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:51:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:51:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:51:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:51:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:51:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:51:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:51:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:51:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:51:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:51:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:51:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:51:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:51:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:51:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:51:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:51:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:51:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:51:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:51:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:51:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:51:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:51:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:51:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:51:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:51:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:51:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:51:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:51:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:51:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:51:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:51:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:51:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:51:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:51:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:51:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:51:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:51:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:51:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:51:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:51:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:51:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:51:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:51:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:51:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:51:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:51:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:51:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:51:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:51:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:51:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:51:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:51:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:51:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:51:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:51:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:51:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:51:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:51:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:51:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:51:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:51:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:51:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:51:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:51:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:51:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:51:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:51:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:51:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:51:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:51:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:51:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:51:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:51:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:51:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:51:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:51:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:51:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:51:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:51:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:52:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:52:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:52:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:52:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:52:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:52:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:52:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:52:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:52:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:52:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:52:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:52:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:52:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:52:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:52:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:52:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:52:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:52:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:52:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:52:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:52:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:52:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:52:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:52:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:52:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:52:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:52:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:52:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:52:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:52:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:52:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:52:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:52:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:52:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:52:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:52:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:52:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:52:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:52:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:52:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:52:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:52:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:52:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:52:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:52:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:52:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:52:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:52:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:52:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:52:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:52:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:52:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:52:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:52:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:52:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:52:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:52:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:52:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:52:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:52:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:52:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:52:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:52:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:52:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:52:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:52:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:52:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:52:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:52:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:52:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:52:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:52:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:52:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:52:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:52:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:52:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:52:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:52:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:52:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:52:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:52:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:52:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:52:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:52:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:52:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:52:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:52:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:52:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:52:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:52:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:52:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:52:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:52:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:52:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:52:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:52:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:52:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:52:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:52:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:52:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:52:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:52:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:52:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:52:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:52:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:52:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:52:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:52:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:52:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:52:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:52:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:52:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:52:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:52:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:52:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:52:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:52:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:52:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:52:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:52:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:52:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:52:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:52:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:52:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:52:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:52:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:52:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:52:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:52:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:52:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:52:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:52:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:52:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:52:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:52:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:52:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:52:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:52:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:52:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:52:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:52:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:52:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:52:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:52:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:52:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:52:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:52:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:52:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:52:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:52:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:52:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:52:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:52:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:52:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:52:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:52:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:52:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:52:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:52:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:52:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:52:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:52:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:52:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:52:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:52:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:52:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:52:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:52:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:52:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:52:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:52:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:52:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:52:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:52:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:52:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:52:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:52:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:52:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:52:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:52:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:52:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:52:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:52:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:52:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:52:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:52:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:52:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:52:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:52:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:52:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:52:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:52:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:52:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:52:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:52:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:52:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:52:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:52:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:52:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:52:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:52:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:52:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:52:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:52:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:52:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:52:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:52:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:52:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:52:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:52:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:52:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:52:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:52:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:52:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:52:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:52:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:53:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:53:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:53:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:53:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:53:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:53:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:53:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:53:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:53:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:53:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:53:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:53:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:53:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:53:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:53:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:53:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:53:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:53:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:53:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:53:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:53:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:53:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:53:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:53:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:53:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:53:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:53:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:53:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:53:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:53:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:53:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:53:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:53:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:53:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:53:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:53:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:53:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:53:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:53:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:53:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:53:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:53:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:53:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:53:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:53:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:53:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:53:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:53:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:53:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:53:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:53:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:53:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:53:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:53:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:53:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:53:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:53:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:53:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:53:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:53:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:53:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:53:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:53:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:53:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:53:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:53:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:53:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:53:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:53:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:53:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:53:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:53:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:53:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:53:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:53:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:53:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:53:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:53:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:53:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:53:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:53:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:53:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:53:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:53:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:53:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:53:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:53:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:53:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:53:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:53:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:53:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:53:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:53:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:53:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:53:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:53:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:53:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:53:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:53:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:53:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:53:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:53:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:53:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:53:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:53:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:53:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:53:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:53:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:53:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:53:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:53:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:53:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:53:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:53:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:53:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:53:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:53:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:53:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:53:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:53:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:53:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:53:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:53:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:53:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:53:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:53:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:53:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:53:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:53:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:53:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:53:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:53:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:53:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:53:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:53:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:53:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:53:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:53:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:53:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:53:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:53:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:53:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:53:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:53:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:53:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:53:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:53:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:53:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:53:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:53:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:53:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:53:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:53:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:53:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:53:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:53:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:53:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:53:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:53:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:53:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:53:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:53:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:53:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:53:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:53:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:53:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:53:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:53:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:53:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:53:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:53:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:53:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:53:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:53:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:53:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:53:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:53:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:53:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:53:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:53:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:53:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:53:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:53:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:53:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:53:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:53:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:53:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:53:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:53:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:53:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:53:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:53:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:53:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:53:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:53:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:53:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:53:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:53:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:53:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:53:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:53:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:53:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:53:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:53:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:53:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:53:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:53:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:53:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:53:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:53:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:53:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:53:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:53:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:53:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:53:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:53:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:54:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:54:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:54:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:54:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:54:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:54:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:54:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:54:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:54:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:54:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:54:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:54:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:54:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:54:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:54:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:54:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:54:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:54:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:54:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:54:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:54:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:54:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:54:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:54:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:54:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:54:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:54:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:54:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:54:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:54:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:54:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:54:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:54:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:54:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:54:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:54:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:54:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:54:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:54:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:54:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:54:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:54:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:54:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:54:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:54:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:54:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:54:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:54:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:54:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:54:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:54:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:54:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:54:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:54:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:54:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:54:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:54:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:54:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:54:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:54:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:54:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:54:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:54:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:54:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:54:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:54:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:54:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:54:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:54:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:54:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:54:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:54:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:54:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:54:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:54:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:54:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:54:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:54:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:54:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:54:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:54:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:54:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:54:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:54:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:54:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:54:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:54:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:54:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:54:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:54:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:54:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:54:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:54:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:54:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:54:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:54:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:54:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:54:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:54:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:54:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:54:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:54:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:54:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:54:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:54:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:54:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:54:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:54:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:54:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:54:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:54:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:54:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:54:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:54:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:54:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:54:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:54:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:54:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:54:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:54:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:54:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:54:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:54:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:54:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:54:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:54:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:56:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:56:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:56:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:56:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:56:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:56:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:56:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:56:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:56:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:56:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:56:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:56:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:56:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:56:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:56:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:56:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:56:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:56:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:56:41 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 12:56:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:56:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:56:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:56:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:56:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:56:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:56:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:56:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:56:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:56:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:56:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:56:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:56:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:56:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:56:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:56:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:56:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:56:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:56:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:56:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:56:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:56:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:56:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:56:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:56:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:56:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:56:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:56:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:56:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:56:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:56:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:56:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:56:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:56:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:56:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:56:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:56:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:56:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:56:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:56:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:56:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:56:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:56:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:56:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:56:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:56:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:56:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:56:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:56:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:56:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:56:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:56:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:56:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:56:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:57:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:57:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:57:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:57:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:57:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:57:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:57:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:57:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:57:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:57:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:57:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:57:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:57:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:57:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:57:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:57:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:57:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:57:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:57:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:57:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:57:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:57:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:57:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:57:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:57:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:57:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:57:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:57:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:57:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:57:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:57:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:57:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:57:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:57:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:57:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:57:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:57:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:57:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:57:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:57:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:57:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:57:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:57:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:57:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:57:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:57:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:57:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:57:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:57:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:57:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:57:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:57:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:57:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:57:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:57:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:57:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:57:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:57:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:57:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:57:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:57:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:57:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:57:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:57:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:57:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:57:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:57:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:57:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:57:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:57:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:57:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:57:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:57:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:57:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:57:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:57:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:57:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:57:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:57:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:57:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:57:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:57:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:57:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:57:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:57:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:57:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:57:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:57:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:57:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:57:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:57:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:57:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:57:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:57:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:57:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:57:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:57:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:57:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:57:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:57:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:57:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:57:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:57:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:57:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:57:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:57:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:57:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:57:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:57:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:57:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:57:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:57:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:57:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:57:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:57:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:57:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:57:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:57:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:57:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:57:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:57:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:57:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:57:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:57:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:57:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:57:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:57:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:57:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:57:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:57:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:57:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:57:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:57:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:57:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:57:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:57:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:57:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:57:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:57:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:57:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:57:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:57:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:57:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:57:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:57:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:57:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:57:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:57:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:57:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:57:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:57:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:57:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:57:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:57:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:57:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:57:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:57:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:57:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:57:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:57:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:57:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:57:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:57:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:57:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:57:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:57:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:57:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:57:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:57:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:57:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:57:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:57:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:57:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:57:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:57:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:57:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:57:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:57:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:57:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:57:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:57:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:57:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:57:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:57:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:57:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:57:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:57:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:57:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:57:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:57:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:57:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:57:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:57:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:57:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:57:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:57:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:57:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:57:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:57:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:57:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:57:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:57:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:57:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:57:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:57:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:57:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:57:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:57:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:57:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:57:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:57:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:57:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:57:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:57:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:57:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:57:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:58:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:58:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:58:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:58:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:58:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:58:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:58:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:58:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:58:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:58:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:58:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:58:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:58:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:58:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:58:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:58:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:58:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:58:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:58:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:58:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:58:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:58:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:58:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:58:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:58:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:58:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:58:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:58:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:58:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:58:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:58:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:58:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:58:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:58:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:58:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:58:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:58:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:58:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:58:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:58:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:58:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:58:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:58:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:58:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:58:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:58:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:58:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:58:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:58:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:58:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:58:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:58:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:58:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:58:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:58:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:58:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:58:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:58:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:58:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:58:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:58:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:58:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:58:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:58:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:58:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:58:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:58:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:58:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:58:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:58:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:58:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:58:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:58:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:58:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:58:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:58:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:58:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:58:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:58:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:58:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:58:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:58:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:58:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:58:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:58:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:58:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:58:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:58:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:58:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:58:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:58:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:58:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:58:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:58:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:58:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:58:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:58:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:58:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:58:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:58:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:58:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:58:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:58:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:58:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:58:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:58:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:58:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:58:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:58:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:58:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:58:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:58:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:58:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:58:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:58:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:58:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:58:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:58:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:58:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:58:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:58:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:58:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:58:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:58:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:58:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:58:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:58:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:58:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:58:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:58:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:58:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:58:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:58:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:58:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:58:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:58:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:58:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:58:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:58:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:58:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:58:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:58:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:58:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:58:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:58:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:58:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:58:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:58:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:58:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:58:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:58:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:58:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:58:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:58:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:58:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:58:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:58:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:58:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:58:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:58:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:58:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:58:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:58:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:58:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:58:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:58:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:58:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:58:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:58:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:58:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:58:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:58:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:58:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:58:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:58:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:58:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:58:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:58:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:58:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:58:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:58:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:58:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:58:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:58:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:58:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:58:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:58:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:58:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:58:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:58:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:58:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:58:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:58:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:58:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:58:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:58:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:58:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:58:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:58:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:58:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:58:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:58:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:58:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:58:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:58:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:58:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:58:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:58:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:58:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:58:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:58:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:58:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:58:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:58:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:58:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:58:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:59:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:59:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:59:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:59:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:59:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:59:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:59:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:59:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:59:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:59:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:59:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:59:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:59:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:59:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:59:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:59:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:59:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:59:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:59:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:59:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:59:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:59:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:59:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:59:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:59:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:59:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:59:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:59:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:59:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:59:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:59:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:59:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:59:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:59:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:59:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:59:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:59:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:59:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:59:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:59:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:59:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:59:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:59:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:59:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:59:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:59:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:59:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:59:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:59:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:59:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:59:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:59:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:59:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:59:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:59:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:59:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:59:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:59:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:59:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:59:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:59:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:59:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:59:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:59:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:59:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:59:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:59:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:59:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:59:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:59:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:59:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:59:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:59:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:59:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:59:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:59:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:59:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:59:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:59:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:59:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:59:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:59:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:59:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:59:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:59:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:59:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:59:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:59:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:59:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:59:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:59:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:59:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:59:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:59:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:59:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:59:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:59:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:59:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:59:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:59:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:59:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:59:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:59:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:59:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:59:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:59:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:59:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:59:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:59:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:59:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:59:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:59:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:59:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:59:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:59:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:59:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:59:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:59:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:59:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:59:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:59:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:59:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:59:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:59:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:59:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:59:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:59:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:59:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:59:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:59:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:59:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:59:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:59:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:59:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:59:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:59:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:59:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:59:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:59:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:59:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:59:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:59:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:59:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:59:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:59:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:59:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:59:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:59:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:59:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:59:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:59:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:59:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:59:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:59:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:59:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:59:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:59:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:59:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:59:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:59:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:59:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:59:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:59:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:59:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:59:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:59:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:59:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:59:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:59:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:59:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:59:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:59:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:59:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:59:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:59:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:59:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:59:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:59:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:59:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:59:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:59:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:59:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:59:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:59:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:59:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:59:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:59:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:59:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:59:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:59:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:59:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:59:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:59:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:59:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:59:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:59:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:59:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:59:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 12:59:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 12:59:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:59:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:59:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:59:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 12:59:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 12:59:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 12:59:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 12:59:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 12:59:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 12:59:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 12:59:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 12:59:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 12:59:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 12:59:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 12:59:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 12:59:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 12:59:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:00:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:00:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:00:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:00:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:00:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:00:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:00:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:00:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:00:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:00:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:00:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:00:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:00:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:00:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:00:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:00:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:00:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:00:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:00:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:00:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:00:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:00:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:00:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:00:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:00:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:00:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:00:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:00:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:00:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:00:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:00:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:00:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:00:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:00:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:00:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:00:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:00:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:00:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:00:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:00:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:00:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:00:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:00:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:00:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:00:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:00:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:00:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:00:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:00:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:00:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:00:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:00:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:00:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:00:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:00:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:00:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:00:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:00:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:00:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:00:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:00:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:00:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:00:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:00:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:00:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:00:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:00:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:00:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:00:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:00:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:00:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:00:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:00:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:00:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:00:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:00:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:00:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:00:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:00:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:00:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:00:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:00:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:00:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:00:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:00:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:00:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:00:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:00:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:00:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:00:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:00:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:00:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:00:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:00:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:00:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:00:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:00:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:00:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:00:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:00:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:00:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:00:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:00:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:00:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:00:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:00:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:00:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:00:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:00:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:00:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:00:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:00:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:00:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:00:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:00:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:00:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:00:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:00:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:00:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:00:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:00:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:00:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:00:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:00:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:00:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:00:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:00:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:00:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:00:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:00:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:00:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:00:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:00:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:00:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:00:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:00:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:00:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:00:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:00:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:00:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:00:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:00:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:00:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:00:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:00:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:00:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:00:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:00:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:00:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:00:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:00:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:00:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:00:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:00:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:00:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:00:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:00:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:00:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:00:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:00:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:00:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:00:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:00:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:00:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:00:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:00:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:00:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:00:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:00:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:00:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:00:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:00:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:00:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:00:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:00:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:00:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:00:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:00:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:00:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:00:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:00:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:00:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:00:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:00:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:00:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:00:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:00:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:00:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:00:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:00:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:00:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:00:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:00:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:00:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:00:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:00:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:00:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:00:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:00:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:00:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:00:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:00:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:00:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:00:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:00:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:00:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:00:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:00:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:00:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:00:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:00:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:00:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:00:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:00:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:00:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:00:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:01:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:01:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:01:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:01:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:01:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:01:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:01:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:01:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:01:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:01:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:01:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:01:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:01:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:01:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:01:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:01:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:01:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:01:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:01:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:01:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:01:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:01:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:01:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:01:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:01:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:01:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:01:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:01:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:01:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:01:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:01:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:01:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:01:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:01:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:01:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:01:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:01:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:01:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:01:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:01:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:01:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:01:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:01:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:01:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:01:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:01:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:01:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:01:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:01:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:01:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:01:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:01:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:01:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:01:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:01:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:01:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:01:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:01:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:01:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:01:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:01:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:01:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:01:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:01:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:01:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:01:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:01:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:01:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:01:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:01:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:01:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:01:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:01:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:01:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:01:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:01:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:01:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:01:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:01:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:01:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:01:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:01:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:01:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:01:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:01:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:01:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:01:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:01:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:01:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:01:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:01:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:01:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:01:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:01:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:01:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:01:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:01:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:01:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:01:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:01:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:01:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:01:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:01:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:01:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:01:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:01:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:01:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:01:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:01:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:01:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:01:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:01:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:01:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:01:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:01:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:01:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:01:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:01:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:01:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:01:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:01:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:01:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:01:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:01:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:01:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:01:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:01:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:01:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:01:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:01:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:01:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:01:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:01:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:01:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:01:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:01:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:01:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:01:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:01:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:01:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:01:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:01:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:01:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:01:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:01:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:01:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:01:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:01:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:01:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:01:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:01:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:01:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:01:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:01:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:01:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:01:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:01:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:01:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:01:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:01:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:01:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:01:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:01:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:01:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:01:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:01:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:01:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:01:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:01:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:01:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:01:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:01:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:01:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:01:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:01:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:01:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:01:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:01:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:01:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:01:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:01:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:01:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:01:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:01:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:01:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:01:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:01:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:01:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:01:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:01:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:01:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:01:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:01:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:01:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:01:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:01:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:01:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:01:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:01:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:01:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:01:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:01:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:01:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:01:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:01:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:01:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:01:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:01:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:01:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:01:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:01:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:01:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:01:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:01:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:01:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:01:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:02:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:02:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:02:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:02:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:02:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:02:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:02:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:02:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:02:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:02:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:02:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:02:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:02:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:02:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:02:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:02:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:02:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:02:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:02:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:02:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:02:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:02:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:02:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:02:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:02:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:02:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:02:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:02:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:02:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:02:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:02:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:02:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:02:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:02:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:02:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:02:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:02:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:02:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:02:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:02:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:02:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:02:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:02:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:02:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:02:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:02:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:02:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:02:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:02:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:02:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:02:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:02:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:02:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:02:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:02:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:02:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:02:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:02:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:02:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:02:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:02:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:02:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:02:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:02:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:02:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:02:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:02:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:02:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:02:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:02:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:02:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:02:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:02:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:02:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:02:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:02:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:02:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:02:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:02:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:02:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:02:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:02:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:02:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:02:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:02:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:02:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:02:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:02:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:02:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:02:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:02:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:02:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:02:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:02:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:02:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:02:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:02:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:02:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:02:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:02:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:02:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:02:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:02:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:02:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:02:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:02:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:02:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:02:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:02:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:02:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:02:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:02:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:02:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:02:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:02:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:02:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:02:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:02:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:02:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:02:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:02:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:02:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:02:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:02:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:02:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:02:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:02:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:02:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:02:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:02:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:02:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:02:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:02:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:02:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:02:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:02:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:02:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:02:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:02:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:02:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:02:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:02:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:02:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:02:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:02:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:02:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:02:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:02:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:02:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:02:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:02:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:02:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:02:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:02:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:02:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:02:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:02:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:02:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:02:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:02:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:02:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:02:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:02:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:02:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:02:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:02:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:02:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:02:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:02:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:02:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:02:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:02:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:02:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:02:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:02:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:02:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:02:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:02:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:02:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:02:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:02:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:02:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:02:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:02:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:02:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:02:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:02:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:02:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:02:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:02:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:02:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:02:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:02:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:02:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:02:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:02:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:02:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:02:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:02:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:02:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:02:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:02:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:02:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:02:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:02:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:02:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:02:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:02:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:02:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:02:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:02:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:02:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:02:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:02:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:02:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:02:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:03:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:03:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:03:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:03:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:03:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:03:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:03:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:03:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:03:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:03:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:03:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:03:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:03:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:03:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:03:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:03:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:03:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:03:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:03:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:03:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:03:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:03:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:03:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:03:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:03:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:03:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:03:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:03:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:03:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:03:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:03:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:03:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:03:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:03:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:03:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:03:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:03:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:03:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:03:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:03:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:03:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:03:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:03:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:03:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:03:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:03:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:03:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:03:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:03:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:03:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:03:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:03:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:03:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:03:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:03:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:03:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:03:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:03:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:03:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:03:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:03:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:03:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:03:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:03:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:03:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:03:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:03:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:03:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:03:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:03:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:03:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:03:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:03:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:03:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:03:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:03:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:03:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:03:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:03:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:03:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:03:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:03:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:03:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:03:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:03:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:03:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:03:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:03:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:03:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:03:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:03:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:03:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:03:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:03:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:03:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:03:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:03:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:03:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:03:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:03:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:03:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:03:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:03:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:03:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:03:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:03:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:03:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:03:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:03:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:03:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:03:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:03:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:03:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:03:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:03:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:03:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:03:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:03:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:03:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:03:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:03:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:03:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:03:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:03:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:03:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:03:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:03:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:03:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:03:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:03:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:03:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:03:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:03:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:03:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:03:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:03:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:03:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:03:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:03:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:03:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:03:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:03:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:03:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:03:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:03:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:03:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:03:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:03:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:03:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:03:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:03:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:03:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:03:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:03:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:03:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:03:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:03:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:03:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:03:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:03:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:03:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:03:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:03:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:03:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:03:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:03:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:03:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:03:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:03:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:03:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:03:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:03:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:03:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:03:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:03:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:03:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:03:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:03:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:03:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:03:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:03:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:03:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:03:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:03:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:03:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:03:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:03:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:03:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:03:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:03:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:03:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:03:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:03:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:03:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:03:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:03:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:03:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:03:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:03:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:03:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:03:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:03:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:03:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:03:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:03:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:03:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:03:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:03:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:03:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:03:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:03:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:03:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:03:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:03:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:03:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:03:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:04:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:04:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:04:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:04:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:04:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:04:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:04:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:04:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:04:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:04:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:04:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:04:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:04:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:04:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:04:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:04:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:04:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:04:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:04:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:04:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:04:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:04:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:04:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:04:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:04:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:04:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:04:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:04:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:04:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:04:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:04:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:04:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:04:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:04:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:04:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:04:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:04:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:04:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:04:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:04:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:04:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:04:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:04:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:04:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:04:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:04:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:04:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:04:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:04:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:04:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:04:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:04:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:04:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:04:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:04:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:04:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:04:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:04:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:04:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:04:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:04:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:04:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:04:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:04:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:04:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:04:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:04:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:04:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:04:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:04:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:04:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:04:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:04:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:04:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:04:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:04:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:04:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:04:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:04:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:04:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:04:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:04:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:04:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:04:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:04:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:04:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:04:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:04:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:04:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:04:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:04:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:04:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:04:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:04:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:04:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:04:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:04:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:04:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:04:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:04:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:04:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:04:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:04:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:04:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:04:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:04:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:04:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:04:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:04:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:04:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:04:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:04:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:04:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:04:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:04:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:04:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:04:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:04:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:04:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:04:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:04:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:04:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:04:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:04:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:04:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:04:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:04:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:04:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:04:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:04:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:04:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:04:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:04:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:04:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:04:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:04:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:04:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:04:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:04:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:04:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:04:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:04:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:04:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:04:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:06:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:06:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:06:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:06:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:06:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:06:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:06:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:06:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:06:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:06:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:06:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:06:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:06:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:06:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:06:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:06:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:06:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:06:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:06:46 rdma06 rsyslogd[1041]: imjournal: 5295 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 13:06:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:06:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:06:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:06:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:06:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:06:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:06:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:06:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:06:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:06:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:06:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:06:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:06:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:06:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:06:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:06:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:06:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:06:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:06:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:06:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:06:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:06:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:06:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:06:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:06:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:06:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:06:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:06:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:06:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:06:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:06:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:06:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:06:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:06:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:06:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:06:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:07:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:07:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:07:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:07:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:07:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:07:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:07:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:07:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:07:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:07:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:07:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:07:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:07:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:07:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:07:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:07:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:07:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:07:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:07:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:07:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:07:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:07:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:07:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:07:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:07:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:07:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:07:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:07:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:07:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:07:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:07:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:07:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:07:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:07:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:07:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:07:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:07:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:07:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:07:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:07:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:07:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:07:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:07:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:07:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:07:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:07:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:07:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:07:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:07:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:07:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:07:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:07:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:07:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:07:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:07:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:07:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:07:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:07:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:07:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:07:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:07:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:07:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:07:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:07:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:07:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:07:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:07:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:07:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:07:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:07:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:07:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:07:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:07:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:07:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:07:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:07:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:07:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:07:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:07:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:07:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:07:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:07:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:07:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:07:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:07:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:07:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:07:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:07:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:07:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:07:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:07:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:07:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:07:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:07:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:07:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:07:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:07:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:07:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:07:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:07:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:07:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:07:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:07:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:07:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:07:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:07:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:07:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:07:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:07:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:07:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:07:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:07:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:07:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:07:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:07:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:07:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:07:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:07:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:07:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:07:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:07:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:07:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:07:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:07:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:07:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:07:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:07:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:07:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:07:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:07:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:07:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:07:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:07:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:07:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:07:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:07:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:07:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:07:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:07:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:07:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:07:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:07:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:07:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:07:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:07:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:07:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:07:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:07:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:07:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:07:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:07:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:07:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:07:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:07:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:07:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:07:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:07:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:07:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:07:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:07:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:07:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:07:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:07:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:07:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:07:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:07:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:07:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:07:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:07:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:07:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:07:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:07:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:07:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:07:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:07:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:07:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:07:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:07:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:07:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:07:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:07:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:07:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:07:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:07:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:07:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:07:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:07:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:07:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:07:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:07:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:07:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:07:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:07:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:07:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:07:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:07:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:07:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:07:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:07:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:07:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:07:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:07:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:07:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:07:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:07:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:07:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:07:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:07:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:07:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:07:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:07:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:07:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:07:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:07:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:07:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:07:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:08:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:08:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:08:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:08:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:08:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:08:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:08:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:08:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:08:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:08:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:08:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:08:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:08:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:08:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:08:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:08:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:08:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:08:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:08:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:08:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:08:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:08:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:08:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:08:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:08:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:08:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:08:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:08:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:08:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:08:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:08:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:08:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:08:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:08:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:08:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:08:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:08:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:08:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:08:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:08:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:08:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:08:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:08:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:08:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:08:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:08:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:08:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:08:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:08:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:08:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:08:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:08:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:08:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:08:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:08:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:08:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:08:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:08:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:08:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:08:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:08:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:08:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:08:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:08:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:08:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:08:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:08:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:08:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:08:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:08:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:08:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:08:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:08:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:08:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:08:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:08:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:08:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:08:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:08:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:08:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:08:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:08:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:08:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:08:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:08:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:08:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:08:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:08:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:08:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:08:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:08:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:08:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:08:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:08:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:08:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:08:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:08:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:08:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:08:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:08:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:08:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:08:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:08:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:08:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:08:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:08:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:08:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:08:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:08:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:08:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:08:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:08:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:08:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:08:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:08:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:08:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:08:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:08:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:08:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:08:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:08:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:08:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:08:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:08:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:08:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:08:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:08:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:08:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:08:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:08:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:08:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:08:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:08:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:08:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:08:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:08:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:08:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:08:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:08:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:08:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:08:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:08:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:08:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:08:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:08:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:08:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:08:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:08:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:08:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:08:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:08:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:08:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:08:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:08:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:08:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:08:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:08:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:08:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:08:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:08:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:08:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:08:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:08:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:08:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:08:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:08:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:08:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:08:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:08:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:08:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:08:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:08:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:08:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:08:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:08:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:08:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:08:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:08:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:08:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:08:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:08:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:08:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:08:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:08:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:08:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:08:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:08:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:08:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:08:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:08:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:08:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:08:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:08:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:08:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:08:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:08:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:08:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:08:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:08:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:08:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:08:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:08:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:08:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:08:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:08:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:08:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:08:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:08:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:08:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:08:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:08:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:08:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:08:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:08:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:08:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:08:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:09:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:09:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:09:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:09:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:09:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:09:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:09:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:09:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:09:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:09:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:09:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:09:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:09:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:09:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:09:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:09:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:09:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:09:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:09:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:09:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:09:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:09:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:09:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:09:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:09:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:09:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:09:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:09:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:09:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:09:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:09:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:09:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:09:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:09:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:09:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:09:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:09:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:09:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:09:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:09:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:09:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:09:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:09:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:09:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:09:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:09:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:09:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:09:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:09:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:09:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:09:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:09:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:09:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:09:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:09:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:09:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:09:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:09:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:09:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:09:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:09:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:09:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:09:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:09:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:09:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:09:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:09:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:09:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:09:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:09:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:09:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:09:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:09:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:09:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:09:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:09:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:09:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:09:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:09:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:09:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:09:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:09:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:09:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:09:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:09:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:09:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:09:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:09:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:09:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:09:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:09:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:09:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:09:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:09:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:09:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:09:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:09:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:09:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:09:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:09:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:09:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:09:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:09:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:09:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:09:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:09:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:09:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:09:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:09:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:09:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:09:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:09:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:09:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:09:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:09:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:09:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:09:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:09:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:09:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:09:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:09:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:09:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:09:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:09:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:09:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:09:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:09:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:09:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:09:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:09:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:09:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:09:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:09:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:09:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:09:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:09:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:09:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:09:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:09:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:09:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:09:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:09:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:09:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:09:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:09:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:09:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:09:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:09:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:09:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:09:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:09:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:09:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:09:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:09:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:09:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:09:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:09:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:09:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:09:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:09:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:09:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:09:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:09:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:09:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:09:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:09:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:09:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:09:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:09:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:09:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:09:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:09:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:09:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:09:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:09:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:09:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:09:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:09:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:09:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:09:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:09:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:09:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:09:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:09:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:09:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:09:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:09:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:09:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:09:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:09:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:09:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:09:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:09:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:09:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:09:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:09:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:09:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:09:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:09:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:09:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:09:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:09:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:09:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:09:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:09:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:09:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:09:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:09:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:09:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:09:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:09:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:09:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:09:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:09:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:09:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:09:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:10:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:10:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:10:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:10:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:10:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:10:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:10:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:10:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:10:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:10:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:10:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:10:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:10:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:10:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:10:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:10:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:10:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:10:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:10:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:10:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:10:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:10:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:10:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:10:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:10:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:10:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:10:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:10:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:10:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:10:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:10:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:10:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:10:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:10:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:10:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:10:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:10:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:10:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:10:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:10:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:10:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:10:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:10:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:10:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:10:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:10:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:10:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:10:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:10:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:10:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:10:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:10:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:10:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:10:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:10:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:10:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:10:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:10:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:10:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:10:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:10:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:10:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:10:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:10:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:10:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:10:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:10:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:10:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:10:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:10:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:10:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:10:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:10:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:10:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:10:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:10:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:10:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:10:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:10:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:10:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:10:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:10:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:10:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:10:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:10:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:10:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:10:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:10:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:10:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:10:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:10:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:10:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:10:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:10:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:10:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:10:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:10:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:10:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:10:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:10:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:10:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:10:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:10:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:10:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:10:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:10:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:10:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:10:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:10:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:10:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:10:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:10:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:10:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:10:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:10:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:10:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:10:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:10:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:10:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:10:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:10:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:10:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:10:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:10:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:10:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:10:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:10:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:10:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:10:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:10:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:10:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:10:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:10:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:10:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:10:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:10:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:10:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:10:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:10:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:10:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:10:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:10:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:10:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:10:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:10:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:10:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:10:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:10:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:10:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:10:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:10:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:10:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:10:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:10:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:10:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:10:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:10:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:10:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:10:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:10:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:10:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:10:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:10:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:10:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:10:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:10:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:10:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:10:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:10:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:10:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:10:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:10:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:10:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:10:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:10:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:10:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:10:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:10:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:10:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:10:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:10:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:10:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:10:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:10:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:10:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:10:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:10:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:10:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:10:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:10:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:10:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:10:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:10:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:10:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:10:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:10:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:10:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:10:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:10:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:10:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:10:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:10:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:10:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:10:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:10:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:10:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:10:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:10:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:10:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:10:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:10:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:10:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:10:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:10:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:10:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:10:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:11:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:11:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:11:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:11:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:11:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:11:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:11:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:11:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:11:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:11:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:11:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:11:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:11:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:11:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:11:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:11:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:11:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:11:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:11:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:11:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:11:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:11:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:11:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:11:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:11:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:11:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:11:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:11:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:11:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:11:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:11:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:11:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:11:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:11:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:11:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:11:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:11:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:11:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:11:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:11:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:11:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:11:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:11:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:11:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:11:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:11:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:11:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:11:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:11:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:11:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:11:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:11:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:11:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:11:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:11:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:11:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:11:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:11:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:11:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:11:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:11:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:11:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:11:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:11:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:11:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:11:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:11:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:11:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:11:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:11:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:11:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:11:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:11:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:11:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:11:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:11:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:11:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:11:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:11:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:11:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:11:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:11:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:11:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:11:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:11:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:11:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:11:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:11:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:11:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:11:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:11:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:11:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:11:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:11:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:11:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:11:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:11:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:11:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:11:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:11:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:11:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:11:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:11:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:11:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:11:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:11:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:11:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:11:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:11:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:11:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:11:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:11:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:11:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:11:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:11:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:11:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:11:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:11:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:11:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:11:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:11:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:11:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:11:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:11:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:11:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:11:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:11:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:11:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:11:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:11:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:11:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:11:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:11:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:11:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:11:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:11:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:11:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:11:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:11:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:11:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:11:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:11:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:11:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:11:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:11:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:11:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:11:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:11:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:11:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:11:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:11:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:11:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:11:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:11:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:11:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:11:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:11:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:11:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:11:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:11:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:11:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:11:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:11:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:11:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:11:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:11:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:11:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:11:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:11:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:11:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:11:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:11:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:11:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:11:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:11:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:11:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:11:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:11:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:11:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:11:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:11:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:11:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:11:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:11:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:11:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:11:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:11:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:11:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:11:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:11:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:11:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:11:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:11:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:11:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:11:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:11:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:11:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:11:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:11:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:11:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:11:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:11:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:11:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:11:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:11:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:11:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:11:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:11:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:11:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:11:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:11:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:11:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:11:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:11:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:11:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:11:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:12:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:12:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:12:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:12:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:12:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:12:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:12:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:12:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:12:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:12:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:12:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:12:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:12:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:12:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:12:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:12:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:12:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:12:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:12:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:12:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:12:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:12:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:12:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:12:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:12:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:12:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:12:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:12:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:12:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:12:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:12:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:12:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:12:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:12:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:12:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:12:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:12:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:12:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:12:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:12:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:12:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:12:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:12:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:12:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:12:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:12:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:12:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:12:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:12:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:12:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:12:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:12:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:12:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:12:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:12:11 rdma06 systemd-journald[790]: Data hash table of /run/log/journal/95379d2f723847b68d9e976dee8967a9/system.journal has a fill level at 75.0 (163751 of 218332 items, 75497472 file size, 461 bytes per hash table item), suggesting rotation.
Apr 17 13:12:11 rdma06 systemd-journald[790]: /run/log/journal/95379d2f723847b68d9e976dee8967a9/system.journal: Journal header limits reached or header out-of-date, rotating.
Apr 17 13:12:11 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 17 13:12:11 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 17 13:12:11 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 17 13:12:11 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 17 13:12:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:12:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:12:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:12:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:12:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:12:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:12:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:12:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:12:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:12:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:12:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:12:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:12:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:12:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:12:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:12:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:12:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:12:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:12:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:12:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:12:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:12:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:12:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:12:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:12:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:12:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:12:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:12:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:12:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:12:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:12:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:12:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:12:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:12:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:12:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:12:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:12:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:12:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:12:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:12:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:12:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:12:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:12:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:12:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:12:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:12:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:12:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:12:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:12:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:12:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:12:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:12:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:12:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:12:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:12:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:12:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:12:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:12:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:12:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:12:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:12:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:12:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:12:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:12:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:12:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:12:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:12:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:12:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:12:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:12:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:12:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:12:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:12:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:12:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:12:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:12:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:12:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:12:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:12:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:12:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:12:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:12:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:12:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:12:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:12:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:12:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:12:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:12:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:12:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:12:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:12:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:12:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:12:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:12:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:12:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:12:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:12:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:12:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:12:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:12:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:12:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:12:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:12:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:12:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:12:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:12:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:12:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:12:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:12:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:12:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:12:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:12:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:12:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:12:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:12:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:12:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:12:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:12:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:12:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:12:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:12:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:12:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:12:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:12:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:12:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:12:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:12:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:12:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:12:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:12:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:12:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:12:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:12:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:12:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:12:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:12:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:12:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:12:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:12:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:12:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:12:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:12:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:12:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:12:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:12:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:12:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:12:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:12:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:12:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:12:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:12:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:12:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:12:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:12:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:12:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:12:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:12:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:12:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:12:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:12:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:12:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:12:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:13:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:13:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:13:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:13:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:13:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:13:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:13:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:13:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:13:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:13:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:13:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:13:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:13:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:13:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:13:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:13:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:13:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:13:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:13:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:13:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:13:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:13:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:13:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:13:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:13:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:13:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:13:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:13:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:13:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:13:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:13:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:13:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:13:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:13:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:13:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:13:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:13:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:13:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:13:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:13:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:13:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:13:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:13:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:13:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:13:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:13:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:13:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:13:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:13:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:13:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:13:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:13:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:13:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:13:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:13:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:13:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:13:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:13:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:13:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:13:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:13:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:13:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:13:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:13:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:13:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:13:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:13:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:13:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:13:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:13:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:13:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:13:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:13:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:13:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:13:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:13:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:13:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:13:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:13:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:13:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:13:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:13:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:13:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:13:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:13:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:13:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:13:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:13:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:13:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:13:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:13:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:13:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:13:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:13:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:13:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:13:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:13:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:13:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:13:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:13:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:13:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:13:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:13:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:13:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:13:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:13:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:13:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:13:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:13:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:13:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:13:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:13:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:13:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:13:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:13:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:13:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:13:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:13:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:13:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:13:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:13:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:13:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:13:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:13:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:13:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:13:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:13:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:13:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:13:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:13:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:13:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:13:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:13:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:13:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:13:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:13:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:13:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:13:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:13:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:13:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:13:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:13:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:13:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:13:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:13:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:13:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:13:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:13:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:13:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:13:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:13:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:13:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:13:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:13:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:13:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:13:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:13:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:13:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:13:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:13:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:13:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:13:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:13:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:13:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:13:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:13:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:13:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:13:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:13:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:13:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:13:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:13:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:13:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:13:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:13:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:13:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:13:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:13:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:13:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:13:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:13:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:13:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:13:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:13:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:13:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:13:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:13:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:13:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:13:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:13:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:13:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:13:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:13:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:13:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:13:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:13:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:13:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:13:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:13:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:13:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:13:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:13:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:13:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:13:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:13:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:13:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:13:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:13:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:13:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:13:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:13:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:13:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:13:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:13:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:13:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:13:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:14:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:14:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:14:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:14:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:14:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:14:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:14:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:14:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:14:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:14:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:14:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:14:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:14:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:14:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:14:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:14:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:14:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:14:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:14:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:14:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:14:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:14:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:14:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:14:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:14:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:14:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:14:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:14:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:14:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:14:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:14:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:14:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:14:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:14:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:14:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:14:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:14:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:14:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:14:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:14:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:14:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:14:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:14:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:14:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:14:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:14:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:14:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:14:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:14:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:14:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:14:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:14:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:14:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:14:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:14:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:14:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:14:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:14:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:14:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:14:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:14:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:14:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:14:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:14:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:14:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:14:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:14:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:14:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:14:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:14:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:14:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:14:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:14:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:14:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:14:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:14:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:14:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:14:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:14:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:14:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:14:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:14:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:14:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:14:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:14:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:14:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:14:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:14:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:14:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:14:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:14:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:14:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:14:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:14:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:14:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:14:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:14:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:14:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:14:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:14:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:14:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:14:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:14:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:14:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:14:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:14:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:14:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:14:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:14:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:14:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:14:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:14:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:14:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:14:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:14:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:14:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:14:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:14:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:14:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:14:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:14:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:14:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:14:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:14:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:14:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:14:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:14:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:14:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:14:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:14:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:14:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:14:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:14:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:14:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:14:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:14:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:14:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:14:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:14:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:14:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:14:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:14:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:14:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:14:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:14:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:14:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:14:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:14:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:14:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:14:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:14:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:14:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:14:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:14:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:14:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:14:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:14:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:14:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:14:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:14:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:14:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:14:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:16:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:16:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:16:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:16:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:16:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:16:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:16:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:16:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:16:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:16:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:16:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:16:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:16:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:16:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:16:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:16:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:16:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:16:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:16:47 rdma06 rsyslogd[1041]: imjournal: 5088 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 13:16:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:16:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:16:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:16:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:16:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:16:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:16:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:16:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:16:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:16:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:16:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:16:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:16:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:16:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:16:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:16:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:16:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:16:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:16:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:16:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:16:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:16:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:16:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:16:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:16:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:16:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:16:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:16:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:16:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:16:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:16:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:16:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:16:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:16:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:16:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:16:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:17:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:17:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:17:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:17:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:17:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:17:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:17:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:17:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:17:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:17:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:17:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:17:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:17:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:17:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:17:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:17:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:17:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:17:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:17:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:17:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:17:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:17:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:17:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:17:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:17:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:17:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:17:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:17:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:17:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:17:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:17:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:17:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:17:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:17:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:17:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:17:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:17:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:17:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:17:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:17:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:17:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:17:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:17:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:17:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:17:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:17:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:17:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:17:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:17:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:17:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:17:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:17:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:17:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:17:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:17:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:17:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:17:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:17:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:17:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:17:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:17:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:17:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:17:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:17:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:17:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:17:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:17:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:17:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:17:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:17:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:17:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:17:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:17:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:17:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:17:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:17:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:17:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:17:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:17:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:17:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:17:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:17:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:17:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:17:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:17:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:17:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:17:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:17:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:17:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:17:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:17:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:17:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:17:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:17:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:17:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:17:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:17:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:17:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:17:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:17:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:17:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:17:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:17:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:17:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:17:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:17:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:17:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:17:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:17:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:17:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:17:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:17:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:17:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:17:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:17:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:17:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:17:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:17:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:17:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:17:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:17:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:17:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:17:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:17:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:17:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:17:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:17:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:17:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:17:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:17:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:17:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:17:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:17:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:17:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:17:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:17:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:17:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:17:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:17:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:17:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:17:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:17:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:17:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:17:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:17:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:17:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:17:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:17:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:17:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:17:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:17:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:17:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:17:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:17:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:17:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:17:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:17:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:17:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:17:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:17:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:17:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:17:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:17:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:17:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:17:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:17:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:17:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:17:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:17:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:17:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:17:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:17:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:17:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:17:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:17:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:17:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:17:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:17:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:17:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:17:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:17:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:17:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:17:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:17:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:17:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:17:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:17:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:17:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:17:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:17:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:17:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:17:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:17:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:17:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:17:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:17:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:17:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:17:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:17:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:17:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:17:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:17:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:17:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:17:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:17:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:17:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:17:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:17:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:17:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:17:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:17:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:17:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:17:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:17:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:17:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:17:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:18:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:18:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:18:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:18:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:18:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:18:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:18:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:18:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:18:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:18:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:18:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:18:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:18:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:18:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:18:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:18:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:18:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:18:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:18:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:18:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:18:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:18:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:18:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:18:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:18:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:18:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:18:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:18:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:18:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:18:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:18:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:18:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:18:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:18:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:18:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:18:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:18:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:18:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:18:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:18:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:18:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:18:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:18:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:18:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:18:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:18:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:18:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:18:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:18:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:18:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:18:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:18:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:18:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:18:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:18:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:18:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:18:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:18:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:18:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:18:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:18:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:18:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:18:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:18:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:18:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:18:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:18:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:18:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:18:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:18:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:18:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:18:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:18:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:18:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:18:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:18:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:18:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:18:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:18:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:18:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:18:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:18:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:18:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:18:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:18:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:18:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:18:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:18:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:18:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:18:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:18:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:18:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:18:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:18:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:18:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:18:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:18:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:18:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:18:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:18:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:18:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:18:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:18:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:18:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:18:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:18:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:18:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:18:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:18:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:18:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:18:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:18:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:18:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:18:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:18:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:18:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:18:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:18:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:18:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:18:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:18:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:18:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:18:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:18:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:18:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:18:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:18:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:18:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:18:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:18:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:18:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:18:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:18:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:18:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:18:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:18:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:18:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:18:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:18:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:18:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:18:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:18:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:18:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:18:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:18:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:18:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:18:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:18:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:18:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:18:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:18:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:18:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:18:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:18:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:18:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:18:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:18:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:18:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:18:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:18:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:18:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:18:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:18:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:18:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:18:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:18:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:18:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:18:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:18:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:18:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:18:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:18:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:18:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:18:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:18:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:18:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:18:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:18:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:18:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:18:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:18:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:18:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:18:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:18:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:18:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:18:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:18:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:18:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:18:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:18:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:18:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:18:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:18:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:18:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:18:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:18:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:18:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:18:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:18:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:18:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:18:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:18:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:18:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:18:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:18:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:18:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:18:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:18:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:18:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:18:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:18:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:18:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:18:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:18:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:18:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:18:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:19:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:19:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:19:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:19:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:19:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:19:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:19:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:19:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:19:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:19:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:19:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:19:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:19:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:19:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:19:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:19:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:19:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:19:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:19:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:19:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:19:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:19:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:19:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:19:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:19:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:19:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:19:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:19:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:19:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:19:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:19:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:19:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:19:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:19:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:19:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:19:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:19:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:19:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:19:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:19:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:19:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:19:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:19:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:19:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:19:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:19:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:19:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:19:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:19:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:19:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:19:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:19:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:19:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:19:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:19:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:19:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:19:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:19:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:19:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:19:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:19:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:19:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:19:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:19:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:19:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:19:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:19:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:19:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:19:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:19:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:19:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:19:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:19:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:19:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:19:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:19:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:19:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:19:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:19:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:19:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:19:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:19:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:19:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:19:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:19:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:19:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:19:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:19:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:19:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:19:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:19:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:19:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:19:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:19:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:19:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:19:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:19:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:19:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:19:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:19:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:19:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:19:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:19:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:19:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:19:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:19:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:19:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:19:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:19:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:19:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:19:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:19:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:19:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:19:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:19:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:19:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:19:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:19:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:19:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:19:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:19:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:19:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:19:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:19:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:19:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:19:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:19:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:19:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:19:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:19:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:19:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:19:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:19:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:19:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:19:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:19:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:19:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:19:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:19:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:19:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:19:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:19:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:19:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:19:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:19:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:19:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:19:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:19:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:19:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:19:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:19:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:19:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:19:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:19:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:19:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:19:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:19:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:19:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:19:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:19:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:19:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:19:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:19:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:19:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:19:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:19:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:19:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:19:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:19:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:19:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:19:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:19:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:19:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:19:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:19:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:19:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:19:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:19:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:19:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:19:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:19:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:19:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:19:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:19:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:19:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:19:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:19:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:19:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:19:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:19:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:19:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:19:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:19:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:19:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:19:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:19:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:19:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:19:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:19:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:19:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:19:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:19:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:19:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:19:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:19:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:19:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:19:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:19:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:19:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:19:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:19:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:19:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:19:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:19:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:19:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:19:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:20:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:20:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:20:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:20:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:20:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:20:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:20:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:20:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:20:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:20:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:20:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:20:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:20:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:20:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:20:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:20:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:20:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:20:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:20:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:20:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:20:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:20:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:20:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:20:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:20:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:20:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:20:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:20:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:20:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:20:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:20:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:20:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:20:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:20:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:20:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:20:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:20:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:20:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:20:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:20:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:20:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:20:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:20:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:20:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:20:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:20:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:20:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:20:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:20:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:20:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:20:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:20:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:20:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:20:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:20:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:20:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:20:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:20:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:20:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:20:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:20:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:20:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:20:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:20:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:20:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:20:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:20:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:20:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:20:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:20:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:20:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:20:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:20:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:20:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:20:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:20:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:20:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:20:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:20:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:20:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:20:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:20:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:20:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:20:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:20:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:20:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:20:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:20:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:20:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:20:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:20:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:20:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:20:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:20:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:20:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:20:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:20:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:20:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:20:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:20:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:20:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:20:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:20:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:20:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:20:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:20:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:20:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:20:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:20:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:20:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:20:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:20:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:20:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:20:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:20:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:20:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:20:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:20:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:20:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:20:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:20:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:20:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:20:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:20:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:20:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:20:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:20:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:20:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:20:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:20:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:20:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:20:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:20:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:20:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:20:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:20:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:20:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:20:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:20:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:20:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:20:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:20:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:20:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:20:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:20:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:20:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:20:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:20:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:20:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:20:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:20:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:20:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:20:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:20:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:20:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:20:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:20:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:20:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:20:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:20:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:20:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:20:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:20:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:20:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:20:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:20:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:20:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:20:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:20:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:20:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:20:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:20:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:20:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:20:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:20:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:20:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:20:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:20:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:20:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:20:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:20:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:20:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:20:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:20:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:20:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:20:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:20:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:20:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:20:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:20:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:20:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:20:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:20:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:20:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:20:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:20:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:20:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:20:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:20:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:20:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:20:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:20:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:20:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:20:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:20:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:20:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:20:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:20:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:20:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:20:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:20:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:20:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:20:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:20:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:20:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:20:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:21:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:21:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:21:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:21:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:21:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:21:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:21:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:21:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:21:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:21:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:21:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:21:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:21:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:21:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:21:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:21:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:21:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:21:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:21:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:21:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:21:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:21:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:21:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:21:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:21:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:21:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:21:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:21:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:21:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:21:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:21:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:21:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:21:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:21:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:21:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:21:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:21:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:21:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:21:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:21:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:21:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:21:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:21:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:21:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:21:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:21:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:21:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:21:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:21:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:21:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:21:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:21:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:21:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:21:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:21:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:21:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:21:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:21:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:21:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:21:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:21:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:21:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:21:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:21:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:21:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:21:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:21:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:21:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:21:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:21:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:21:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:21:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:21:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:21:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:21:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:21:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:21:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:21:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:21:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:21:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:21:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:21:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:21:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:21:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:21:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:21:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:21:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:21:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:21:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:21:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:21:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:21:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:21:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:21:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:21:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:21:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:21:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:21:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:21:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:21:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:21:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:21:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:21:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:21:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:21:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:21:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:21:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:21:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:21:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:21:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:21:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:21:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:21:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:21:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:21:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:21:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:21:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:21:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:21:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:21:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:21:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:21:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:21:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:21:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:21:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:21:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:21:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:21:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:21:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:21:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:21:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:21:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:21:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:21:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:21:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:21:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:21:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:21:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:21:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:21:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:21:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:21:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:21:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:21:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:21:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:21:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:21:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:21:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:21:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:21:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:21:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:21:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:21:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:21:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:21:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:21:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:21:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:21:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:21:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:21:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:21:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:21:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:21:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:21:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:21:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:21:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:21:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:21:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:21:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:21:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:21:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:21:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:21:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:21:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:21:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:21:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:21:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:21:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:21:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:21:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:21:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:21:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:21:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:21:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:21:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:21:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:21:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:21:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:21:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:21:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:21:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:21:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:21:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:21:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:21:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:21:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:21:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:21:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:21:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:21:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:21:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:21:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:21:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:21:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:21:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:21:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:21:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:21:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:21:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:21:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:21:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:21:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:21:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:21:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:21:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:21:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:22:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:22:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:22:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:22:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:22:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:22:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:22:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:22:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:22:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:22:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:22:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:22:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:22:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:22:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:22:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:22:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:22:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:22:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:22:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:22:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:22:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:22:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:22:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:22:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:22:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:22:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:22:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:22:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:22:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:22:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:22:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:22:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:22:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:22:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:22:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:22:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:22:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:22:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:22:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:22:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:22:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:22:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:22:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:22:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:22:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:22:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:22:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:22:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:22:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:22:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:22:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:22:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:22:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:22:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:22:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:22:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:22:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:22:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:22:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:22:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:22:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:22:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:22:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:22:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:22:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:22:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:22:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:22:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:22:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:22:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:22:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:22:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:22:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:22:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:22:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:22:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:22:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:22:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:22:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:22:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:22:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:22:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:22:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:22:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:22:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:22:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:22:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:22:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:22:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:22:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:22:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:22:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:22:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:22:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:22:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:22:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:22:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:22:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:22:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:22:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:22:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:22:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:22:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:22:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:22:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:22:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:22:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:22:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:22:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:22:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:22:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:22:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:22:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:22:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:22:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:22:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:22:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:22:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:22:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:22:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:22:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:22:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:22:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:22:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:22:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:22:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:22:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:22:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:22:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:22:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:22:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:22:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:22:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:22:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:22:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:22:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:22:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:22:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:22:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:22:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:22:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:22:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:22:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:22:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:22:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:22:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:22:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:22:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:22:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:22:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:22:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:22:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:22:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:22:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:22:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:22:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:22:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:22:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:22:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:22:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:22:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:22:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:22:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:22:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:22:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:22:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:22:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:22:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:22:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:22:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:22:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:22:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:22:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:22:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:22:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:22:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:22:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:22:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:22:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:22:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:22:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:22:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:22:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:22:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:22:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:22:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:22:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:22:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:22:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:22:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:22:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:22:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:22:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:22:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:22:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:22:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:22:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:22:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:22:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:22:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:22:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:22:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:22:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:22:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:22:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:22:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:22:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:22:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:22:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:22:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:22:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:22:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:22:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:22:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:22:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:22:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:23:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:23:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:23:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:23:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:23:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:23:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:23:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:23:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:23:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:23:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:23:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:23:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:23:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:23:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:23:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:23:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:23:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:23:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:23:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:23:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:23:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:23:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:23:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:23:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:23:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:23:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:23:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:23:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:23:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:23:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:23:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:23:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:23:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:23:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:23:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:23:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:23:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:23:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:23:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:23:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:23:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:23:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:23:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:23:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:23:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:23:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:23:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:23:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:23:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:23:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:23:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:23:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:23:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:23:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:23:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:23:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:23:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:23:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:23:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:23:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:23:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:23:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:23:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:23:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:23:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:23:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:23:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:23:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:23:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:23:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:23:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:23:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:23:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:23:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:23:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:23:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:23:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:23:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:23:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:23:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:23:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:23:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:23:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:23:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:23:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:23:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:23:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:23:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:23:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:23:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:23:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:23:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:23:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:23:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:23:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:23:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:23:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:23:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:23:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:23:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:23:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:23:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:23:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:23:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:23:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:23:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:23:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:23:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:23:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:23:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:23:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:23:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:23:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:23:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:23:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:23:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:23:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:23:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:23:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:23:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:23:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:23:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:23:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:23:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:23:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:23:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:23:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:23:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:23:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:23:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:23:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:23:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:23:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:23:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:23:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:23:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:23:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:23:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:23:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:23:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:23:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:23:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:23:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:23:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:23:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:23:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:23:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:23:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:23:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:23:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:23:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:23:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:23:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:23:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:23:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:23:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:23:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:23:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:23:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:23:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:23:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:23:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:23:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:23:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:23:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:23:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:23:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:23:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:23:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:23:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:23:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:23:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:23:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:23:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:23:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:23:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:23:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:23:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:23:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:23:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:23:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:23:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:23:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:23:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:23:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:23:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:23:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:23:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:23:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:23:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:23:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:23:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:23:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:23:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:23:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:23:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:23:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:23:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:23:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:23:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:23:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:23:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:23:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:23:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:23:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:23:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:23:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:23:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:23:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:23:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:23:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:23:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:23:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:23:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:23:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:23:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:24:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:24:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:24:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:24:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:24:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:24:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:24:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:24:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:24:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:24:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:24:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:24:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:24:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:24:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:24:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:24:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:24:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:24:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:24:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:24:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:24:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:24:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:24:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:24:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:24:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:24:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:24:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:24:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:24:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:24:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:24:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:24:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:24:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:24:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:24:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:24:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:24:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:24:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:24:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:24:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:24:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:24:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:24:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:24:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:24:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:24:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:24:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:24:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:24:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:24:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:24:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:24:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:24:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:24:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:24:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:24:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:24:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:24:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:24:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:24:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:24:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:24:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:24:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:24:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:24:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:24:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:24:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:24:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:24:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:24:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:24:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:24:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:24:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:24:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:24:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:24:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:24:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:24:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:24:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:24:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:24:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:24:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:24:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:24:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:24:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:24:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:24:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:24:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:24:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:24:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:24:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:24:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:24:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:24:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:24:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:24:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:24:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:24:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:24:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:24:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:24:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:24:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:24:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:24:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:24:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:24:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:24:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:24:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:24:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:24:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:24:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:24:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:24:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:24:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:24:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:24:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:24:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:24:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:24:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:24:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:24:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:24:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:24:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:24:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:24:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:24:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:24:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:24:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:24:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:24:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:24:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:24:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:24:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:24:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:24:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:24:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:24:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:24:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:24:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:24:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:24:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:24:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:24:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:24:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:24:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:24:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:24:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:24:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:24:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:24:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:24:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:24:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:24:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:24:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:24:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:24:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:24:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:24:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:24:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:24:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:24:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:24:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:26:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:26:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:26:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:26:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:26:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:26:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:26:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:26:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:26:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:26:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:26:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:26:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:26:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:26:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:26:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:26:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:26:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:26:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:26:52 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 13:26:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:26:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:26:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:26:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:26:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:26:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:26:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:26:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:26:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:26:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:26:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:26:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:26:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:26:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:26:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:26:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:26:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:26:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:27:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:27:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:27:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:27:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:27:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:27:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:27:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:27:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:27:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:27:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:27:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:27:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:27:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:27:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:27:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:27:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:27:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:27:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:27:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:27:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:27:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:27:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:27:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:27:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:27:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:27:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:27:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:27:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:27:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:27:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:27:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:27:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:27:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:27:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:27:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:27:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:27:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:27:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:27:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:27:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:27:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:27:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:27:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:27:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:27:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:27:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:27:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:27:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:27:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:27:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:27:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:27:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:27:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:27:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:27:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:27:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:27:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:27:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:27:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:27:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:27:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:27:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:27:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:27:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:27:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:27:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:27:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:27:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:27:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:27:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:27:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:27:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:27:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:27:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:27:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:27:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:27:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:27:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:27:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:27:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:27:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:27:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:27:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:27:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:27:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:27:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:27:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:27:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:27:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:27:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:27:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:27:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:27:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:27:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:27:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:27:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:27:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:27:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:27:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:27:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:27:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:27:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:27:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:27:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:27:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:27:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:27:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:27:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:27:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:27:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:27:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:27:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:27:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:27:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:27:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:27:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:27:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:27:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:27:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:27:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:27:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:27:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:27:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:27:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:27:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:27:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:27:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:27:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:27:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:27:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:27:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:27:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:27:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:27:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:27:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:27:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:27:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:27:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:27:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:27:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:27:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:27:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:27:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:27:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:27:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:27:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:27:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:27:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:27:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:27:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:27:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:27:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:27:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:27:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:27:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:27:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:27:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:27:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:27:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:27:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:27:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:27:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:27:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:27:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:27:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:27:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:27:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:27:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:27:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:27:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:27:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:27:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:27:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:27:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:27:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:27:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:27:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:27:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:27:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:27:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:27:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:27:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:27:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:27:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:27:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:27:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:27:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:27:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:27:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:27:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:27:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:27:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:27:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:27:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:27:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:27:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:27:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:27:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:27:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:27:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:27:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:27:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:27:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:27:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:27:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:27:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:27:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:27:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:27:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:27:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:27:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:27:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:27:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:27:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:27:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:27:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:28:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:28:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:28:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:28:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:28:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:28:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:28:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:28:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:28:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:28:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:28:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:28:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:28:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:28:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:28:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:28:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:28:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:28:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:28:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:28:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:28:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:28:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:28:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:28:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:28:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:28:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:28:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:28:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:28:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:28:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:28:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:28:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:28:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:28:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:28:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:28:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:28:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:28:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:28:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:28:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:28:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:28:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:28:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:28:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:28:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:28:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:28:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:28:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:28:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:28:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:28:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:28:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:28:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:28:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:28:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:28:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:28:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:28:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:28:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:28:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:28:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:28:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:28:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:28:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:28:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:28:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:28:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:28:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:28:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:28:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:28:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:28:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:28:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:28:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:28:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:28:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:28:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:28:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:28:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:28:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:28:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:28:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:28:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:28:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:28:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:28:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:28:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:28:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:28:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:28:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:28:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:28:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:28:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:28:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:28:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:28:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:28:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:28:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:28:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:28:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:28:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:28:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:28:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:28:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:28:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:28:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:28:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:28:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:28:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:28:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:28:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:28:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:28:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:28:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:28:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:28:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:28:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:28:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:28:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:28:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:28:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:28:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:28:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:28:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:28:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:28:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:28:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:28:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:28:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:28:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:28:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:28:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:28:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:28:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:28:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:28:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:28:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:28:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:28:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:28:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:28:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:28:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:28:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:28:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:28:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:28:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:28:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:28:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:28:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:28:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:28:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:28:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:28:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:28:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:28:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:28:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:28:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:28:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:28:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:28:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:28:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:28:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:28:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:28:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:28:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:28:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:28:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:28:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:28:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:28:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:28:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:28:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:28:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:28:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:28:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:28:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:28:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:28:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:28:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:28:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:28:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:28:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:28:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:28:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:28:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:28:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:28:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:28:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:28:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:28:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:28:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:28:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:28:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:28:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:28:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:28:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:28:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:28:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:28:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:28:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:28:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:28:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:28:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:28:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:28:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:28:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:28:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:28:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:28:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:28:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:28:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:28:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:28:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:28:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:28:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:28:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:29:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:29:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:29:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:29:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:29:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:29:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:29:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:29:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:29:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:29:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:29:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:29:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:29:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:29:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:29:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:29:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:29:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:29:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:29:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:29:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:29:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:29:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:29:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:29:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:29:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:29:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:29:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:29:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:29:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:29:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:29:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:29:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:29:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:29:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:29:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:29:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:29:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:29:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:29:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:29:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:29:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:29:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:29:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:29:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:29:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:29:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:29:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:29:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:29:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:29:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:29:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:29:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:29:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:29:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:29:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:29:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:29:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:29:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:29:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:29:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:29:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:29:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:29:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:29:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:29:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:29:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:29:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:29:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:29:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:29:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:29:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:29:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:29:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:29:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:29:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:29:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:29:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:29:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:29:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:29:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:29:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:29:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:29:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:29:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:29:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:29:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:29:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:29:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:29:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:29:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:29:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:29:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:29:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:29:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:29:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:29:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:29:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:29:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:29:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:29:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:29:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:29:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:29:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:29:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:29:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:29:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:29:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:29:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:29:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:29:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:29:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:29:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:29:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:29:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:29:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:29:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:29:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:29:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:29:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:29:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:29:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:29:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:29:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:29:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:29:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:29:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:29:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:29:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:29:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:29:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:29:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:29:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:29:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:29:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:29:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:29:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:29:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:29:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:29:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:29:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:29:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:29:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:29:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:29:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:29:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:29:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:29:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:29:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:29:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:29:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:29:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:29:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:29:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:29:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:29:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:29:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:29:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:29:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:29:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:29:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:29:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:29:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:29:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:29:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:29:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:29:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:29:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:29:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:29:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:29:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:29:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:29:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:29:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:29:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:29:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:29:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:29:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:29:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:29:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:29:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:29:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:29:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:29:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:29:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:29:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:29:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:29:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:29:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:29:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:29:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:29:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:29:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:29:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:29:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:29:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:29:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:29:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:29:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:29:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:29:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:29:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:29:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:29:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:29:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:29:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:29:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:29:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:29:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:29:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:29:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:29:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:29:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:29:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:29:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:29:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:29:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:30:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:30:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:30:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:30:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:30:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:30:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:30:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:30:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:30:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:30:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:30:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:30:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:30:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:30:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:30:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:30:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:30:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:30:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:30:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:30:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:30:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:30:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:30:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:30:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:30:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:30:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:30:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:30:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:30:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:30:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:30:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:30:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:30:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:30:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:30:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:30:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:30:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:30:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:30:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:30:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:30:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:30:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:30:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:30:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:30:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:30:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:30:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:30:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:30:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:30:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:30:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:30:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:30:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:30:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:30:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:30:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:30:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:30:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:30:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:30:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:30:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:30:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:30:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:30:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:30:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:30:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:30:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:30:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:30:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:30:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:30:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:30:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:30:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:30:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:30:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:30:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:30:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:30:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:30:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:30:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:30:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:30:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:30:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:30:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:30:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:30:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:30:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:30:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:30:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:30:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:30:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:30:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:30:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:30:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:30:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:30:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:30:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:30:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:30:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:30:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:30:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:30:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:30:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:30:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:30:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:30:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:30:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:30:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:30:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:30:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:30:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:30:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:30:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:30:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:30:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:30:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:30:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:30:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:30:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:30:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:30:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:30:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:30:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:30:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:30:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:30:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:30:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:30:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:30:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:30:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:30:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:30:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:30:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:30:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:30:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:30:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:30:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:30:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:30:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:30:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:30:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:30:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:30:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:30:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:30:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:30:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:30:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:30:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:30:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:30:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:30:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:30:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:30:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:30:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:30:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:30:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:30:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:30:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:30:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:30:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:30:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:30:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:30:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:30:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:30:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:30:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:30:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:30:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:30:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:30:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:30:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:30:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:30:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:30:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:30:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:30:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:30:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:30:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:30:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:30:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:30:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:30:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:30:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:30:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:30:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:30:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:30:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:30:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:30:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:30:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:30:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:30:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:30:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:30:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:30:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:30:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:30:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:30:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:30:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:30:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:30:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:30:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:30:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:30:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:30:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:30:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:30:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:30:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:30:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:30:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:30:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:30:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:30:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:30:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:30:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:30:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:31:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:31:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:31:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:31:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:31:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:31:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:31:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:31:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:31:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:31:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:31:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:31:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:31:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:31:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:31:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:31:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:31:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:31:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:31:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:31:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:31:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:31:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:31:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:31:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:31:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:31:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:31:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:31:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:31:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:31:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:31:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:31:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:31:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:31:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:31:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:31:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:31:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:31:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:31:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:31:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:31:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:31:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:31:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:31:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:31:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:31:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:31:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:31:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:31:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:31:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:31:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:31:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:31:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:31:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:31:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:31:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:31:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:31:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:31:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:31:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:31:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:31:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:31:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:31:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:31:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:31:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:31:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:31:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:31:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:31:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:31:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:31:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:31:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:31:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:31:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:31:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:31:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:31:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:31:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:31:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:31:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:31:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:31:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:31:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:31:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:31:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:31:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:31:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:31:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:31:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:31:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:31:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:31:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:31:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:31:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:31:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:31:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:31:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:31:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:31:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:31:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:31:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:31:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:31:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:31:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:31:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:31:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:31:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:31:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:31:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:31:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:31:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:31:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:31:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:31:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:31:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:31:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:31:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:31:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:31:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:31:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:31:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:31:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:31:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:31:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:31:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:31:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:31:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:31:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:31:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:31:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:31:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:31:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:31:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:31:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:31:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:31:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:31:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:31:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:31:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:31:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:31:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:31:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:31:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:31:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:31:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:31:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:31:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:31:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:31:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:31:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:31:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:31:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:31:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:31:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:31:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:31:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:31:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:31:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:31:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:31:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:31:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:31:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:31:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:31:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:31:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:31:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:31:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:31:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:31:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:31:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:31:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:31:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:31:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:31:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:31:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:31:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:31:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:31:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:31:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:31:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:31:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:31:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:31:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:31:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:31:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:31:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:31:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:31:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:31:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:31:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:31:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:31:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:31:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:31:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:31:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:31:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:31:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:31:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:31:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:31:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:31:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:31:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:31:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:31:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:31:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:31:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:31:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:31:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:31:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:31:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:31:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:31:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:31:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:31:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:31:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:32:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:32:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:32:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:32:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:32:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:32:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:32:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:32:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:32:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:32:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:32:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:32:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:32:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:32:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:32:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:32:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:32:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:32:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:32:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:32:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:32:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:32:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:32:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:32:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:32:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:32:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:32:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:32:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:32:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:32:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:32:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:32:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:32:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:32:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:32:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:32:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:32:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:32:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:32:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:32:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:32:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:32:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:32:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:32:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:32:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:32:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:32:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:32:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:32:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:32:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:32:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:32:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:32:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:32:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:32:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:32:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:32:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:32:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:32:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:32:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:32:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:32:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:32:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:32:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:32:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:32:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:32:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:32:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:32:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:32:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:32:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:32:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:32:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:32:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:32:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:32:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:32:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:32:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:32:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:32:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:32:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:32:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:32:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:32:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:32:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:32:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:32:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:32:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:32:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:32:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:32:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:32:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:32:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:32:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:32:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:32:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:32:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:32:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:32:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:32:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:32:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:32:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:32:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:32:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:32:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:32:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:32:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:32:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:32:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:32:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:32:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:32:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:32:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:32:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:32:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:32:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:32:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:32:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:32:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:32:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:32:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:32:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:32:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:32:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:32:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:32:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:32:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:32:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:32:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:32:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:32:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:32:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:32:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:32:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:32:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:32:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:32:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:32:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:32:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:32:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:32:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:32:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:32:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:32:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:32:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:32:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:32:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:32:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:32:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:32:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:32:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:32:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:32:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:32:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:32:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:32:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:32:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:32:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:32:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:32:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:32:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:32:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:32:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:32:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:32:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:32:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:32:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:32:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:32:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:32:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:32:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:32:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:32:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:32:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:32:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:32:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:32:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:32:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:32:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:32:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:32:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:32:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:32:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:32:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:32:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:32:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:32:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:32:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:32:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:32:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:32:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:32:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:32:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:32:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:32:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:32:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:32:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:32:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:32:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:32:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:32:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:32:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:32:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:32:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:32:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:32:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:32:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:32:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:32:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:32:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:32:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:32:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:32:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:32:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:32:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:32:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:33:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:33:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:33:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:33:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:33:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:33:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:33:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:33:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:33:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:33:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:33:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:33:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:33:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:33:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:33:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:33:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:33:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:33:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:33:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:33:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:33:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:33:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:33:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:33:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:33:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:33:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:33:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:33:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:33:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:33:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:33:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:33:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:33:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:33:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:33:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:33:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:33:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:33:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:33:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:33:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:33:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:33:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:33:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:33:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:33:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:33:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:33:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:33:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:33:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:33:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:33:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:33:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:33:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:33:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:33:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:33:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:33:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:33:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:33:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:33:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:33:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:33:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:33:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:33:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:33:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:33:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:33:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:33:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:33:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:33:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:33:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:33:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:33:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:33:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:33:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:33:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:33:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:33:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:33:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:33:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:33:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:33:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:33:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:33:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:33:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:33:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:33:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:33:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:33:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:33:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:33:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:33:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:33:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:33:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:33:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:33:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:33:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:33:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:33:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:33:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:33:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:33:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:33:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:33:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:33:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:33:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:33:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:33:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:33:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:33:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:33:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:33:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:33:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:33:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:33:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:33:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:33:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:33:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:33:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:33:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:33:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:33:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:33:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:33:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:33:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:33:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:33:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:33:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:33:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:33:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:33:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:33:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:33:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:33:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:33:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:33:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:33:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:33:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:33:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:33:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:33:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:33:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:33:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:33:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:33:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:33:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:33:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:33:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:33:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:33:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:33:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:33:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:33:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:33:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:33:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:33:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:33:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:33:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:33:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:33:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:33:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:33:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:33:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:33:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:33:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:33:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:33:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:33:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:33:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:33:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:33:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:33:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:33:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:33:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:33:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:33:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:33:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:33:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:33:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:33:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:33:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:33:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:33:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:33:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:33:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:33:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:33:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:33:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:33:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:33:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:33:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:33:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:33:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:33:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:33:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:33:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:33:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:33:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:33:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:33:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:33:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:33:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:33:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:33:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:33:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:33:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:33:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:33:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:33:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:33:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:33:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:33:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:33:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:33:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:33:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:33:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:34:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:34:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:34:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:34:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:34:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:34:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:34:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:34:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:34:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:34:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:34:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:34:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:34:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:34:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:34:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:34:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:34:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:34:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:34:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:34:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:34:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:34:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:34:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:34:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:34:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:34:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:34:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:34:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:34:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:34:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:34:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:34:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:34:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:34:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:34:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:34:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:34:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:34:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:34:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:34:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:34:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:34:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:34:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:34:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:34:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:34:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:34:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:34:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:34:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:34:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:34:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:34:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:34:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:34:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:34:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:34:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:34:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:34:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:34:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:34:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:34:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:34:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:34:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:34:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:34:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:34:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:34:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:34:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:34:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:34:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:34:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:34:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:34:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:34:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:34:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:34:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:34:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:34:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:34:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:34:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:34:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:34:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:34:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:34:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:34:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:34:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:34:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:34:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:34:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:34:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:34:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:34:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:34:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:34:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:34:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:34:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:34:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:34:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:34:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:34:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:34:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:34:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:34:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:34:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:34:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:34:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:34:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:34:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:34:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:34:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:34:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:34:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:34:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:34:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:34:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:34:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:34:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:34:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:34:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:34:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:34:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:34:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:34:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:34:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:34:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:34:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:34:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:34:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:34:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:34:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:34:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:34:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:34:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:34:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:34:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:34:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:34:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:34:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:34:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:34:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:34:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:34:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:34:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:34:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:34:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:34:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:34:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:34:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:34:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:34:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:34:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:34:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:34:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:34:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:34:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:34:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:34:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:34:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:34:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:34:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:34:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:34:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:34:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:34:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:34:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:34:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:34:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:34:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:34:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:34:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:34:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:34:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:34:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:34:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:34:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:34:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:34:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:34:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:34:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:34:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:36:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:36:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:36:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:36:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:36:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:36:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:36:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:36:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:36:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:36:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:36:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:36:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:36:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:36:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:36:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:36:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:36:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:36:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:36:57 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 13:37:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:37:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:37:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:37:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:37:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:37:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:37:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:37:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:37:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:37:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:37:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:37:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:37:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:37:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:37:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:37:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:37:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:37:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:37:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:37:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:37:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:37:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:37:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:37:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:37:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:37:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:37:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:37:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:37:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:37:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:37:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:37:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:37:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:37:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:37:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:37:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:37:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:37:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:37:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:37:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:37:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:37:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:37:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:37:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:37:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:37:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:37:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:37:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:37:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:37:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:37:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:37:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:37:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:37:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:37:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:37:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:37:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:37:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:37:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:37:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:37:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:37:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:37:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:37:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:37:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:37:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:37:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:37:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:37:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:37:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:37:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:37:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:37:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:37:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:37:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:37:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:37:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:37:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:37:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:37:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:37:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:37:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:37:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:37:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:37:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:37:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:37:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:37:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:37:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:37:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:37:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:37:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:37:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:37:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:37:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:37:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:37:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:37:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:37:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:37:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:37:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:37:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:37:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:37:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:37:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:37:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:37:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:37:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:37:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:37:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:37:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:37:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:37:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:37:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:37:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:37:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:37:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:37:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:37:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:37:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:37:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:37:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:37:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:37:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:37:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:37:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:37:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:37:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:37:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:37:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:37:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:37:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:37:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:37:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:37:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:37:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:37:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:37:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:37:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:37:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:37:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:37:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:37:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:37:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:37:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:37:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:37:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:37:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:37:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:37:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:37:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:37:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:37:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:37:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:37:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:37:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:37:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:37:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:37:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:37:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:37:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:37:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:37:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:37:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:37:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:37:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:37:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:37:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:37:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:37:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:37:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:37:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:37:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:37:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:37:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:37:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:37:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:37:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:37:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:37:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:37:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:37:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:37:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:37:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:37:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:37:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:37:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:37:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:37:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:37:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:37:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:37:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:37:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:37:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:37:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:37:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:37:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:37:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:37:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:37:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:37:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:37:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:37:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:37:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:37:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:37:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:37:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:37:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:37:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:37:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:37:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:37:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:37:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:37:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:37:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:37:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:38:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:38:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:38:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:38:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:38:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:38:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:38:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:38:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:38:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:38:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:38:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:38:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:38:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:38:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:38:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:38:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:38:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:38:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:38:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:38:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:38:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:38:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:38:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:38:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:38:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:38:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:38:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:38:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:38:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:38:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:38:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:38:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:38:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:38:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:38:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:38:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:38:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:38:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:38:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:38:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:38:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:38:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:38:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:38:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:38:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:38:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:38:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:38:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:38:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:38:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:38:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:38:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:38:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:38:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:38:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:38:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:38:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:38:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:38:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:38:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:38:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:38:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:38:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:38:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:38:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:38:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:38:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:38:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:38:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:38:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:38:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:38:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:38:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:38:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:38:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:38:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:38:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:38:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:38:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:38:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:38:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:38:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:38:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:38:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:38:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:38:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:38:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:38:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:38:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:38:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:38:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:38:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:38:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:38:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:38:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:38:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:38:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:38:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:38:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:38:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:38:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:38:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:38:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:38:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:38:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:38:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:38:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:38:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:38:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:38:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:38:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:38:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:38:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:38:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:38:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:38:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:38:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:38:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:38:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:38:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:38:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:38:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:38:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:38:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:38:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:38:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:38:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:38:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:38:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:38:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:38:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:38:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:38:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:38:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:38:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:38:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:38:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:38:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:38:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:38:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:38:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:38:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:38:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:38:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:38:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:38:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:38:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:38:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:38:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:38:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:38:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:38:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:38:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:38:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:38:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:38:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:38:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:38:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:38:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:38:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:38:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:38:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:38:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:38:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:38:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:38:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:38:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:38:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:38:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:38:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:38:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:38:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:38:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:38:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:38:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:38:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:38:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:38:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:38:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:38:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:38:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:38:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:38:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:38:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:38:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:38:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:38:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:38:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:38:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:38:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:38:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:38:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:38:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:38:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:38:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:38:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:38:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:38:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:38:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:38:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:38:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:38:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:38:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:38:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:38:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:38:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:38:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:38:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:38:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:38:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:38:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:38:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:38:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:38:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:38:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:38:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:39:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:39:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:39:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:39:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:39:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:39:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:39:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:39:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:39:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:39:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:39:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:39:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:39:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:39:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:39:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:39:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:39:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:39:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:39:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:39:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:39:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:39:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:39:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:39:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:39:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:39:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:39:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:39:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:39:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:39:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:39:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:39:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:39:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:39:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:39:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:39:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:39:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:39:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:39:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:39:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:39:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:39:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:39:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:39:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:39:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:39:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:39:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:39:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:39:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:39:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:39:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:39:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:39:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:39:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:39:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:39:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:39:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:39:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:39:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:39:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:39:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:39:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:39:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:39:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:39:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:39:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:39:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:39:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:39:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:39:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:39:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:39:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:39:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:39:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:39:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:39:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:39:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:39:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:39:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:39:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:39:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:39:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:39:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:39:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:39:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:39:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:39:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:39:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:39:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:39:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:39:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:39:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:39:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:39:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:39:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:39:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:39:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:39:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:39:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:39:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:39:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:39:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:39:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:39:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:39:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:39:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:39:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:39:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:39:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:39:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:39:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:39:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:39:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:39:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:39:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:39:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:39:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:39:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:39:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:39:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:39:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:39:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:39:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:39:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:39:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:39:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:39:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:39:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:39:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:39:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:39:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:39:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:39:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:39:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:39:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:39:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:39:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:39:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:39:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:39:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:39:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:39:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:39:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:39:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:39:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:39:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:39:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:39:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:39:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:39:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:39:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:39:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:39:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:39:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:39:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:39:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:39:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:39:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:39:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:39:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:39:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:39:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:39:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:39:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:39:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:39:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:39:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:39:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:39:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:39:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:39:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:39:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:39:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:39:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:39:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:39:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:39:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:39:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:39:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:39:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:39:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:39:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:39:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:39:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:39:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:39:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:39:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:39:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:39:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:39:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:39:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:39:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:39:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:39:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:39:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:39:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:39:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:39:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:39:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:39:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:39:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:39:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:39:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:39:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:39:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:39:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:39:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:39:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:39:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:39:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:39:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:39:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:39:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:39:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:39:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:39:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:40:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:40:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:40:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:40:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:40:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:40:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:40:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:40:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:40:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:40:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:40:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:40:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:40:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:40:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:40:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:40:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:40:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:40:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:40:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:40:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:40:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:40:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:40:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:40:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:40:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:40:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:40:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:40:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:40:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:40:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:40:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:40:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:40:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:40:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:40:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:40:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:40:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:40:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:40:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:40:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:40:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:40:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:40:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:40:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:40:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:40:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:40:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:40:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:40:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:40:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:40:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:40:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:40:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:40:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:40:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:40:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:40:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:40:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:40:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:40:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:40:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:40:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:40:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:40:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:40:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:40:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:40:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:40:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:40:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:40:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:40:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:40:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:40:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:40:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:40:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:40:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:40:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:40:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:40:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:40:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:40:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:40:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:40:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:40:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:40:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:40:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:40:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:40:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:40:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:40:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:40:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:40:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:40:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:40:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:40:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:40:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:40:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:40:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:40:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:40:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:40:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:40:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:40:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:40:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:40:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:40:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:40:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:40:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:40:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:40:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:40:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:40:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:40:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:40:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:40:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:40:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:40:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:40:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:40:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:40:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:40:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:40:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:40:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:40:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:40:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:40:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:40:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:40:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:40:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:40:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:40:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:40:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:40:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:40:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:40:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:40:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:40:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:40:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:40:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:40:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:40:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:40:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:40:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:40:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:40:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:40:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:40:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:40:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:40:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:40:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:40:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:40:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:40:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:40:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:40:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:40:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:40:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:40:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:40:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:40:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:40:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:40:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:40:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:40:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:40:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:40:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:40:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:40:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:40:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:40:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:40:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:40:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:40:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:40:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:40:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:40:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:40:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:40:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:40:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:40:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:40:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:40:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:40:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:40:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:40:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:40:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:40:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:40:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:40:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:40:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:40:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:40:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:40:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:40:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:40:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:40:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:40:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:40:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:40:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:40:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:40:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:40:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:40:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:40:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:40:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:40:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:40:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:40:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:40:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:40:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:40:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:40:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:40:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:40:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:40:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:40:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:41:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:41:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:41:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:41:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:41:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:41:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:41:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:41:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:41:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:41:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:41:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:41:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:41:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:41:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:41:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:41:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:41:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:41:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:41:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:41:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:41:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:41:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:41:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:41:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:41:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:41:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:41:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:41:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:41:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:41:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:41:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:41:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:41:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:41:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:41:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:41:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:41:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:41:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:41:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:41:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:41:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:41:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:41:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:41:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:41:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:41:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:41:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:41:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:41:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:41:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:41:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:41:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:41:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:41:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:41:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:41:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:41:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:41:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:41:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:41:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:41:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:41:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:41:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:41:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:41:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:41:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:41:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:41:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:41:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:41:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:41:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:41:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:41:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:41:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:41:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:41:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:41:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:41:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:41:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:41:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:41:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:41:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:41:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:41:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:41:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:41:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:41:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:41:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:41:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:41:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:41:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:41:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:41:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:41:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:41:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:41:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:41:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:41:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:41:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:41:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:41:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:41:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:41:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:41:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:41:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:41:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:41:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:41:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:41:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:41:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:41:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:41:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:41:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:41:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:41:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:41:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:41:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:41:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:41:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:41:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:41:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:41:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:41:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:41:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:41:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:41:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:41:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:41:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:41:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:41:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:41:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:41:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:41:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:41:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:41:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:41:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:41:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:41:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:41:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:41:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:41:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:41:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:41:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:41:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:41:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:41:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:41:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:41:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:41:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:41:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:41:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:41:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:41:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:41:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:41:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:41:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:41:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:41:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:41:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:41:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:41:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:41:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:41:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:41:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:41:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:41:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:41:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:41:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:41:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:41:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:41:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:41:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:41:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:41:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:41:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:41:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:41:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:41:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:41:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:41:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:41:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:41:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:41:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:41:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:41:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:41:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:41:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:41:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:41:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:41:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:41:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:41:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:41:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:41:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:41:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:41:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:41:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:41:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:41:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:41:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:41:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:41:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:41:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:41:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:41:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:41:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:41:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:41:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:41:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:41:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:41:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:41:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:41:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:41:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:41:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:41:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:42:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:42:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:42:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:42:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:42:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:42:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:42:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:42:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:42:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:42:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:42:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:42:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:42:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:42:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:42:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:42:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:42:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:42:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:42:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:42:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:42:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:42:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:42:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:42:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:42:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:42:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:42:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:42:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:42:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:42:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:42:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:42:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:42:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:42:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:42:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:42:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:42:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:42:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:42:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:42:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:42:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:42:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:42:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:42:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:42:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:42:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:42:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:42:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:42:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:42:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:42:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:42:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:42:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:42:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:42:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:42:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:42:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:42:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:42:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:42:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:42:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:42:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:42:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:42:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:42:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:42:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:42:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:42:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:42:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:42:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:42:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:42:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:42:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:42:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:42:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:42:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:42:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:42:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:42:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:42:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:42:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:42:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:42:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:42:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:42:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:42:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:42:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:42:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:42:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:42:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:42:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:42:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:42:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:42:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:42:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:42:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:42:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:42:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:42:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:42:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:42:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:42:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:42:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:42:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:42:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:42:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:42:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:42:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:42:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:42:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:42:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:42:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:42:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:42:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:42:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:42:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:42:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:42:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:42:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:42:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:42:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:42:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:42:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:42:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:42:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:42:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:42:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:42:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:42:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:42:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:42:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:42:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:42:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:42:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:42:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:42:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:42:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:42:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:42:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:42:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:42:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:42:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:42:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:42:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:42:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:42:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:42:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:42:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:42:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:42:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:42:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:42:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:42:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:42:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:42:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:42:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:42:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:42:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:42:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:42:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:42:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:42:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:42:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:42:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:42:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:42:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:42:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:42:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:42:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:42:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:42:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:42:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:42:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:42:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:42:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:42:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:42:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:42:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:42:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:42:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:42:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:42:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:42:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:42:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:42:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:42:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:42:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:42:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:42:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:42:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:42:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:42:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:42:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:42:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:42:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:42:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:42:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:42:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:42:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:42:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:42:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:42:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:42:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:42:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:42:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:42:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:42:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:42:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:42:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:42:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:42:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:42:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:42:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:42:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:42:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:42:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:43:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:43:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:43:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:43:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:43:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:43:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:43:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:43:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:43:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:43:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:43:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:43:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:43:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:43:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:43:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:43:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:43:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:43:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:43:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:43:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:43:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:43:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:43:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:43:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:43:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:43:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:43:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:43:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:43:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:43:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:43:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:43:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:43:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:43:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:43:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:43:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:43:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:43:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:43:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:43:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:43:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:43:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:43:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:43:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:43:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:43:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:43:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:43:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:43:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:43:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:43:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:43:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:43:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:43:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:43:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:43:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:43:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:43:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:43:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:43:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:43:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:43:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:43:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:43:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:43:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:43:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:43:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:43:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:43:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:43:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:43:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:43:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:43:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:43:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:43:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:43:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:43:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:43:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:43:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:43:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:43:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:43:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:43:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:43:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:43:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:43:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:43:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:43:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:43:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:43:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:43:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:43:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:43:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:43:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:43:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:43:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:43:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:43:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:43:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:43:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:43:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:43:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:43:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:43:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:43:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:43:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:43:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:43:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:43:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:43:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:43:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:43:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:43:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:43:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:43:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:43:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:43:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:43:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:43:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:43:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:43:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:43:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:43:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:43:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:43:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:43:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:43:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:43:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:43:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:43:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:43:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:43:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:43:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:43:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:43:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:43:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:43:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:43:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:43:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:43:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:43:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:43:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:43:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:43:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:43:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:43:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:43:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:43:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:43:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:43:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:43:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:43:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:43:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:43:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:43:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:43:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:43:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:43:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:43:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:43:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:43:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:43:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:43:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:43:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:43:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:43:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:43:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:43:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:43:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:43:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:43:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:43:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:43:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:43:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:43:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:43:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:43:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:43:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:43:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:43:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:43:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:43:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:43:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:43:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:43:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:43:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:43:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:43:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:43:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:43:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:43:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:43:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:43:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:43:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:43:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:43:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:43:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:43:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:43:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:43:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:43:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:43:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:43:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:43:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:43:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:43:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:43:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:43:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:43:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:43:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:43:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:43:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:43:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:43:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:43:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:43:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:44:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:44:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:44:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:44:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:44:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:44:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:44:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:44:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:44:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:44:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:44:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:44:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:44:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:44:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:44:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:44:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:44:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:44:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:44:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:44:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:44:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:44:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:44:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:44:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:44:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:44:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:44:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:44:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:44:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:44:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:44:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:44:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:44:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:44:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:44:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:44:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:44:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:44:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:44:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:44:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:44:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:44:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:44:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:44:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:44:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:44:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:44:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:44:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:44:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:44:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:44:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:44:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:44:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:44:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:44:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:44:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:44:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:44:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:44:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:44:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:44:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:44:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:44:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:44:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:44:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:44:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:44:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:44:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:44:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:44:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:44:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:44:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:44:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:44:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:44:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:44:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:44:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:44:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:44:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:44:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:44:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:44:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:44:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:44:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:44:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:44:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:44:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:44:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:44:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:44:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:44:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:44:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:44:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:44:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:44:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:44:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:44:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:44:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:44:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:44:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:44:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:44:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:44:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:44:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:44:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:44:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:44:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:44:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:44:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:44:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:44:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:44:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:44:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:44:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:44:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:44:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:44:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:44:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:44:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:44:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:44:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:44:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:44:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:44:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:44:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:44:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:44:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:44:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:44:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:44:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:44:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:44:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:44:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:44:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:44:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:44:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:44:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:44:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:44:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:44:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:44:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:44:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:44:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:44:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:44:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:44:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:44:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:44:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:44:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:44:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:44:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:44:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:44:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:44:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:44:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:44:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:44:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:44:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:44:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:44:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:44:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:44:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:44:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:44:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:44:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:44:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:44:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:44:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:44:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:44:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:44:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:44:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:44:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:44:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:44:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:44:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:44:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:44:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:44:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:44:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:44:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:44:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:44:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:44:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:44:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:44:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:44:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:44:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:44:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:44:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:44:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:44:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:44:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:44:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:44:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:44:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:44:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:44:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:46:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:46:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:46:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:46:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:46:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:46:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:46:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:46:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:46:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:46:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:46:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:46:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:46:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:46:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:46:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:46:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:46:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:46:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:46:58 rdma06 rsyslogd[1041]: imjournal: 5082 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 13:47:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:47:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:47:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:47:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:47:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:47:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:47:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:47:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:47:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:47:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:47:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:47:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:47:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:47:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:47:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:47:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:47:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:47:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:47:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:47:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:47:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:47:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:47:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:47:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:47:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:47:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:47:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:47:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:47:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:47:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:47:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:47:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:47:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:47:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:47:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:47:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:47:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:47:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:47:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:47:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:47:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:47:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:47:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:47:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:47:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:47:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:47:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:47:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:47:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:47:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:47:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:47:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:47:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:47:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:47:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:47:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:47:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:47:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:47:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:47:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:47:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:47:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:47:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:47:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:47:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:47:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:47:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:47:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:47:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:47:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:47:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:47:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:47:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:47:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:47:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:47:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:47:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:47:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:47:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:47:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:47:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:47:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:47:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:47:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:47:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:47:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:47:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:47:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:47:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:47:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:47:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:47:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:47:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:47:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:47:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:47:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:47:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:47:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:47:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:47:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:47:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:47:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:47:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:47:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:47:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:47:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:47:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:47:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:47:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:47:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:47:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:47:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:47:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:47:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:47:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:47:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:47:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:47:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:47:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:47:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:47:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:47:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:47:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:47:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:47:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:47:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:47:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:47:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:47:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:47:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:47:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:47:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:47:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:47:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:47:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:47:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:47:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:47:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:47:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:47:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:47:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:47:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:47:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:47:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:47:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:47:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:47:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:47:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:47:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:47:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:47:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:47:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:47:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:47:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:47:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:47:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:47:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:47:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:47:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:47:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:47:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:47:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:47:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:47:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:47:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:47:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:47:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:47:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:47:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:47:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:47:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:47:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:47:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:47:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:47:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:47:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:47:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:47:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:47:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:47:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:47:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:47:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:47:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:47:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:47:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:47:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:47:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:47:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:47:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:47:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:47:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:47:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:47:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:47:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:47:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:47:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:47:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:47:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:47:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:47:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:47:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:47:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:47:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:47:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:47:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:47:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:47:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:47:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:47:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:47:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:47:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:47:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:47:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:47:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:47:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:47:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:48:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:48:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:48:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:48:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:48:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:48:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:48:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:48:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:48:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:48:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:48:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:48:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:48:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:48:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:48:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:48:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:48:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:48:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:48:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:48:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:48:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:48:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:48:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:48:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:48:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:48:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:48:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:48:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:48:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:48:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:48:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:48:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:48:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:48:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:48:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:48:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:48:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:48:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:48:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:48:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:48:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:48:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:48:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:48:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:48:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:48:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:48:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:48:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:48:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:48:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:48:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:48:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:48:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:48:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:48:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:48:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:48:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:48:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:48:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:48:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:48:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:48:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:48:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:48:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:48:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:48:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:48:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:48:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:48:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:48:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:48:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:48:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:48:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:48:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:48:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:48:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:48:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:48:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:48:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:48:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:48:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:48:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:48:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:48:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:48:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:48:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:48:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:48:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:48:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:48:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:48:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:48:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:48:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:48:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:48:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:48:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:48:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:48:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:48:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:48:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:48:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:48:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:48:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:48:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:48:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:48:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:48:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:48:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:48:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:48:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:48:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:48:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:48:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:48:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:48:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:48:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:48:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:48:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:48:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:48:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:48:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:48:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:48:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:48:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:48:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:48:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:48:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:48:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:48:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:48:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:48:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:48:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:48:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:48:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:48:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:48:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:48:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:48:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:48:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:48:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:48:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:48:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:48:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:48:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:48:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:48:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:48:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:48:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:48:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:48:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:48:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:48:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:48:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:48:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:48:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:48:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:48:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:48:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:48:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:48:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:48:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:48:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:48:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:48:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:48:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:48:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:48:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:48:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:48:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:48:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:48:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:48:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:48:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:48:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:48:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:48:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:48:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:48:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:48:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:48:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:48:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:48:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:48:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:48:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:48:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:48:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:48:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:48:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:48:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:48:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:48:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:48:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:48:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:48:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:48:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:48:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:48:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:48:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:48:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:48:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:48:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:48:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:48:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:48:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:48:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:48:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:48:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:48:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:48:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:48:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:48:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:48:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:48:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:48:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:48:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:48:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:49:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:49:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:49:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:49:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:49:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:49:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:49:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:49:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:49:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:49:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:49:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:49:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:49:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:49:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:49:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:49:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:49:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:49:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:49:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:49:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:49:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:49:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:49:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:49:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:49:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:49:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:49:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:49:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:49:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:49:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:49:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:49:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:49:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:49:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:49:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:49:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:49:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:49:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:49:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:49:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:49:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:49:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:49:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:49:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:49:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:49:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:49:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:49:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:49:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:49:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:49:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:49:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:49:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:49:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:49:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:49:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:49:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:49:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:49:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:49:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:49:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:49:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:49:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:49:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:49:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:49:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:49:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:49:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:49:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:49:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:49:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:49:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:49:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:49:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:49:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:49:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:49:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:49:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:49:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:49:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:49:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:49:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:49:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:49:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:49:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:49:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:49:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:49:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:49:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:49:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:49:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:49:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:49:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:49:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:49:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:49:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:49:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:49:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:49:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:49:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:49:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:49:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:49:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:49:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:49:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:49:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:49:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:49:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:49:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:49:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:49:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:49:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:49:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:49:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:49:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:49:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:49:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:49:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:49:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:49:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:49:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:49:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:49:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:49:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:49:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:49:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:49:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:49:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:49:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:49:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:49:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:49:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:49:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:49:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:49:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:49:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:49:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:49:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:49:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:49:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:49:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:49:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:49:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:49:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:49:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:49:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:49:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:49:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:49:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:49:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:49:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:49:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:49:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:49:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:49:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:49:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:49:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:49:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:49:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:49:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:49:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:49:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:49:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:49:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:49:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:49:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:49:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:49:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:49:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:49:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:49:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:49:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:49:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:49:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:49:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:49:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:49:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:49:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:49:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:49:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:49:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:49:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:49:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:49:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:49:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:49:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:49:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:49:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:49:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:49:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:49:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:49:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:49:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:49:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:49:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:49:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:49:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:49:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:49:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:49:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:49:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:49:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:49:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:49:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:49:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:49:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:49:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:49:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:49:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:49:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:49:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:49:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:49:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:49:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:49:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:49:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:50:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:50:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:50:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:50:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:50:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:50:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:50:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:50:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:50:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:50:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:50:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:50:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:50:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:50:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:50:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:50:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:50:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:50:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:50:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:50:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:50:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:50:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:50:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:50:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:50:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:50:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:50:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:50:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:50:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:50:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:50:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:50:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:50:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:50:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:50:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:50:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:50:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:50:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:50:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:50:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:50:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:50:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:50:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:50:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:50:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:50:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:50:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:50:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:50:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:50:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:50:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:50:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:50:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:50:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:50:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:50:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:50:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:50:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:50:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:50:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:50:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:50:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:50:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:50:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:50:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:50:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:50:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:50:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:50:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:50:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:50:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:50:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:50:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:50:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:50:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:50:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:50:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:50:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:50:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:50:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:50:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:50:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:50:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:50:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:50:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:50:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:50:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:50:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:50:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:50:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:50:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:50:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:50:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:50:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:50:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:50:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:50:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:50:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:50:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:50:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:50:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:50:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:50:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:50:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:50:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:50:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:50:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:50:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:50:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:50:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:50:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:50:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:50:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:50:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:50:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:50:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:50:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:50:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:50:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:50:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:50:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:50:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:50:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:50:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:50:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:50:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:50:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:50:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:50:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:50:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:50:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:50:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:50:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:50:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:50:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:50:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:50:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:50:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:50:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:50:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:50:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:50:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:50:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:50:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:50:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:50:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:50:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:50:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:50:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:50:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:50:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:50:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:50:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:50:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:50:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:50:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:50:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:50:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:50:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:50:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:50:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:50:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:50:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:50:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:50:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:50:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:50:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:50:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:50:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:50:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:50:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:50:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:50:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:50:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:50:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:50:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:50:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:50:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:50:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:50:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:50:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:50:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:50:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:50:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:50:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:50:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:50:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:50:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:50:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:50:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:50:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:50:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:50:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:50:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:50:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:50:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:50:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:50:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:50:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:50:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:50:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:50:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:50:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:50:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:50:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:50:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:50:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:50:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:50:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:50:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:50:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:50:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:50:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:50:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:50:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:50:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:51:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:51:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:51:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:51:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:51:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:51:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:51:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:51:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:51:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:51:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:51:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:51:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:51:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:51:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:51:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:51:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:51:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:51:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:51:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:51:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:51:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:51:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:51:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:51:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:51:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:51:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:51:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:51:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:51:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:51:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:51:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:51:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:51:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:51:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:51:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:51:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:51:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:51:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:51:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:51:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:51:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:51:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:51:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:51:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:51:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:51:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:51:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:51:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:51:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:51:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:51:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:51:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:51:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:51:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:51:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:51:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:51:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:51:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:51:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:51:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:51:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:51:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:51:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:51:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:51:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:51:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:51:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:51:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:51:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:51:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:51:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:51:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:51:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:51:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:51:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:51:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:51:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:51:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:51:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:51:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:51:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:51:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:51:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:51:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:51:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:51:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:51:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:51:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:51:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:51:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:51:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:51:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:51:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:51:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:51:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:51:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:51:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:51:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:51:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:51:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:51:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:51:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:51:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:51:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:51:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:51:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:51:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:51:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:51:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:51:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:51:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:51:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:51:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:51:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:51:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:51:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:51:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:51:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:51:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:51:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:51:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:51:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:51:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:51:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:51:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:51:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:51:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:51:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:51:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:51:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:51:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:51:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:51:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:51:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:51:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:51:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:51:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:51:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:51:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:51:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:51:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:51:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:51:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:51:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:51:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:51:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:51:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:51:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:51:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:51:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:51:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:51:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:51:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:51:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:51:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:51:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:51:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:51:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:51:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:51:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:51:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:51:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:51:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:51:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:51:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:51:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:51:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:51:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:51:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:51:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:51:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:51:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:51:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:51:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:51:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:51:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:51:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:51:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:51:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:51:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:51:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:51:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:51:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:51:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:51:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:51:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:51:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:51:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:51:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:51:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:51:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:51:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:51:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:51:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:51:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:51:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:51:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:51:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:51:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:51:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:51:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:51:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:51:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:51:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:51:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:51:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:51:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:51:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:51:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:51:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:51:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:51:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:51:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:51:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:51:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:51:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:52:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:52:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:52:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:52:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:52:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:52:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:52:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:52:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:52:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:52:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:52:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:52:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:52:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:52:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:52:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:52:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:52:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:52:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:52:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:52:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:52:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:52:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:52:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:52:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:52:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:52:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:52:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:52:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:52:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:52:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:52:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:52:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:52:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:52:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:52:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:52:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:52:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:52:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:52:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:52:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:52:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:52:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:52:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:52:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:52:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:52:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:52:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:52:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:52:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:52:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:52:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:52:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:52:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:52:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:52:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:52:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:52:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:52:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:52:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:52:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:52:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:52:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:52:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:52:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:52:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:52:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:52:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:52:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:52:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:52:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:52:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:52:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:52:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:52:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:52:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:52:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:52:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:52:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:52:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:52:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:52:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:52:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:52:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:52:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:52:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:52:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:52:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:52:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:52:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:52:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:52:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:52:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:52:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:52:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:52:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:52:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:52:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:52:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:52:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:52:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:52:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:52:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:52:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:52:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:52:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:52:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:52:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:52:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:52:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:52:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:52:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:52:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:52:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:52:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:52:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:52:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:52:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:52:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:52:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:52:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:52:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:52:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:52:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:52:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:52:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:52:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:52:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:52:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:52:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:52:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:52:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:52:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:52:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:52:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:52:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:52:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:52:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:52:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:52:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:52:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:52:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:52:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:52:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:52:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:52:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:52:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:52:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:52:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:52:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:52:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:52:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:52:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:52:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:52:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:52:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:52:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:52:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:52:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:52:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:52:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:52:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:52:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:52:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:52:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:52:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:52:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:52:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:52:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:52:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:52:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:52:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:52:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:52:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:52:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:52:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:52:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:52:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:52:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:52:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:52:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:52:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:52:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:52:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:52:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:52:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:52:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:52:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:52:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:52:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:52:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:52:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:52:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:52:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:52:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:52:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:52:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:52:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:52:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:52:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:52:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:52:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:52:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:52:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:52:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:52:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:52:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:52:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:52:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:52:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:52:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:52:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:52:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:52:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:52:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:52:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:52:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:53:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:53:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:53:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:53:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:53:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:53:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:53:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:53:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:53:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:53:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:53:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:53:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:53:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:53:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:53:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:53:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:53:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:53:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:53:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:53:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:53:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:53:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:53:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:53:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:53:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:53:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:53:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:53:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:53:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:53:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:53:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:53:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:53:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:53:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:53:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:53:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:53:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:53:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:53:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:53:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:53:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:53:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:53:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:53:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:53:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:53:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:53:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:53:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:53:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:53:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:53:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:53:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:53:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:53:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:53:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:53:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:53:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:53:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:53:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:53:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:53:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:53:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:53:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:53:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:53:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:53:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:53:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:53:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:53:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:53:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:53:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:53:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:53:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:53:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:53:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:53:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:53:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:53:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:53:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:53:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:53:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:53:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:53:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:53:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:53:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:53:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:53:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:53:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:53:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:53:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:53:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:53:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:53:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:53:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:53:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:53:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:53:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:53:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:53:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:53:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:53:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:53:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:53:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:53:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:53:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:53:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:53:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:53:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:53:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:53:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:53:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:53:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:53:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:53:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:53:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:53:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:53:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:53:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:53:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:53:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:53:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:53:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:53:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:53:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:53:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:53:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:53:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:53:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:53:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:53:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:53:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:53:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:53:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:53:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:53:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:53:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:53:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:53:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:53:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:53:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:53:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:53:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:53:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:53:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:53:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:53:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:53:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:53:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:53:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:53:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:53:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:53:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:53:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:53:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:53:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:53:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:53:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:53:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:53:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:53:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:53:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:53:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:53:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:53:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:53:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:53:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:53:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:53:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:53:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:53:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:53:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:53:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:53:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:53:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:53:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:53:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:53:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:53:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:53:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:53:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:53:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:53:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:53:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:53:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:53:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:53:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:53:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:53:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:53:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:53:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:53:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:53:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:53:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:53:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:53:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:53:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:53:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:53:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:53:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:53:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:53:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:53:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:53:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:53:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:53:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:53:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:53:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:53:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:53:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:53:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:53:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:53:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:53:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:53:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:53:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:53:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:54:00 rdma06 systemd[1]: Starting dnf makecache...
Apr 17 13:54:01 rdma06 dnf[1157222]: Updating Subscription Management repositories.
Apr 17 13:54:01 rdma06 dnf[1157222]: Unable to read consumer identity
Apr 17 13:54:01 rdma06 dnf[1157222]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 17 13:54:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:54:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:54:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:54:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:54:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:54:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:54:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:54:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:54:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:54:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:54:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:54:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:54:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:54:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:54:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:54:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:54:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:54:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:54:03 rdma06 dnf[1157222]: Copr repo for fmf owned by psss                 1.7 kB/s | 3.3 kB     00:01
Apr 17 13:54:03 rdma06 dnf[1157222]: beaker-AppStream                                 27 kB/s | 168  B     00:00
Apr 17 13:54:03 rdma06 dnf[1157222]: Errors during downloading metadata for repository 'beaker-AppStream':
Apr 17 13:54:03 rdma06 dnf[1157222]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 13:54:03 rdma06 dnf[1157222]: Error: Failed to download metadata for repo 'beaker-AppStream': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 13:54:03 rdma06 dnf[1157222]: beaker-AppStream-debuginfo                       55 kB/s | 168  B     00:00
Apr 17 13:54:03 rdma06 dnf[1157222]: Errors during downloading metadata for repository 'beaker-AppStream-debuginfo':
Apr 17 13:54:03 rdma06 dnf[1157222]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 13:54:03 rdma06 dnf[1157222]: Error: Failed to download metadata for repo 'beaker-AppStream-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 13:54:03 rdma06 dnf[1157222]: beaker-BaseOS                                    57 kB/s | 168  B     00:00
Apr 17 13:54:03 rdma06 dnf[1157222]: Errors during downloading metadata for repository 'beaker-BaseOS':
Apr 17 13:54:03 rdma06 dnf[1157222]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 13:54:03 rdma06 dnf[1157222]: Error: Failed to download metadata for repo 'beaker-BaseOS': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 13:54:03 rdma06 dnf[1157222]: beaker-BaseOS-debuginfo                          58 kB/s | 168  B     00:00
Apr 17 13:54:03 rdma06 dnf[1157222]: Errors during downloading metadata for repository 'beaker-BaseOS-debuginfo':
Apr 17 13:54:03 rdma06 dnf[1157222]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 13:54:03 rdma06 dnf[1157222]: Error: Failed to download metadata for repo 'beaker-BaseOS-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 13:54:03 rdma06 dnf[1157222]: beaker-CRB                                       56 kB/s | 168  B     00:00
Apr 17 13:54:03 rdma06 dnf[1157222]: Errors during downloading metadata for repository 'beaker-CRB':
Apr 17 13:54:03 rdma06 dnf[1157222]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 13:54:03 rdma06 dnf[1157222]: Error: Failed to download metadata for repo 'beaker-CRB': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 13:54:03 rdma06 dnf[1157222]: beaker-CRB-debuginfo                             58 kB/s | 168  B     00:00
Apr 17 13:54:03 rdma06 dnf[1157222]: Errors during downloading metadata for repository 'beaker-CRB-debuginfo':
Apr 17 13:54:03 rdma06 dnf[1157222]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 13:54:03 rdma06 dnf[1157222]: Error: Failed to download metadata for repo 'beaker-CRB-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 13:54:03 rdma06 dnf[1157222]: beaker-HighAvailability                          58 kB/s | 168  B     00:00
Apr 17 13:54:03 rdma06 dnf[1157222]: Errors during downloading metadata for repository 'beaker-HighAvailability':
Apr 17 13:54:03 rdma06 dnf[1157222]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 13:54:03 rdma06 dnf[1157222]: Error: Failed to download metadata for repo 'beaker-HighAvailability': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 13:54:03 rdma06 dnf[1157222]: beaker-HighAvailability-debuginfo                58 kB/s | 168  B     00:00
Apr 17 13:54:03 rdma06 dnf[1157222]: Errors during downloading metadata for repository 'beaker-HighAvailability-debuginfo':
Apr 17 13:54:03 rdma06 dnf[1157222]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 13:54:03 rdma06 dnf[1157222]: Error: Failed to download metadata for repo 'beaker-HighAvailability-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 13:54:03 rdma06 dnf[1157222]: beaker-NFV                                       59 kB/s | 168  B     00:00
Apr 17 13:54:03 rdma06 dnf[1157222]: Errors during downloading metadata for repository 'beaker-NFV':
Apr 17 13:54:03 rdma06 dnf[1157222]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 13:54:03 rdma06 dnf[1157222]: Error: Failed to download metadata for repo 'beaker-NFV': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 13:54:03 rdma06 dnf[1157222]: beaker-NFV-debuginfo                             57 kB/s | 168  B     00:00
Apr 17 13:54:03 rdma06 dnf[1157222]: Errors during downloading metadata for repository 'beaker-NFV-debuginfo':
Apr 17 13:54:03 rdma06 dnf[1157222]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 13:54:03 rdma06 dnf[1157222]: Error: Failed to download metadata for repo 'beaker-NFV-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 13:54:03 rdma06 dnf[1157222]: beaker-RT                                        59 kB/s | 168  B     00:00
Apr 17 13:54:03 rdma06 dnf[1157222]: Errors during downloading metadata for repository 'beaker-RT':
Apr 17 13:54:03 rdma06 dnf[1157222]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 13:54:03 rdma06 dnf[1157222]: Error: Failed to download metadata for repo 'beaker-RT': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 13:54:03 rdma06 dnf[1157222]: beaker-RT-debuginfo                              54 kB/s | 168  B     00:00
Apr 17 13:54:03 rdma06 dnf[1157222]: Errors during downloading metadata for repository 'beaker-RT-debuginfo':
Apr 17 13:54:03 rdma06 dnf[1157222]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 13:54:03 rdma06 dnf[1157222]: Error: Failed to download metadata for repo 'beaker-RT-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 13:54:03 rdma06 dnf[1157222]: beaker-ResilientStorage                          61 kB/s | 168  B     00:00
Apr 17 13:54:03 rdma06 dnf[1157222]: Errors during downloading metadata for repository 'beaker-ResilientStorage':
Apr 17 13:54:03 rdma06 dnf[1157222]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 13:54:03 rdma06 dnf[1157222]: Error: Failed to download metadata for repo 'beaker-ResilientStorage': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 13:54:03 rdma06 dnf[1157222]: beaker-ResilientStorage-debuginfo                55 kB/s | 168  B     00:00
Apr 17 13:54:03 rdma06 dnf[1157222]: Errors during downloading metadata for repository 'beaker-ResilientStorage-debuginfo':
Apr 17 13:54:03 rdma06 dnf[1157222]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 13:54:03 rdma06 dnf[1157222]: Error: Failed to download metadata for repo 'beaker-ResilientStorage-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 13:54:03 rdma06 dnf[1157222]: beaker-SAP                                       57 kB/s | 168  B     00:00
Apr 17 13:54:03 rdma06 dnf[1157222]: Errors during downloading metadata for repository 'beaker-SAP':
Apr 17 13:54:03 rdma06 dnf[1157222]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 13:54:03 rdma06 dnf[1157222]: Error: Failed to download metadata for repo 'beaker-SAP': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 13:54:03 rdma06 dnf[1157222]: beaker-SAP-debuginfo                             55 kB/s | 168  B     00:00
Apr 17 13:54:03 rdma06 dnf[1157222]: Errors during downloading metadata for repository 'beaker-SAP-debuginfo':
Apr 17 13:54:03 rdma06 dnf[1157222]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 13:54:03 rdma06 dnf[1157222]: Error: Failed to download metadata for repo 'beaker-SAP-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 13:54:03 rdma06 dnf[1157222]: beaker-SAPHANA                                   56 kB/s | 168  B     00:00
Apr 17 13:54:03 rdma06 dnf[1157222]: Errors during downloading metadata for repository 'beaker-SAPHANA':
Apr 17 13:54:03 rdma06 dnf[1157222]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 13:54:03 rdma06 dnf[1157222]: Error: Failed to download metadata for repo 'beaker-SAPHANA': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 13:54:03 rdma06 dnf[1157222]: beaker-SAPHANA-debuginfo                         63 kB/s | 168  B     00:00
Apr 17 13:54:03 rdma06 dnf[1157222]: Errors during downloading metadata for repository 'beaker-SAPHANA-debuginfo':
Apr 17 13:54:03 rdma06 dnf[1157222]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 13:54:03 rdma06 dnf[1157222]: Error: Failed to download metadata for repo 'beaker-SAPHANA-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 13:54:04 rdma06 dnf[1157222]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr 17 13:54:04 rdma06 dnf[1157222]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr 17 13:54:04 rdma06 dnf[1157222]: Ignoring repositories: beaker-AppStream, beaker-AppStream-debuginfo, beaker-BaseOS, beaker-BaseOS-debuginfo, beaker-CRB, beaker-CRB-debuginfo, beaker-HighAvailability, beaker-HighAvailability-debuginfo, beaker-NFV, beaker-NFV-debuginfo, beaker-RT, beaker-RT-debuginfo, beaker-ResilientStorage, beaker-ResilientStorage-debuginfo, beaker-SAP, beaker-SAP-debuginfo, beaker-SAPHANA, beaker-SAPHANA-debuginfo
Apr 17 13:54:04 rdma06 dnf[1157222]: Metadata cache created.
Apr 17 13:54:04 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 17 13:54:04 rdma06 systemd[1]: Finished dnf makecache.
Apr 17 13:54:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:54:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:54:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:54:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:54:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:54:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:54:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:54:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:54:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:54:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:54:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:54:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:54:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:54:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:54:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:54:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:54:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:54:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:54:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:54:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:54:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:54:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:54:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:54:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:54:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:54:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:54:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:54:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:54:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:54:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:54:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:54:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:54:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:54:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:54:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:54:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:54:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:54:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:54:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:54:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:54:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:54:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:54:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:54:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:54:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:54:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:54:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:54:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:54:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:54:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:54:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:54:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:54:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:54:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:54:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:54:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:54:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:54:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:54:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:54:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:54:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:54:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:54:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:54:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:54:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:54:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:54:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:54:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:54:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:54:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:54:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:54:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:54:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:54:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:54:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:54:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:54:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:54:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:54:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:54:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:54:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:54:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:54:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:54:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:54:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:54:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:54:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:54:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:54:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:54:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:54:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:54:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:54:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:54:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:54:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:54:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:54:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:54:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:54:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:54:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:54:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:54:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:54:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:54:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:54:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:54:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:54:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:54:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:54:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:54:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:54:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:54:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:54:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:54:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:54:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:54:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:54:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:54:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:54:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:54:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:54:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:54:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:54:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:54:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:54:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:54:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:54:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:54:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:54:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:54:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:54:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:54:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:54:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:54:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:54:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:54:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:54:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:54:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:54:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:54:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:54:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:54:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:54:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:54:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:54:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:54:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:54:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:54:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:54:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:54:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:54:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:54:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:54:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:54:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:54:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:54:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:54:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:54:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:54:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:54:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:54:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:54:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:54:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:54:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:54:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:54:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:54:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:54:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:54:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:54:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:54:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:54:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:54:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:54:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:54:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:54:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:54:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:57:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:57:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:57:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:57:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:57:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:57:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:57:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:57:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:57:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:57:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:57:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:57:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:57:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:57:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:57:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:57:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:57:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:57:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:57:03 rdma06 rsyslogd[1041]: imjournal: 5374 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 13:57:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:57:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:57:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:57:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:57:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:57:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:57:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:57:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:57:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:57:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:57:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:57:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:57:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:57:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:57:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:57:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:57:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:57:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:57:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:57:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:57:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:57:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:57:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:57:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:57:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:57:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:57:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:57:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:57:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:57:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:57:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:57:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:57:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:57:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:57:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:57:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:57:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:57:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:57:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:57:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:57:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:57:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:57:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:57:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:57:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:57:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:57:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:57:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:57:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:57:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:57:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:57:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:57:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:57:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:57:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:57:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:57:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:57:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:57:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:57:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:57:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:57:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:57:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:57:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:57:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:57:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:57:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:57:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:57:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:57:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:57:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:57:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:57:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:57:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:57:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:57:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:57:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:57:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:57:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:57:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:57:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:57:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:57:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:57:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:57:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:57:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:57:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:57:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:57:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:57:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:57:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:57:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:57:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:57:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:57:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:57:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:57:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:57:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:57:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:57:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:57:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:57:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:57:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:57:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:57:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:57:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:57:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:57:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:57:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:57:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:57:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:57:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:57:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:57:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:57:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:57:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:57:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:57:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:57:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:57:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:57:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:57:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:57:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:57:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:57:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:57:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:57:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:57:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:57:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:57:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:57:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:57:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:57:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:57:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:57:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:57:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:57:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:57:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:57:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:57:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:57:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:57:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:57:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:57:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:57:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:57:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:57:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:57:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:57:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:57:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:57:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:57:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:57:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:57:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:57:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:57:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:57:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:57:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:57:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:57:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:57:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:57:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:57:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:57:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:57:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:57:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:57:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:57:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:57:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:57:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:57:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:57:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:57:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:57:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:57:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:57:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:57:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:57:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:57:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:57:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:57:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:57:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:57:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:57:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:57:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:57:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:57:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:57:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:57:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:57:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:57:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:57:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:57:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:57:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:57:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:57:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:57:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:57:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:58:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:58:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:58:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:58:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:58:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:58:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:58:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:58:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:58:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:58:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:58:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:58:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:58:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:58:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:58:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:58:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:58:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:58:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:58:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:58:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:58:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:58:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:58:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:58:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:58:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:58:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:58:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:58:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:58:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:58:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:58:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:58:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:58:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:58:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:58:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:58:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:58:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:58:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:58:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:58:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:58:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:58:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:58:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:58:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:58:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:58:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:58:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:58:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:58:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:58:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:58:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:58:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:58:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:58:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:58:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:58:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:58:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:58:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:58:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:58:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:58:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:58:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:58:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:58:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:58:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:58:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:58:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:58:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:58:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:58:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:58:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:58:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:58:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:58:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:58:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:58:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:58:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:58:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:58:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:58:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:58:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:58:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:58:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:58:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:58:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:58:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:58:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:58:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:58:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:58:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:58:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:58:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:58:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:58:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:58:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:58:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:58:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:58:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:58:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:58:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:58:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:58:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:58:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:58:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:58:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:58:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:58:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:58:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:58:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:58:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:58:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:58:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:58:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:58:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:58:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:58:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:58:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:58:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:58:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:58:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:58:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:58:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:58:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:58:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:58:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:58:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:58:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:58:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:58:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:58:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:58:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:58:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:58:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:58:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:58:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:58:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:58:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:58:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:58:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:58:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:58:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:58:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:58:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:58:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:58:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:58:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:58:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:58:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:58:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:58:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:58:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:58:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:58:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:58:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:58:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:58:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:58:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:58:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:58:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:58:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:58:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:58:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:58:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:58:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:58:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:58:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:58:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:58:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:58:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:58:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:58:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:58:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:58:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:58:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:58:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:58:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:58:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:58:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:58:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:58:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:58:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:58:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:58:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:58:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:58:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:58:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:58:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:58:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:58:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:58:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:58:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:58:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:58:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:58:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:58:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:58:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:58:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:58:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:58:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:58:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:58:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:58:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:58:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:58:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:58:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:58:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:58:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:58:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:58:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:58:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:58:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:58:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:58:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:58:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:58:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:58:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:59:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:59:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:59:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:59:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:59:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:59:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:59:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:59:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:59:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:59:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:59:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:59:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:59:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:59:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:59:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:59:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:59:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:59:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:59:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:59:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:59:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:59:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:59:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:59:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:59:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:59:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:59:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:59:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:59:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:59:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:59:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:59:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:59:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:59:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:59:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:59:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:59:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:59:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:59:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:59:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:59:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:59:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:59:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:59:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:59:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:59:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:59:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:59:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:59:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:59:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:59:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:59:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:59:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:59:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:59:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:59:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:59:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:59:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:59:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:59:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:59:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:59:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:59:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:59:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:59:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:59:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:59:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:59:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:59:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:59:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:59:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:59:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:59:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:59:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:59:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:59:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:59:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:59:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:59:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:59:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:59:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:59:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:59:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:59:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:59:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:59:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:59:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:59:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:59:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:59:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:59:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:59:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:59:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:59:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:59:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:59:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:59:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:59:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:59:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:59:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:59:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:59:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:59:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:59:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:59:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:59:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:59:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:59:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:59:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:59:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:59:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:59:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:59:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:59:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:59:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:59:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:59:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:59:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:59:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:59:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:59:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:59:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:59:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:59:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:59:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:59:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:59:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:59:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:59:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:59:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:59:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:59:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:59:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:59:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:59:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:59:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:59:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:59:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:59:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:59:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:59:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:59:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:59:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:59:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:59:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:59:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:59:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:59:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:59:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:59:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:59:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:59:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:59:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:59:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:59:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:59:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:59:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:59:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:59:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:59:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:59:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:59:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:59:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:59:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:59:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:59:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:59:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:59:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:59:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:59:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:59:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:59:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:59:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:59:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:59:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:59:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:59:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:59:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:59:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:59:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:59:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:59:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:59:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:59:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:59:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:59:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:59:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:59:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:59:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:59:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:59:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:59:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:59:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:59:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:59:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:59:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:59:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:59:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 13:59:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 13:59:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:59:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:59:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:59:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 13:59:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 13:59:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 13:59:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 13:59:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 13:59:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 13:59:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 13:59:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 13:59:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 13:59:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 13:59:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 13:59:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 13:59:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 13:59:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:00:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:00:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:00:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:00:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:00:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:00:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:00:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:00:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:00:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:00:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:00:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:00:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:00:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:00:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:00:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:00:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:00:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:00:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:00:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:00:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:00:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:00:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:00:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:00:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:00:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:00:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:00:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:00:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:00:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:00:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:00:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:00:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:00:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:00:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:00:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:00:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:00:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:00:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:00:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:00:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:00:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:00:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:00:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:00:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:00:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:00:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:00:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:00:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:00:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:00:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:00:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:00:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:00:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:00:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:00:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:00:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:00:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:00:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:00:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:00:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:00:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:00:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:00:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:00:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:00:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:00:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:00:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:00:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:00:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:00:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:00:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:00:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:00:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:00:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:00:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:00:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:00:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:00:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:00:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:00:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:00:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:00:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:00:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:00:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:00:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:00:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:00:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:00:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:00:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:00:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:00:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:00:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:00:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:00:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:00:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:00:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:00:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:00:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:00:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:00:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:00:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:00:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:00:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:00:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:00:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:00:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:00:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:00:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:00:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:00:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:00:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:00:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:00:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:00:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:00:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:00:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:00:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:00:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:00:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:00:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:00:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:00:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:00:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:00:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:00:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:00:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:00:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:00:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:00:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:00:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:00:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:00:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:00:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:00:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:00:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:00:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:00:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:00:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:00:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:00:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:00:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:00:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:00:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:00:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:00:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:00:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:00:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:00:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:00:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:00:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:00:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:00:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:00:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:00:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:00:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:00:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:00:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:00:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:00:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:00:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:00:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:00:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:00:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:00:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:00:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:00:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:00:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:00:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:00:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:00:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:00:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:00:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:00:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:00:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:00:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:00:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:00:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:00:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:00:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:00:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:00:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:00:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:00:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:00:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:00:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:00:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:00:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:00:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:00:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:00:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:00:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:00:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:00:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:00:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:00:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:00:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:00:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:00:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:00:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:00:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:00:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:00:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:00:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:00:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:00:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:00:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:00:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:00:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:00:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:00:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:00:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:00:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:00:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:00:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:00:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:00:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:01:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:01:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:01:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:01:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:01:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:01:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:01:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:01:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:01:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:01:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:01:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:01:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:01:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:01:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:01:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:01:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:01:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:01:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:01:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:01:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:01:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:01:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:01:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:01:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:01:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:01:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:01:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:01:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:01:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:01:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:01:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:01:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:01:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:01:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:01:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:01:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:01:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:01:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:01:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:01:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:01:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:01:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:01:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:01:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:01:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:01:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:01:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:01:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:01:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:01:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:01:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:01:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:01:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:01:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:01:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:01:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:01:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:01:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:01:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:01:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:01:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:01:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:01:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:01:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:01:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:01:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:01:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:01:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:01:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:01:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:01:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:01:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:01:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:01:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:01:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:01:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:01:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:01:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:01:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:01:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:01:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:01:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:01:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:01:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:01:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:01:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:01:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:01:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:01:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:01:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:01:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:01:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:01:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:01:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:01:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:01:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:01:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:01:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:01:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:01:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:01:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:01:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:01:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:01:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:01:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:01:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:01:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:01:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:01:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:01:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:01:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:01:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:01:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:01:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:01:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:01:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:01:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:01:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:01:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:01:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:01:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:01:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:01:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:01:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:01:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:01:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:01:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:01:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:01:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:01:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:01:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:01:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:01:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:01:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:01:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:01:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:01:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:01:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:01:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:01:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:01:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:01:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:01:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:01:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:01:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:01:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:01:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:01:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:01:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:01:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:01:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:01:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:01:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:01:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:01:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:01:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:01:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:01:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:01:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:01:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:01:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:01:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:01:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:01:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:01:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:01:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:01:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:01:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:01:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:01:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:01:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:01:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:01:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:01:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:01:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:01:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:01:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:01:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:01:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:01:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:01:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:01:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:01:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:01:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:01:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:01:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:01:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:01:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:01:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:01:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:01:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:01:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:01:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:01:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:01:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:01:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:01:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:01:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:01:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:01:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:01:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:01:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:01:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:01:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:01:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:01:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:01:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:01:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:01:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:01:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:01:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:01:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:01:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:01:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:01:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:01:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:02:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:02:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:02:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:02:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:02:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:02:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:02:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:02:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:02:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:02:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:02:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:02:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:02:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:02:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:02:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:02:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:02:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:02:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:02:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:02:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:02:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:02:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:02:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:02:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:02:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:02:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:02:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:02:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:02:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:02:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:02:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:02:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:02:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:02:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:02:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:02:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:02:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:02:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:02:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:02:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:02:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:02:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:02:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:02:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:02:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:02:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:02:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:02:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:02:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:02:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:02:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:02:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:02:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:02:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:02:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:02:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:02:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:02:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:02:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:02:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:02:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:02:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:02:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:02:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:02:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:02:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:02:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:02:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:02:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:02:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:02:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:02:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:02:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:02:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:02:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:02:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:02:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:02:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:02:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:02:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:02:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:02:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:02:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:02:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:02:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:02:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:02:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:02:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:02:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:02:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:02:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:02:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:02:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:02:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:02:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:02:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:02:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:02:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:02:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:02:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:02:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:02:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:02:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:02:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:02:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:02:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:02:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:02:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:02:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:02:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:02:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:02:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:02:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:02:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:02:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:02:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:02:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:02:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:02:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:02:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:02:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:02:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:02:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:02:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:02:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:02:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:02:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:02:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:02:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:02:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:02:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:02:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:02:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:02:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:02:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:02:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:02:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:02:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:02:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:02:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:02:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:02:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:02:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:02:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:02:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:02:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:02:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:02:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:02:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:02:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:02:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:02:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:02:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:02:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:02:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:02:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:02:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:02:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:02:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:02:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:02:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:02:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:02:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:02:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:02:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:02:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:02:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:02:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:02:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:02:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:02:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:02:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:02:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:02:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:02:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:02:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:02:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:02:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:02:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:02:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:02:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:02:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:02:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:02:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:02:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:02:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:02:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:02:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:02:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:02:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:02:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:02:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:02:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:02:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:02:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:02:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:02:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:02:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:02:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:02:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:02:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:02:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:02:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:02:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:02:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:02:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:02:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:02:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:02:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:02:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:02:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:02:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:02:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:02:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:02:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:02:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:03:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:03:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:03:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:03:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:03:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:03:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:03:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:03:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:03:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:03:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:03:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:03:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:03:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:03:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:03:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:03:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:03:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:03:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:03:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:03:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:03:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:03:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:03:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:03:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:03:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:03:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:03:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:03:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:03:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:03:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:03:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:03:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:03:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:03:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:03:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:03:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:03:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:03:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:03:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:03:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:03:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:03:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:03:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:03:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:03:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:03:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:03:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:03:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:03:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:03:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:03:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:03:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:03:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:03:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:03:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:03:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:03:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:03:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:03:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:03:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:03:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:03:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:03:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:03:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:03:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:03:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:03:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:03:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:03:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:03:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:03:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:03:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:03:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:03:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:03:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:03:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:03:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:03:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:03:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:03:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:03:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:03:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:03:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:03:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:03:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:03:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:03:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:03:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:03:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:03:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:03:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:03:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:03:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:03:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:03:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:03:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:03:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:03:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:03:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:03:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:03:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:03:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:03:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:03:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:03:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:03:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:03:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:03:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:03:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:03:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:03:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:03:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:03:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:03:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:03:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:03:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:03:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:03:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:03:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:03:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:03:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:03:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:03:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:03:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:03:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:03:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:03:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:03:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:03:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:03:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:03:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:03:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:03:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:03:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:03:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:03:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:03:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:03:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:03:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:03:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:03:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:03:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:03:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:03:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:03:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:03:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:03:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:03:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:03:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:03:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:03:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:03:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:03:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:03:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:03:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:03:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:03:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:03:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:03:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:03:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:03:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:03:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:03:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:03:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:03:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:03:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:03:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:03:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:03:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:03:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:03:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:03:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:03:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:03:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:03:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:03:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:03:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:03:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:03:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:03:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:03:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:03:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:03:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:03:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:03:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:03:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:03:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:03:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:03:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:03:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:03:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:03:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:03:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:03:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:03:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:03:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:03:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:03:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:03:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:03:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:03:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:03:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:03:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:03:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:03:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:03:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:03:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:03:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:03:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:03:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:03:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:03:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:03:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:03:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:03:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:03:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:04:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:04:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:04:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:04:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:04:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:04:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:04:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:04:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:04:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:04:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:04:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:04:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:04:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:04:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:04:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:04:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:04:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:04:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:04:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:04:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:04:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:04:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:04:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:04:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:04:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:04:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:04:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:04:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:04:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:04:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:04:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:04:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:04:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:04:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:04:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:04:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:04:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:04:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:04:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:04:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:04:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:04:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:04:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:04:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:04:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:04:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:04:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:04:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:04:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:04:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:04:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:04:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:04:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:04:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:04:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:04:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:04:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:04:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:04:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:04:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:04:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:04:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:04:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:04:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:04:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:04:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:04:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:04:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:04:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:04:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:04:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:04:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:04:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:04:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:04:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:04:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:04:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:04:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:04:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:04:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:04:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:04:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:04:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:04:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:04:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:04:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:04:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:04:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:04:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:04:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:04:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:04:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:04:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:04:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:04:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:04:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:04:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:04:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:04:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:04:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:04:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:04:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:04:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:04:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:04:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:04:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:04:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:04:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:04:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:04:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:04:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:04:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:04:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:04:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:04:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:04:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:04:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:04:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:04:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:04:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:04:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:04:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:04:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:04:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:04:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:04:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:04:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:04:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:04:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:04:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:04:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:04:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:04:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:04:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:04:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:04:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:04:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:04:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:04:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:04:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:04:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:04:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:04:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:04:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:04:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:04:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:04:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:04:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:04:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:04:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:04:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:04:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:04:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:04:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:04:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:04:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:04:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:04:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:04:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:04:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:04:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:04:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:04:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:04:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:04:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:04:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:04:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:04:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:04:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:04:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:04:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:04:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:04:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:04:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:04:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:04:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:04:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:04:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:04:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:04:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:04:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:04:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:04:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:04:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:04:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:04:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:04:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:04:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:04:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:04:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:04:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:04:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:04:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:04:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:04:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:04:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:04:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:04:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:04:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:04:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:04:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:04:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:04:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:04:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:04:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:04:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:04:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:04:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:04:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:04:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:04:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:04:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:04:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:04:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:04:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:04:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:07:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:07:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:07:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:07:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:07:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:07:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:07:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:07:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:07:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:07:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:07:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:07:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:07:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:07:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:07:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:07:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:07:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:07:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:07:08 rdma06 rsyslogd[1041]: imjournal: 5295 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 14:07:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:07:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:07:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:07:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:07:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:07:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:07:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:07:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:07:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:07:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:07:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:07:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:07:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:07:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:07:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:07:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:07:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:07:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:07:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:07:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:07:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:07:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:07:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:07:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:07:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:07:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:07:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:07:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:07:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:07:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:07:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:07:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:07:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:07:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:07:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:07:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:07:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:07:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:07:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:07:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:07:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:07:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:07:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:07:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:07:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:07:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:07:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:07:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:07:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:07:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:07:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:07:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:07:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:07:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:07:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:07:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:07:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:07:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:07:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:07:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:07:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:07:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:07:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:07:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:07:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:07:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:07:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:07:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:07:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:07:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:07:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:07:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:07:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:07:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:07:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:07:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:07:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:07:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:07:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:07:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:07:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:07:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:07:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:07:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:07:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:07:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:07:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:07:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:07:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:07:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:07:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:07:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:07:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:07:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:07:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:07:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:07:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:07:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:07:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:07:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:07:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:07:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:07:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:07:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:07:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:07:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:07:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:07:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:07:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:07:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:07:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:07:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:07:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:07:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:07:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:07:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:07:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:07:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:07:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:07:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:07:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:07:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:07:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:07:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:07:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:07:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:07:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:07:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:07:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:07:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:07:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:07:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:07:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:07:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:07:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:07:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:07:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:07:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:07:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:07:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:07:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:07:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:07:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:07:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:07:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:07:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:07:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:07:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:07:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:07:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:07:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:07:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:07:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:07:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:07:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:07:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:07:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:07:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:07:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:07:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:07:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:07:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:07:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:07:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:07:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:07:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:07:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:07:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:07:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:07:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:07:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:07:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:07:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:07:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:07:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:07:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:07:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:07:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:07:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:07:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:08:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:08:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:08:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:08:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:08:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:08:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:08:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:08:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:08:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:08:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:08:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:08:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:08:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:08:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:08:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:08:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:08:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:08:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:08:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:08:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:08:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:08:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:08:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:08:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:08:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:08:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:08:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:08:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:08:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:08:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:08:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:08:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:08:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:08:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:08:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:08:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:08:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:08:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:08:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:08:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:08:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:08:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:08:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:08:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:08:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:08:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:08:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:08:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:08:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:08:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:08:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:08:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:08:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:08:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:08:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:08:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:08:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:08:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:08:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:08:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:08:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:08:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:08:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:08:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:08:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:08:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:08:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:08:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:08:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:08:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:08:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:08:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:08:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:08:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:08:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:08:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:08:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:08:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:08:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:08:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:08:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:08:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:08:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:08:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:08:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:08:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:08:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:08:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:08:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:08:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:08:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:08:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:08:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:08:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:08:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:08:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:08:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:08:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:08:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:08:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:08:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:08:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:08:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:08:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:08:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:08:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:08:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:08:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:08:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:08:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:08:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:08:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:08:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:08:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:08:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:08:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:08:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:08:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:08:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:08:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:08:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:08:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:08:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:08:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:08:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:08:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:08:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:08:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:08:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:08:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:08:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:08:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:08:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:08:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:08:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:08:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:08:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:08:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:08:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:08:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:08:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:08:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:08:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:08:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:08:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:08:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:08:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:08:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:08:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:08:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:08:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:08:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:08:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:08:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:08:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:08:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:08:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:08:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:08:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:08:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:08:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:08:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:08:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:08:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:08:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:08:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:08:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:08:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:08:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:08:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:08:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:08:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:08:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:08:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:08:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:08:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:08:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:08:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:08:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:08:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:08:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:08:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:08:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:08:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:08:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:08:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:08:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:08:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:08:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:08:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:08:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:08:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:08:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:08:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:08:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:08:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:08:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:08:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:08:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:08:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:08:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:08:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:08:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:08:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:08:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:08:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:08:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:08:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:08:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:08:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:08:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:08:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:08:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:08:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:08:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:08:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:09:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:09:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:09:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:09:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:09:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:09:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:09:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:09:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:09:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:09:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:09:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:09:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:09:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:09:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:09:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:09:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:09:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:09:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:09:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:09:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:09:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:09:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:09:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:09:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:09:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:09:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:09:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:09:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:09:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:09:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:09:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:09:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:09:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:09:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:09:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:09:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:09:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:09:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:09:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:09:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:09:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:09:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:09:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:09:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:09:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:09:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:09:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:09:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:09:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:09:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:09:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:09:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:09:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:09:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:09:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:09:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:09:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:09:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:09:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:09:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:09:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:09:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:09:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:09:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:09:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:09:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:09:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:09:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:09:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:09:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:09:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:09:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:09:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:09:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:09:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:09:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:09:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:09:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:09:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:09:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:09:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:09:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:09:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:09:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:09:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:09:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:09:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:09:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:09:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:09:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:09:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:09:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:09:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:09:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:09:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:09:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:09:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:09:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:09:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:09:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:09:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:09:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:09:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:09:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:09:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:09:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:09:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:09:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:09:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:09:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:09:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:09:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:09:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:09:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:09:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:09:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:09:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:09:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:09:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:09:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:09:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:09:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:09:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:09:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:09:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:09:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:09:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:09:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:09:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:09:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:09:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:09:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:09:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:09:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:09:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:09:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:09:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:09:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:09:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:09:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:09:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:09:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:09:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:09:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:09:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:09:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:09:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:09:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:09:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:09:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:09:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:09:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:09:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:09:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:09:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:09:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:09:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:09:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:09:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:09:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:09:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:09:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:09:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:09:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:09:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:09:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:09:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:09:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:09:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:09:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:09:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:09:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:09:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:09:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:09:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:09:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:09:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:09:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:09:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:09:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:09:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:09:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:09:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:09:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:09:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:09:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:09:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:09:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:09:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:09:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:09:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:09:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:09:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:09:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:09:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:09:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:09:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:09:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:09:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:09:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:09:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:09:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:09:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:09:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:09:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:09:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:09:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:09:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:09:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:09:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:09:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:09:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:09:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:09:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:09:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:09:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:10:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:10:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:10:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:10:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:10:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:10:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:10:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:10:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:10:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:10:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:10:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:10:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:10:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:10:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:10:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:10:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:10:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:10:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:10:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:10:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:10:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:10:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:10:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:10:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:10:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:10:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:10:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:10:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:10:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:10:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:10:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:10:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:10:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:10:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:10:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:10:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:10:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:10:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:10:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:10:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:10:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:10:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:10:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:10:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:10:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:10:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:10:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:10:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:10:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:10:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:10:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:10:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:10:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:10:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:10:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:10:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:10:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:10:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:10:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:10:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:10:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:10:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:10:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:10:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:10:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:10:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:10:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:10:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:10:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:10:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:10:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:10:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:10:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:10:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:10:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:10:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:10:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:10:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:10:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:10:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:10:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:10:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:10:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:10:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:10:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:10:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:10:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:10:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:10:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:10:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:10:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:10:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:10:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:10:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:10:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:10:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:10:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:10:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:10:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:10:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:10:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:10:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:10:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:10:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:10:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:10:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:10:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:10:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:10:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:10:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:10:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:10:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:10:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:10:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:10:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:10:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:10:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:10:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:10:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:10:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:10:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:10:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:10:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:10:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:10:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:10:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:10:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:10:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:10:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:10:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:10:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:10:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:10:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:10:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:10:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:10:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:10:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:10:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:10:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:10:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:10:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:10:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:10:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:10:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:10:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:10:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:10:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:10:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:10:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:10:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:10:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:10:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:10:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:10:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:10:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:10:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:10:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:10:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:10:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:10:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:10:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:10:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:10:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:10:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:10:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:10:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:10:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:10:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:10:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:10:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:10:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:10:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:10:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:10:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:10:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:10:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:10:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:10:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:10:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:10:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:10:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:10:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:10:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:10:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:10:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:10:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:10:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:10:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:10:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:10:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:10:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:10:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:10:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:10:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:10:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:10:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:10:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:10:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:10:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:10:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:10:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:10:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:10:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:10:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:10:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:10:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:10:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:10:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:10:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:10:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:10:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:10:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:10:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:10:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:10:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:10:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:11:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:11:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:11:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:11:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:11:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:11:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:11:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:11:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:11:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:11:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:11:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:11:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:11:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:11:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:11:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:11:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:11:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:11:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:11:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:11:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:11:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:11:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:11:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:11:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:11:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:11:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:11:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:11:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:11:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:11:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:11:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:11:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:11:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:11:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:11:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:11:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:11:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:11:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:11:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:11:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:11:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:11:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:11:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:11:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:11:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:11:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:11:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:11:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:11:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:11:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:11:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:11:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:11:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:11:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:11:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:11:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:11:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:11:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:11:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:11:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:11:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:11:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:11:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:11:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:11:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:11:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:11:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:11:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:11:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:11:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:11:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:11:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:11:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:11:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:11:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:11:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:11:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:11:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:11:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:11:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:11:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:11:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:11:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:11:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:11:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:11:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:11:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:11:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:11:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:11:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:11:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:11:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:11:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:11:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:11:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:11:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:11:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:11:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:11:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:11:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:11:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:11:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:11:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:11:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:11:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:11:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:11:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:11:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:11:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:11:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:11:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:11:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:11:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:11:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:11:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:11:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:11:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:11:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:11:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:11:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:11:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:11:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:11:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:11:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:11:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:11:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:11:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:11:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:11:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:11:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:11:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:11:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:11:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:11:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:11:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:11:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:11:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:11:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:11:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:11:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:11:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:11:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:11:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:11:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:11:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:11:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:11:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:11:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:11:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:11:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:11:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:11:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:11:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:11:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:11:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:11:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:11:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:11:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:11:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:11:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:11:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:11:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:11:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:11:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:11:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:11:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:11:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:11:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:11:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:11:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:11:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:11:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:11:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:11:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:11:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:11:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:11:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:11:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:11:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:11:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:11:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:11:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:11:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:11:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:11:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:11:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:11:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:11:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:11:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:11:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:11:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:11:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:11:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:11:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:11:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:11:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:11:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:11:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:11:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:11:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:11:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:11:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:11:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:11:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:11:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:11:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:11:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:11:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:11:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:11:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:11:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:11:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:11:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:11:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:11:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:11:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:12:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:12:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:12:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:12:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:12:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:12:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:12:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:12:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:12:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:12:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:12:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:12:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:12:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:12:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:12:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:12:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:12:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:12:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:12:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:12:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:12:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:12:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:12:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:12:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:12:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:12:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:12:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:12:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:12:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:12:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:12:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:12:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:12:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:12:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:12:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:12:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:12:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:12:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:12:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:12:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:12:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:12:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:12:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:12:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:12:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:12:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:12:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:12:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:12:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:12:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:12:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:12:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:12:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:12:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:12:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:12:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:12:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:12:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:12:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:12:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:12:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:12:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:12:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:12:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:12:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:12:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:12:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:12:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:12:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:12:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:12:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:12:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:12:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:12:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:12:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:12:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:12:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:12:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:12:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:12:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:12:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:12:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:12:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:12:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:12:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:12:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:12:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:12:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:12:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:12:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:12:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:12:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:12:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:12:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:12:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:12:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:12:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:12:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:12:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:12:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:12:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:12:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:12:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:12:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:12:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:12:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:12:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:12:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:12:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:12:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:12:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:12:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:12:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:12:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:12:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:12:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:12:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:12:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:12:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:12:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:12:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:12:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:12:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:12:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:12:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:12:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:12:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:12:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:12:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:12:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:12:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:12:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:12:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:12:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:12:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:12:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:12:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:12:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:12:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:12:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:12:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:12:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:12:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:12:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:12:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:12:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:12:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:12:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:12:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:12:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:12:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:12:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:12:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:12:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:12:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:12:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:12:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:12:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:12:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:12:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:12:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:12:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:12:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:12:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:12:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:12:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:12:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:12:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:12:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:12:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:12:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:12:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:12:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:12:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:12:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:12:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:12:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:12:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:12:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:12:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:12:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:12:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:12:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:12:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:12:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:12:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:12:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:12:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:12:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:12:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:12:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:12:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:12:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:12:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:12:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:12:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:12:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:12:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:12:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:12:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:12:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:12:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:12:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:12:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:12:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:12:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:12:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:12:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:12:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:12:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:12:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:12:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:12:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:12:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:12:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:12:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:13:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:13:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:13:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:13:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:13:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:13:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:13:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:13:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:13:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:13:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:13:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:13:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:13:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:13:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:13:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:13:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:13:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:13:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:13:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:13:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:13:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:13:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:13:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:13:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:13:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:13:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:13:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:13:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:13:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:13:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:13:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:13:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:13:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:13:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:13:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:13:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:13:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:13:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:13:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:13:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:13:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:13:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:13:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:13:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:13:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:13:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:13:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:13:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:13:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:13:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:13:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:13:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:13:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:13:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:13:13 rdma06 systemd-journald[790]: Data hash table of /run/log/journal/95379d2f723847b68d9e976dee8967a9/system.journal has a fill level at 75.0 (163750 of 218332 items, 75497472 file size, 461 bytes per hash table item), suggesting rotation.
Apr 17 14:13:13 rdma06 systemd-journald[790]: /run/log/journal/95379d2f723847b68d9e976dee8967a9/system.journal: Journal header limits reached or header out-of-date, rotating.
Apr 17 14:13:13 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 17 14:13:14 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 17 14:13:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:13:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:13:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:13:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:13:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:13:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:13:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:13:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:13:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:13:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:13:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:13:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:13:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:13:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:13:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:13:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:13:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:13:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:13:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:13:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:13:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:13:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:13:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:13:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:13:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:13:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:13:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:13:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:13:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:13:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:13:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:13:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:13:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:13:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:13:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:13:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:13:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:13:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:13:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:13:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:13:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:13:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:13:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:13:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:13:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:13:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:13:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:13:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:13:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:13:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:13:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:13:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:13:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:13:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:13:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:13:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:13:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:13:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:13:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:13:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:13:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:13:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:13:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:13:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:13:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:13:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:13:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:13:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:13:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:13:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:13:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:13:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:13:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:13:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:13:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:13:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:13:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:13:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:13:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:13:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:13:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:13:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:13:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:13:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:13:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:13:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:13:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:13:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:13:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:13:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:13:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:13:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:13:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:13:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:13:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:13:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:13:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:13:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:13:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:13:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:13:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:13:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:13:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:13:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:13:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:13:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:13:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:13:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:13:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:13:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:13:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:13:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:13:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:13:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:13:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:13:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:13:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:13:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:13:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:13:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:13:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:13:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:13:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:13:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:13:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:13:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:13:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:13:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:13:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:13:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:13:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:13:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:13:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:13:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:13:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:13:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:13:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:13:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:13:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:13:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:13:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:13:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:13:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:13:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:13:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:13:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:13:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:13:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:13:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:13:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:13:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:13:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:13:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:13:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:13:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:13:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:13:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:13:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:13:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:13:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:13:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:13:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:14:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:14:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:14:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:14:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:14:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:14:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:14:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:14:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:14:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:14:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:14:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:14:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:14:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:14:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:14:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:14:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:14:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:14:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:14:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:14:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:14:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:14:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:14:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:14:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:14:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:14:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:14:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:14:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:14:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:14:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:14:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:14:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:14:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:14:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:14:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:14:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:14:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:14:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:14:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:14:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:14:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:14:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:14:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:14:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:14:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:14:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:14:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:14:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:14:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:14:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:14:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:14:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:14:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:14:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:14:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:14:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:14:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:14:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:14:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:14:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:14:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:14:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:14:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:14:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:14:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:14:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:14:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:14:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:14:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:14:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:14:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:14:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:14:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:14:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:14:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:14:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:14:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:14:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:14:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:14:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:14:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:14:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:14:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:14:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:14:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:14:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:14:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:14:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:14:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:14:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:14:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:14:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:14:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:14:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:14:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:14:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:14:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:14:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:14:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:14:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:14:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:14:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:14:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:14:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:14:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:14:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:14:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:14:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:14:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:14:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:14:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:14:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:14:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:14:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:14:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:14:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:14:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:14:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:14:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:14:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:14:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:14:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:14:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:14:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:14:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:14:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:14:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:14:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:14:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:14:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:14:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:14:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:14:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:14:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:14:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:14:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:14:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:14:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:14:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:14:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:14:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:14:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:14:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:14:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:14:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:14:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:14:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:14:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:14:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:14:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:14:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:14:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:14:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:14:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:14:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:14:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:14:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:14:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:14:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:14:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:14:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:14:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:14:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:14:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:14:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:14:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:14:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:14:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:14:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:14:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:14:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:14:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:14:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:14:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:14:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:14:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:14:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:14:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:14:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:14:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:14:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:14:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:14:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:14:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:14:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:14:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:14:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:14:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:14:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:14:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:14:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:14:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:14:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:14:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:14:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:14:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:14:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:14:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:14:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:14:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:14:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:14:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:14:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:14:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:14:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:14:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:14:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:14:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:14:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:14:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:14:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:14:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:14:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:14:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:14:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:14:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:15:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:15:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:15:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:15:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:15:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:15:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:15:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:15:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:15:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:15:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:15:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:15:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:15:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:15:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:15:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:15:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:15:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:15:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:17:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:17:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:17:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:17:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:17:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:17:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:17:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:17:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:17:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:17:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:17:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:17:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:17:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:17:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:17:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:17:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:17:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:17:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:17:09 rdma06 rsyslogd[1041]: imjournal: 5086 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 14:17:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:17:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:17:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:17:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:17:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:17:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:17:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:17:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:17:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:17:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:17:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:17:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:17:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:17:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:17:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:17:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:17:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:17:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:17:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:17:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:17:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:17:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:17:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:17:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:17:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:17:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:17:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:17:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:17:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:17:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:17:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:17:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:17:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:17:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:17:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:17:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:17:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:17:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:17:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:17:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:17:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:17:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:17:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:17:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:17:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:17:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:17:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:17:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:17:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:17:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:17:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:17:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:17:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:17:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:17:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:17:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:17:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:17:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:17:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:17:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:17:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:17:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:17:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:17:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:17:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:17:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:17:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:17:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:17:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:17:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:17:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:17:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:17:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:17:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:17:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:17:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:17:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:17:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:17:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:17:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:17:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:17:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:17:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:17:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:17:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:17:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:17:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:17:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:17:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:17:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:17:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:17:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:17:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:17:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:17:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:17:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:17:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:17:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:17:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:17:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:17:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:17:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:17:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:17:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:17:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:17:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:17:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:17:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:17:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:17:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:17:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:17:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:17:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:17:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:17:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:17:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:17:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:17:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:17:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:17:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:17:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:17:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:17:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:17:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:17:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:17:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:17:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:17:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:17:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:17:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:17:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:17:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:17:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:17:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:17:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:17:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:17:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:17:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:17:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:17:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:17:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:17:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:17:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:17:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:17:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:17:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:17:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:17:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:17:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:17:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:17:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:17:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:17:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:17:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:17:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:17:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:17:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:17:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:17:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:17:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:17:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:17:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:17:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:17:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:17:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:17:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:17:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:17:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:17:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:17:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:17:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:17:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:17:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:17:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:17:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:17:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:17:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:17:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:17:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:17:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:18:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:18:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:18:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:18:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:18:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:18:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:18:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:18:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:18:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:18:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:18:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:18:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:18:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:18:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:18:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:18:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:18:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:18:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:18:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:18:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:18:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:18:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:18:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:18:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:18:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:18:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:18:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:18:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:18:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:18:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:18:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:18:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:18:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:18:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:18:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:18:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:18:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:18:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:18:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:18:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:18:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:18:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:18:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:18:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:18:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:18:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:18:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:18:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:18:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:18:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:18:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:18:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:18:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:18:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:18:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:18:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:18:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:18:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:18:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:18:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:18:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:18:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:18:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:18:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:18:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:18:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:18:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:18:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:18:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:18:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:18:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:18:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:18:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:18:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:18:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:18:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:18:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:18:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:18:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:18:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:18:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:18:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:18:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:18:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:18:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:18:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:18:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:18:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:18:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:18:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:18:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:18:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:18:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:18:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:18:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:18:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:18:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:18:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:18:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:18:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:18:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:18:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:18:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:18:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:18:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:18:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:18:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:18:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:18:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:18:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:18:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:18:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:18:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:18:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:18:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:18:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:18:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:18:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:18:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:18:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:18:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:18:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:18:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:18:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:18:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:18:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:18:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:18:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:18:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:18:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:18:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:18:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:18:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:18:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:18:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:18:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:18:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:18:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:18:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:18:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:18:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:18:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:18:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:18:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:18:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:18:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:18:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:18:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:18:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:18:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:18:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:18:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:18:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:18:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:18:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:18:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:18:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:18:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:18:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:18:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:18:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:18:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:18:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:18:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:18:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:18:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:18:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:18:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:18:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:18:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:18:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:18:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:18:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:18:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:18:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:18:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:18:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:18:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:18:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:18:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:18:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:18:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:18:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:18:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:18:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:18:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:18:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:18:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:18:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:18:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:18:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:18:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:18:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:18:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:18:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:18:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:18:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:18:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:18:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:18:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:18:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:18:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:18:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:18:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:18:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:18:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:18:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:18:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:18:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:18:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:18:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:18:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:18:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:18:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:18:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:18:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:19:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:19:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:19:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:19:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:19:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:19:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:19:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:19:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:19:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:19:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:19:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:19:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:19:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:19:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:19:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:19:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:19:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:19:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:19:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:19:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:19:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:19:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:19:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:19:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:19:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:19:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:19:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:19:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:19:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:19:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:19:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:19:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:19:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:19:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:19:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:19:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:19:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:19:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:19:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:19:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:19:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:19:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:19:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:19:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:19:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:19:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:19:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:19:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:19:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:19:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:19:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:19:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:19:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:19:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:19:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:19:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:19:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:19:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:19:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:19:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:19:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:19:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:19:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:19:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:19:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:19:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:19:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:19:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:19:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:19:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:19:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:19:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:19:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:19:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:19:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:19:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:19:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:19:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:19:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:19:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:19:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:19:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:19:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:19:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:19:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:19:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:19:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:19:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:19:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:19:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:19:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:19:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:19:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:19:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:19:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:19:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:19:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:19:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:19:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:19:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:19:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:19:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:19:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:19:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:19:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:19:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:19:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:19:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:19:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:19:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:19:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:19:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:19:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:19:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:19:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:19:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:19:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:19:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:19:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:19:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:19:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:19:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:19:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:19:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:19:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:19:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:19:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:19:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:19:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:19:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:19:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:19:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:19:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:19:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:19:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:19:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:19:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:19:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:19:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:19:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:19:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:19:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:19:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:19:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:19:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:19:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:19:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:19:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:19:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:19:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:19:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:19:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:19:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:19:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:19:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:19:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:19:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:19:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:19:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:19:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:19:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:19:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:19:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:19:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:19:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:19:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:19:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:19:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:19:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:19:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:19:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:19:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:19:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:19:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:19:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:19:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:19:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:19:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:19:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:19:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:19:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:19:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:19:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:19:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:19:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:19:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:19:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:19:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:19:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:19:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:19:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:19:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:19:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:19:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:19:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:19:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:19:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:19:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:19:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:19:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:19:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:19:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:19:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:19:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:19:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:19:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:19:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:19:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:19:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:19:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:19:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:19:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:19:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:19:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:19:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:19:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:20:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:20:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:20:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:20:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:20:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:20:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:20:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:20:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:20:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:20:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:20:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:20:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:20:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:20:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:20:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:20:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:20:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:20:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:20:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:20:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:20:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:20:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:20:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:20:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:20:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:20:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:20:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:20:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:20:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:20:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:20:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:20:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:20:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:20:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:20:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:20:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:20:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:20:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:20:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:20:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:20:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:20:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:20:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:20:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:20:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:20:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:20:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:20:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:20:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:20:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:20:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:20:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:20:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:20:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:20:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:20:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:20:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:20:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:20:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:20:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:20:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:20:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:20:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:20:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:20:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:20:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:20:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:20:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:20:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:20:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:20:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:20:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:20:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:20:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:20:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:20:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:20:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:20:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:20:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:20:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:20:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:20:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:20:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:20:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:20:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:20:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:20:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:20:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:20:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:20:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:20:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:20:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:20:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:20:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:20:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:20:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:20:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:20:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:20:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:20:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:20:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:20:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:20:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:20:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:20:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:20:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:20:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:20:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:20:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:20:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:20:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:20:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:20:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:20:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:20:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:20:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:20:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:20:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:20:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:20:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:20:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:20:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:20:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:20:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:20:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:20:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:20:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:20:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:20:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:20:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:20:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:20:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:20:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:20:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:20:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:20:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:20:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:20:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:20:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:20:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:20:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:20:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:20:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:20:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:20:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:20:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:20:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:20:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:20:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:20:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:20:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:20:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:20:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:20:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:20:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:20:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:20:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:20:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:20:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:20:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:20:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:20:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:20:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:20:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:20:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:20:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:20:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:20:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:20:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:20:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:20:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:20:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:20:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:20:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:20:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:20:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:20:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:20:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:20:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:20:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:20:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:20:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:20:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:20:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:20:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:20:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:20:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:20:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:20:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:20:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:20:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:20:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:20:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:20:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:20:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:20:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:20:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:20:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:20:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:20:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:20:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:20:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:20:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:20:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:20:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:20:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:20:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:20:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:20:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:20:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:20:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:20:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:20:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:20:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:20:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:20:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:21:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:21:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:21:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:21:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:21:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:21:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:21:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:21:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:21:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:21:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:21:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:21:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:21:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:21:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:21:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:21:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:21:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:21:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:21:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:21:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:21:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:21:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:21:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:21:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:21:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:21:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:21:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:21:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:21:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:21:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:21:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:21:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:21:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:21:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:21:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:21:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:21:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:21:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:21:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:21:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:21:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:21:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:21:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:21:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:21:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:21:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:21:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:21:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:21:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:21:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:21:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:21:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:21:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:21:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:21:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:21:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:21:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:21:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:21:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:21:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:21:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:21:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:21:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:21:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:21:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:21:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:21:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:21:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:21:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:21:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:21:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:21:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:21:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:21:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:21:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:21:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:21:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:21:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:21:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:21:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:21:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:21:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:21:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:21:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:21:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:21:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:21:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:21:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:21:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:21:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:21:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:21:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:21:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:21:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:21:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:21:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:21:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:21:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:21:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:21:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:21:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:21:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:21:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:21:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:21:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:21:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:21:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:21:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:21:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:21:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:21:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:21:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:21:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:21:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:21:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:21:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:21:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:21:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:21:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:21:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:21:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:21:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:21:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:21:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:21:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:21:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:21:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:21:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:21:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:21:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:21:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:21:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:21:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:21:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:21:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:21:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:21:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:21:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:21:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:21:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:21:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:21:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:21:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:21:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:21:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:21:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:21:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:21:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:21:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:21:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:21:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:21:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:21:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:21:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:21:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:21:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:21:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:21:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:21:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:21:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:21:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:21:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:21:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:21:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:21:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:21:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:21:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:21:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:21:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:21:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:21:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:21:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:21:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:21:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:21:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:21:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:21:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:21:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:21:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:21:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:21:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:21:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:21:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:21:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:21:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:21:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:21:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:21:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:21:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:21:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:21:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:21:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:21:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:21:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:21:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:21:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:21:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:21:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:21:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:21:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:21:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:21:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:21:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:21:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:21:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:21:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:21:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:21:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:21:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:21:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:21:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:21:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:21:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:21:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:21:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:21:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:22:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:22:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:22:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:22:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:22:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:22:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:22:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:22:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:22:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:22:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:22:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:22:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:22:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:22:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:22:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:22:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:22:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:22:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:22:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:22:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:22:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:22:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:22:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:22:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:22:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:22:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:22:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:22:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:22:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:22:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:22:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:22:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:22:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:22:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:22:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:22:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:22:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:22:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:22:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:22:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:22:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:22:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:22:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:22:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:22:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:22:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:22:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:22:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:22:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:22:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:22:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:22:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:22:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:22:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:22:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:22:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:22:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:22:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:22:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:22:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:22:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:22:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:22:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:22:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:22:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:22:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:22:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:22:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:22:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:22:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:22:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:22:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:22:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:22:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:22:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:22:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:22:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:22:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:22:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:22:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:22:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:22:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:22:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:22:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:22:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:22:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:22:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:22:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:22:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:22:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:22:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:22:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:22:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:22:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:22:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:22:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:22:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:22:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:22:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:22:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:22:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:22:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:22:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:22:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:22:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:22:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:22:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:22:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:22:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:22:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:22:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:22:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:22:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:22:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:22:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:22:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:22:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:22:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:22:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:22:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:22:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:22:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:22:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:22:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:22:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:22:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:22:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:22:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:22:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:22:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:22:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:22:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:22:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:22:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:22:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:22:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:22:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:22:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:22:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:22:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:22:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:22:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:22:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:22:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:22:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:22:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:22:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:22:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:22:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:22:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:22:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:22:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:22:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:22:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:22:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:22:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:22:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:22:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:22:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:22:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:22:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:22:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:22:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:22:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:22:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:22:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:22:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:22:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:22:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:22:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:22:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:22:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:22:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:22:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:22:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:22:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:22:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:22:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:22:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:22:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:22:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:22:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:22:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:22:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:22:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:22:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:22:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:22:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:22:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:22:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:22:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:22:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:22:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:22:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:22:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:22:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:22:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:22:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:22:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:22:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:22:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:22:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:22:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:22:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:22:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:22:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:22:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:22:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:22:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:22:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:22:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:22:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:22:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:22:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:22:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:22:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:23:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:23:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:23:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:23:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:23:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:23:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:23:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:23:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:23:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:23:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:23:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:23:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:23:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:23:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:23:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:23:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:23:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:23:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:23:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:23:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:23:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:23:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:23:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:23:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:23:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:23:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:23:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:23:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:23:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:23:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:23:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:23:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:23:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:23:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:23:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:23:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:23:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:23:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:23:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:23:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:23:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:23:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:23:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:23:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:23:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:23:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:23:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:23:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:23:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:23:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:23:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:23:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:23:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:23:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:23:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:23:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:23:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:23:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:23:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:23:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:23:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:23:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:23:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:23:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:23:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:23:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:23:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:23:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:23:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:23:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:23:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:23:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:23:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:23:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:23:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:23:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:23:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:23:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:23:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:23:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:23:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:23:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:23:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:23:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:23:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:23:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:23:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:23:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:23:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:23:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:23:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:23:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:23:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:23:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:23:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:23:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:23:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:23:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:23:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:23:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:23:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:23:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:23:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:23:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:23:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:23:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:23:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:23:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:23:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:23:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:23:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:23:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:23:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:23:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:23:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:23:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:23:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:23:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:23:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:23:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:23:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:23:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:23:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:23:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:23:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:23:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:23:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:23:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:23:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:23:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:23:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:23:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:23:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:23:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:23:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:23:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:23:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:23:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:23:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:23:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:23:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:23:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:23:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:23:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:23:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:23:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:23:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:23:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:23:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:23:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:23:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:23:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:23:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:23:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:23:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:23:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:23:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:23:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:23:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:23:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:23:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:23:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:23:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:23:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:23:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:23:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:24:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:24:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:24:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:24:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:24:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:24:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:24:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:24:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:24:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:24:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:24:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:24:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:24:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:24:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:24:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:24:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:24:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:24:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:24:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:24:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:24:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:24:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:24:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:24:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:24:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:24:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:24:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:24:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:24:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:24:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:24:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:24:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:24:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:24:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:24:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:24:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:24:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:24:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:24:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:24:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:24:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:24:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:24:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:24:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:24:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:24:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:24:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:24:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:24:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:24:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:24:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:24:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:24:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:24:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:24:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:24:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:24:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:24:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:24:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:24:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:24:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:24:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:24:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:24:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:24:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:24:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:24:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:24:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:24:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:24:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:24:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:24:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:24:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:24:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:24:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:24:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:24:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:24:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:24:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:24:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:24:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:24:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:24:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:24:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:24:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:24:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:24:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:24:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:24:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:24:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:24:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:24:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:24:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:24:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:24:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:24:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:24:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:24:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:24:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:24:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:24:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:24:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:24:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:24:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:24:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:24:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:24:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:24:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:24:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:24:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:24:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:24:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:24:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:24:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:24:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:24:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:24:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:24:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:24:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:24:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:24:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:24:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:24:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:24:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:24:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:24:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:24:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:24:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:24:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:24:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:24:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:24:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:24:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:24:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:24:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:24:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:24:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:24:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:24:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:24:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:24:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:24:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:24:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:24:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:24:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:24:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:24:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:24:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:24:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:24:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:24:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:24:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:24:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:24:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:24:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:24:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:24:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:24:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:24:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:24:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:24:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:24:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:24:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:24:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:24:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:24:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:24:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:24:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:24:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:24:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:24:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:24:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:24:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:24:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:24:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:24:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:24:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:24:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:24:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:24:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:24:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:24:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:24:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:24:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:24:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:24:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:24:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:24:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:24:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:24:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:24:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:24:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:24:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:24:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:24:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:24:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:25:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:25:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:25:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:25:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:25:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:25:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:25:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:25:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:25:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:25:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:25:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:25:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:25:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:25:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:25:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:25:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:25:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:25:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:27:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:27:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:27:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:27:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:27:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:27:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:27:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:27:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:27:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:27:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:27:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:27:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:27:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:27:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:27:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:27:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:27:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:27:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:27:14 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 14:27:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:27:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:27:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:27:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:27:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:27:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:27:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:27:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:27:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:27:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:27:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:27:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:27:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:27:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:27:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:27:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:27:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:27:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:27:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:27:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:27:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:27:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:27:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:27:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:27:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:27:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:27:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:27:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:27:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:27:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:27:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:27:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:27:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:27:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:27:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:27:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:27:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:27:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:27:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:27:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:27:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:27:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:27:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:27:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:27:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:27:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:27:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:27:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:27:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:27:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:27:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:27:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:27:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:27:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:27:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:27:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:27:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:27:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:27:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:27:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:27:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:27:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:27:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:27:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:27:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:27:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:27:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:27:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:27:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:27:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:27:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:27:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:27:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:27:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:27:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:27:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:27:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:27:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:27:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:27:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:27:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:27:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:27:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:27:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:27:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:27:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:27:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:27:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:27:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:27:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:27:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:27:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:27:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:27:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:27:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:27:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:27:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:27:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:27:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:27:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:27:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:27:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:27:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:27:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:27:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:27:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:27:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:27:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:27:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:27:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:27:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:27:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:27:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:27:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:27:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:27:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:27:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:27:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:27:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:27:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:27:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:27:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:27:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:27:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:27:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:27:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:27:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:27:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:27:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:27:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:27:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:27:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:27:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:27:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:27:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:27:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:27:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:27:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:27:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:27:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:27:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:27:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:27:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:27:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:27:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:27:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:27:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:27:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:27:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:27:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:27:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:27:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:27:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:27:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:27:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:27:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:27:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:27:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:27:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:27:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:27:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:27:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:28:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:28:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:28:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:28:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:28:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:28:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:28:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:28:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:28:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:28:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:28:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:28:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:28:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:28:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:28:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:28:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:28:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:28:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:28:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:28:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:28:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:28:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:28:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:28:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:28:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:28:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:28:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:28:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:28:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:28:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:28:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:28:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:28:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:28:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:28:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:28:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:28:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:28:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:28:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:28:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:28:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:28:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:28:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:28:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:28:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:28:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:28:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:28:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:28:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:28:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:28:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:28:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:28:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:28:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:28:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:28:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:28:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:28:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:28:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:28:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:28:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:28:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:28:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:28:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:28:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:28:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:28:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:28:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:28:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:28:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:28:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:28:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:28:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:28:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:28:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:28:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:28:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:28:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:28:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:28:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:28:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:28:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:28:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:28:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:28:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:28:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:28:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:28:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:28:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:28:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:28:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:28:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:28:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:28:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:28:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:28:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:28:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:28:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:28:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:28:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:28:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:28:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:28:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:28:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:28:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:28:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:28:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:28:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:28:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:28:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:28:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:28:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:28:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:28:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:28:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:28:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:28:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:28:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:28:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:28:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:28:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:28:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:28:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:28:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:28:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:28:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:28:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:28:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:28:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:28:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:28:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:28:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:28:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:28:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:28:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:28:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:28:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:28:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:28:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:28:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:28:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:28:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:28:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:28:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:28:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:28:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:28:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:28:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:28:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:28:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:28:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:28:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:28:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:28:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:28:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:28:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:28:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:28:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:28:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:28:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:28:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:28:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:28:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:28:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:28:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:28:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:28:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:28:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:28:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:28:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:28:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:28:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:28:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:28:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:28:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:28:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:28:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:28:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:28:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:28:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:28:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:28:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:28:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:28:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:28:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:28:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:28:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:28:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:28:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:28:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:28:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:28:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:28:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:28:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:28:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:28:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:28:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:28:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:28:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:28:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:28:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:28:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:28:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:28:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:28:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:28:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:28:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:28:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:28:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:28:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:28:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:28:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:28:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:28:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:28:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:28:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:29:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:29:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:29:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:29:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:29:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:29:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:29:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:29:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:29:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:29:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:29:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:29:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:29:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:29:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:29:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:29:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:29:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:29:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:29:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:29:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:29:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:29:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:29:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:29:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:29:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:29:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:29:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:29:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:29:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:29:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:29:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:29:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:29:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:29:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:29:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:29:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:29:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:29:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:29:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:29:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:29:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:29:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:29:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:29:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:29:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:29:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:29:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:29:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:29:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:29:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:29:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:29:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:29:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:29:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:29:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:29:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:29:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:29:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:29:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:29:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:29:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:29:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:29:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:29:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:29:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:29:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:29:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:29:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:29:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:29:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:29:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:29:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:29:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:29:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:29:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:29:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:29:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:29:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:29:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:29:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:29:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:29:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:29:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:29:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:29:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:29:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:29:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:29:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:29:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:29:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:29:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:29:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:29:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:29:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:29:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:29:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:29:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:29:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:29:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:29:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:29:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:29:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:29:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:29:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:29:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:29:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:29:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:29:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:29:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:29:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:29:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:29:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:29:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:29:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:29:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:29:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:29:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:29:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:29:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:29:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:29:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:29:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:29:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:29:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:29:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:29:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:29:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:29:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:29:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:29:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:29:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:29:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:29:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:29:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:29:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:29:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:29:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:29:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:29:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:29:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:29:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:29:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:29:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:29:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:29:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:29:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:29:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:29:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:29:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:29:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:29:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:29:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:29:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:29:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:29:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:29:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:29:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:29:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:29:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:29:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:29:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:29:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:29:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:29:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:29:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:29:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:29:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:29:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:29:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:29:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:29:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:29:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:29:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:29:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:29:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:29:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:29:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:29:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:29:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:29:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:29:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:29:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:29:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:29:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:29:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:29:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:29:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:29:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:29:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:29:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:29:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:29:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:29:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:29:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:29:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:29:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:29:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:29:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:29:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:29:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:29:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:29:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:29:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:29:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:29:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:29:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:29:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:29:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:29:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:29:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:29:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:29:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:29:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:29:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:29:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:29:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:30:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:30:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:30:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:30:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:30:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:30:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:30:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:30:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:30:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:30:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:30:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:30:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:30:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:30:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:30:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:30:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:30:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:30:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:30:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:30:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:30:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:30:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:30:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:30:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:30:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:30:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:30:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:30:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:30:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:30:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:30:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:30:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:30:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:30:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:30:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:30:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:30:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:30:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:30:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:30:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:30:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:30:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:30:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:30:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:30:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:30:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:30:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:30:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:30:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:30:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:30:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:30:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:30:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:30:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:30:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:30:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:30:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:30:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:30:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:30:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:30:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:30:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:30:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:30:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:30:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:30:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:30:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:30:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:30:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:30:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:30:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:30:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:30:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:30:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:30:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:30:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:30:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:30:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:30:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:30:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:30:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:30:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:30:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:30:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:30:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:30:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:30:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:30:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:30:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:30:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:30:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:30:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:30:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:30:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:30:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:30:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:30:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:30:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:30:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:30:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:30:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:30:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:30:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:30:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:30:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:30:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:30:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:30:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:30:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:30:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:30:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:30:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:30:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:30:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:30:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:30:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:30:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:30:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:30:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:30:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:30:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:30:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:30:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:30:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:30:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:30:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:30:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:30:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:30:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:30:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:30:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:30:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:30:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:30:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:30:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:30:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:30:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:30:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:30:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:30:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:30:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:30:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:30:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:30:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:30:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:30:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:30:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:30:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:30:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:30:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:30:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:30:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:30:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:30:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:30:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:30:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:30:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:30:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:30:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:30:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:30:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:30:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:30:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:30:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:30:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:30:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:30:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:30:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:30:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:30:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:30:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:30:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:30:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:30:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:30:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:30:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:30:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:30:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:30:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:30:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:30:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:30:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:30:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:30:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:30:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:30:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:30:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:30:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:30:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:30:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:30:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:30:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:30:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:30:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:30:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:30:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:30:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:30:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:30:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:30:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:30:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:30:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:30:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:30:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:30:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:30:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:30:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:30:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:30:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:30:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:30:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:30:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:30:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:30:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:30:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:30:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:31:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:31:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:31:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:31:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:31:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:31:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:31:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:31:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:31:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:31:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:31:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:31:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:31:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:31:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:31:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:31:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:31:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:31:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:31:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:31:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:31:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:31:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:31:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:31:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:31:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:31:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:31:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:31:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:31:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:31:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:31:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:31:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:31:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:31:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:31:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:31:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:31:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:31:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:31:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:31:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:31:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:31:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:31:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:31:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:31:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:31:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:31:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:31:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:31:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:31:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:31:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:31:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:31:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:31:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:31:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:31:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:31:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:31:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:31:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:31:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:31:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:31:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:31:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:31:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:31:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:31:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:31:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:31:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:31:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:31:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:31:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:31:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:31:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:31:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:31:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:31:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:31:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:31:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:31:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:31:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:31:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:31:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:31:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:31:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:31:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:31:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:31:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:31:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:31:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:31:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:31:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:31:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:31:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:31:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:31:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:31:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:31:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:31:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:31:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:31:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:31:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:31:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:31:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:31:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:31:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:31:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:31:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:31:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:31:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:31:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:31:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:31:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:31:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:31:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:31:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:31:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:31:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:31:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:31:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:31:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:31:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:31:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:31:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:31:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:31:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:31:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:31:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:31:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:31:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:31:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:31:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:31:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:31:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:31:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:31:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:31:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:31:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:31:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:31:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:31:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:31:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:31:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:31:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:31:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:31:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:31:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:31:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:31:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:31:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:31:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:31:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:31:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:31:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:31:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:31:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:31:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:31:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:31:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:31:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:31:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:31:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:31:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:31:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:31:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:31:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:31:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:31:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:31:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:31:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:31:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:31:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:31:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:31:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:31:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:31:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:31:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:31:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:31:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:31:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:31:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:31:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:31:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:31:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:31:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:31:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:31:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:31:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:31:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:31:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:31:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:31:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:31:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:31:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:31:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:31:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:31:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:31:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:31:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:31:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:31:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:31:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:31:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:31:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:31:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:31:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:31:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:32:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:32:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:32:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:32:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:32:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:32:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:32:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:32:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:32:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:32:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:32:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:32:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:32:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:32:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:32:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:32:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:32:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:32:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:32:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:32:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:32:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:32:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:32:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:32:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:32:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:32:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:32:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:32:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:32:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:32:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:32:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:32:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:32:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:32:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:32:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:32:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:32:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:32:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:32:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:32:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:32:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:32:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:32:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:32:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:32:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:32:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:32:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:32:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:32:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:32:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:32:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:32:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:32:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:32:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:32:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:32:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:32:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:32:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:32:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:32:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:32:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:32:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:32:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:32:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:32:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:32:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:32:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:32:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:32:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:32:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:32:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:32:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:32:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:32:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:32:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:32:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:32:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:32:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:32:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:32:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:32:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:32:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:32:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:32:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:32:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:32:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:32:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:32:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:32:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:32:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:32:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:32:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:32:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:32:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:32:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:32:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:32:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:32:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:32:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:32:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:32:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:32:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:32:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:32:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:32:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:32:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:32:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:32:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:32:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:32:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:32:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:32:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:32:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:32:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:32:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:32:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:32:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:32:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:32:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:32:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:32:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:32:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:32:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:32:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:32:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:32:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:32:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:32:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:32:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:32:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:32:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:32:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:32:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:32:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:32:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:32:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:32:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:32:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:32:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:32:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:32:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:32:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:32:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:32:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:32:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:32:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:32:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:32:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:32:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:32:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:32:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:32:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:32:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:32:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:32:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:32:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:32:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:32:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:32:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:32:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:32:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:32:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:32:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:32:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:32:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:32:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:32:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:32:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:32:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:32:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:32:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:32:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:32:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:32:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:32:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:32:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:32:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:32:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:32:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:32:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:32:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:32:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:32:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:32:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:32:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:32:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:32:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:32:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:32:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:32:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:32:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:32:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:32:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:32:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:32:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:32:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:32:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:32:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:32:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:32:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:32:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:32:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:32:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:32:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:32:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:32:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:32:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:32:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:32:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:32:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:32:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:32:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:32:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:32:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:32:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:32:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:33:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:33:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:33:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:33:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:33:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:33:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:33:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:33:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:33:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:33:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:33:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:33:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:33:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:33:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:33:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:33:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:33:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:33:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:33:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:33:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:33:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:33:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:33:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:33:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:33:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:33:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:33:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:33:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:33:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:33:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:33:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:33:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:33:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:33:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:33:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:33:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:33:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:33:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:33:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:33:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:33:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:33:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:33:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:33:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:33:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:33:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:33:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:33:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:33:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:33:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:33:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:33:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:33:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:33:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:33:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:33:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:33:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:33:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:33:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:33:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:33:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:33:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:33:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:33:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:33:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:33:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:33:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:33:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:33:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:33:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:33:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:33:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:33:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:33:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:33:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:33:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:33:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:33:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:33:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:33:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:33:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:33:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:33:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:33:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:33:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:33:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:33:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:33:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:33:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:33:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:33:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:33:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:33:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:33:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:33:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:33:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:33:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:33:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:33:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:33:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:33:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:33:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:33:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:33:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:33:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:33:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:33:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:33:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:33:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:33:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:33:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:33:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:33:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:33:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:33:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:33:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:33:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:33:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:33:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:33:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:33:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:33:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:33:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:33:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:33:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:33:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:33:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:33:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:33:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:33:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:33:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:33:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:33:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:33:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:33:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:33:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:33:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:33:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:33:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:33:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:33:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:33:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:33:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:33:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:33:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:33:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:33:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:33:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:33:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:33:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:33:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:33:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:33:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:33:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:33:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:33:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:33:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:33:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:33:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:33:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:33:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:33:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:33:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:33:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:33:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:33:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:33:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:33:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:33:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:33:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:33:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:33:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:33:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:33:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:33:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:33:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:33:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:33:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:33:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:33:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:33:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:33:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:33:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:33:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:33:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:33:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:33:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:33:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:33:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:33:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:33:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:33:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:33:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:33:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:33:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:33:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:33:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:33:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:33:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:33:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:33:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:33:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:33:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:33:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:33:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:33:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:33:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:33:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:33:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:33:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:33:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:33:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:33:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:33:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:33:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:33:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:34:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:34:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:34:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:34:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:34:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:34:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:34:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:34:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:34:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:34:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:34:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:34:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:34:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:34:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:34:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:34:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:34:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:34:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:34:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:34:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:34:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:34:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:34:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:34:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:34:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:34:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:34:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:34:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:34:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:34:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:34:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:34:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:34:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:34:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:34:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:34:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:34:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:34:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:34:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:34:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:34:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:34:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:34:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:34:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:34:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:34:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:34:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:34:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:34:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:34:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:34:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:34:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:34:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:34:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:34:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:34:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:34:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:34:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:34:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:34:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:34:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:34:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:34:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:34:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:34:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:34:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:34:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:34:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:34:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:34:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:34:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:34:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:34:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:34:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:34:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:34:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:34:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:34:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:34:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:34:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:34:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:34:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:34:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:34:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:34:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:34:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:34:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:34:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:34:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:34:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:34:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:34:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:34:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:34:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:34:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:34:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:34:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:34:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:34:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:34:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:34:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:34:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:34:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:34:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:34:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:34:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:34:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:34:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:34:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:34:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:34:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:34:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:34:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:34:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:34:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:34:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:34:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:34:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:34:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:34:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:34:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:34:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:34:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:34:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:34:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:34:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:34:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:34:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:34:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:34:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:34:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:34:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:34:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:34:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:34:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:34:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:34:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:34:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:34:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:34:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:34:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:34:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:34:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:34:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:34:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:34:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:34:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:34:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:34:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:34:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:34:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:34:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:34:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:34:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:34:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:34:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:34:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:34:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:34:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:34:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:34:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:34:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:34:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:34:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:34:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:34:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:34:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:34:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:34:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:34:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:34:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:34:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:34:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:34:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:34:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:34:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:34:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:34:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:34:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:34:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:34:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:34:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:34:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:34:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:34:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:34:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:34:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:34:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:34:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:34:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:34:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:34:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:34:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:34:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:34:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:34:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:34:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:34:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:34:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:34:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:34:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:34:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:34:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:34:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:34:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:34:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:34:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:34:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:34:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:34:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:34:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:34:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:34:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:34:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:34:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:34:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:35:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:35:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:35:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:35:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:35:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:35:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:35:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:35:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:35:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:35:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:35:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:35:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:35:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:35:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:35:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:35:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:35:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:35:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:35:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:35:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:35:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:35:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:35:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:35:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:35:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:35:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:35:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:35:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:35:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:35:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:35:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:35:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:35:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:35:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:35:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:35:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:37:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:37:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:37:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:37:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:37:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:37:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:37:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:37:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:37:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:37:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:37:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:37:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:37:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:37:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:37:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:37:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:37:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:37:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:37:19 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 14:37:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:37:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:37:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:37:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:37:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:37:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:37:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:37:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:37:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:37:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:37:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:37:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:37:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:37:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:37:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:37:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:37:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:37:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:37:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:37:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:37:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:37:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:37:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:37:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:37:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:37:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:37:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:37:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:37:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:37:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:37:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:37:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:37:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:37:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:37:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:37:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:37:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:37:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:37:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:37:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:37:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:37:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:37:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:37:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:37:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:37:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:37:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:37:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:37:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:37:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:37:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:37:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:37:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:37:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:37:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:37:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:37:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:37:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:37:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:37:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:37:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:37:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:37:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:37:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:37:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:37:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:37:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:37:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:37:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:37:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:37:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:37:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:37:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:37:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:37:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:37:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:37:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:37:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:37:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:37:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:37:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:37:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:37:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:37:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:37:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:37:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:37:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:37:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:37:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:37:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:37:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:37:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:37:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:37:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:37:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:37:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:37:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:37:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:37:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:37:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:37:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:37:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:37:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:37:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:37:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:37:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:37:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:37:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:37:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:37:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:37:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:37:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:37:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:37:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:37:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:37:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:37:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:37:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:37:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:37:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:37:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:37:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:37:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:37:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:37:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:37:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:37:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:37:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:37:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:37:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:37:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:37:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:37:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:37:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:37:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:37:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:37:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:37:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:37:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:37:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:37:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:37:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:37:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:37:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:38:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:38:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:38:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:38:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:38:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:38:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:38:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:38:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:38:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:38:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:38:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:38:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:38:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:38:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:38:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:38:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:38:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:38:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:38:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:38:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:38:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:38:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:38:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:38:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:38:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:38:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:38:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:38:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:38:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:38:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:38:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:38:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:38:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:38:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:38:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:38:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:38:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:38:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:38:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:38:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:38:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:38:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:38:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:38:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:38:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:38:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:38:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:38:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:38:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:38:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:38:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:38:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:38:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:38:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:38:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:38:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:38:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:38:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:38:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:38:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:38:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:38:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:38:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:38:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:38:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:38:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:38:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:38:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:38:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:38:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:38:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:38:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:38:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:38:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:38:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:38:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:38:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:38:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:38:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:38:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:38:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:38:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:38:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:38:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:38:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:38:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:38:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:38:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:38:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:38:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:38:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:38:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:38:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:38:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:38:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:38:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:38:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:38:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:38:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:38:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:38:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:38:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:38:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:38:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:38:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:38:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:38:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:38:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:38:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:38:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:38:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:38:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:38:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:38:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:38:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:38:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:38:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:38:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:38:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:38:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:38:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:38:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:38:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:38:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:38:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:38:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:38:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:38:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:38:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:38:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:38:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:38:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:38:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:38:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:38:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:38:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:38:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:38:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:38:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:38:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:38:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:38:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:38:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:38:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:38:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:38:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:38:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:38:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:38:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:38:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:38:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:38:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:38:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:38:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:38:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:38:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:38:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:38:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:38:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:38:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:38:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:38:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:38:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:38:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:38:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:38:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:38:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:38:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:38:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:38:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:38:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:38:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:38:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:38:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:38:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:38:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:38:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:38:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:38:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:38:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:38:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:38:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:38:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:38:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:38:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:38:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:38:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:38:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:38:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:38:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:38:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:38:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:38:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:38:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:38:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:38:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:38:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:38:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:38:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:38:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:38:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:38:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:38:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:38:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:38:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:38:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:38:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:38:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:38:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:38:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:38:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:38:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:38:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:38:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:38:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:38:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:39:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:39:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:39:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:39:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:39:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:39:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:39:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:39:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:39:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:39:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:39:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:39:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:39:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:39:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:39:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:39:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:39:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:39:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:39:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:39:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:39:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:39:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:39:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:39:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:39:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:39:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:39:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:39:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:39:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:39:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:39:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:39:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:39:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:39:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:39:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:39:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:39:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:39:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:39:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:39:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:39:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:39:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:39:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:39:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:39:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:39:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:39:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:39:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:39:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:39:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:39:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:39:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:39:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:39:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:39:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:39:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:39:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:39:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:39:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:39:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:39:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:39:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:39:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:39:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:39:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:39:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:39:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:39:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:39:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:39:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:39:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:39:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:39:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:39:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:39:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:39:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:39:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:39:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:39:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:39:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:39:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:39:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:39:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:39:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:39:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:39:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:39:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:39:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:39:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:39:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:39:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:39:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:39:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:39:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:39:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:39:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:39:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:39:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:39:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:39:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:39:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:39:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:39:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:39:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:39:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:39:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:39:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:39:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:39:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:39:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:39:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:39:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:39:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:39:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:39:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:39:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:39:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:39:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:39:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:39:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:39:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:39:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:39:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:39:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:39:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:39:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:39:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:39:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:39:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:39:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:39:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:39:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:39:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:39:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:39:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:39:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:39:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:39:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:39:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:39:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:39:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:39:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:39:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:39:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:39:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:39:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:39:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:39:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:39:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:39:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:39:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:39:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:39:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:39:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:39:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:39:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:40:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:40:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:40:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:40:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:40:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:40:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:40:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:40:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:40:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:40:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:40:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:40:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:40:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:40:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:40:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:40:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:40:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:40:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:40:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:40:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:40:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:40:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:40:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:40:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:40:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:40:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:40:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:40:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:40:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:40:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:40:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:40:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:40:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:40:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:40:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:40:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:40:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:40:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:40:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:40:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:40:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:40:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:40:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:40:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:40:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:40:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:40:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:40:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:40:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:40:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:40:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:40:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:40:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:40:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:40:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:40:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:40:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:40:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:40:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:40:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:40:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:40:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:40:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:40:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:40:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:40:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:40:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:40:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:40:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:40:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:40:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:40:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:40:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:40:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:40:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:40:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:40:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:40:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:40:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:40:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:40:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:40:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:40:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:40:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:40:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:40:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:40:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:40:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:40:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:40:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:40:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:40:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:40:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:40:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:40:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:40:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:40:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:40:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:40:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:40:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:40:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:40:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:40:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:40:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:40:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:40:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:40:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:40:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:40:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:40:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:40:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:40:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:40:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:40:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:40:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:40:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:40:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:40:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:40:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:40:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:40:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:40:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:40:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:40:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:40:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:40:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:40:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:40:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:40:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:40:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:40:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:40:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:40:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:40:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:40:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:40:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:40:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:40:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:40:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:40:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:40:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:40:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:40:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:40:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:40:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:40:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:40:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:40:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:40:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:40:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:40:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:40:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:40:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:40:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:40:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:40:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:40:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:40:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:40:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:40:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:40:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:40:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:40:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:40:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:40:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:40:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:40:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:40:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:40:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:40:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:40:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:40:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:40:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:40:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:40:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:40:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:40:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:40:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:40:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:40:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:40:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:40:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:40:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:40:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:40:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:40:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:40:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:40:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:40:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:40:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:40:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:40:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:40:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:40:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:40:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:40:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:40:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:40:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:40:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:40:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:40:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:40:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:40:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:40:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:40:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:40:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:40:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:40:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:40:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:40:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:40:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:40:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:40:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:40:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:40:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:40:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:41:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:41:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:41:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:41:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:41:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:41:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:41:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:41:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:41:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:41:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:41:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:41:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:41:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:41:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:41:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:41:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:41:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:41:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:41:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:41:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:41:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:41:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:41:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:41:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:41:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:41:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:41:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:41:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:41:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:41:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:41:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:41:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:41:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:41:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:41:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:41:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:41:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:41:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:41:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:41:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:41:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:41:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:41:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:41:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:41:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:41:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:41:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:41:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:41:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:41:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:41:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:41:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:41:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:41:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:41:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:41:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:41:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:41:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:41:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:41:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:41:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:41:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:41:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:41:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:41:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:41:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:41:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:41:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:41:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:41:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:41:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:41:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:41:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:41:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:41:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:41:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:41:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:41:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:41:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:41:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:41:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:41:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:41:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:41:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:41:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:41:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:41:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:41:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:41:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:41:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:41:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:41:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:41:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:41:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:41:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:41:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:41:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:41:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:41:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:41:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:41:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:41:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:41:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:41:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:41:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:41:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:41:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:41:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:41:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:41:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:41:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:41:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:41:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:41:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:41:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:41:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:41:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:41:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:41:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:41:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:41:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:41:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:41:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:41:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:41:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:41:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:41:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:41:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:41:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:41:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:41:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:41:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:41:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:41:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:41:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:41:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:41:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:41:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:41:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:41:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:41:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:41:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:41:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:41:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:41:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:41:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:41:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:41:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:41:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:41:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:41:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:41:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:41:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:41:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:41:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:41:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:41:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:41:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:41:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:41:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:41:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:41:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:41:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:41:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:41:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:41:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:41:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:41:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:41:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:41:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:41:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:41:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:41:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:41:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:41:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:41:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:41:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:41:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:41:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:41:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:41:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:41:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:41:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:41:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:41:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:41:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:41:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:41:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:41:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:41:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:41:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:41:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:41:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:41:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:41:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:41:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:41:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:41:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:41:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:41:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:41:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:41:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:41:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:41:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:41:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:41:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:41:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:41:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:41:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:41:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:41:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:41:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:41:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:41:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:41:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:41:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:42:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:42:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:42:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:42:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:42:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:42:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:42:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:42:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:42:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:42:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:42:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:42:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:42:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:42:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:42:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:42:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:42:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:42:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:42:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:42:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:42:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:42:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:42:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:42:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:42:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:42:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:42:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:42:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:42:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:42:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:42:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:42:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:42:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:42:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:42:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:42:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:42:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:42:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:42:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:42:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:42:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:42:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:42:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:42:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:42:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:42:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:42:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:42:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:42:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:42:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:42:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:42:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:42:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:42:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:42:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:42:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:42:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:42:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:42:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:42:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:42:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:42:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:42:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:42:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:42:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:42:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:42:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:42:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:42:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:42:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:42:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:42:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:42:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:42:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:42:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:42:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:42:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:42:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:42:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:42:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:42:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:42:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:42:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:42:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:42:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:42:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:42:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:42:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:42:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:42:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:42:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:42:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:42:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:42:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:42:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:42:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:42:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:42:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:42:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:42:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:42:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:42:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:42:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:42:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:42:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:42:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:42:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:42:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:42:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:42:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:42:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:42:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:42:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:42:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:42:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:42:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:42:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:42:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:42:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:42:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:42:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:42:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:42:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:42:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:42:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:42:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:42:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:42:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:42:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:42:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:42:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:42:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:42:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:42:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:42:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:42:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:42:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:42:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:42:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:42:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:42:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:42:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:42:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:42:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:42:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:42:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:42:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:42:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:42:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:42:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:42:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:42:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:42:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:42:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:42:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:42:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:42:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:42:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:42:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:42:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:42:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:42:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:42:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:42:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:42:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:42:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:42:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:42:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:42:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:42:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:42:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:42:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:42:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:42:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:42:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:42:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:42:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:42:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:42:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:42:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:42:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:42:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:42:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:42:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:42:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:42:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:42:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:42:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:42:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:42:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:42:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:42:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:42:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:42:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:42:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:42:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:42:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:42:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:42:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:42:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:42:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:42:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:42:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:42:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:42:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:42:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:42:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:42:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:42:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:42:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:42:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:42:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:42:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:42:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:42:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:42:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:43:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:43:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:43:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:43:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:43:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:43:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:43:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:43:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:43:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:43:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:43:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:43:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:43:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:43:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:43:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:43:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:43:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:43:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:43:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:43:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:43:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:43:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:43:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:43:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:43:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:43:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:43:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:43:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:43:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:43:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:43:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:43:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:43:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:43:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:43:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:43:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:43:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:43:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:43:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:43:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:43:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:43:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:43:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:43:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:43:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:43:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:43:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:43:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:43:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:43:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:43:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:43:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:43:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:43:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:43:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:43:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:43:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:43:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:43:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:43:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:43:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:43:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:43:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:43:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:43:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:43:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:43:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:43:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:43:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:43:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:43:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:43:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:43:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:43:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:43:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:43:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:43:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:43:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:43:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:43:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:43:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:43:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:43:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:43:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:43:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:43:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:43:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:43:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:43:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:43:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:43:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:43:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:43:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:43:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:43:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:43:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:43:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:43:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:43:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:43:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:43:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:43:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:43:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:43:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:43:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:43:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:43:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:43:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:43:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:43:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:43:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:43:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:43:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:43:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:43:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:43:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:43:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:43:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:43:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:43:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:43:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:43:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:43:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:43:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:43:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:43:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:43:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:43:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:43:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:43:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:43:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:43:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:43:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:43:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:43:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:43:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:43:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:43:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:43:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:43:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:43:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:43:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:43:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:43:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:43:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:43:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:43:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:43:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:43:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:43:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:43:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:43:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:43:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:43:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:43:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:43:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:43:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:43:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:43:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:43:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:43:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:43:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:43:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:43:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:43:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:43:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:43:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:43:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:43:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:43:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:43:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:43:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:43:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:43:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:43:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:43:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:43:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:43:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:43:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:43:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:43:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:43:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:43:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:43:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:43:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:43:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:43:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:43:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:43:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:43:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:43:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:43:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:43:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:43:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:43:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:43:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:43:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:43:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:43:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:43:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:43:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:43:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:43:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:43:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:43:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:43:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:43:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:43:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:43:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:43:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:43:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:43:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:43:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:43:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:43:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:43:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:44:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:44:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:44:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:44:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:44:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:44:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:44:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:44:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:44:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:44:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:44:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:44:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:44:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:44:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:44:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:44:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:44:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:44:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:44:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:44:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:44:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:44:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:44:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:44:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:44:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:44:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:44:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:44:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:44:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:44:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:44:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:44:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:44:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:44:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:44:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:44:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:44:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:44:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:44:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:44:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:44:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:44:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:44:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:44:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:44:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:44:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:44:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:44:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:44:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:44:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:44:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:44:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:44:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:44:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:44:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:44:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:44:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:44:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:44:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:44:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:44:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:44:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:44:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:44:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:44:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:44:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:44:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:44:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:44:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:44:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:44:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:44:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:44:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:44:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:44:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:44:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:44:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:44:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:44:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:44:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:44:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:44:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:44:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:44:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:44:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:44:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:44:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:44:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:44:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:44:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:44:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:44:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:44:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:44:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:44:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:44:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:44:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:44:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:44:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:44:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:44:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:44:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:44:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:44:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:44:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:44:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:44:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:44:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:44:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:44:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:44:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:44:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:44:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:44:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:44:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:44:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:44:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:44:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:44:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:44:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:44:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:44:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:44:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:44:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:44:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:44:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:44:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:44:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:44:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:44:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:44:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:44:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:44:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:44:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:44:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:44:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:44:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:44:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:44:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:44:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:44:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:44:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:44:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:44:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:44:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:44:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:44:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:44:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:44:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:44:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:44:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:44:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:44:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:44:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:44:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:44:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:44:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:44:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:44:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:44:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:44:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:44:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:44:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:44:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:44:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:44:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:44:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:44:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:44:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:44:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:44:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:44:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:44:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:44:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:44:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:44:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:44:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:44:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:44:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:44:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:44:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:44:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:44:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:44:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:44:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:44:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:44:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:44:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:44:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:44:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:44:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:44:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:44:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:44:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:44:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:44:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:44:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:44:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:44:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:44:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:44:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:44:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:44:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:44:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:44:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:44:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:44:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:44:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:44:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:44:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:44:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:44:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:44:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:44:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:44:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:44:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:45:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:45:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:45:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:45:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:45:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:45:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:45:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:45:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:45:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:45:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:45:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:45:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:45:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:45:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:45:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:45:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:45:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:45:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:45:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:45:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:45:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:45:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:45:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:45:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:45:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:45:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:45:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:45:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:45:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:45:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:45:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:45:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:45:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:45:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:45:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:45:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:45:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:45:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:45:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:45:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:45:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:45:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:45:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:45:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:45:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:45:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:45:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:45:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:45:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:45:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:45:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:45:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:45:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:45:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:47:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:47:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:47:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:47:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:47:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:47:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:47:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:47:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:47:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:47:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:47:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:47:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:47:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:47:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:47:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:47:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:47:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:47:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:47:25 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 14:47:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:47:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:47:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:47:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:47:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:47:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:47:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:47:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:47:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:47:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:47:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:47:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:47:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:47:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:47:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:47:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:47:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:47:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:47:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:47:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:47:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:47:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:47:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:47:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:47:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:47:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:47:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:47:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:47:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:47:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:47:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:47:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:47:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:47:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:47:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:47:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:47:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:47:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:47:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:47:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:47:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:47:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:47:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:47:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:47:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:47:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:47:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:47:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:47:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:47:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:47:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:47:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:47:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:47:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:47:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:47:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:47:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:47:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:47:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:47:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:47:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:47:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:47:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:47:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:47:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:47:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:47:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:47:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:47:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:47:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:47:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:47:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:47:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:47:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:47:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:47:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:47:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:47:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:47:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:47:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:47:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:47:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:47:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:47:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:47:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:47:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:48:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:48:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:48:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:48:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:48:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:48:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:48:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:48:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:48:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:48:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:48:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:48:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:48:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:48:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:48:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:48:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:48:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:48:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:48:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:48:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:48:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:48:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:48:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:48:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:48:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:48:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:48:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:48:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:48:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:48:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:48:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:48:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:48:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:48:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:48:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:48:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:48:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:48:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:48:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:48:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:48:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:48:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:48:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:48:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:48:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:48:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:48:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:48:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:48:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:48:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:48:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:48:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:48:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:48:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:48:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:48:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:48:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:48:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:48:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:48:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:48:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:48:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:48:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:48:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:48:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:48:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:48:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:48:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:48:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:48:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:48:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:48:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:48:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:48:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:48:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:48:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:48:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:48:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:48:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:48:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:48:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:48:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:48:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:48:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:48:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:48:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:48:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:48:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:48:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:48:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:48:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:48:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:48:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:48:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:48:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:48:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:48:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:48:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:48:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:48:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:48:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:48:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:48:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:48:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:48:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:48:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:48:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:48:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:48:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:48:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:48:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:48:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:48:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:48:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:48:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:48:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:48:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:48:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:48:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:48:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:48:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:48:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:48:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:48:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:48:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:48:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:48:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:48:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:48:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:48:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:48:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:48:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:48:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:48:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:48:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:48:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:48:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:48:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:48:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:48:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:48:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:48:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:48:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:48:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:48:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:48:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:48:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:48:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:48:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:48:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:48:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:48:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:48:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:48:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:48:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:48:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:48:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:48:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:48:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:48:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:48:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:48:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:48:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:48:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:48:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:48:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:48:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:48:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:48:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:48:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:48:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:48:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:48:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:48:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:48:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:48:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:48:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:48:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:48:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:48:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:48:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:48:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:48:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:48:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:48:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:48:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:48:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:48:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:48:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:48:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:48:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:48:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:48:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:48:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:48:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:48:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:48:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:48:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:48:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:48:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:48:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:48:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:48:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:48:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:48:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:48:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:48:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:48:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:48:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:48:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:48:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:48:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:48:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:48:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:48:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:48:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:49:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:49:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:49:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:49:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:49:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:49:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:49:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:49:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:49:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:49:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:49:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:49:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:49:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:49:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:49:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:49:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:49:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:49:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:49:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:49:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:49:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:49:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:49:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:49:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:49:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:49:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:49:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:49:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:49:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:49:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:49:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:49:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:49:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:49:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:49:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:49:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:49:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:49:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:49:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:49:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:49:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:49:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:49:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:49:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:49:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:49:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:49:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:49:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:49:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:49:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:49:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:49:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:49:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:49:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:49:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:49:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:49:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:49:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:49:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:49:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:49:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:49:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:49:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:49:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:49:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:49:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:49:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:49:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:49:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:49:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:49:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:49:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:49:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:49:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:49:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:49:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:49:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:49:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:49:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:49:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:49:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:49:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:49:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:49:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:49:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:49:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:49:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:49:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:49:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:49:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:49:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:49:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:49:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:49:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:49:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:49:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:49:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:49:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:49:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:49:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:49:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:49:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:49:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:49:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:49:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:49:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:49:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:49:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:49:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:49:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:49:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:49:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:49:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:49:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:49:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:49:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:49:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:49:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:49:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:49:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:49:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:49:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:49:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:49:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:49:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:49:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:49:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:49:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:49:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:49:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:49:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:49:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:49:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:49:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:49:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:49:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:49:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:49:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:49:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:49:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:49:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:49:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:49:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:49:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:49:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:49:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:49:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:49:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:49:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:49:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:49:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:49:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:49:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:49:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:49:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:49:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:49:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:49:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:49:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:49:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:49:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:49:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:49:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:49:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:49:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:49:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:49:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:49:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:49:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:49:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:49:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:49:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:49:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:49:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:49:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:49:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:49:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:49:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:49:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:49:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:49:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:49:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:49:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:49:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:49:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:49:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:49:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:49:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:49:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:49:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:49:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:49:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:49:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:49:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:49:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:49:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:49:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:49:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:49:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:49:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:49:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:49:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:49:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:49:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:49:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:49:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:50:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:50:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:50:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:50:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:50:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:50:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:50:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:50:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:50:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:50:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:50:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:50:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:50:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:50:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:50:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:50:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:50:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:50:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:50:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:50:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:50:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:50:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:50:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:50:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:50:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:50:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:50:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:50:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:50:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:50:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:50:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:50:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:50:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:50:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:50:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:50:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:50:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:50:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:50:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:50:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:50:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:50:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:50:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:50:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:50:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:50:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:50:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:50:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:50:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:50:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:50:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:50:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:50:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:50:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:50:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:50:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:50:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:50:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:50:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:50:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:50:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:50:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:50:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:50:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:50:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:50:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:50:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:50:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:50:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:50:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:50:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:50:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:50:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:50:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:50:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:50:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:50:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:50:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:50:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:50:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:50:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:50:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:50:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:50:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:50:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:50:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:50:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:50:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:50:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:50:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:50:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:50:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:50:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:50:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:50:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:50:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:50:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:50:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:50:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:50:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:50:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:50:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:50:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:50:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:50:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:50:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:50:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:50:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:50:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:50:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:50:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:50:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:50:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:50:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:50:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:50:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:50:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:50:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:50:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:50:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:50:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:50:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:50:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:50:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:50:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:50:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:50:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:50:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:50:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:50:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:50:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:50:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:50:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:50:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:50:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:50:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:50:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:50:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:50:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:50:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:50:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:50:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:50:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:50:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:50:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:50:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:50:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:50:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:50:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:50:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:50:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:50:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:50:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:50:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:50:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:50:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:50:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:50:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:50:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:50:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:50:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:50:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:50:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:50:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:50:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:50:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:50:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:50:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:50:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:50:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:50:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:50:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:50:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:50:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:50:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:50:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:50:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:50:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:50:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:50:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:50:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:50:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:50:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:50:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:50:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:50:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:50:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:50:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:50:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:50:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:50:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:50:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:50:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:50:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:50:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:50:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:50:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:50:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:50:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:50:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:50:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:50:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:50:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:50:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:50:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:50:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:50:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:50:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:50:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:50:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:50:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:50:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:50:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:50:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:50:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:50:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:51:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:51:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:51:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:51:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:51:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:51:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:51:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:51:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:51:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:51:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:51:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:51:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:51:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:51:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:51:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:51:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:51:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:51:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:51:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:51:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:51:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:51:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:51:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:51:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:51:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:51:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:51:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:51:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:51:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:51:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:51:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:51:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:51:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:51:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:51:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:51:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:51:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:51:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:51:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:51:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:51:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:51:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:51:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:51:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:51:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:51:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:51:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:51:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:51:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:51:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:51:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:51:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:51:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:51:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:51:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:51:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:51:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:51:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:51:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:51:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:51:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:51:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:51:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:51:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:51:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:51:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:51:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:51:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:51:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:51:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:51:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:51:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:51:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:51:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:51:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:51:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:51:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:51:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:51:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:51:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:51:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:51:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:51:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:51:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:51:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:51:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:51:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:51:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:51:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:51:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:51:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:51:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:51:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:51:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:51:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:51:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:51:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:51:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:51:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:51:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:51:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:51:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:51:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:51:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:51:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:51:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:51:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:51:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:51:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:51:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:51:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:51:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:51:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:51:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:51:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:51:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:51:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:51:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:51:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:51:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:51:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:51:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:51:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:51:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:51:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:51:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:51:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:51:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:51:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:51:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:51:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:51:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:51:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:51:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:51:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:51:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:51:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:51:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:51:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:51:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:51:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:51:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:51:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:51:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:51:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:51:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:51:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:51:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:51:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:51:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:51:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:51:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:51:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:51:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:51:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:51:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:51:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:51:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:51:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:51:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:51:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:51:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:51:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:51:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:51:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:51:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:51:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:51:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:51:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:51:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:51:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:51:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:51:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:51:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:51:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:51:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:51:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:51:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:51:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:51:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:51:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:51:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:51:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:51:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:51:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:51:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:51:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:51:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:51:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:51:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:51:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:51:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:51:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:51:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:51:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:51:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:51:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:51:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:51:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:51:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:51:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:51:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:51:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:51:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:51:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:51:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:51:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:51:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:51:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:51:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:51:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:51:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:51:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:51:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:51:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:51:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:52:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:52:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:52:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:52:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:52:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:52:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:52:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:52:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:52:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:52:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:52:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:52:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:52:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:52:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:52:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:52:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:52:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:52:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:52:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:52:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:52:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:52:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:52:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:52:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:52:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:52:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:52:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:52:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:52:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:52:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:52:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:52:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:52:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:52:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:52:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:52:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:52:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:52:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:52:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:52:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:52:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:52:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:52:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:52:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:52:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:52:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:52:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:52:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:52:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:52:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:52:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:52:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:52:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:52:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:52:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:52:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:52:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:52:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:52:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:52:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:52:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:52:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:52:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:52:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:52:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:52:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:52:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:52:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:52:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:52:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:52:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:52:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:52:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:52:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:52:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:52:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:52:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:52:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:52:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:52:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:52:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:52:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:52:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:52:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:52:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:52:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:52:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:52:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:52:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:52:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:52:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:52:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:52:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:52:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:52:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:52:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:52:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:52:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:52:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:52:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:52:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:52:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:52:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:52:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:52:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:52:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:52:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:52:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:52:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:52:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:52:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:52:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:52:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:52:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:52:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:52:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:52:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:52:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:52:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:52:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:52:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:52:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:52:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:52:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:52:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:52:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:52:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:52:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:52:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:52:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:52:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:52:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:52:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:52:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:52:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:52:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:52:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:52:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:52:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:52:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:52:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:52:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:52:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:52:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:52:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:52:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:52:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:52:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:52:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:52:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:52:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:52:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:52:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:52:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:52:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:52:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:52:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:52:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:52:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:52:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:52:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:52:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:52:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:52:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:52:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:52:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:52:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:52:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:52:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:52:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:52:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:52:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:52:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:52:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:52:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:52:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:52:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:52:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:52:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:52:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:52:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:52:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:52:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:52:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:52:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:52:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:52:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:52:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:52:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:52:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:52:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:52:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:52:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:52:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:52:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:52:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:52:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:52:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:52:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:52:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:52:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:52:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:52:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:52:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:52:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:52:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:52:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:52:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:52:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:52:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:52:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:52:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:52:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:52:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:52:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:52:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:53:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:53:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:53:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:53:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:53:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:53:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:53:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:53:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:53:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:53:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:53:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:53:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:53:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:53:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:53:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:53:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:53:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:53:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:53:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:53:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:53:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:53:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:53:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:53:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:53:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:53:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:53:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:53:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:53:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:53:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:53:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:53:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:53:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:53:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:53:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:53:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:53:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:53:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:53:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:53:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:53:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:53:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:53:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:53:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:53:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:53:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:53:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:53:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:53:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:53:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:53:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:53:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:53:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:53:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:53:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:53:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:53:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:53:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:53:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:53:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:53:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:53:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:53:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:53:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:53:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:53:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:53:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:53:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:53:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:53:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:53:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:53:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:53:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:53:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:53:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:53:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:53:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:53:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:53:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:53:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:53:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:53:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:53:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:53:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:53:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:53:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:53:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:53:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:53:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:53:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:53:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:53:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:53:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:53:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:53:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:53:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:53:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:53:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:53:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:53:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:53:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:53:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:53:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:53:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:53:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:53:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:53:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:53:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:53:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:53:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:53:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:53:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:53:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:53:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:53:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:53:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:53:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:53:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:53:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:53:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:53:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:53:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:53:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:53:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:53:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:53:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:53:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:53:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:53:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:53:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:53:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:53:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:53:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:53:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:53:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:53:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:54:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:54:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:54:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:54:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:54:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:54:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:54:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:54:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:54:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:54:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:54:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:54:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:54:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:54:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:54:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:54:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:54:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:54:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:54:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:54:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:54:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:54:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:54:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:54:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:54:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:54:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:54:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:54:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:54:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:54:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:54:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:54:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:54:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:54:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:54:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:54:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:54:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:54:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:54:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:54:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:54:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:54:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:54:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:54:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:54:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:54:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:54:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:54:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:54:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:54:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:54:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:54:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:54:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:54:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:54:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:54:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:54:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:54:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:54:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:54:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:54:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:54:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:54:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:54:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:54:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:54:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:54:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:54:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:54:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:54:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:54:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:54:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:54:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:54:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:54:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:54:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:54:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:54:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:54:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:54:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:54:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:54:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:54:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:54:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:54:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:54:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:54:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:54:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:54:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:54:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:54:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:54:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:54:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:54:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:54:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:54:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:54:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:54:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:54:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:54:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:54:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:54:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:54:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:54:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:54:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:54:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:54:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:54:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:54:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:54:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:54:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:54:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:54:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:54:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:54:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:54:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:54:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:54:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:54:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:54:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:54:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:54:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:54:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:54:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:54:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:54:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:54:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:54:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:54:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:54:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:54:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:54:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:54:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:54:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:54:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:54:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:54:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:54:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:54:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:54:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:54:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:54:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:54:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:54:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:54:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:54:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:54:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:54:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:54:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:54:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:54:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:54:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:54:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:54:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:54:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:54:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:54:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:54:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:55:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:55:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:55:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:55:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:55:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:55:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:55:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:55:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:55:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:55:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:55:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:55:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:55:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:55:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:55:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:55:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:55:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:55:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:55:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:55:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:55:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:55:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:55:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:55:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:55:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:55:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:55:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:55:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:55:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:55:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:55:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:55:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:55:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:55:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:55:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:55:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:55:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:55:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:55:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:55:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:55:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:55:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:55:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:55:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:55:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:55:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:55:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:55:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:55:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:55:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:55:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:55:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:55:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:55:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:55:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:55:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:55:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:55:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:55:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:55:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:55:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:55:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:55:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:55:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:55:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:55:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:55:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:55:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:55:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:55:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:55:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:55:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:55:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:55:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:55:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:55:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:55:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:55:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:55:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:55:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:55:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:55:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:55:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:55:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:55:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:55:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:55:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:55:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:55:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:55:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:57:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:57:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:57:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:57:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:57:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:57:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:57:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:57:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:57:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:57:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:57:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:57:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:57:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:57:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:57:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:57:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:57:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:57:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:57:30 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 14:57:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:57:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:57:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:57:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:57:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:57:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:57:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:57:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:57:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:57:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:57:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:57:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:57:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:57:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:57:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:57:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:57:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:57:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:57:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:57:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:57:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:57:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:57:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:57:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:57:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:57:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:57:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:57:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:57:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:57:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:57:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:57:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:57:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:57:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:57:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:57:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:57:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:57:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:57:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:57:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:57:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:57:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:57:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:57:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:57:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:57:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:57:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:57:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:57:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:57:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:57:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:57:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:57:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:57:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:57:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:57:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:57:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:57:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:57:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:57:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:57:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:57:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:57:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:57:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:57:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:57:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:57:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:57:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:57:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:57:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:57:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:57:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:57:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:57:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:57:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:57:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:57:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:57:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:57:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:57:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:57:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:57:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:57:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:57:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:57:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:57:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:57:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:57:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:57:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:57:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:58:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:58:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:58:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:58:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:58:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:58:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:58:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:58:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:58:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:58:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:58:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:58:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:58:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:58:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:58:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:58:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:58:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:58:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:58:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:58:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:58:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:58:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:58:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:58:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:58:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:58:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:58:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:58:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:58:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:58:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:58:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:58:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:58:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:58:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:58:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:58:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:58:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:58:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:58:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:58:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:58:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:58:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:58:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:58:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:58:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:58:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:58:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:58:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:58:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:58:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:58:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:58:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:58:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:58:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:58:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:58:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:58:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:58:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:58:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:58:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:58:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:58:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:58:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:58:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:58:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:58:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:58:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:58:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:58:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:58:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:58:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:58:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:58:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:58:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:58:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:58:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:58:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:58:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:58:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:58:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:58:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:58:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:58:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:58:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:58:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:58:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:58:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:58:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:58:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:58:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:58:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:58:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:58:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:58:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:58:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:58:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:58:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:58:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:58:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:58:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:58:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:58:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:58:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:58:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:58:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:58:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:58:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:58:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:58:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:58:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:58:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:58:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:58:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:58:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:58:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:58:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:58:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:58:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:58:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:58:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:58:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:58:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:58:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:58:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:58:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:58:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:58:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:58:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:58:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:58:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:58:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:58:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:58:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:58:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:58:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:58:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:58:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:58:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:58:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:58:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:58:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:58:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:58:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:58:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:58:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:58:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:58:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:58:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:58:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:58:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:58:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:58:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:58:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:58:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:58:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:58:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:58:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:58:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:58:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:58:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:58:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:58:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:58:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:58:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:58:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:58:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:58:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:58:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:58:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:58:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:58:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:58:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:58:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:58:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:58:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:58:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:58:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:58:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:58:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:58:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:58:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:58:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:58:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:58:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:58:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:58:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:58:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:58:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:58:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:58:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:58:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:58:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:58:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:58:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:58:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:58:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:58:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:58:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:58:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:58:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:58:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:58:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:58:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:58:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:58:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:58:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:58:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:58:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:58:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:58:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:58:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:58:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:58:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:58:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:58:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:58:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:59:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:59:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:59:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:59:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:59:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:59:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:59:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:59:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:59:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:59:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:59:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:59:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:59:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:59:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:59:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:59:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:59:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:59:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:59:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:59:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:59:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:59:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:59:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:59:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:59:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:59:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:59:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:59:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:59:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:59:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:59:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:59:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:59:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:59:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:59:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:59:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:59:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:59:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:59:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:59:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:59:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:59:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:59:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:59:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:59:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:59:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:59:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:59:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:59:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:59:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:59:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:59:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:59:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:59:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:59:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:59:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:59:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:59:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:59:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:59:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:59:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:59:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:59:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:59:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:59:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:59:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:59:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:59:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:59:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:59:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:59:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:59:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:59:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:59:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:59:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:59:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:59:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:59:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:59:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:59:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:59:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:59:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:59:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:59:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:59:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:59:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:59:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:59:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:59:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:59:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:59:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:59:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:59:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:59:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:59:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:59:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:59:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:59:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:59:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:59:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:59:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:59:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:59:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:59:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:59:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:59:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:59:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:59:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:59:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:59:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:59:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:59:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:59:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:59:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:59:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:59:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:59:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:59:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:59:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:59:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:59:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:59:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:59:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:59:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:59:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:59:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:59:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:59:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:59:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:59:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:59:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:59:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:59:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:59:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:59:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:59:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:59:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:59:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:59:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:59:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:59:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:59:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:59:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:59:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:59:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:59:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:59:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:59:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:59:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:59:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:59:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:59:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:59:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:59:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:59:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:59:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:59:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:59:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:59:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:59:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:59:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:59:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:59:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:59:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:59:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:59:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:59:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:59:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:59:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:59:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:59:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:59:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:59:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:59:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:59:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:59:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:59:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:59:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:59:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:59:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:59:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:59:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:59:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:59:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:59:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:59:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:59:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:59:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:59:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:59:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:59:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:59:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:59:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:59:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:59:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:59:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:59:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:59:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 14:59:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 14:59:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:59:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:59:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:59:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 14:59:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 14:59:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 14:59:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 14:59:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 14:59:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 14:59:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 14:59:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 14:59:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 14:59:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 14:59:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 14:59:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 14:59:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 14:59:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:00:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:00:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:00:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:00:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:00:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:00:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:00:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:00:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:00:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:00:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:00:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:00:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:00:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:00:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:00:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:00:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:00:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:00:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:00:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:00:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:00:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:00:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:00:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:00:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:00:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:00:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:00:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:00:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:00:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:00:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:00:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:00:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:00:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:00:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:00:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:00:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:00:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:00:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:00:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:00:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:00:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:00:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:00:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:00:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:00:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:00:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:00:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:00:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:00:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:00:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:00:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:00:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:00:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:00:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:00:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:00:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:00:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:00:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:00:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:00:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:00:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:00:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:00:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:00:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:00:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:00:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:00:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:00:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:00:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:00:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:00:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:00:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:00:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:00:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:00:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:00:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:00:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:00:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:00:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:00:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:00:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:00:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:00:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:00:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:00:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:00:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:00:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:00:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:00:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:00:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:00:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:00:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:00:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:00:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:00:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:00:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:00:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:00:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:00:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:00:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:00:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:00:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:00:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:00:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:00:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:00:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:00:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:00:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:00:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:00:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:00:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:00:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:00:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:00:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:00:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:00:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:00:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:00:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:00:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:00:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:00:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:00:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:00:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:00:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:00:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:00:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:00:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:00:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:00:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:00:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:00:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:00:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:00:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:00:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:00:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:00:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:00:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:00:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:00:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:00:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:00:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:00:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:00:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:00:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:00:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:00:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:00:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:00:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:00:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:00:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:00:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:00:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:00:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:00:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:00:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:00:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:00:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:00:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:00:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:00:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:00:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:00:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:00:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:00:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:00:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:00:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:00:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:00:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:00:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:00:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:00:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:00:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:00:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:00:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:00:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:00:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:00:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:00:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:00:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:00:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:00:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:00:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:00:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:00:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:00:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:00:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:00:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:00:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:00:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:00:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:00:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:00:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:00:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:00:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:00:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:00:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:00:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:00:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:00:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:00:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:00:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:00:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:00:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:00:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:00:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:00:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:00:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:00:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:00:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:00:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:00:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:00:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:00:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:00:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:00:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:00:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:01:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:01:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:01:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:01:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:01:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:01:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:01:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:01:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:01:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:01:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:01:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:01:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:01:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:01:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:01:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:01:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:01:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:01:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:01:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:01:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:01:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:01:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:01:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:01:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:01:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:01:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:01:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:01:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:01:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:01:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:01:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:01:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:01:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:01:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:01:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:01:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:01:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:01:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:01:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:01:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:01:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:01:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:01:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:01:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:01:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:01:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:01:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:01:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:01:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:01:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:01:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:01:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:01:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:01:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:01:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:01:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:01:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:01:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:01:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:01:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:01:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:01:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:01:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:01:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:01:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:01:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:01:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:01:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:01:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:01:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:01:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:01:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:01:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:01:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:01:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:01:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:01:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:01:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:01:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:01:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:01:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:01:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:01:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:01:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:01:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:01:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:01:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:01:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:01:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:01:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:01:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:01:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:01:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:01:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:01:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:01:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:01:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:01:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:01:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:01:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:01:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:01:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:01:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:01:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:01:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:01:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:01:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:01:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:01:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:01:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:01:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:01:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:01:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:01:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:01:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:01:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:01:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:01:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:01:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:01:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:01:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:01:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:01:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:01:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:01:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:01:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:01:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:01:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:01:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:01:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:01:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:01:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:01:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:01:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:01:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:01:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:01:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:01:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:01:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:01:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:01:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:01:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:01:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:01:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:01:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:01:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:01:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:01:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:01:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:01:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:01:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:01:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:01:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:01:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:01:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:01:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:02:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:02:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:02:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:02:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:02:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:02:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:02:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:02:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:02:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:02:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:02:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:02:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:02:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:02:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:02:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:02:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:02:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:02:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:02:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:02:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:02:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:02:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:02:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:02:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:02:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:02:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:02:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:02:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:02:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:02:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:02:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:02:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:02:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:02:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:02:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:02:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:02:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:02:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:02:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:02:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:02:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:02:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:02:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:02:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:02:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:02:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:02:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:02:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:02:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:02:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:02:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:02:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:02:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:02:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:02:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:02:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:02:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:02:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:02:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:02:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:02:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:02:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:02:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:02:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:02:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:02:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:02:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:02:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:02:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:02:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:02:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:02:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:02:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:02:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:02:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:02:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:02:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:02:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:02:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:02:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:02:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:02:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:02:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:02:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:02:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:02:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:02:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:02:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:02:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:02:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:02:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:02:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:02:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:02:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:02:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:02:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:02:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:02:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:02:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:02:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:02:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:02:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:02:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:02:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:02:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:02:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:02:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:02:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:02:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:02:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:02:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:02:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:02:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:02:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:02:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:02:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:02:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:02:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:02:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:02:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:02:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:02:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:02:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:02:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:02:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:02:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:02:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:02:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:02:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:02:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:02:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:02:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:02:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:02:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:02:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:02:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:02:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:02:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:02:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:02:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:02:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:02:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:02:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:02:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:02:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:02:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:02:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:02:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:02:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:02:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:02:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:02:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:02:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:02:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:02:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:02:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:02:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:02:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:02:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:02:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:02:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:02:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:02:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:02:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:02:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:02:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:02:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:02:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:02:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:02:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:02:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:02:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:02:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:02:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:02:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:02:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:02:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:02:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:02:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:02:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:02:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:02:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:02:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:02:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:02:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:02:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:02:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:02:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:02:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:02:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:02:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:02:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:02:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:02:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:02:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:02:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:02:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:02:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:02:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:02:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:02:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:02:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:02:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:02:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:02:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:02:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:02:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:02:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:02:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:02:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:02:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:02:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:02:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:02:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:02:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:02:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:03:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:03:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:03:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:03:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:03:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:03:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:03:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:03:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:03:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:03:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:03:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:03:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:03:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:03:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:03:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:03:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:03:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:03:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:03:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:03:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:03:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:03:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:03:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:03:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:03:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:03:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:03:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:03:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:03:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:03:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:03:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:03:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:03:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:03:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:03:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:03:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:03:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:03:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:03:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:03:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:03:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:03:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:03:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:03:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:03:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:03:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:03:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:03:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:03:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:03:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:03:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:03:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:03:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:03:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:03:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:03:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:03:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:03:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:03:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:03:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:03:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:03:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:03:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:03:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:03:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:03:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:03:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:03:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:03:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:03:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:03:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:03:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:03:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:03:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:03:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:03:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:03:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:03:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:03:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:03:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:03:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:03:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:03:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:03:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:03:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:03:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:03:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:03:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:03:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:03:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:03:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:03:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:03:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:03:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:03:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:03:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:03:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:03:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:03:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:03:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:03:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:03:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:03:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:03:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:03:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:03:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:03:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:03:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:03:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:03:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:03:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:03:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:03:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:03:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:03:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:03:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:03:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:03:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:03:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:03:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:03:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:03:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:03:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:03:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:03:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:03:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:03:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:03:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:03:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:03:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:03:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:03:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:03:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:03:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:03:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:03:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:03:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:03:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:03:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:03:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:03:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:03:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:03:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:03:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:03:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:03:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:03:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:03:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:03:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:03:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:03:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:03:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:03:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:03:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:03:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:03:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:03:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:03:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:03:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:03:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:03:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:03:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:03:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:03:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:03:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:03:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:03:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:03:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:03:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:03:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:03:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:03:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:03:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:03:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:03:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:03:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:03:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:03:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:03:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:03:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:03:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:03:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:03:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:03:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:03:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:03:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:03:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:03:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:03:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:03:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:03:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:03:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:03:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:03:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:03:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:03:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:03:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:03:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:03:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:03:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:03:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:03:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:03:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:03:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:03:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:03:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:03:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:03:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:03:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:03:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:03:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:03:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:03:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:03:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:03:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:03:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:04:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:04:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:04:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:04:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:04:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:04:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:04:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:04:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:04:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:04:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:04:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:04:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:04:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:04:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:04:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:04:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:04:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:04:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:04:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:04:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:04:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:04:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:04:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:04:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:04:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:04:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:04:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:04:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:04:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:04:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:04:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:04:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:04:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:04:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:04:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:04:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:04:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:04:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:04:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:04:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:04:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:04:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:04:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:04:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:04:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:04:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:04:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:04:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:04:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:04:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:04:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:04:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:04:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:04:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:04:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:04:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:04:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:04:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:04:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:04:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:04:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:04:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:04:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:04:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:04:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:04:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:04:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:04:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:04:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:04:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:04:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:04:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:04:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:04:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:04:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:04:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:04:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:04:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:04:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:04:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:04:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:04:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:04:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:04:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:04:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:04:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:04:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:04:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:04:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:04:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:04:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:04:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:04:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:04:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:04:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:04:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:04:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:04:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:04:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:04:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:04:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:04:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:04:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:04:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:04:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:04:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:04:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:04:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:04:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:04:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:04:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:04:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:04:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:04:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:04:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:04:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:04:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:04:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:04:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:04:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:04:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:04:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:04:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:04:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:04:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:04:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:04:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:04:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:04:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:04:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:04:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:04:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:04:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:04:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:04:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:04:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:04:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:04:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:04:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:04:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:04:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:04:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:04:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:04:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:04:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:04:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:04:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:04:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:04:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:04:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:04:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:04:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:04:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:04:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:04:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:04:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:04:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:04:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:04:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:04:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:04:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:04:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:04:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:04:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:04:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:04:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:04:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:04:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:04:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:04:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:04:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:04:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:04:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:04:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:04:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:04:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:04:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:04:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:04:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:04:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:04:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:04:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:04:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:04:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:04:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:04:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:04:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:04:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:04:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:04:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:04:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:04:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:04:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:04:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:04:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:04:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:04:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:04:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:04:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:04:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:04:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:04:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:04:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:04:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:04:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:04:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:04:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:04:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:04:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:04:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:04:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:04:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:04:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:04:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:04:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:04:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:05:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:05:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:05:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:05:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:05:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:05:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:05:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:05:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:05:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:05:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:05:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:05:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:05:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:05:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:05:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:05:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:05:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:05:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:05:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:05:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:05:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:05:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:05:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:05:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:05:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:05:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:05:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:05:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:05:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:05:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:05:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:05:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:05:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:05:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:05:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:05:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:05:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:05:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:05:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:05:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:05:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:05:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:05:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:05:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:05:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:05:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:05:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:05:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:05:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:05:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:05:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:05:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:05:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:05:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:05:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:05:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:05:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:05:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:05:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:05:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:05:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:05:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:05:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:05:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:05:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:05:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:05:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:05:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:05:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:05:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:05:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:05:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:05:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:05:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:05:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:05:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:05:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:05:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:05:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:05:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:05:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:05:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:05:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:05:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:05:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:05:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:05:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:05:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:05:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:05:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:05:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:05:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:05:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:05:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:05:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:05:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:05:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:05:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:05:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:05:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:05:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:05:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:05:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:05:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:05:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:05:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:05:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:05:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:07:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:07:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:07:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:07:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:07:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:07:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:07:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:07:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:07:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:07:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:07:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:07:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:07:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:07:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:07:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:07:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:07:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:07:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:07:35 rdma06 rsyslogd[1041]: imjournal: 5295 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 15:07:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:07:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:07:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:07:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:07:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:07:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:07:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:07:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:07:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:07:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:07:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:07:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:07:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:07:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:07:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:07:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:07:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:07:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:07:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:07:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:07:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:07:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:07:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:07:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:07:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:07:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:07:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:07:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:07:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:07:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:07:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:07:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:07:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:07:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:07:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:07:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:07:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:07:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:07:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:07:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:07:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:07:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:07:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:07:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:07:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:07:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:07:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:07:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:07:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:07:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:07:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:07:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:07:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:07:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:07:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:07:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:07:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:07:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:07:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:07:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:07:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:07:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:07:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:07:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:07:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:07:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:07:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:07:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:07:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:07:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:07:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:07:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:08:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:08:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:08:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:08:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:08:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:08:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:08:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:08:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:08:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:08:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:08:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:08:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:08:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:08:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:08:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:08:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:08:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:08:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:08:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:08:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:08:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:08:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:08:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:08:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:08:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:08:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:08:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:08:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:08:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:08:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:08:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:08:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:08:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:08:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:08:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:08:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:08:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:08:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:08:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:08:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:08:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:08:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:08:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:08:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:08:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:08:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:08:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:08:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:08:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:08:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:08:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:08:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:08:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:08:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:08:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:08:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:08:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:08:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:08:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:08:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:08:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:08:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:08:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:08:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:08:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:08:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:08:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:08:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:08:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:08:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:08:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:08:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:08:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:08:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:08:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:08:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:08:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:08:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:08:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:08:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:08:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:08:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:08:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:08:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:08:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:08:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:08:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:08:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:08:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:08:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:08:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:08:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:08:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:08:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:08:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:08:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:08:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:08:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:08:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:08:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:08:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:08:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:08:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:08:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:08:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:08:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:08:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:08:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:08:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:08:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:08:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:08:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:08:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:08:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:08:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:08:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:08:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:08:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:08:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:08:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:08:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:08:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:08:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:08:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:08:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:08:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:08:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:08:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:08:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:08:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:08:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:08:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:08:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:08:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:08:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:08:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:08:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:08:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:08:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:08:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:08:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:08:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:08:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:08:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:08:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:08:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:08:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:08:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:08:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:08:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:08:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:08:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:08:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:08:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:08:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:08:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:08:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:08:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:08:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:08:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:08:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:08:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:08:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:08:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:08:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:08:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:08:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:08:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:08:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:08:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:08:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:08:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:08:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:08:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:08:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:08:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:08:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:08:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:08:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:08:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:08:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:08:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:08:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:08:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:08:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:08:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:08:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:08:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:08:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:08:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:08:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:08:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:08:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:08:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:08:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:08:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:08:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:08:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:08:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:08:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:08:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:08:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:08:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:08:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:08:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:08:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:08:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:08:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:08:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:08:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:08:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:08:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:08:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:08:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:08:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:08:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:09:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:09:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:09:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:09:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:09:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:09:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:09:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:09:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:09:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:09:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:09:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:09:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:09:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:09:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:09:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:09:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:09:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:09:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:09:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:09:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:09:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:09:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:09:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:09:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:09:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:09:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:09:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:09:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:09:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:09:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:09:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:09:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:09:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:09:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:09:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:09:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:09:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:09:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:09:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:09:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:09:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:09:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:09:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:09:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:09:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:09:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:09:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:09:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:09:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:09:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:09:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:09:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:09:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:09:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:09:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:09:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:09:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:09:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:09:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:09:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:09:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:09:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:09:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:09:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:09:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:09:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:09:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:09:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:09:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:09:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:09:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:09:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:09:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:09:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:09:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:09:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:09:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:09:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:09:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:09:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:09:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:09:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:09:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:09:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:09:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:09:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:09:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:09:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:09:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:09:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:09:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:09:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:09:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:09:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:09:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:09:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:09:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:09:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:09:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:09:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:09:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:09:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:09:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:09:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:09:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:09:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:09:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:09:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:09:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:09:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:09:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:09:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:09:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:09:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:09:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:09:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:09:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:09:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:09:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:09:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:09:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:09:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:09:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:09:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:09:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:09:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:09:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:09:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:09:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:09:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:09:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:09:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:09:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:09:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:09:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:09:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:09:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:09:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:09:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:09:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:09:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:09:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:09:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:09:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:09:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:09:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:09:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:09:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:09:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:09:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:09:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:09:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:09:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:09:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:09:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:09:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:09:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:09:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:09:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:09:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:09:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:09:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:09:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:09:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:09:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:09:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:09:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:09:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:09:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:09:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:09:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:09:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:09:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:09:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:09:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:09:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:09:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:09:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:09:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:09:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:09:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:09:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:09:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:09:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:09:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:09:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:09:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:09:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:09:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:09:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:09:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:09:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:09:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:09:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:09:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:09:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:09:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:09:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:09:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:09:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:09:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:09:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:09:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:09:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:09:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:09:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:09:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:09:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:09:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:09:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:09:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:09:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:09:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:09:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:09:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:09:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:10:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:10:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:10:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:10:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:10:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:10:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:10:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:10:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:10:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:10:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:10:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:10:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:10:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:10:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:10:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:10:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:10:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:10:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:10:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:10:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:10:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:10:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:10:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:10:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:10:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:10:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:10:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:10:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:10:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:10:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:10:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:10:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:10:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:10:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:10:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:10:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:10:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:10:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:10:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:10:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:10:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:10:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:10:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:10:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:10:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:10:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:10:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:10:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:10:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:10:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:10:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:10:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:10:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:10:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:10:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:10:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:10:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:10:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:10:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:10:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:10:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:10:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:10:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:10:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:10:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:10:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:10:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:10:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:10:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:10:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:10:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:10:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:10:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:10:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:10:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:10:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:10:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:10:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:10:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:10:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:10:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:10:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:10:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:10:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:10:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:10:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:10:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:10:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:10:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:10:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:10:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:10:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:10:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:10:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:10:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:10:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:10:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:10:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:10:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:10:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:10:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:10:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:10:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:10:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:10:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:10:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:10:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:10:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:10:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:10:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:10:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:10:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:10:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:10:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:10:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:10:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:10:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:10:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:10:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:10:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:10:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:10:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:10:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:10:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:10:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:10:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:10:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:10:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:10:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:10:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:10:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:10:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:10:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:10:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:10:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:10:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:10:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:10:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:10:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:10:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:10:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:10:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:10:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:10:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:10:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:10:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:10:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:10:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:10:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:10:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:10:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:10:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:10:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:10:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:10:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:10:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:10:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:10:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:10:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:10:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:10:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:10:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:10:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:10:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:10:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:10:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:10:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:10:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:10:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:10:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:10:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:10:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:10:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:10:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:10:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:10:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:10:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:10:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:10:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:10:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:10:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:10:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:10:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:10:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:10:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:10:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:10:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:10:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:10:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:10:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:10:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:10:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:10:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:10:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:10:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:10:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:10:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:10:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:10:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:10:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:10:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:10:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:10:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:10:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:10:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:10:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:10:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:10:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:10:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:10:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:10:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:10:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:10:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:10:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:10:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:10:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:11:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:11:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:11:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:11:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:11:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:11:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:11:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:11:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:11:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:11:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:11:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:11:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:11:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:11:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:11:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:11:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:11:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:11:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:11:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:11:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:11:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:11:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:11:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:11:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:11:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:11:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:11:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:11:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:11:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:11:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:11:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:11:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:11:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:11:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:11:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:11:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:11:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:11:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:11:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:11:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:11:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:11:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:11:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:11:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:11:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:11:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:11:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:11:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:11:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:11:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:11:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:11:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:11:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:11:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:11:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:11:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:11:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:11:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:11:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:11:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:11:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:11:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:11:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:11:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:11:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:11:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:11:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:11:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:11:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:11:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:11:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:11:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:11:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:11:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:11:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:11:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:11:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:11:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:11:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:11:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:11:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:11:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:11:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:11:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:11:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:11:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:11:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:11:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:11:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:11:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:11:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:11:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:11:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:11:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:11:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:11:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:11:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:11:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:11:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:11:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:11:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:11:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:11:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:11:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:11:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:11:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:11:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:11:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:11:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:11:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:11:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:11:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:11:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:11:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:11:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:11:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:11:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:11:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:11:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:11:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:11:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:11:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:11:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:11:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:11:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:11:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:11:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:11:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:11:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:11:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:11:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:11:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:11:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:11:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:11:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:11:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:11:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:11:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:11:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:11:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:11:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:11:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:11:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:11:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:11:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:11:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:11:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:11:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:11:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:11:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:11:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:11:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:11:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:11:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:11:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:11:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:11:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:11:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:11:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:11:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:11:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:11:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:11:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:11:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:11:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:11:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:11:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:11:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:11:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:11:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:11:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:11:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:11:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:11:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:11:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:11:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:11:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:11:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:11:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:11:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:11:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:11:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:11:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:11:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:11:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:11:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:12:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:12:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:12:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:12:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:12:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:12:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:12:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:12:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:12:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:12:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:12:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:12:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:12:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:12:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:12:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:12:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:12:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:12:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:12:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:12:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:12:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:12:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:12:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:12:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:12:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:12:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:12:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:12:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:12:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:12:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:12:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:12:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:12:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:12:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:12:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:12:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:12:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:12:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:12:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:12:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:12:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:12:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:12:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:12:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:12:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:12:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:12:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:12:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:12:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:12:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:12:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:12:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:12:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:12:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:12:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:12:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:12:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:12:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:12:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:12:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:12:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:12:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:12:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:12:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:12:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:12:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:12:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:12:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:12:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:12:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:12:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:12:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:12:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:12:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:12:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:12:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:12:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:12:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:12:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:12:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:12:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:12:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:12:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:12:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:12:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:12:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:12:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:12:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:12:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:12:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:12:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:12:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:12:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:12:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:12:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:12:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:12:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:12:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:12:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:12:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:12:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:12:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:12:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:12:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:12:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:12:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:12:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:12:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:12:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:12:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:12:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:12:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:12:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:12:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:12:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:12:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:12:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:12:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:12:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:12:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:12:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:12:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:12:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:12:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:12:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:12:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:12:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:12:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:12:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:12:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:12:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:12:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:12:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:12:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:12:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:12:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:12:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:12:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:12:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:12:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:12:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:12:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:12:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:12:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:12:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:12:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:12:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:12:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:12:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:12:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:12:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:12:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:12:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:12:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:12:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:12:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:12:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:12:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:12:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:12:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:12:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:12:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:12:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:12:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:12:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:12:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:12:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:12:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:12:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:12:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:12:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:12:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:12:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:12:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:12:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:12:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:12:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:12:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:12:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:12:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:12:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:12:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:12:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:12:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:12:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:12:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:12:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:12:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:12:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:12:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:12:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:12:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:12:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:12:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:12:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:12:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:12:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:12:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:12:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:12:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:12:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:12:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:12:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:12:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:12:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:12:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:12:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:12:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:12:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:12:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:12:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:12:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:12:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:12:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:12:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:12:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:13:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:13:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:13:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:13:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:13:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:13:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:13:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:13:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:13:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:13:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:13:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:13:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:13:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:13:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:13:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:13:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:13:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:13:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:13:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:13:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:13:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:13:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:13:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:13:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:13:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:13:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:13:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:13:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:13:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:13:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:13:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:13:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:13:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:13:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:13:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:13:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:13:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:13:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:13:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:13:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:13:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:13:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:13:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:13:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:13:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:13:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:13:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:13:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:13:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:13:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:13:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:13:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:13:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:13:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:13:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:13:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:13:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:13:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:13:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:13:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:13:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:13:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:13:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:13:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:13:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:13:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:13:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:13:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:13:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:13:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:13:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:13:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:13:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:13:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:13:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:13:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:13:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:13:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:13:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:13:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:13:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:13:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:13:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:13:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:13:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:13:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:13:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:13:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:13:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:13:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:13:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:13:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:13:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:13:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:13:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:13:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:13:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:13:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:13:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:13:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:13:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:13:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:13:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:13:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:13:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:13:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:13:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:13:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:13:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:13:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:13:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:13:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:13:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:13:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:13:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:13:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:13:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:13:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:13:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:13:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:13:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:13:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:13:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:13:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:13:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:13:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:13:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:13:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:13:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:13:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:13:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:13:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:13:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:13:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:13:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:13:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:13:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:13:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:13:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:13:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:13:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:13:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:13:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:13:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:13:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:13:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:13:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:13:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:13:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:13:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:13:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:13:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:13:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:13:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:13:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:13:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:13:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:13:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:13:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:13:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:13:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:13:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:13:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:13:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:13:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:13:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:13:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:13:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:13:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:13:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:13:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:13:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:13:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:13:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:13:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:13:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:13:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:13:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:13:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:13:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:13:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:13:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:13:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:13:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:13:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:13:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:13:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:13:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:13:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:13:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:13:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:13:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:13:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:13:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:13:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:13:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:13:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:13:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:13:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:13:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:13:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:13:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:13:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:13:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:13:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:13:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:13:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:13:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:13:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:13:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:13:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:13:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:13:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:13:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:13:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:13:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:14:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:14:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:14:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:14:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:14:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:14:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:14:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:14:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:14:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:14:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:14:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:14:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:14:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:14:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:14:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:14:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:14:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:14:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:14:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:14:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:14:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:14:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:14:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:14:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:14:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:14:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:14:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:14:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:14:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:14:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:14:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:14:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:14:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:14:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:14:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:14:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:14:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:14:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:14:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:14:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:14:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:14:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:14:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:14:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:14:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:14:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:14:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:14:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:14:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:14:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:14:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:14:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:14:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:14:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:14:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:14:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:14:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:14:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:14:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:14:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:14:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:14:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:14:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:14:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:14:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:14:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:14:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:14:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:14:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:14:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:14:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:14:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:14:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:14:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:14:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:14:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:14:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:14:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:14:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:14:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:14:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:14:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:14:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:14:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:14:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:14:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:14:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:14:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:14:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:14:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:14:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:14:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:14:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:14:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:14:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:14:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:14:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:14:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:14:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:14:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:14:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:14:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:14:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:14:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:14:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:14:25 rdma06 systemd-journald[790]: Data hash table of /run/log/journal/95379d2f723847b68d9e976dee8967a9/system.journal has a fill level at 75.0 (163751 of 218332 items, 75497472 file size, 461 bytes per hash table item), suggesting rotation.
Apr 17 15:14:25 rdma06 systemd-journald[790]: /run/log/journal/95379d2f723847b68d9e976dee8967a9/system.journal: Journal header limits reached or header out-of-date, rotating.
Apr 17 15:14:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:14:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:14:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:14:25 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 17 15:14:25 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 17 15:14:25 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 17 15:14:26 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 17 15:14:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:14:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:14:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:14:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:14:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:14:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:14:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:14:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:14:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:14:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:14:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:14:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:14:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:14:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:14:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:14:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:14:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:14:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:14:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:14:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:14:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:14:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:14:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:14:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:14:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:14:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:14:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:14:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:14:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:14:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:14:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:14:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:14:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:14:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:14:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:14:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:14:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:14:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:14:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:14:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:14:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:14:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:14:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:14:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:14:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:14:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:14:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:14:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:14:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:14:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:14:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:14:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:14:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:14:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:14:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:14:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:14:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:14:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:14:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:14:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:14:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:14:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:14:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:14:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:14:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:14:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:14:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:14:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:14:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:14:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:14:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:14:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:14:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:14:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:14:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:14:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:14:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:14:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:14:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:14:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:14:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:14:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:14:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:14:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:14:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:14:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:14:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:14:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:14:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:14:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:14:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:14:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:14:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:14:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:14:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:14:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:14:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:14:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:14:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:14:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:14:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:14:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:14:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:14:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:14:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:14:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:14:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:14:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:15:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:15:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:15:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:15:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:15:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:15:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:15:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:15:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:15:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:15:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:15:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:15:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:15:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:15:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:15:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:15:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:15:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:15:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:15:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:15:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:15:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:15:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:15:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:15:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:15:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:15:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:15:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:15:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:15:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:15:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:15:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:15:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:15:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:15:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:15:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:15:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:15:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:15:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:15:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:15:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:15:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:15:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:15:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:15:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:15:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:15:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:15:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:15:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:15:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:15:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:15:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:15:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:15:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:15:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:15:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:15:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:15:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:15:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:15:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:15:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:15:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:15:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:15:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:15:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:15:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:15:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:15:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:15:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:15:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:15:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:15:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:15:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:15:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:15:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:15:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:15:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:15:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:15:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:15:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:15:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:15:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:15:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:15:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:15:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:15:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:15:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:15:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:15:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:15:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:15:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:15:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:15:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:15:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:15:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:15:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:15:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:15:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:15:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:15:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:15:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:15:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:15:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:15:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:15:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:15:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:15:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:15:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:15:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:15:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:15:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:15:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:15:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:15:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:15:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:15:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:15:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:15:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:15:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:15:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:15:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:15:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:15:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:15:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:15:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:15:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:15:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:17:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:17:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:17:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:17:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:17:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:17:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:17:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:17:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:17:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:17:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:17:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:17:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:17:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:17:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:17:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:17:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:17:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:17:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:17:36 rdma06 rsyslogd[1041]: imjournal: 5088 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 15:17:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:17:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:17:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:17:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:17:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:17:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:17:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:17:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:17:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:17:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:17:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:17:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:17:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:17:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:17:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:17:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:17:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:17:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:17:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:17:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:17:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:17:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:17:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:17:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:17:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:17:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:17:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:17:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:17:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:17:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:17:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:17:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:17:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:17:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:17:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:17:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:17:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:17:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:17:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:17:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:17:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:17:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:17:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:17:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:17:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:17:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:17:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:17:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:17:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:17:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:17:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:17:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:17:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:17:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:17:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:17:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:17:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:17:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:17:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:17:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:17:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:17:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:17:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:17:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:17:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:17:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:17:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:17:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:17:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:17:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:17:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:17:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:18:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:18:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:18:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:18:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:18:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:18:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:18:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:18:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:18:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:18:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:18:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:18:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:18:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:18:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:18:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:18:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:18:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:18:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:18:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:18:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:18:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:18:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:18:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:18:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:18:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:18:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:18:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:18:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:18:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:18:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:18:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:18:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:18:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:18:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:18:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:18:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:18:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:18:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:18:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:18:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:18:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:18:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:18:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:18:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:18:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:18:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:18:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:18:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:18:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:18:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:18:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:18:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:18:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:18:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:18:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:18:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:18:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:18:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:18:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:18:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:18:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:18:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:18:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:18:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:18:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:18:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:18:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:18:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:18:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:18:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:18:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:18:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:18:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:18:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:18:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:18:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:18:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:18:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:18:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:18:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:18:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:18:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:18:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:18:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:18:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:18:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:18:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:18:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:18:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:18:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:18:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:18:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:18:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:18:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:18:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:18:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:18:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:18:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:18:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:18:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:18:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:18:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:18:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:18:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:18:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:18:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:18:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:18:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:18:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:18:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:18:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:18:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:18:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:18:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:18:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:18:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:18:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:18:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:18:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:18:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:18:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:18:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:18:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:18:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:18:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:18:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:18:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:18:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:18:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:18:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:18:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:18:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:18:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:18:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:18:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:18:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:18:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:18:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:18:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:18:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:18:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:18:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:18:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:18:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:18:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:18:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:18:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:18:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:18:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:18:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:18:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:18:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:18:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:18:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:18:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:18:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:18:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:18:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:18:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:18:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:18:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:18:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:18:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:18:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:18:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:18:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:18:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:18:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:18:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:18:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:18:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:18:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:18:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:18:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:18:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:18:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:18:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:18:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:18:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:18:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:18:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:18:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:18:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:18:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:18:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:18:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:18:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:18:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:18:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:18:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:18:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:18:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:18:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:18:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:18:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:18:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:18:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:18:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:18:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:18:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:18:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:18:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:18:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:18:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:18:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:18:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:18:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:18:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:18:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:18:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:18:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:18:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:18:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:18:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:18:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:18:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:19:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:19:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:19:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:19:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:19:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:19:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:19:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:19:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:19:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:19:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:19:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:19:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:19:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:19:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:19:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:19:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:19:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:19:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:19:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:19:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:19:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:19:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:19:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:19:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:19:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:19:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:19:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:19:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:19:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:19:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:19:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:19:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:19:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:19:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:19:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:19:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:19:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:19:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:19:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:19:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:19:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:19:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:19:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:19:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:19:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:19:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:19:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:19:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:19:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:19:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:19:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:19:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:19:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:19:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:19:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:19:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:19:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:19:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:19:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:19:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:19:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:19:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:19:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:19:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:19:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:19:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:19:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:19:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:19:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:19:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:19:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:19:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:19:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:19:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:19:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:19:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:19:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:19:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:19:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:19:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:19:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:19:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:19:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:19:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:19:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:19:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:19:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:19:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:19:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:19:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:19:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:19:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:19:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:19:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:19:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:19:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:19:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:19:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:19:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:19:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:19:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:19:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:19:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:19:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:19:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:19:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:19:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:19:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:19:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:19:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:19:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:19:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:19:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:19:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:19:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:19:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:19:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:19:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:19:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:19:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:19:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:19:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:19:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:19:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:19:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:19:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:19:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:19:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:19:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:19:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:19:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:19:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:19:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:19:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:19:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:19:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:19:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:19:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:19:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:19:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:19:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:19:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:19:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:19:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:19:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:19:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:19:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:19:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:19:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:19:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:19:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:19:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:19:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:19:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:19:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:19:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:19:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:19:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:19:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:19:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:19:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:19:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:19:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:19:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:19:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:19:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:19:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:19:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:19:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:19:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:19:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:19:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:19:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:19:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:19:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:19:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:19:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:19:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:19:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:19:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:19:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:19:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:19:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:19:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:19:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:19:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:19:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:19:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:19:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:19:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:19:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:19:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:19:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:19:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:19:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:19:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:19:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:19:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:19:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:19:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:19:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:19:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:19:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:19:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:19:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:19:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:19:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:19:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:19:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:19:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:19:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:19:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:19:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:19:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:19:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:19:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:20:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:20:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:20:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:20:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:20:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:20:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:20:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:20:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:20:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:20:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:20:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:20:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:20:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:20:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:20:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:20:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:20:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:20:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:20:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:20:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:20:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:20:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:20:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:20:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:20:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:20:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:20:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:20:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:20:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:20:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:20:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:20:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:20:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:20:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:20:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:20:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:20:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:20:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:20:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:20:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:20:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:20:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:20:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:20:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:20:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:20:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:20:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:20:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:20:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:20:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:20:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:20:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:20:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:20:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:20:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:20:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:20:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:20:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:20:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:20:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:20:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:20:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:20:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:20:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:20:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:20:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:20:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:20:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:20:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:20:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:20:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:20:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:20:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:20:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:20:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:20:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:20:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:20:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:20:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:20:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:20:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:20:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:20:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:20:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:20:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:20:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:20:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:20:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:20:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:20:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:20:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:20:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:20:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:20:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:20:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:20:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:20:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:20:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:20:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:20:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:20:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:20:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:20:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:20:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:20:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:20:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:20:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:20:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:20:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:20:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:20:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:20:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:20:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:20:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:20:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:20:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:20:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:20:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:20:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:20:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:20:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:20:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:20:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:20:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:20:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:20:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:20:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:20:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:20:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:20:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:20:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:20:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:20:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:20:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:20:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:20:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:20:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:20:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:20:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:20:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:20:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:20:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:20:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:20:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:20:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:20:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:20:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:20:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:20:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:20:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:20:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:20:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:20:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:20:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:20:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:20:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:20:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:20:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:20:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:20:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:20:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:20:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:20:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:20:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:20:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:20:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:20:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:20:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:20:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:20:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:20:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:20:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:20:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:20:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:20:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:20:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:20:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:20:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:20:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:20:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:20:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:20:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:20:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:20:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:20:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:20:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:20:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:20:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:20:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:20:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:20:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:20:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:20:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:20:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:20:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:20:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:20:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:20:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:20:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:20:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:20:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:20:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:20:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:20:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:20:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:20:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:20:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:20:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:20:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:20:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:20:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:20:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:20:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:20:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:20:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:20:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:21:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:21:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:21:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:21:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:21:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:21:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:21:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:21:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:21:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:21:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:21:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:21:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:21:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:21:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:21:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:21:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:21:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:21:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:21:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:21:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:21:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:21:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:21:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:21:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:21:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:21:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:21:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:21:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:21:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:21:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:21:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:21:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:21:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:21:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:21:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:21:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:21:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:21:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:21:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:21:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:21:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:21:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:21:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:21:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:21:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:21:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:21:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:21:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:21:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:21:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:21:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:21:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:21:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:21:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:21:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:21:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:21:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:21:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:21:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:21:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:21:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:21:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:21:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:21:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:21:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:21:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:21:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:21:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:21:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:21:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:21:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:21:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:21:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:21:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:21:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:21:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:21:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:21:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:21:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:21:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:21:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:21:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:21:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:21:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:21:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:21:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:21:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:21:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:21:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:21:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:21:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:21:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:21:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:21:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:21:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:21:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:21:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:21:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:21:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:21:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:21:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:21:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:21:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:21:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:21:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:21:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:21:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:21:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:21:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:21:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:21:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:21:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:21:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:21:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:21:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:21:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:21:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:21:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:21:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:21:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:21:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:21:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:21:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:21:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:21:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:21:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:21:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:21:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:21:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:21:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:21:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:21:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:21:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:21:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:21:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:21:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:21:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:21:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:21:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:21:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:21:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:21:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:21:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:21:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:21:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:21:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:21:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:21:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:21:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:21:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:21:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:21:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:21:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:21:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:21:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:21:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:21:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:21:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:21:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:21:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:21:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:21:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:21:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:21:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:21:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:21:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:21:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:21:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:21:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:21:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:21:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:21:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:21:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:21:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:21:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:21:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:21:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:21:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:21:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:21:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:21:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:21:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:21:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:21:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:21:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:21:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:21:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:21:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:21:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:21:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:21:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:21:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:21:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:21:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:21:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:21:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:21:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:21:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:21:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:21:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:21:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:21:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:21:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:21:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:21:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:21:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:21:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:21:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:21:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:21:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:21:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:21:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:21:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:21:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:21:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:21:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:22:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:22:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:22:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:22:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:22:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:22:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:22:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:22:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:22:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:22:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:22:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:22:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:22:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:22:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:22:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:22:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:22:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:22:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:22:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:22:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:22:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:22:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:22:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:22:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:22:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:22:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:22:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:22:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:22:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:22:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:22:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:22:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:22:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:22:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:22:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:22:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:22:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:22:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:22:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:22:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:22:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:22:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:22:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:22:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:22:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:22:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:22:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:22:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:22:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:22:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:22:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:22:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:22:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:22:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:22:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:22:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:22:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:22:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:22:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:22:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:22:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:22:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:22:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:22:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:22:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:22:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:22:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:22:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:22:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:22:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:22:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:22:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:22:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:22:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:22:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:22:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:22:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:22:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:22:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:22:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:22:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:22:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:22:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:22:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:22:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:22:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:22:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:22:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:22:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:22:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:22:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:22:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:22:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:22:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:22:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:22:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:22:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:22:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:22:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:22:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:22:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:22:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:22:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:22:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:22:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:22:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:22:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:22:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:22:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:22:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:22:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:22:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:22:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:22:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:22:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:22:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:22:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:22:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:22:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:22:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:22:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:22:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:22:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:22:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:22:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:22:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:22:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:22:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:22:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:22:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:22:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:22:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:22:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:22:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:22:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:22:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:22:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:22:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:22:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:22:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:22:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:22:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:22:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:22:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:22:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:22:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:22:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:22:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:22:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:22:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:22:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:22:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:22:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:22:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:22:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:22:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:22:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:22:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:22:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:22:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:22:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:22:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:22:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:22:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:22:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:22:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:22:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:22:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:22:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:22:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:22:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:22:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:22:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:22:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:22:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:22:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:22:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:22:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:22:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:22:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:22:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:22:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:22:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:22:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:22:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:22:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:22:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:22:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:22:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:22:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:22:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:22:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:22:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:22:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:22:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:22:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:22:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:22:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:22:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:22:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:22:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:22:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:22:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:22:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:22:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:22:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:22:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:22:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:22:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:22:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:22:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:22:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:22:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:22:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:22:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:22:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:23:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:23:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:23:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:23:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:23:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:23:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:23:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:23:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:23:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:23:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:23:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:23:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:23:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:23:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:23:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:23:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:23:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:23:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:23:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:23:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:23:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:23:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:23:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:23:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:23:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:23:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:23:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:23:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:23:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:23:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:23:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:23:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:23:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:23:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:23:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:23:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:23:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:23:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:23:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:23:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:23:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:23:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:23:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:23:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:23:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:23:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:23:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:23:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:23:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:23:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:23:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:23:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:23:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:23:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:23:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:23:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:23:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:23:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:23:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:23:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:23:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:23:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:23:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:23:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:23:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:23:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:23:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:23:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:23:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:23:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:23:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:23:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:23:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:23:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:23:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:23:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:23:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:23:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:23:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:23:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:23:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:23:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:23:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:23:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:23:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:23:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:23:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:23:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:23:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:23:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:23:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:23:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:23:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:23:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:23:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:23:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:23:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:23:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:23:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:23:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:23:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:23:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:23:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:23:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:23:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:23:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:23:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:23:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:23:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:23:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:23:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:23:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:23:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:23:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:23:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:23:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:23:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:23:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:23:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:23:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:23:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:23:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:23:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:23:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:23:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:23:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:23:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:23:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:23:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:23:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:23:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:23:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:23:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:23:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:23:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:23:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:23:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:23:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:23:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:23:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:23:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:23:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:23:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:23:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:23:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:23:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:23:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:23:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:23:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:23:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:23:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:23:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:23:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:23:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:23:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:23:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:23:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:23:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:23:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:23:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:23:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:23:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:23:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:23:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:23:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:23:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:23:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:23:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:23:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:23:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:23:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:23:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:23:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:23:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:23:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:23:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:23:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:23:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:23:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:23:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:23:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:23:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:23:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:23:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:23:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:23:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:23:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:23:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:23:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:23:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:23:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:23:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:23:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:23:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:23:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:23:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:23:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:23:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:23:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:23:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:23:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:23:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:23:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:23:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:23:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:23:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:23:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:23:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:23:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:23:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:23:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:23:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:23:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:23:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:23:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:23:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:24:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:24:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:24:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:24:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:24:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:24:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:24:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:24:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:24:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:24:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:24:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:24:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:24:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:24:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:24:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:24:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:24:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:24:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:24:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:24:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:24:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:24:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:24:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:24:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:24:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:24:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:24:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:24:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:24:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:24:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:24:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:24:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:24:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:24:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:24:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:24:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:24:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:24:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:24:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:24:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:24:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:24:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:24:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:24:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:24:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:24:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:24:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:24:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:24:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:24:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:24:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:24:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:24:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:24:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:24:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:24:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:24:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:24:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:24:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:24:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:24:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:24:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:24:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:24:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:24:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:24:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:24:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:24:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:24:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:24:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:24:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:24:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:24:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:24:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:24:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:24:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:24:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:24:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:24:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:24:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:24:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:24:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:24:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:24:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:24:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:24:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:24:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:24:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:24:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:24:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:24:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:24:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:24:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:24:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:24:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:24:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:24:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:24:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:24:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:24:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:24:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:24:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:24:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:24:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:24:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:24:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:24:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:24:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:24:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:24:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:24:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:24:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:24:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:24:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:24:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:24:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:24:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:24:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:24:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:24:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:24:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:24:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:24:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:24:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:24:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:24:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:24:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:24:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:24:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:24:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:24:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:24:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:24:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:24:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:24:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:24:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:24:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:24:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:24:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:24:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:24:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:24:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:24:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:24:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:24:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:24:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:24:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:24:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:24:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:24:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:24:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:24:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:24:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:24:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:24:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:24:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:24:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:24:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:24:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:24:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:24:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:24:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:24:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:24:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:24:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:24:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:24:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:24:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:24:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:24:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:24:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:24:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:24:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:24:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:24:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:24:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:24:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:24:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:24:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:24:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:24:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:24:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:24:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:24:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:24:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:24:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:24:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:24:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:24:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:24:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:24:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:24:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:24:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:24:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:24:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:24:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:24:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:24:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:24:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:24:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:24:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:24:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:24:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:24:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:24:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:24:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:24:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:24:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:24:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:24:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:24:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:24:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:24:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:24:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:24:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:24:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:25:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:25:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:25:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:25:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:25:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:25:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:25:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:25:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:25:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:25:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:25:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:25:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:25:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:25:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:25:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:25:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:25:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:25:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:25:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:25:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:25:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:25:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:25:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:25:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:25:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:25:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:25:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:25:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:25:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:25:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:25:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:25:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:25:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:25:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:25:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:25:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:25:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:25:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:25:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:25:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:25:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:25:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:25:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:25:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:25:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:25:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:25:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:25:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:25:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:25:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:25:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:25:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:25:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:25:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:25:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:25:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:25:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:25:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:25:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:25:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:25:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:25:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:25:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:25:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:25:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:25:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:25:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:25:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:25:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:25:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:25:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:25:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:25:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:25:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:25:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:25:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:25:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:25:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:25:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:25:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:25:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:25:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:25:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:25:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:25:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:25:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:25:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:25:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:25:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:25:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:25:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:25:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:25:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:25:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:25:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:25:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:25:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:25:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:25:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:25:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:25:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:25:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:25:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:25:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:25:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:25:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:25:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:25:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:25:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:25:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:25:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:25:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:25:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:25:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:25:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:25:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:25:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:25:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:25:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:25:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:25:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:25:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:25:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:25:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:25:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:25:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:27:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:27:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:27:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:27:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:27:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:27:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:27:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:27:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:27:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:27:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:27:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:27:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:27:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:27:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:27:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:27:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:27:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:27:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:27:41 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 15:27:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:27:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:27:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:27:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:27:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:27:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:27:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:27:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:27:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:27:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:27:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:27:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:27:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:27:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:27:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:27:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:27:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:27:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:27:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:27:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:27:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:27:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:27:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:27:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:27:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:27:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:27:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:27:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:27:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:27:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:27:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:27:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:27:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:27:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:27:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:27:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:27:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:27:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:27:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:27:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:27:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:27:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:27:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:27:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:27:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:27:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:27:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:27:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:27:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:27:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:27:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:27:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:27:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:27:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:28:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:28:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:28:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:28:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:28:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:28:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:28:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:28:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:28:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:28:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:28:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:28:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:28:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:28:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:28:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:28:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:28:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:28:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:28:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:28:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:28:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:28:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:28:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:28:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:28:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:28:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:28:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:28:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:28:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:28:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:28:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:28:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:28:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:28:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:28:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:28:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:28:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:28:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:28:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:28:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:28:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:28:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:28:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:28:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:28:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:28:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:28:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:28:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:28:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:28:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:28:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:28:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:28:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:28:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:28:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:28:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:28:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:28:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:28:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:28:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:28:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:28:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:28:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:28:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:28:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:28:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:28:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:28:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:28:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:28:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:28:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:28:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:28:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:28:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:28:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:28:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:28:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:28:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:28:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:28:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:28:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:28:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:28:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:28:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:28:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:28:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:28:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:28:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:28:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:28:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:28:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:28:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:28:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:28:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:28:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:28:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:28:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:28:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:28:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:28:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:28:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:28:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:28:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:28:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:28:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:28:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:28:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:28:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:28:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:28:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:28:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:28:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:28:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:28:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:28:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:28:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:28:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:28:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:28:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:28:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:28:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:28:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:28:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:28:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:28:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:28:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:28:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:28:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:28:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:28:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:28:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:28:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:28:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:28:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:28:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:28:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:28:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:28:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:28:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:28:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:28:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:28:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:28:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:28:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:28:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:28:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:28:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:28:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:28:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:28:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:28:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:28:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:28:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:28:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:28:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:28:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:28:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:28:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:28:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:28:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:28:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:28:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:28:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:28:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:28:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:28:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:28:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:28:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:28:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:28:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:28:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:28:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:28:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:28:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:28:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:28:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:28:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:28:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:28:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:28:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:28:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:28:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:28:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:28:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:28:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:28:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:28:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:28:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:28:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:28:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:28:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:28:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:28:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:28:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:28:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:28:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:28:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:28:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:28:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:28:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:28:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:28:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:28:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:28:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:28:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:28:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:28:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:28:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:28:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:28:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:28:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:28:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:28:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:28:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:28:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:28:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:29:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:29:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:29:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:29:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:29:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:29:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:29:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:29:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:29:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:29:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:29:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:29:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:29:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:29:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:29:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:29:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:29:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:29:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:29:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:29:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:29:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:29:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:29:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:29:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:29:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:29:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:29:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:29:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:29:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:29:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:29:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:29:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:29:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:29:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:29:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:29:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:29:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:29:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:29:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:29:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:29:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:29:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:29:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:29:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:29:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:29:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:29:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:29:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:29:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:29:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:29:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:29:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:29:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:29:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:29:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:29:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:29:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:29:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:29:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:29:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:29:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:29:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:29:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:29:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:29:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:29:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:29:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:29:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:29:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:29:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:29:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:29:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:29:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:29:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:29:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:29:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:29:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:29:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:29:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:29:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:29:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:29:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:29:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:29:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:29:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:29:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:29:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:29:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:29:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:29:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:29:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:29:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:29:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:29:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:29:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:29:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:29:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:29:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:29:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:29:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:29:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:29:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:29:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:29:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:29:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:29:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:29:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:29:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:29:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:29:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:29:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:29:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:29:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:29:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:29:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:29:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:29:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:29:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:29:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:29:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:29:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:29:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:29:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:29:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:29:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:29:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:29:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:29:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:29:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:29:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:29:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:29:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:29:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:29:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:29:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:29:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:29:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:29:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:29:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:29:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:29:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:29:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:29:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:29:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:29:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:29:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:29:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:29:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:29:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:29:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:29:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:29:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:29:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:29:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:29:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:29:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:29:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:29:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:29:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:29:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:29:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:29:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:29:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:29:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:29:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:29:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:29:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:29:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:29:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:29:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:29:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:29:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:29:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:29:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:29:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:29:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:29:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:29:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:29:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:29:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:29:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:29:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:29:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:29:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:29:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:29:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:29:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:29:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:29:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:29:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:29:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:29:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:29:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:29:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:29:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:29:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:29:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:29:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:29:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:29:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:29:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:29:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:29:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:29:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:29:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:29:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:29:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:29:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:29:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:29:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:29:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:29:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:29:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:29:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:29:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:29:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:30:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:30:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:30:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:30:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:30:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:30:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:30:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:30:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:30:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:30:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:30:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:30:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:30:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:30:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:30:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:30:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:30:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:30:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:30:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:30:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:30:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:30:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:30:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:30:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:30:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:30:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:30:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:30:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:30:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:30:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:30:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:30:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:30:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:30:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:30:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:30:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:30:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:30:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:30:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:30:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:30:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:30:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:30:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:30:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:30:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:30:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:30:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:30:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:30:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:30:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:30:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:30:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:30:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:30:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:30:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:30:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:30:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:30:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:30:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:30:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:30:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:30:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:30:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:30:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:30:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:30:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:30:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:30:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:30:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:30:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:30:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:30:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:30:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:30:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:30:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:30:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:30:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:30:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:30:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:30:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:30:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:30:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:30:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:30:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:30:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:30:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:30:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:30:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:30:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:30:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:30:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:30:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:30:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:30:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:30:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:30:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:30:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:30:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:30:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:30:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:30:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:30:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:30:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:30:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:30:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:30:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:30:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:30:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:30:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:30:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:30:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:30:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:30:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:30:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:30:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:30:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:30:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:30:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:30:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:30:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:30:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:30:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:30:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:30:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:30:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:30:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:30:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:30:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:30:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:30:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:30:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:30:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:30:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:30:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:30:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:30:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:30:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:30:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:30:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:30:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:30:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:30:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:30:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:30:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:30:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:30:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:30:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:30:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:30:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:30:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:30:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:30:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:30:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:30:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:30:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:30:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:30:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:30:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:30:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:30:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:30:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:30:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:30:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:30:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:30:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:30:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:30:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:30:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:30:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:30:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:30:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:30:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:30:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:30:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:30:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:30:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:30:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:30:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:30:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:30:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:30:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:30:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:30:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:30:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:30:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:30:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:30:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:30:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:30:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:30:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:30:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:30:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:30:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:30:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:30:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:30:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:30:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:30:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:30:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:30:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:30:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:30:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:30:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:30:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:30:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:30:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:30:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:30:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:30:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:30:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:30:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:30:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:30:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:30:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:30:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:30:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:31:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:31:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:31:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:31:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:31:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:31:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:31:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:31:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:31:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:31:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:31:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:31:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:31:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:31:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:31:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:31:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:31:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:31:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:31:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:31:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:31:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:31:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:31:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:31:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:31:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:31:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:31:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:31:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:31:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:31:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:31:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:31:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:31:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:31:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:31:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:31:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:31:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:31:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:31:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:31:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:31:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:31:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:31:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:31:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:31:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:31:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:31:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:31:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:31:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:31:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:31:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:31:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:31:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:31:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:31:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:31:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:31:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:31:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:31:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:31:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:31:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:31:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:31:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:31:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:31:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:31:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:31:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:31:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:31:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:31:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:31:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:31:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:31:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:31:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:31:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:31:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:31:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:31:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:31:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:31:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:31:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:31:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:31:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:31:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:31:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:31:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:31:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:31:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:31:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:31:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:31:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:31:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:31:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:31:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:31:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:31:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:31:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:31:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:31:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:31:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:31:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:31:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:31:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:31:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:31:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:31:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:31:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:31:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:31:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:31:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:31:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:31:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:31:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:31:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:31:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:31:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:31:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:31:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:31:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:31:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:31:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:31:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:31:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:31:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:31:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:31:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:31:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:31:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:31:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:31:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:31:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:31:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:31:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:31:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:31:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:31:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:31:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:31:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:31:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:31:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:31:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:31:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:31:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:31:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:31:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:31:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:31:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:31:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:31:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:31:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:31:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:31:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:31:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:31:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:31:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:31:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:31:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:31:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:31:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:31:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:31:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:31:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:31:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:31:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:31:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:31:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:32:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:32:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:32:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:32:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:32:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:32:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:32:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:32:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:32:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:32:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:32:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:32:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:32:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:32:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:32:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:32:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:32:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:32:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:32:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:32:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:32:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:32:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:32:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:32:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:32:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:32:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:32:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:32:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:32:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:32:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:32:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:32:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:32:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:32:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:32:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:32:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:32:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:32:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:32:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:32:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:32:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:32:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:32:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:32:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:32:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:32:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:32:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:32:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:32:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:32:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:32:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:32:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:32:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:32:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:32:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:32:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:32:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:32:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:32:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:32:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:32:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:32:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:32:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:32:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:32:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:32:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:32:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:32:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:32:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:32:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:32:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:32:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:32:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:32:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:32:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:32:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:32:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:32:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:32:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:32:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:32:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:32:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:32:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:32:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:32:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:32:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:32:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:32:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:32:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:32:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:32:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:32:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:32:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:32:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:32:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:32:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:32:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:32:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:32:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:32:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:32:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:32:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:32:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:32:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:32:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:32:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:32:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:32:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:32:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:32:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:32:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:32:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:32:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:32:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:32:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:32:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:32:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:32:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:32:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:32:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:32:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:32:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:32:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:32:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:32:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:32:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:32:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:32:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:32:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:32:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:32:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:32:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:32:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:32:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:32:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:32:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:32:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:32:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:32:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:32:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:32:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:32:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:32:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:32:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:32:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:32:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:32:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:32:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:32:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:32:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:32:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:32:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:32:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:32:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:32:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:32:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:32:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:32:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:32:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:32:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:32:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:32:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:32:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:32:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:32:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:32:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:32:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:32:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:32:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:32:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:32:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:32:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:32:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:32:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:32:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:32:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:32:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:32:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:32:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:32:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:32:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:32:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:32:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:32:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:32:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:32:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:32:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:32:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:32:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:32:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:32:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:32:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:32:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:32:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:32:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:32:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:32:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:32:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:32:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:32:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:32:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:32:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:32:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:32:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:32:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:32:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:32:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:32:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:32:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:32:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:32:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:32:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:32:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:32:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:32:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:32:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:33:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:33:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:33:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:33:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:33:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:33:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:33:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:33:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:33:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:33:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:33:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:33:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:33:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:33:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:33:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:33:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:33:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:33:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:33:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:33:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:33:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:33:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:33:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:33:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:33:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:33:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:33:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:33:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:33:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:33:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:33:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:33:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:33:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:33:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:33:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:33:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:33:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:33:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:33:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:33:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:33:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:33:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:33:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:33:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:33:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:33:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:33:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:33:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:33:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:33:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:33:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:33:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:33:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:33:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:33:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:33:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:33:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:33:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:33:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:33:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:33:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:33:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:33:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:33:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:33:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:33:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:33:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:33:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:33:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:33:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:33:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:33:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:33:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:33:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:33:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:33:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:33:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:33:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:33:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:33:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:33:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:33:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:33:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:33:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:33:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:33:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:33:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:33:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:33:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:33:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:33:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:33:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:33:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:33:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:33:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:33:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:33:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:33:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:33:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:33:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:33:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:33:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:33:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:33:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:33:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:33:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:33:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:33:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:33:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:33:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:33:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:33:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:33:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:33:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:33:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:33:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:33:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:33:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:33:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:33:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:33:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:33:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:33:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:33:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:33:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:33:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:33:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:33:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:33:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:33:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:33:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:33:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:33:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:33:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:33:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:33:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:33:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:33:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:33:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:33:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:33:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:33:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:33:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:33:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:33:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:33:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:33:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:33:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:33:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:33:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:33:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:33:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:33:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:33:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:33:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:33:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:33:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:33:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:33:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:33:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:33:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:33:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:33:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:33:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:33:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:33:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:33:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:33:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:33:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:33:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:33:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:33:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:33:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:33:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:33:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:33:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:33:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:33:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:33:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:33:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:33:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:33:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:33:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:33:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:33:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:33:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:33:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:33:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:33:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:33:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:33:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:33:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:33:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:33:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:33:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:33:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:33:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:33:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:33:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:33:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:33:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:33:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:33:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:33:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:33:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:33:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:33:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:33:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:33:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:33:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:33:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:33:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:33:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:33:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:33:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:33:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:34:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:34:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:34:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:34:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:34:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:34:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:34:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:34:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:34:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:34:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:34:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:34:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:34:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:34:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:34:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:34:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:34:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:34:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:34:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:34:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:34:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:34:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:34:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:34:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:34:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:34:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:34:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:34:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:34:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:34:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:34:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:34:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:34:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:34:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:34:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:34:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:34:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:34:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:34:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:34:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:34:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:34:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:34:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:34:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:34:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:34:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:34:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:34:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:34:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:34:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:34:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:34:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:34:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:34:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:34:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:34:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:34:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:34:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:34:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:34:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:34:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:34:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:34:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:34:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:34:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:34:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:34:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:34:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:34:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:34:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:34:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:34:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:34:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:34:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:34:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:34:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:34:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:34:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:34:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:34:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:34:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:34:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:34:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:34:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:34:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:34:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:34:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:34:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:34:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:34:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:34:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:34:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:34:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:34:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:34:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:34:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:34:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:34:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:34:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:34:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:34:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:34:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:34:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:34:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:34:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:34:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:34:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:34:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:34:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:34:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:34:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:34:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:34:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:34:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:34:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:34:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:34:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:34:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:34:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:34:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:34:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:34:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:34:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:34:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:34:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:34:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:34:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:34:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:34:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:34:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:34:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:34:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:34:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:34:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:34:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:34:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:34:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:34:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:34:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:34:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:34:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:34:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:34:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:34:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:34:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:34:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:34:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:34:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:34:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:34:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:34:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:34:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:34:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:34:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:34:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:34:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:34:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:34:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:34:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:34:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:34:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:34:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:34:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:34:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:34:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:34:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:34:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:34:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:34:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:34:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:34:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:34:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:34:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:34:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:34:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:34:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:34:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:34:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:34:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:34:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:34:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:34:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:34:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:34:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:34:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:34:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:34:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:34:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:34:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:34:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:34:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:34:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:34:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:34:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:34:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:34:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:34:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:34:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:34:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:34:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:34:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:34:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:34:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:34:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:34:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:34:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:34:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:34:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:34:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:34:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:34:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:34:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:34:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:34:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:34:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:34:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:35:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:35:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:35:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:35:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:35:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:35:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:35:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:35:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:35:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:35:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:35:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:35:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:35:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:35:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:35:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:35:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:35:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:35:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:35:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:35:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:35:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:35:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:35:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:35:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:35:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:35:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:35:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:35:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:35:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:35:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:35:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:35:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:35:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:35:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:35:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:35:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:35:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:35:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:35:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:35:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:35:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:35:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:35:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:35:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:35:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:35:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:35:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:35:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:35:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:35:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:35:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:35:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:35:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:35:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:35:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:35:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:35:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:35:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:35:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:35:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:35:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:35:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:35:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:35:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:35:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:35:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:35:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:35:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:35:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:35:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:35:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:35:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:35:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:35:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:35:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:35:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:35:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:35:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:35:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:35:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:35:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:35:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:35:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:35:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:35:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:35:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:35:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:35:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:35:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:35:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:35:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:35:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:35:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:35:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:35:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:35:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:35:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:35:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:35:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:35:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:35:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:35:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:35:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:35:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:35:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:35:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:35:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:35:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:35:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:35:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:35:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:35:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:35:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:35:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:35:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:35:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:35:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:35:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:35:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:35:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:35:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:35:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:35:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:35:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:35:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:35:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:35:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:35:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:35:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:35:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:35:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:35:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:35:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:35:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:35:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:35:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:35:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:35:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:35:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:35:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:35:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:35:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:35:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:35:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:37:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:37:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:37:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:37:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:37:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:37:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:37:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:37:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:37:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:37:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:37:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:37:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:37:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:37:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:37:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:37:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:37:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:37:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:37:46 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 15:37:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:37:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:37:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:37:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:37:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:37:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:37:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:37:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:37:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:37:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:37:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:37:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:37:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:37:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:37:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:37:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:37:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:37:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:37:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:37:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:37:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:37:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:37:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:37:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:37:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:37:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:37:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:37:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:37:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:37:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:37:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:37:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:37:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:37:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:37:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:37:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:38:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:38:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:38:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:38:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:38:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:38:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:38:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:38:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:38:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:38:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:38:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:38:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:38:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:38:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:38:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:38:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:38:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:38:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:38:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:38:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:38:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:38:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:38:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:38:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:38:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:38:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:38:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:38:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:38:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:38:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:38:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:38:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:38:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:38:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:38:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:38:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:38:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:38:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:38:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:38:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:38:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:38:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:38:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:38:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:38:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:38:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:38:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:38:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:38:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:38:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:38:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:38:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:38:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:38:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:38:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:38:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:38:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:38:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:38:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:38:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:38:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:38:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:38:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:38:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:38:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:38:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:38:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:38:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:38:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:38:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:38:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:38:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:38:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:38:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:38:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:38:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:38:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:38:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:38:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:38:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:38:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:38:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:38:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:38:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:38:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:38:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:38:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:38:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:38:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:38:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:38:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:38:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:38:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:38:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:38:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:38:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:38:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:38:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:38:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:38:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:38:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:38:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:38:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:38:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:38:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:38:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:38:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:38:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:38:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:38:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:38:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:38:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:38:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:38:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:38:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:38:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:38:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:38:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:38:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:38:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:38:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:38:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:38:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:38:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:38:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:38:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:38:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:38:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:38:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:38:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:38:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:38:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:38:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:38:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:38:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:38:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:38:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:38:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:38:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:38:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:38:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:38:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:38:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:38:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:38:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:38:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:38:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:38:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:38:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:38:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:38:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:38:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:38:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:38:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:38:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:38:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:38:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:38:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:38:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:38:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:38:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:38:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:38:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:38:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:38:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:38:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:38:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:38:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:38:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:38:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:38:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:38:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:38:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:38:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:38:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:38:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:38:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:38:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:38:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:38:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:38:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:38:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:38:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:38:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:38:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:38:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:38:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:38:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:38:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:38:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:38:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:38:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:38:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:38:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:38:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:38:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:38:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:38:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:38:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:38:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:38:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:38:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:38:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:38:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:38:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:38:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:38:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:38:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:38:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:38:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:38:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:38:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:38:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:38:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:38:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:38:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:39:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:39:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:39:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:39:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:39:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:39:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:39:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:39:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:39:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:39:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:39:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:39:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:39:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:39:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:39:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:39:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:39:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:39:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:39:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:39:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:39:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:39:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:39:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:39:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:39:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:39:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:39:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:39:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:39:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:39:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:39:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:39:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:39:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:39:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:39:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:39:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:39:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:39:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:39:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:39:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:39:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:39:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:39:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:39:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:39:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:39:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:39:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:39:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:39:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:39:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:39:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:39:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:39:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:39:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:39:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:39:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:39:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:39:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:39:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:39:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:39:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:39:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:39:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:39:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:39:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:39:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:39:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:39:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:39:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:39:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:39:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:39:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:39:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:39:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:39:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:39:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:39:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:39:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:39:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:39:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:39:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:39:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:39:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:39:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:39:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:39:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:39:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:39:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:39:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:39:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:39:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:39:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:39:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:39:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:39:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:39:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:39:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:39:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:39:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:39:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:39:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:39:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:39:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:39:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:39:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:39:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:39:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:39:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:39:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:39:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:39:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:39:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:39:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:39:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:39:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:39:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:39:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:39:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:39:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:39:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:39:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:39:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:39:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:39:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:39:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:39:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:39:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:39:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:39:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:39:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:39:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:39:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:39:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:39:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:39:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:39:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:39:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:39:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:39:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:39:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:39:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:39:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:39:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:39:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:39:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:39:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:39:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:39:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:39:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:39:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:39:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:39:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:39:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:39:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:39:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:39:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:39:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:39:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:39:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:39:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:39:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:39:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:39:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:39:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:39:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:39:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:39:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:39:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:39:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:39:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:39:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:39:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:39:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:39:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:39:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:39:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:39:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:39:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:39:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:39:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:39:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:39:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:39:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:39:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:39:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:39:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:39:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:39:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:39:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:39:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:39:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:39:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:39:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:39:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:39:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:39:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:39:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:39:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:39:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:39:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:39:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:39:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:39:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:39:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:39:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:39:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:39:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:39:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:39:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:39:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:39:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:39:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:39:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:39:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:39:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:39:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:40:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:40:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:40:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:40:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:40:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:40:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:40:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:40:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:40:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:40:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:40:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:40:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:40:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:40:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:40:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:40:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:40:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:40:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:40:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:40:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:40:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:40:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:40:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:40:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:40:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:40:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:40:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:40:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:40:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:40:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:40:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:40:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:40:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:40:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:40:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:40:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:40:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:40:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:40:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:40:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:40:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:40:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:40:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:40:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:40:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:40:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:40:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:40:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:40:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:40:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:40:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:40:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:40:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:40:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:40:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:40:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:40:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:40:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:40:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:40:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:40:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:40:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:40:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:40:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:40:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:40:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:40:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:40:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:40:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:40:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:40:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:40:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:40:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:40:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:40:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:40:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:40:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:40:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:40:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:40:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:40:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:40:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:40:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:40:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:40:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:40:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:40:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:40:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:40:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:40:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:40:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:40:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:40:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:40:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:40:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:40:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:40:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:40:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:40:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:40:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:40:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:40:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:40:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:40:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:40:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:40:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:40:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:40:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:40:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:40:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:40:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:40:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:40:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:40:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:40:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:40:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:40:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:40:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:40:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:40:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:40:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:40:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:40:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:40:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:40:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:40:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:40:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:40:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:40:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:40:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:40:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:40:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:40:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:40:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:40:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:40:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:40:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:40:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:40:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:40:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:40:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:40:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:40:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:40:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:40:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:40:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:40:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:40:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:40:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:40:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:40:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:40:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:40:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:40:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:40:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:40:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:40:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:40:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:40:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:40:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:40:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:40:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:40:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:40:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:40:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:40:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:40:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:40:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:40:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:40:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:40:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:40:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:40:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:40:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:40:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:40:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:40:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:40:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:40:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:40:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:40:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:40:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:40:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:40:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:40:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:40:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:40:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:40:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:40:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:40:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:40:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:40:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:40:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:40:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:40:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:40:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:40:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:40:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:40:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:40:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:40:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:40:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:40:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:40:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:40:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:40:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:40:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:40:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:40:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:40:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:40:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:40:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:40:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:40:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:40:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:40:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:41:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:41:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:41:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:41:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:41:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:41:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:41:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:41:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:41:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:41:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:41:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:41:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:41:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:41:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:41:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:41:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:41:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:41:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:41:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:41:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:41:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:41:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:41:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:41:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:41:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:41:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:41:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:41:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:41:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:41:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:41:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:41:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:41:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:41:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:41:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:41:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:41:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:41:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:41:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:41:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:41:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:41:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:41:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:41:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:41:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:41:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:41:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:41:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:41:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:41:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:41:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:41:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:41:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:41:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:41:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:41:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:41:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:41:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:41:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:41:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:41:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:41:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:41:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:41:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:41:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:41:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:41:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:41:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:41:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:41:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:41:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:41:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:41:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:41:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:41:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:41:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:41:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:41:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:41:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:41:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:41:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:41:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:41:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:41:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:41:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:41:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:41:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:41:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:41:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:41:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:41:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:41:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:41:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:41:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:41:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:41:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:41:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:41:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:41:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:41:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:41:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:41:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:41:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:41:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:41:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:41:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:41:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:41:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:41:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:41:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:41:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:41:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:41:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:41:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:41:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:41:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:41:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:41:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:41:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:41:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:41:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:41:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:41:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:41:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:41:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:41:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:41:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:41:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:41:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:41:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:41:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:41:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:41:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:41:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:41:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:41:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:41:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:41:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:41:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:41:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:41:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:41:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:41:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:41:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:41:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:41:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:41:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:41:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:41:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:41:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:41:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:41:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:41:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:41:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:41:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:41:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:41:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:41:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:41:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:41:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:41:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:41:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:41:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:41:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:41:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:41:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:41:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:41:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:41:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:41:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:41:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:41:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:41:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:41:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:41:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:41:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:41:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:41:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:41:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:41:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:41:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:41:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:41:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:41:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:41:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:41:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:41:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:41:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:41:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:41:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:41:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:41:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:41:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:41:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:41:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:41:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:41:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:41:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:41:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:41:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:41:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:41:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:41:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:41:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:41:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:41:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:41:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:41:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:41:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:41:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:41:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:41:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:41:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:41:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:41:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:41:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:42:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:42:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:42:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:42:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:42:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:42:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:42:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:42:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:42:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:42:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:42:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:42:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:42:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:42:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:42:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:42:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:42:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:42:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:42:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:42:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:42:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:42:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:42:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:42:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:42:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:42:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:42:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:42:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:42:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:42:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:42:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:42:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:42:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:42:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:42:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:42:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:42:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:42:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:42:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:42:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:42:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:42:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:42:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:42:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:42:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:42:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:42:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:42:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:42:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:42:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:42:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:42:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:42:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:42:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:42:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:42:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:42:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:42:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:42:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:42:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:42:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:42:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:42:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:42:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:42:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:42:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:42:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:42:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:42:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:42:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:42:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:42:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:42:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:42:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:42:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:42:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:42:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:42:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:42:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:42:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:42:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:42:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:42:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:42:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:42:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:42:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:42:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:42:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:42:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:42:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:42:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:42:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:42:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:42:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:42:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:42:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:42:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:42:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:42:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:42:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:42:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:42:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:42:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:42:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:42:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:42:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:42:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:42:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:42:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:42:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:42:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:42:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:42:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:42:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:42:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:42:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:42:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:42:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:42:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:42:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:42:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:42:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:42:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:42:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:42:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:42:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:42:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:42:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:42:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:42:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:42:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:42:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:42:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:42:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:42:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:42:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:42:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:42:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:42:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:42:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:42:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:42:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:42:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:42:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:42:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:42:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:42:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:42:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:42:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:42:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:42:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:42:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:42:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:42:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:42:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:42:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:42:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:42:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:42:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:42:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:42:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:42:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:42:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:42:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:42:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:42:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:42:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:42:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:42:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:42:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:42:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:42:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:42:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:42:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:42:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:42:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:42:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:42:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:42:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:42:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:42:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:42:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:42:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:42:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:42:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:42:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:42:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:42:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:42:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:42:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:42:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:42:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:42:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:42:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:42:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:42:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:42:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:42:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:42:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:42:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:42:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:42:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:42:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:42:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:42:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:42:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:42:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:42:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:42:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:42:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:42:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:42:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:42:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:42:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:42:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:42:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:43:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:43:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:43:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:43:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:43:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:43:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:43:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:43:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:43:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:43:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:43:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:43:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:43:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:43:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:43:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:43:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:43:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:43:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:43:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:43:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:43:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:43:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:43:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:43:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:43:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:43:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:43:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:43:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:43:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:43:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:43:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:43:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:43:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:43:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:43:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:43:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:43:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:43:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:43:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:43:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:43:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:43:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:43:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:43:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:43:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:43:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:43:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:43:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:43:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:43:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:43:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:43:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:43:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:43:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:43:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:43:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:43:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:43:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:43:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:43:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:43:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:43:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:43:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:43:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:43:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:43:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:43:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:43:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:43:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:43:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:43:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:43:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:43:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:43:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:43:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:43:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:43:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:43:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:43:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:43:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:43:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:43:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:43:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:43:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:43:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:43:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:43:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:43:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:43:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:43:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:43:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:43:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:43:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:43:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:43:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:43:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:43:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:43:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:43:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:43:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:43:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:43:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:43:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:43:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:43:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:43:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:43:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:43:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:43:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:43:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:43:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:43:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:43:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:43:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:43:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:43:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:43:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:43:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:43:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:43:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:43:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:43:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:43:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:43:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:43:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:43:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:43:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:43:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:43:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:43:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:43:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:43:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:43:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:43:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:43:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:43:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:43:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:43:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:43:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:43:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:43:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:43:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:43:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:43:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:43:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:43:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:43:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:43:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:43:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:43:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:43:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:43:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:43:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:43:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:43:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:43:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:43:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:43:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:43:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:43:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:43:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:43:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:43:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:43:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:43:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:43:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:43:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:43:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:43:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:43:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:43:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:43:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:43:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:43:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:43:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:43:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:43:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:43:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:43:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:43:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:43:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:43:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:43:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:43:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:43:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:43:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:43:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:43:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:43:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:43:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:43:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:43:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:43:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:43:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:43:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:43:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:43:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:43:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:43:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:43:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:43:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:43:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:43:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:43:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:43:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:43:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:43:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:43:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:43:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:43:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:43:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:43:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:43:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:43:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:43:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:43:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:44:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:44:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:44:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:44:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:44:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:44:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:44:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:44:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:44:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:44:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:44:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:44:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:44:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:44:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:44:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:44:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:44:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:44:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:44:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:44:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:44:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:44:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:44:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:44:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:44:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:44:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:44:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:44:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:44:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:44:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:44:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:44:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:44:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:44:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:44:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:44:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:44:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:44:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:44:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:44:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:44:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:44:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:44:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:44:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:44:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:44:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:44:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:44:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:44:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:44:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:44:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:44:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:44:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:44:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:44:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:44:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:44:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:44:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:44:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:44:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:44:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:44:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:44:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:44:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:44:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:44:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:44:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:44:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:44:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:44:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:44:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:44:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:44:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:44:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:44:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:44:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:44:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:44:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:44:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:44:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:44:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:44:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:44:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:44:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:44:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:44:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:44:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:44:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:44:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:44:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:44:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:44:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:44:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:44:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:44:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:44:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:44:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:44:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:44:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:44:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:44:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:44:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:44:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:44:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:44:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:44:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:44:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:44:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:44:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:44:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:44:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:44:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:44:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:44:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:44:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:44:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:44:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:44:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:44:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:44:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:44:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:44:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:44:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:44:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:44:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:44:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:44:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:44:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:44:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:44:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:44:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:44:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:44:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:44:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:44:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:44:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:44:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:44:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:44:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:44:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:44:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:44:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:44:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:44:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:44:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:44:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:44:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:44:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:44:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:44:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:44:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:44:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:44:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:44:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:44:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:44:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:44:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:44:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:44:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:44:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:44:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:44:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:44:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:44:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:44:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:44:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:44:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:44:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:44:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:44:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:44:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:44:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:44:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:44:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:44:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:44:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:44:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:44:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:44:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:44:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:44:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:44:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:44:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:44:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:44:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:44:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:44:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:44:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:44:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:44:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:44:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:44:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:44:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:44:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:44:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:44:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:44:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:44:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:44:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:44:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:44:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:44:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:44:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:44:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:44:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:44:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:44:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:44:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:44:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:44:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:44:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:44:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:44:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:44:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:44:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:44:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:45:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:45:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:45:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:45:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:45:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:45:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:45:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:45:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:45:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:45:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:45:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:45:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:45:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:45:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:45:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:45:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:45:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:45:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:45:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:45:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:45:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:45:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:45:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:45:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:45:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:45:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:45:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:45:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:45:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:45:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:45:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:45:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:45:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:45:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:45:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:45:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:45:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:45:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:45:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:45:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:45:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:45:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:45:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:45:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:45:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:45:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:45:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:45:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:45:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:45:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:45:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:45:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:45:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:45:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:45:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:45:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:45:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:45:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:45:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:45:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:45:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:45:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:45:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:45:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:45:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:45:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:45:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:45:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:45:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:45:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:45:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:45:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:45:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:45:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:45:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:45:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:45:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:45:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:45:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:45:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:45:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:45:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:45:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:45:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:45:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:45:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:45:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:45:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:45:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:45:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:45:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:45:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:45:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:45:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:45:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:45:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:45:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:45:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:45:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:45:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:45:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:45:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:45:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:45:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:45:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:45:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:45:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:45:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:45:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:45:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:45:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:45:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:45:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:45:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:45:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:45:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:45:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:45:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:45:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:45:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:45:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:45:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:45:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:45:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:45:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:45:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:45:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:45:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:45:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:45:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:45:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:45:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:45:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:45:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:45:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:45:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:45:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:45:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:45:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:45:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:45:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:45:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:45:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:45:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:45:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:45:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:45:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:45:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:45:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:45:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:45:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:45:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:45:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:45:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:45:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:45:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:45:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:45:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:45:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:45:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:45:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:45:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:47:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:47:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:47:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:47:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:47:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:47:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:47:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:47:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:47:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:47:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:47:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:47:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:47:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:47:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:47:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:47:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:47:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:47:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:47:47 rdma06 rsyslogd[1041]: imjournal: 5082 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 15:47:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:47:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:47:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:47:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:47:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:47:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:47:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:47:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:47:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:47:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:47:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:47:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:47:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:47:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:47:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:47:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:47:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:47:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:47:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:47:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:47:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:47:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:47:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:47:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:47:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:47:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:47:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:47:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:47:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:47:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:47:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:47:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:47:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:47:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:47:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:47:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:48:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:48:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:48:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:48:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:48:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:48:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:48:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:48:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:48:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:48:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:48:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:48:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:48:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:48:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:48:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:48:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:48:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:48:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:48:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:48:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:48:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:48:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:48:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:48:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:48:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:48:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:48:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:48:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:48:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:48:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:48:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:48:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:48:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:48:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:48:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:48:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:48:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:48:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:48:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:48:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:48:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:48:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:48:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:48:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:48:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:48:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:48:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:48:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:48:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:48:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:48:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:48:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:48:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:48:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:48:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:48:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:48:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:48:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:48:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:48:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:48:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:48:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:48:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:48:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:48:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:48:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:48:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:48:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:48:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:48:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:48:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:48:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:48:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:48:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:48:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:48:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:48:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:48:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:48:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:48:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:48:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:48:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:48:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:48:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:48:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:48:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:48:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:48:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:48:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:48:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:48:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:48:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:48:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:48:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:48:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:48:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:48:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:48:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:48:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:48:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:48:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:48:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:48:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:48:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:48:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:48:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:48:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:48:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:48:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:48:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:48:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:48:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:48:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:48:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:48:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:48:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:48:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:48:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:48:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:48:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:48:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:48:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:48:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:48:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:48:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:48:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:48:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:48:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:48:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:48:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:48:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:48:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:48:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:48:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:48:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:48:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:48:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:48:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:48:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:48:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:48:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:48:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:48:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:48:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:48:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:48:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:48:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:48:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:48:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:48:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:48:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:48:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:48:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:48:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:48:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:48:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:48:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:48:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:48:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:48:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:48:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:48:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:48:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:48:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:48:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:48:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:48:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:48:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:48:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:48:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:48:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:48:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:48:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:48:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:48:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:48:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:48:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:48:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:48:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:48:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:48:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:48:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:48:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:48:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:48:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:48:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:48:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:48:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:48:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:48:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:48:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:48:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:48:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:48:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:48:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:48:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:48:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:48:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:48:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:48:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:48:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:48:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:48:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:48:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:48:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:48:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:48:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:48:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:48:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:48:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:48:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:48:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:48:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:48:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:48:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:48:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:49:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:49:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:49:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:49:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:49:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:49:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:49:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:49:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:49:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:49:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:49:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:49:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:49:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:49:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:49:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:49:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:49:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:49:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:49:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:49:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:49:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:49:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:49:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:49:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:49:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:49:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:49:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:49:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:49:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:49:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:49:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:49:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:49:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:49:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:49:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:49:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:49:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:49:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:49:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:49:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:49:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:49:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:49:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:49:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:49:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:49:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:49:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:49:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:49:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:49:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:49:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:49:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:49:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:49:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:49:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:49:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:49:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:49:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:49:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:49:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:49:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:49:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:49:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:49:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:49:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:49:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:49:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:49:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:49:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:49:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:49:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:49:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:49:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:49:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:49:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:49:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:49:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:49:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:49:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:49:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:49:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:49:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:49:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:49:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:49:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:49:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:49:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:49:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:49:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:49:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:49:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:49:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:49:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:49:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:49:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:49:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:49:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:49:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:49:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:49:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:49:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:49:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:49:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:49:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:49:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:49:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:49:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:49:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:49:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:49:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:49:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:49:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:49:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:49:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:49:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:49:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:49:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:49:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:49:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:49:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:49:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:49:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:49:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:49:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:49:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:49:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:49:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:49:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:49:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:49:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:49:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:49:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:49:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:49:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:49:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:49:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:49:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:49:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:49:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:49:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:49:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:49:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:49:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:49:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:49:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:49:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:49:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:49:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:49:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:49:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:49:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:49:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:49:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:49:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:49:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:49:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:49:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:49:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:49:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:49:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:49:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:49:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:49:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:49:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:49:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:49:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:49:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:49:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:49:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:49:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:49:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:49:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:49:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:49:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:49:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:49:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:49:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:49:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:49:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:49:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:49:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:49:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:49:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:49:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:49:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:49:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:49:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:49:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:49:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:49:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:49:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:49:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:49:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:49:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:49:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:49:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:49:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:49:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:49:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:49:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:49:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:49:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:49:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:49:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:49:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:49:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:49:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:49:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:49:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:49:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:49:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:49:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:49:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:49:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:49:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:49:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:50:00 rdma06 systemd[1]: Starting dnf makecache...
Apr 17 15:50:01 rdma06 dnf[1160098]: Updating Subscription Management repositories.
Apr 17 15:50:01 rdma06 dnf[1160098]: Unable to read consumer identity
Apr 17 15:50:01 rdma06 dnf[1160098]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 17 15:50:01 rdma06 dnf[1160098]: Metadata cache refreshed recently.
Apr 17 15:50:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 17 15:50:01 rdma06 systemd[1]: Finished dnf makecache.
Apr 17 15:50:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:50:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:50:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:50:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:50:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:50:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:50:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:50:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:50:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:50:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:50:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:50:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:50:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:50:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:50:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:50:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:50:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:50:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:50:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:50:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:50:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:50:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:50:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:50:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:50:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:50:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:50:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:50:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:50:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:50:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:50:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:50:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:50:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:50:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:50:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:50:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:50:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:50:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:50:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:50:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:50:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:50:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:50:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:50:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:50:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:50:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:50:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:50:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:50:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:50:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:50:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:50:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:50:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:50:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:50:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:50:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:50:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:50:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:50:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:50:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:50:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:50:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:50:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:50:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:50:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:50:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:50:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:50:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:50:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:50:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:50:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:50:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:50:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:50:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:50:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:50:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:50:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:50:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:50:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:50:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:50:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:50:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:50:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:50:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:50:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:50:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:50:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:50:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:50:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:50:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:50:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:50:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:50:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:50:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:50:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:50:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:50:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:50:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:50:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:50:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:50:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:50:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:50:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:50:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:50:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:50:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:50:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:50:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:50:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:50:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:50:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:50:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:50:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:50:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:50:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:50:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:50:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:50:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:50:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:50:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:50:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:50:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:50:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:50:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:50:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:50:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:50:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:50:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:50:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:50:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:50:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:50:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:50:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:50:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:50:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:50:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:50:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:50:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:50:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:50:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:50:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:50:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:50:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:50:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:50:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:50:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:50:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:50:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:50:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:50:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:50:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:50:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:50:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:50:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:50:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:50:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:50:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:50:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:50:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:50:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:50:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:50:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:50:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:50:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:50:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:50:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:50:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:50:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:50:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:50:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:50:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:50:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:50:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:50:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:50:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:50:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:50:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:50:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:50:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:50:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:50:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:50:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:50:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:50:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:50:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:50:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:50:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:50:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:50:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:50:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:50:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:50:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:50:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:50:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:50:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:50:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:50:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:50:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:50:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:50:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:50:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:50:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:50:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:50:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:50:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:50:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:50:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:50:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:50:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:50:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:50:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:50:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:50:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:50:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:50:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:50:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:51:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:51:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:51:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:51:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:51:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:51:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:51:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:51:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:51:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:51:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:51:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:51:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:51:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:51:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:51:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:51:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:51:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:51:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:51:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:51:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:51:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:51:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:51:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:51:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:51:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:51:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:51:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:51:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:51:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:51:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:51:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:51:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:51:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:51:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:51:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:51:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:51:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:51:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:51:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:51:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:51:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:51:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:51:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:51:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:51:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:51:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:51:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:51:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:51:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:51:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:51:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:51:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:51:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:51:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:51:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:51:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:51:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:51:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:51:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:51:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:51:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:51:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:51:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:51:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:51:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:51:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:51:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:51:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:51:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:51:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:51:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:51:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:51:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:51:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:51:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:51:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:51:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:51:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:51:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:51:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:51:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:51:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:51:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:51:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:51:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:51:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:51:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:51:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:51:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:51:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:51:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:51:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:51:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:51:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:51:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:51:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:51:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:51:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:51:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:51:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:51:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:51:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:51:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:51:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:51:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:51:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:51:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:51:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:51:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:51:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:51:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:51:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:51:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:51:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:51:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:51:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:51:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:51:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:51:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:51:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:51:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:51:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:51:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:51:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:51:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:51:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:51:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:51:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:51:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:51:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:51:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:51:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:51:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:51:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:51:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:51:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:51:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:51:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:51:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:51:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:51:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:51:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:51:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:51:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:51:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:51:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:51:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:51:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:51:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:51:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:51:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:51:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:51:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:51:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:51:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:51:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:51:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:51:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:51:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:51:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:51:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:51:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:51:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:51:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:51:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:51:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:51:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:51:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:51:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:51:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:51:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:51:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:51:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:51:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:51:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:51:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:51:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:51:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:51:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:51:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:51:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:51:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:51:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:51:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:51:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:51:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:51:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:51:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:51:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:51:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:51:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:51:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:51:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:51:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:51:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:51:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:51:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:51:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:51:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:51:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:51:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:51:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:51:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:51:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:51:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:51:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:51:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:51:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:51:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:51:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:51:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:51:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:51:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:51:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:51:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:51:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:52:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:52:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:52:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:52:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:52:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:52:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:52:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:52:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:52:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:52:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:52:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:52:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:52:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:52:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:52:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:52:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:52:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:52:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:52:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:52:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:52:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:52:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:52:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:52:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:52:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:52:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:52:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:52:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:52:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:52:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:52:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:52:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:52:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:52:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:52:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:52:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:52:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:52:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:52:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:52:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:52:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:52:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:52:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:52:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:52:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:52:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:52:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:52:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:52:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:52:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:52:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:52:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:52:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:52:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:52:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:52:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:52:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:52:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:52:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:52:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:52:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:52:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:52:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:52:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:52:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:52:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:52:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:52:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:52:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:52:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:52:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:52:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:52:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:52:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:52:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:52:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:52:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:52:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:52:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:52:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:52:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:52:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:52:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:52:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:52:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:52:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:52:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:52:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:52:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:52:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:52:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:52:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:52:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:52:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:52:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:52:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:52:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:52:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:52:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:52:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:52:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:52:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:52:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:52:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:52:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:52:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:52:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:52:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:52:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:52:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:52:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:52:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:52:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:52:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:52:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:52:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:52:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:52:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:52:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:52:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:52:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:52:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:52:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:52:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:52:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:52:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:52:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:52:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:52:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:52:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:52:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:52:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:52:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:52:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:52:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:52:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:52:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:52:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:52:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:52:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:52:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:52:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:52:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:52:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:52:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:52:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:52:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:52:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:52:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:52:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:52:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:52:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:52:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:52:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:52:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:52:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:52:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:52:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:52:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:52:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:52:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:52:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:52:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:52:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:52:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:52:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:52:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:52:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:52:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:52:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:52:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:52:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:52:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:52:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:52:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:52:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:52:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:52:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:52:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:52:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:52:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:52:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:52:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:52:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:52:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:52:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:52:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:52:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:52:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:52:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:52:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:52:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:52:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:52:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:52:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:52:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:52:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:52:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:52:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:52:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:52:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:52:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:52:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:52:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:52:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:52:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:52:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:52:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:52:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:52:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:52:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:52:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:52:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:52:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:52:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:52:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:53:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:53:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:53:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:53:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:53:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:53:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:53:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:53:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:53:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:53:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:53:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:53:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:53:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:53:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:53:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:53:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:53:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:53:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:53:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:53:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:53:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:53:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:53:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:53:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:53:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:53:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:53:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:53:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:53:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:53:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:53:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:53:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:53:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:53:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:53:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:53:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:53:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:53:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:53:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:53:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:53:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:53:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:53:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:53:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:53:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:53:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:53:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:53:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:53:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:53:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:53:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:53:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:53:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:53:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:53:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:53:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:53:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:53:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:53:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:53:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:53:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:53:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:53:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:53:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:53:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:53:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:53:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:53:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:53:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:53:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:53:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:53:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:53:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:53:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:53:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:53:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:53:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:53:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:53:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:53:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:53:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:53:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:53:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:53:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:53:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:53:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:53:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:53:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:53:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:53:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:53:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:53:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:53:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:53:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:53:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:53:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:53:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:53:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:53:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:53:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:53:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:53:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:53:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:53:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:53:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:53:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:53:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:53:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:53:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:53:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:53:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:53:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:53:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:53:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:53:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:53:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:53:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:53:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:53:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:53:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:53:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:53:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:53:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:53:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:53:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:53:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:53:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:53:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:53:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:53:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:53:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:53:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:53:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:53:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:53:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:53:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:53:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:53:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:53:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:53:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:53:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:53:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:53:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:53:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:53:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:53:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:53:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:53:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:53:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:53:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:53:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:53:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:53:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:53:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:53:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:53:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:53:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:53:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:53:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:53:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:53:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:53:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:53:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:53:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:53:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:53:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:53:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:53:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:53:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:53:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:53:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:53:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:53:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:53:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:53:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:53:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:53:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:53:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:53:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:53:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:53:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:53:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:53:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:53:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:53:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:53:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:53:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:53:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:53:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:53:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:53:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:53:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:53:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:53:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:53:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:53:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:53:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:53:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:53:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:53:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:53:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:53:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:53:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:53:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:53:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:53:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:53:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:53:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:53:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:53:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:53:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:53:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:53:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:53:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:53:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:53:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:54:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:54:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:54:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:54:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:54:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:54:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:54:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:54:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:54:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:54:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:54:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:54:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:54:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:54:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:54:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:54:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:54:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:54:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:54:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:54:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:54:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:54:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:54:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:54:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:54:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:54:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:54:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:54:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:54:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:54:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:54:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:54:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:54:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:54:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:54:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:54:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:54:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:54:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:54:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:54:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:54:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:54:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:54:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:54:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:54:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:54:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:54:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:54:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:54:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:54:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:54:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:54:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:54:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:54:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:54:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:54:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:54:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:54:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:54:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:54:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:54:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:54:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:54:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:54:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:54:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:54:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:54:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:54:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:54:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:54:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:54:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:54:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:54:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:54:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:54:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:54:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:54:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:54:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:54:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:54:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:54:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:54:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:54:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:54:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:54:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:54:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:54:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:54:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:54:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:54:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:54:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:54:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:54:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:54:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:54:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:54:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:54:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:54:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:54:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:54:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:54:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:54:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:54:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:54:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:54:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:54:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:54:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:54:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:54:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:54:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:54:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:54:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:54:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:54:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:54:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:54:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:54:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:54:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:54:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:54:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:54:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:54:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:54:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:54:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:54:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:54:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:54:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:54:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:54:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:54:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:54:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:54:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:54:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:54:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:54:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:54:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:54:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:54:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:54:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:54:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:54:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:54:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:54:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:54:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:54:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:54:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:54:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:54:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:54:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:54:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:54:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:54:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:54:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:54:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:54:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:54:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:54:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:54:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:54:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:54:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:54:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:54:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:54:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:54:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:54:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:54:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:54:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:54:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:54:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:54:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:54:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:54:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:54:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:54:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:54:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:54:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:54:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:54:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:54:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:54:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:54:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:54:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:54:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:54:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:54:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:54:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:54:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:54:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:54:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:54:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:54:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:54:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:54:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:54:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:54:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:54:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:54:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:54:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:54:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:54:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:54:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:54:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:54:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:54:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:54:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:54:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:54:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:54:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:54:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:54:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:54:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:54:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:54:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:54:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:54:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:54:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:55:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:55:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:55:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:55:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:55:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:55:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:55:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:55:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:55:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:55:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:55:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:55:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:55:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:55:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:55:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:55:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:55:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:55:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:55:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:55:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:55:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:55:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:55:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:55:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:55:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:55:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:55:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:55:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:55:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:55:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:55:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:55:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:55:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:55:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:55:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:55:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:55:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:55:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:55:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:55:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:55:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:55:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:55:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:55:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:55:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:55:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:55:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:55:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:55:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:55:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:55:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:55:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:55:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:55:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:55:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:55:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:55:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:55:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:55:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:55:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:55:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:55:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:55:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:55:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:55:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:55:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:55:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:55:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:55:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:55:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:55:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:55:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:55:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:55:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:55:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:55:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:55:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:55:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:55:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:55:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:55:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:55:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:55:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:55:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:55:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:55:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:55:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:55:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:55:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:55:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:55:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:55:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:55:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:55:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:55:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:55:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:55:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:55:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:55:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:55:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:55:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:55:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:55:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:55:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:55:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:55:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:55:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:55:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:55:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:55:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:55:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:55:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:55:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:55:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:55:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:55:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:55:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:55:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:55:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:55:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:55:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:55:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:55:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:55:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:55:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:55:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:55:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:55:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:55:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:55:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:55:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:55:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:55:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:55:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:55:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:55:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:55:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:55:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:55:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:55:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:55:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:55:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:55:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:55:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:55:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:55:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:55:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:55:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:55:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:55:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:55:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:55:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:55:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:55:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:55:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:55:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:55:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:55:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:55:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:55:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:55:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:55:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:57:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:57:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:57:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:57:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:57:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:57:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:57:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:57:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:57:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:57:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:57:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:57:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:57:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:57:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:57:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:57:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:57:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:57:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:57:52 rdma06 rsyslogd[1041]: imjournal: 5298 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 15:57:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:57:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:57:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:57:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:57:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:57:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:57:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:57:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:57:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:57:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:57:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:57:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:57:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:57:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:57:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:57:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:57:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:57:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:58:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:58:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:58:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:58:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:58:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:58:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:58:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:58:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:58:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:58:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:58:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:58:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:58:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:58:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:58:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:58:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:58:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:58:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:58:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:58:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:58:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:58:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:58:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:58:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:58:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:58:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:58:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:58:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:58:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:58:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:58:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:58:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:58:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:58:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:58:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:58:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:58:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:58:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:58:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:58:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:58:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:58:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:58:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:58:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:58:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:58:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:58:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:58:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:58:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:58:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:58:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:58:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:58:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:58:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:58:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:58:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:58:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:58:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:58:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:58:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:58:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:58:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:58:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:58:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:58:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:58:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:58:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:58:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:58:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:58:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:58:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:58:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:58:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:58:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:58:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:58:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:58:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:58:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:58:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:58:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:58:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:58:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:58:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:58:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:58:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:58:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:58:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:58:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:58:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:58:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:58:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:58:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:58:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:58:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:58:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:58:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:58:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:58:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:58:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:58:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:58:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:58:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:58:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:58:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:58:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:58:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:58:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:58:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:58:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:58:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:58:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:58:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:58:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:58:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:58:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:58:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:58:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:58:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:58:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:58:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:58:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:58:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:58:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:58:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:58:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:58:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:58:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:58:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:58:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:58:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:58:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:58:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:58:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:58:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:58:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:58:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:58:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:58:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:58:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:58:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:58:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:58:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:58:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:58:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:58:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:58:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:58:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:58:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:58:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:58:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:58:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:58:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:58:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:58:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:58:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:58:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:58:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:58:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:58:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:58:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:58:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:58:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:58:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:58:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:58:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:58:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:58:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:58:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:58:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:58:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:58:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:58:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:58:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:58:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:58:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:58:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:58:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:58:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:58:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:58:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:58:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:58:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:58:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:58:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:58:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:58:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:58:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:58:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:58:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:58:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:58:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:58:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:58:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:58:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:58:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:58:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:58:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:58:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:58:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:58:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:58:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:58:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:58:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:58:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:58:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:58:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:58:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:58:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:58:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:58:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:58:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:58:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:58:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:58:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:58:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:58:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:59:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:59:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:59:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:59:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:59:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:59:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:59:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:59:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:59:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:59:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:59:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:59:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:59:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:59:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:59:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:59:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:59:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:59:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:59:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:59:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:59:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:59:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:59:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:59:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:59:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:59:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:59:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:59:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:59:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:59:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:59:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:59:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:59:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:59:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:59:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:59:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:59:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:59:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:59:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:59:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:59:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:59:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:59:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:59:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:59:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:59:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:59:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:59:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:59:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:59:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:59:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:59:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:59:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:59:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:59:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:59:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:59:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:59:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:59:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:59:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:59:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:59:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:59:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:59:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:59:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:59:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:59:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:59:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:59:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:59:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:59:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:59:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:59:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:59:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:59:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:59:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:59:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:59:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:59:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:59:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:59:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:59:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:59:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:59:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:59:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:59:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:59:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:59:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:59:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:59:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:59:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:59:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:59:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:59:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:59:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:59:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:59:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:59:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:59:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:59:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:59:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:59:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:59:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:59:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:59:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:59:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:59:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:59:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:59:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:59:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:59:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:59:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:59:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:59:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:59:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:59:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:59:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:59:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:59:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:59:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:59:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:59:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:59:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:59:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:59:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:59:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:59:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:59:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:59:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:59:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:59:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:59:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:59:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:59:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:59:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:59:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:59:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:59:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:59:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:59:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:59:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:59:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:59:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:59:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:59:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:59:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:59:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:59:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:59:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:59:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:59:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:59:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:59:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:59:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:59:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:59:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:59:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:59:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:59:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:59:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:59:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:59:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:59:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:59:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:59:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:59:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:59:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:59:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:59:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:59:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:59:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:59:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:59:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:59:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:59:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:59:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:59:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:59:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:59:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:59:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:59:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:59:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:59:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:59:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:59:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:59:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:59:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:59:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:59:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:59:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:59:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:59:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:59:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:59:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:59:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:59:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:59:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:59:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 15:59:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 15:59:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:59:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:59:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:59:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 15:59:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 15:59:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 15:59:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 15:59:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 15:59:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 15:59:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 15:59:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 15:59:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 15:59:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 15:59:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 15:59:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 15:59:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 15:59:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:00:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:00:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:00:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:00:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:00:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:00:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:00:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:00:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:00:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:00:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:00:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:00:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:00:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:00:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:00:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:00:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:00:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:00:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:00:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:00:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:00:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:00:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:00:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:00:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:00:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:00:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:00:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:00:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:00:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:00:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:00:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:00:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:00:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:00:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:00:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:00:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:00:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:00:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:00:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:00:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:00:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:00:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:00:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:00:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:00:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:00:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:00:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:00:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:00:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:00:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:00:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:00:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:00:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:00:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:00:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:00:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:00:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:00:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:00:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:00:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:00:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:00:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:00:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:00:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:00:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:00:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:00:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:00:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:00:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:00:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:00:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:00:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:00:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:00:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:00:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:00:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:00:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:00:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:00:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:00:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:00:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:00:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:00:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:00:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:00:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:00:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:00:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:00:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:00:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:00:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:00:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:00:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:00:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:00:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:00:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:00:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:00:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:00:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:00:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:00:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:00:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:00:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:00:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:00:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:00:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:00:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:00:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:00:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:00:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:00:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:00:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:00:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:00:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:00:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:00:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:00:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:00:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:00:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:00:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:00:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:00:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:00:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:00:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:00:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:00:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:00:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:00:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:00:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:00:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:00:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:00:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:00:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:00:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:00:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:00:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:00:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:00:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:00:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:00:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:00:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:00:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:00:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:00:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:00:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:00:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:00:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:00:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:00:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:00:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:00:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:00:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:00:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:00:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:00:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:00:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:00:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:00:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:00:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:00:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:00:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:00:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:00:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:00:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:00:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:00:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:00:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:00:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:00:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:00:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:00:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:00:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:00:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:00:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:00:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:00:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:00:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:00:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:00:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:00:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:00:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:00:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:00:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:00:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:00:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:00:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:00:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:00:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:00:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:00:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:00:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:00:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:00:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:00:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:00:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:00:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:00:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:00:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:00:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:00:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:00:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:00:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:00:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:00:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:00:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:00:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:00:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:00:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:00:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:00:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:00:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:00:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:00:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:00:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:00:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:00:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:00:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:01:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:01:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:01:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:01:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:01:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:01:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:01:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:01:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:01:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:01:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:01:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:01:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:01:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:01:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:01:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:01:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:01:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:01:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:01:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:01:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:01:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:01:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:01:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:01:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:01:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:01:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:01:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:01:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:01:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:01:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:01:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:01:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:01:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:01:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:01:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:01:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:01:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:01:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:01:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:01:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:01:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:01:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:01:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:01:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:01:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:01:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:01:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:01:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:01:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:01:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:01:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:01:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:01:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:01:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:01:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:01:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:01:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:01:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:01:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:01:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:01:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:01:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:01:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:01:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:01:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:01:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:01:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:01:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:01:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:01:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:01:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:01:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:01:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:01:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:01:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:01:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:01:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:01:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:01:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:01:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:01:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:01:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:01:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:01:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:01:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:01:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:01:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:01:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:01:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:01:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:01:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:01:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:01:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:01:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:01:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:01:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:01:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:01:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:01:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:01:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:01:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:01:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:01:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:01:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:01:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:01:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:01:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:01:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:01:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:01:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:01:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:01:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:01:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:01:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:01:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:01:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:01:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:01:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:01:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:01:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:01:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:01:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:01:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:01:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:01:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:01:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:01:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:01:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:01:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:01:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:01:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:01:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:01:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:01:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:01:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:01:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:01:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:01:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:01:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:01:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:01:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:01:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:01:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:01:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:01:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:01:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:01:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:01:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:01:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:01:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:01:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:01:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:01:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:01:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:01:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:01:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:01:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:01:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:01:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:01:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:01:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:01:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:01:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:01:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:01:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:01:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:01:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:01:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:01:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:01:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:01:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:01:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:01:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:01:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:01:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:01:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:01:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:01:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:01:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:01:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:01:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:01:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:01:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:01:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:01:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:01:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:01:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:01:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:01:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:01:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:01:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:01:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:01:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:01:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:01:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:01:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:01:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:01:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:01:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:01:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:01:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:01:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:01:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:01:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:01:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:01:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:01:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:01:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:01:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:01:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:01:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:01:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:01:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:01:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:01:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:01:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:02:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:02:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:02:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:02:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:02:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:02:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:02:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:02:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:02:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:02:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:02:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:02:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:02:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:02:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:02:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:02:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:02:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:02:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:02:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:02:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:02:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:02:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:02:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:02:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:02:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:02:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:02:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:02:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:02:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:02:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:02:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:02:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:02:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:02:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:02:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:02:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:02:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:02:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:02:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:02:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:02:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:02:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:02:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:02:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:02:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:02:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:02:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:02:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:02:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:02:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:02:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:02:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:02:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:02:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:02:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:02:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:02:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:02:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:02:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:02:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:02:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:02:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:02:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:02:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:02:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:02:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:02:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:02:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:02:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:02:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:02:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:02:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:02:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:02:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:02:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:02:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:02:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:02:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:02:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:02:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:02:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:02:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:02:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:02:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:02:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:02:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:02:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:02:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:02:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:02:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:02:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:02:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:02:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:02:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:02:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:02:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:02:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:02:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:02:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:02:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:02:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:02:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:02:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:02:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:02:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:02:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:02:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:02:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:02:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:02:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:02:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:02:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:02:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:02:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:02:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:02:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:02:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:02:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:02:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:02:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:02:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:02:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:02:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:02:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:02:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:02:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:02:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:02:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:02:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:02:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:02:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:02:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:02:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:02:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:02:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:02:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:02:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:02:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:02:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:02:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:02:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:02:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:02:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:02:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:02:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:02:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:02:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:02:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:02:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:02:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:02:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:02:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:02:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:02:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:02:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:02:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:02:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:02:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:02:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:02:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:02:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:02:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:02:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:02:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:02:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:02:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:02:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:02:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:02:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:02:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:02:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:02:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:02:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:02:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:02:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:02:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:02:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:02:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:02:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:02:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:02:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:02:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:02:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:02:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:02:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:02:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:02:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:02:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:02:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:02:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:02:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:02:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:02:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:02:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:02:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:02:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:02:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:02:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:02:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:02:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:02:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:02:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:02:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:02:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:02:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:02:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:02:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:02:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:02:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:02:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:02:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:02:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:02:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:02:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:02:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:02:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:03:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:03:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:03:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:03:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:03:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:03:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:03:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:03:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:03:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:03:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:03:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:03:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:03:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:03:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:03:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:03:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:03:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:03:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:03:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:03:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:03:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:03:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:03:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:03:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:03:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:03:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:03:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:03:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:03:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:03:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:03:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:03:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:03:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:03:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:03:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:03:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:03:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:03:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:03:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:03:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:03:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:03:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:03:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:03:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:03:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:03:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:03:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:03:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:03:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:03:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:03:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:03:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:03:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:03:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:03:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:03:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:03:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:03:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:03:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:03:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:03:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:03:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:03:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:03:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:03:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:03:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:03:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:03:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:03:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:03:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:03:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:03:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:03:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:03:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:03:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:03:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:03:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:03:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:03:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:03:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:03:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:03:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:03:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:03:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:03:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:03:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:03:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:03:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:03:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:03:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:03:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:03:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:03:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:03:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:03:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:03:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:03:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:03:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:03:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:03:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:03:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:03:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:03:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:03:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:03:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:03:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:03:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:03:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:03:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:03:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:03:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:03:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:03:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:03:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:03:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:03:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:03:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:03:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:03:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:03:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:03:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:03:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:03:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:03:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:03:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:03:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:03:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:03:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:03:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:03:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:03:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:03:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:03:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:03:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:03:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:03:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:03:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:03:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:03:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:03:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:03:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:03:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:03:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:03:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:03:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:03:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:03:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:03:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:03:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:03:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:03:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:03:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:03:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:03:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:03:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:03:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:03:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:03:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:03:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:03:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:03:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:03:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:03:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:03:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:03:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:03:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:03:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:03:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:03:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:03:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:03:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:03:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:03:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:03:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:03:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:03:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:03:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:03:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:03:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:03:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:03:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:03:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:03:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:03:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:03:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:03:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:03:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:03:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:03:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:03:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:03:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:03:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:03:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:03:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:03:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:03:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:03:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:03:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:03:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:03:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:03:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:03:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:03:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:03:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:03:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:03:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:03:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:03:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:03:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:03:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:03:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:03:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:03:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:03:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:03:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:03:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:04:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:04:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:04:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:04:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:04:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:04:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:04:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:04:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:04:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:04:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:04:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:04:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:04:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:04:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:04:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:04:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:04:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:04:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:04:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:04:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:04:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:04:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:04:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:04:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:04:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:04:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:04:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:04:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:04:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:04:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:04:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:04:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:04:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:04:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:04:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:04:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:04:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:04:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:04:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:04:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:04:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:04:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:04:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:04:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:04:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:04:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:04:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:04:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:04:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:04:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:04:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:04:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:04:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:04:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:04:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:04:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:04:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:04:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:04:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:04:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:04:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:04:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:04:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:04:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:04:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:04:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:04:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:04:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:04:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:04:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:04:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:04:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:04:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:04:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:04:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:04:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:04:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:04:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:04:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:04:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:04:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:04:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:04:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:04:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:04:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:04:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:04:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:04:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:04:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:04:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:04:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:04:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:04:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:04:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:04:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:04:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:04:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:04:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:04:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:04:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:04:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:04:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:04:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:04:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:04:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:04:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:04:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:04:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:04:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:04:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:04:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:04:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:04:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:04:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:04:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:04:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:04:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:04:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:04:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:04:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:04:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:04:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:04:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:04:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:04:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:04:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:04:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:04:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:04:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:04:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:04:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:04:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:04:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:04:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:04:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:04:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:04:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:04:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:04:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:04:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:04:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:04:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:04:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:04:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:04:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:04:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:04:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:04:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:04:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:04:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:04:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:04:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:04:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:04:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:04:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:04:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:04:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:04:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:04:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:04:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:04:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:04:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:04:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:04:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:04:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:04:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:04:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:04:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:04:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:04:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:04:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:04:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:04:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:04:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:04:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:04:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:04:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:04:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:04:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:04:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:04:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:04:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:04:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:04:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:04:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:04:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:04:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:04:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:04:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:04:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:04:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:04:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:04:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:04:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:04:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:04:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:04:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:04:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:04:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:04:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:04:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:04:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:04:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:04:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:04:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:04:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:04:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:04:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:04:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:04:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:04:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:04:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:04:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:04:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:04:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:04:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:05:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:05:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:05:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:05:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:05:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:05:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:05:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:05:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:05:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:05:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:05:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:05:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:05:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:05:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:05:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:05:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:05:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:05:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:05:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:05:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:05:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:05:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:05:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:05:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:05:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:05:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:05:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:05:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:05:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:05:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:05:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:05:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:05:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:05:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:05:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:05:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:05:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:05:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:05:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:05:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:05:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:05:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:05:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:05:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:05:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:05:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:05:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:05:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:05:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:05:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:05:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:05:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:05:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:05:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:05:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:05:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:05:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:05:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:05:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:05:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:05:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:05:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:05:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:05:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:05:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:05:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:05:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:05:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:05:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:05:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:05:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:05:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:05:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:05:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:05:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:05:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:05:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:05:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:05:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:05:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:05:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:05:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:05:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:05:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:05:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:05:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:05:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:05:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:05:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:05:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:05:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:05:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:05:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:05:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:05:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:05:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:05:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:05:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:05:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:05:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:05:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:05:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:05:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:05:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:05:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:05:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:05:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:05:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:05:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:05:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:05:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:05:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:05:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:05:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:05:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:05:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:05:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:05:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:05:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:05:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:05:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:05:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:05:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:05:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:05:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:05:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:05:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:05:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:05:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:05:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:05:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:05:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:05:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:05:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:05:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:05:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:05:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:05:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:05:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:05:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:05:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:05:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:05:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:05:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:05:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:05:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:05:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:05:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:05:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:05:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:05:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:05:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:05:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:05:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:05:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:05:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:05:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:05:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:05:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:05:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:05:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:05:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:05:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:05:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:05:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:05:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:05:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:05:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:05:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:05:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:05:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:05:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:05:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:05:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:05:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:05:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:05:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:05:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:05:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:05:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:07:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:07:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:07:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:07:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:07:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:07:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:07:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:07:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:07:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:07:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:07:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:07:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:07:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:07:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:07:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:07:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:07:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:07:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:07:57 rdma06 rsyslogd[1041]: imjournal: 5295 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 16:08:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:08:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:08:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:08:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:08:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:08:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:08:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:08:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:08:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:08:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:08:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:08:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:08:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:08:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:08:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:08:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:08:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:08:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:08:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:08:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:08:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:08:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:08:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:08:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:08:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:08:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:08:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:08:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:08:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:08:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:08:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:08:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:08:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:08:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:08:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:08:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:08:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:08:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:08:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:08:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:08:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:08:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:08:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:08:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:08:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:08:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:08:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:08:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:08:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:08:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:08:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:08:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:08:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:08:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:08:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:08:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:08:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:08:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:08:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:08:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:08:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:08:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:08:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:08:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:08:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:08:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:08:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:08:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:08:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:08:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:08:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:08:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:08:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:08:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:08:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:08:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:08:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:08:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:08:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:08:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:08:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:08:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:08:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:08:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:08:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:08:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:08:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:08:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:08:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:08:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:08:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:08:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:08:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:08:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:08:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:08:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:08:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:08:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:08:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:08:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:08:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:08:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:08:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:08:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:08:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:08:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:08:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:08:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:08:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:08:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:08:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:08:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:08:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:08:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:08:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:08:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:08:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:08:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:08:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:08:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:08:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:08:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:08:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:08:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:08:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:08:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:08:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:08:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:08:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:08:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:08:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:08:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:08:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:08:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:08:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:08:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:08:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:08:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:08:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:08:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:08:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:08:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:08:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:08:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:08:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:08:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:08:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:08:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:08:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:08:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:08:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:08:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:08:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:08:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:08:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:08:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:08:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:08:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:08:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:08:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:08:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:08:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:08:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:08:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:08:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:08:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:08:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:08:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:08:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:08:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:08:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:08:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:08:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:08:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:08:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:08:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:08:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:08:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:08:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:08:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:08:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:08:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:08:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:08:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:08:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:08:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:08:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:08:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:08:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:08:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:08:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:08:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:08:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:08:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:08:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:08:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:08:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:08:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:08:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:08:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:08:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:08:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:08:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:08:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:08:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:08:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:08:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:08:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:08:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:08:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:08:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:08:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:08:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:08:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:08:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:08:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:09:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:09:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:09:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:09:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:09:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:09:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:09:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:09:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:09:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:09:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:09:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:09:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:09:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:09:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:09:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:09:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:09:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:09:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:09:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:09:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:09:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:09:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:09:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:09:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:09:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:09:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:09:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:09:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:09:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:09:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:09:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:09:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:09:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:09:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:09:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:09:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:09:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:09:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:09:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:09:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:09:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:09:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:09:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:09:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:09:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:09:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:09:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:09:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:09:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:09:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:09:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:09:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:09:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:09:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:09:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:09:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:09:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:09:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:09:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:09:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:09:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:09:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:09:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:09:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:09:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:09:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:09:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:09:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:09:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:09:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:09:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:09:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:09:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:09:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:09:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:09:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:09:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:09:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:09:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:09:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:09:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:09:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:09:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:09:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:09:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:09:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:09:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:09:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:09:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:09:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:09:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:09:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:09:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:09:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:09:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:09:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:09:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:09:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:09:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:09:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:09:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:09:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:09:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:09:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:09:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:09:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:09:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:09:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:09:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:09:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:09:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:09:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:09:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:09:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:09:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:09:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:09:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:09:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:09:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:09:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:09:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:09:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:09:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:09:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:09:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:09:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:09:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:09:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:09:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:09:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:09:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:09:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:09:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:09:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:09:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:09:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:09:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:09:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:09:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:09:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:09:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:09:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:09:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:09:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:09:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:09:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:09:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:09:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:09:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:09:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:09:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:09:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:09:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:09:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:09:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:09:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:09:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:09:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:09:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:09:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:09:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:09:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:09:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:09:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:09:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:09:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:09:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:09:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:09:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:09:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:09:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:09:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:09:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:09:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:09:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:09:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:09:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:09:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:09:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:09:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:09:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:09:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:09:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:09:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:09:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:09:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:09:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:09:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:09:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:09:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:09:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:09:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:09:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:09:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:09:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:09:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:09:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:09:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:09:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:09:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:09:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:09:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:09:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:09:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:09:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:09:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:09:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:09:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:09:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:09:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:09:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:09:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:09:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:09:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:09:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:09:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:10:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:10:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:10:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:10:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:10:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:10:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:10:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:10:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:10:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:10:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:10:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:10:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:10:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:10:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:10:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:10:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:10:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:10:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:10:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:10:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:10:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:10:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:10:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:10:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:10:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:10:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:10:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:10:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:10:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:10:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:10:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:10:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:10:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:10:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:10:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:10:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:10:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:10:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:10:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:10:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:10:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:10:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:10:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:10:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:10:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:10:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:10:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:10:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:10:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:10:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:10:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:10:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:10:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:10:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:10:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:10:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:10:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:10:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:10:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:10:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:10:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:10:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:10:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:10:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:10:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:10:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:10:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:10:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:10:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:10:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:10:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:10:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:10:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:10:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:10:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:10:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:10:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:10:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:10:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:10:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:10:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:10:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:10:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:10:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:10:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:10:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:10:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:10:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:10:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:10:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:10:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:10:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:10:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:10:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:10:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:10:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:10:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:10:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:10:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:10:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:10:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:10:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:10:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:10:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:10:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:10:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:10:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:10:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:10:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:10:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:10:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:10:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:10:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:10:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:10:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:10:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:10:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:10:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:10:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:10:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:10:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:10:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:10:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:10:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:10:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:10:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:10:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:10:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:10:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:10:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:10:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:10:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:10:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:10:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:10:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:10:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:10:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:10:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:10:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:10:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:10:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:10:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:10:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:10:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:10:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:10:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:10:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:10:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:10:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:10:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:10:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:10:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:10:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:10:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:10:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:10:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:10:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:10:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:10:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:10:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:10:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:10:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:10:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:10:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:10:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:10:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:10:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:10:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:10:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:10:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:10:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:10:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:10:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:10:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:10:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:10:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:10:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:10:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:10:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:10:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:10:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:10:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:10:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:10:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:10:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:10:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:10:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:10:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:10:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:10:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:10:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:10:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:10:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:10:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:10:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:10:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:10:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:10:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:10:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:10:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:10:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:10:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:10:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:10:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:10:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:10:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:10:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:10:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:10:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:10:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:10:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:10:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:10:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:10:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:10:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:10:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:11:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:11:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:11:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:11:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:11:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:11:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:11:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:11:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:11:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:11:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:11:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:11:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:11:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:11:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:11:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:11:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:11:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:11:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:11:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:11:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:11:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:11:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:11:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:11:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:11:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:11:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:11:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:11:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:11:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:11:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:11:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:11:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:11:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:11:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:11:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:11:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:11:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:11:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:11:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:11:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:11:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:11:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:11:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:11:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:11:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:11:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:11:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:11:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:11:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:11:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:11:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:11:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:11:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:11:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:11:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:11:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:11:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:11:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:11:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:11:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:11:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:11:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:11:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:11:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:11:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:11:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:11:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:11:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:11:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:11:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:11:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:11:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:11:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:11:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:11:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:11:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:11:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:11:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:11:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:11:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:11:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:11:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:11:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:11:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:11:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:11:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:11:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:11:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:11:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:11:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:11:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:11:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:11:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:11:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:11:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:11:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:11:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:11:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:11:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:11:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:11:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:11:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:11:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:11:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:11:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:11:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:11:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:11:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:11:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:11:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:11:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:11:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:11:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:11:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:11:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:11:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:11:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:11:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:11:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:11:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:11:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:11:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:11:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:11:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:11:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:11:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:11:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:11:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:11:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:11:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:11:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:11:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:11:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:11:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:11:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:11:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:11:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:11:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:11:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:11:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:11:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:11:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:11:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:11:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:11:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:11:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:11:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:11:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:11:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:11:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:11:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:11:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:11:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:11:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:11:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:11:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:11:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:11:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:11:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:11:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:11:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:11:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:11:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:11:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:11:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:11:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:11:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:11:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:11:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:11:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:11:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:11:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:11:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:11:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:11:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:11:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:11:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:11:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:11:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:11:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:11:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:11:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:11:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:11:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:11:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:11:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:11:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:11:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:11:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:11:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:11:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:11:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:11:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:11:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:11:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:11:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:11:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:11:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:11:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:11:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:11:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:11:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:11:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:11:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:11:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:11:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:11:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:11:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:11:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:11:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:11:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:11:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:11:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:11:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:11:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:11:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:12:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:12:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:12:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:12:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:12:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:12:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:12:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:12:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:12:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:12:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:12:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:12:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:12:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:12:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:12:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:12:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:12:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:12:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:12:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:12:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:12:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:12:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:12:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:12:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:12:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:12:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:12:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:12:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:12:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:12:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:12:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:12:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:12:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:12:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:12:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:12:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:12:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:12:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:12:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:12:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:12:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:12:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:12:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:12:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:12:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:12:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:12:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:12:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:12:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:12:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:12:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:12:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:12:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:12:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:12:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:12:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:12:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:12:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:12:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:12:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:12:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:12:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:12:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:12:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:12:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:12:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:12:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:12:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:12:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:12:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:12:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:12:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:12:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:12:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:12:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:12:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:12:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:12:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:12:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:12:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:12:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:12:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:12:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:12:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:12:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:12:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:12:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:12:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:12:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:12:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:12:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:12:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:12:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:12:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:12:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:12:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:12:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:12:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:12:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:12:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:12:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:12:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:12:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:12:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:12:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:12:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:12:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:12:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:12:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:12:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:12:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:12:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:12:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:12:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:12:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:12:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:12:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:12:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:12:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:12:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:12:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:12:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:12:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:12:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:12:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:12:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:12:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:12:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:12:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:12:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:12:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:12:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:12:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:12:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:12:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:12:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:12:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:12:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:12:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:12:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:12:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:12:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:12:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:12:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:12:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:12:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:12:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:12:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:12:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:12:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:12:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:12:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:12:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:12:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:12:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:12:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:12:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:12:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:12:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:12:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:12:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:12:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:12:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:12:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:12:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:12:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:12:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:12:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:12:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:12:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:12:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:12:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:12:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:12:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:12:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:12:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:12:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:12:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:12:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:12:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:12:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:12:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:12:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:12:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:12:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:12:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:12:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:12:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:12:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:12:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:12:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:12:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:12:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:12:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:12:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:12:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:12:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:12:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:12:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:12:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:12:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:12:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:12:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:12:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:12:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:12:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:12:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:12:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:12:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:12:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:12:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:12:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:12:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:12:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:12:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:12:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:13:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:13:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:13:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:13:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:13:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:13:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:13:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:13:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:13:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:13:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:13:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:13:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:13:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:13:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:13:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:13:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:13:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:13:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:13:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:13:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:13:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:13:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:13:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:13:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:13:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:13:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:13:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:13:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:13:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:13:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:13:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:13:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:13:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:13:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:13:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:13:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:13:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:13:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:13:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:13:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:13:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:13:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:13:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:13:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:13:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:13:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:13:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:13:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:13:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:13:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:13:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:13:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:13:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:13:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:13:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:13:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:13:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:13:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:13:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:13:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:13:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:13:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:13:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:13:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:13:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:13:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:13:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:13:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:13:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:13:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:13:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:13:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:13:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:13:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:13:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:13:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:13:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:13:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:13:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:13:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:13:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:13:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:13:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:13:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:13:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:13:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:13:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:13:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:13:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:13:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:13:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:13:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:13:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:13:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:13:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:13:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:13:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:13:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:13:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:13:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:13:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:13:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:13:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:13:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:13:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:13:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:13:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:13:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:13:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:13:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:13:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:13:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:13:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:13:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:13:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:13:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:13:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:13:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:13:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:13:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:13:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:13:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:13:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:13:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:13:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:13:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:13:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:13:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:13:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:13:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:13:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:13:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:13:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:13:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:13:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:13:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:13:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:13:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:13:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:13:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:13:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:13:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:13:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:13:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:13:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:13:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:13:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:13:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:13:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:13:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:13:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:13:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:13:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:13:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:13:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:13:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:13:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:13:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:13:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:13:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:13:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:13:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:13:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:13:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:13:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:13:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:13:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:13:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:13:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:13:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:13:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:13:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:13:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:13:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:13:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:13:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:13:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:13:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:13:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:13:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:13:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:13:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:13:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:13:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:13:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:13:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:13:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:13:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:13:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:13:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:13:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:13:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:13:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:13:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:13:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:13:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:13:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:13:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:13:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:13:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:13:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:13:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:13:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:13:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:13:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:13:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:13:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:13:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:13:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:13:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:13:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:13:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:13:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:13:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:13:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:13:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:14:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:14:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:14:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:14:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:14:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:14:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:14:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:14:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:14:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:14:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:14:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:14:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:14:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:14:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:14:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:14:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:14:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:14:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:14:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:14:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:14:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:14:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:14:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:14:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:14:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:14:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:14:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:14:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:14:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:14:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:14:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:14:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:14:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:14:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:14:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:14:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:14:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:14:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:14:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:14:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:14:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:14:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:14:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:14:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:14:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:14:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:14:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:14:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:14:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:14:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:14:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:14:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:14:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:14:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:14:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:14:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:14:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:14:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:14:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:14:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:14:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:14:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:14:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:14:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:14:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:14:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:14:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:14:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:14:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:14:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:14:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:14:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:14:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:14:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:14:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:14:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:14:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:14:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:14:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:14:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:14:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:14:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:14:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:14:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:14:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:14:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:14:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:14:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:14:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:14:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:14:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:14:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:14:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:14:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:14:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:14:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:14:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:14:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:14:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:14:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:14:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:14:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:14:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:14:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:14:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:14:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:14:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:14:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:14:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:14:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:14:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:14:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:14:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:14:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:14:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:14:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:14:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:14:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:14:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:14:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:14:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:14:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:14:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:14:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:14:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:14:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:14:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:14:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:14:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:14:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:14:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:14:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:14:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:14:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:14:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:14:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:14:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:14:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:14:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:14:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:14:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:14:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:14:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:14:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:14:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:14:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:14:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:14:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:14:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:14:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:14:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:14:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:14:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:14:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:14:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:14:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:14:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:14:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:14:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:14:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:14:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:14:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:14:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:14:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:14:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:14:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:14:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:14:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:14:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:14:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:14:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:14:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:14:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:14:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:14:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:14:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:14:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:14:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:14:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:14:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:14:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:14:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:14:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:14:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:14:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:14:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:14:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:14:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:14:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:14:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:14:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:14:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:14:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:14:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:14:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:14:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:14:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:14:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:14:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:14:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:14:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:14:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:14:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:14:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:14:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:14:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:14:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:14:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:14:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:14:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:14:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:14:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:14:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:14:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:14:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:14:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:15:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:15:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:15:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:15:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:15:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:15:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:15:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:15:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:15:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:15:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:15:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:15:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:15:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:15:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:15:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:15:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:15:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:15:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:15:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:15:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:15:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:15:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:15:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:15:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:15:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:15:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:15:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:15:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:15:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:15:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:15:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:15:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:15:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:15:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:15:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:15:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:15:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:15:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:15:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:15:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:15:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:15:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:15:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:15:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:15:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:15:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:15:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:15:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:15:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:15:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:15:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:15:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:15:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:15:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:15:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:15:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:15:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:15:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:15:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:15:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:15:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:15:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:15:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:15:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:15:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:15:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:15:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:15:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:15:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:15:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:15:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:15:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:15:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:15:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:15:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:15:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:15:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:15:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:15:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:15:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:15:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:15:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:15:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:15:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:15:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:15:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:15:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:15:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:15:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:15:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:15:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:15:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:15:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:15:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:15:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:15:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:15:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:15:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:15:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:15:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:15:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:15:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:15:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:15:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:15:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:15:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:15:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:15:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:15:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:15:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:15:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:15:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:15:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:15:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:15:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:15:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:15:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:15:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:15:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:15:32 rdma06 systemd-journald[790]: Data hash table of /run/log/journal/95379d2f723847b68d9e976dee8967a9/system.journal has a fill level at 75.0 (163751 of 218332 items, 75497472 file size, 461 bytes per hash table item), suggesting rotation.
Apr 17 16:15:32 rdma06 systemd-journald[790]: /run/log/journal/95379d2f723847b68d9e976dee8967a9/system.journal: Journal header limits reached or header out-of-date, rotating.
Apr 17 16:15:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:15:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:15:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:15:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:15:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:15:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:15:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:15:32 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 17 16:15:32 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 17 16:15:33 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 17 16:15:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:15:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:15:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:15:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:15:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:15:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:15:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:15:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:15:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:15:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:15:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:15:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:15:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:15:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:15:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:15:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:15:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:15:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:15:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:15:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:15:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:15:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:15:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:15:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:15:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:15:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:15:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:15:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:15:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:15:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:15:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:15:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:15:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:15:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:15:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:15:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:15:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:15:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:15:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:15:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:15:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:15:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:15:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:15:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:15:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:15:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:15:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:15:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:15:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:15:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:15:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:15:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:15:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:15:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:15:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:15:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:15:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:15:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:15:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:15:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:15:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:15:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:15:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:15:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:15:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:15:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:15:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:15:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:15:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:15:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:15:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:15:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:17:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:17:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:17:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:17:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:17:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:17:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:17:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:17:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:17:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:17:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:17:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:17:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:17:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:17:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:17:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:17:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:17:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:17:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:17:58 rdma06 rsyslogd[1041]: imjournal: 5087 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 16:18:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:18:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:18:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:18:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:18:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:18:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:18:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:18:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:18:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:18:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:18:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:18:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:18:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:18:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:18:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:18:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:18:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:18:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:18:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:18:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:18:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:18:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:18:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:18:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:18:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:18:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:18:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:18:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:18:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:18:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:18:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:18:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:18:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:18:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:18:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:18:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:18:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:18:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:18:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:18:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:18:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:18:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:18:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:18:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:18:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:18:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:18:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:18:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:18:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:18:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:18:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:18:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:18:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:18:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:18:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:18:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:18:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:18:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:18:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:18:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:18:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:18:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:18:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:18:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:18:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:18:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:18:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:18:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:18:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:18:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:18:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:18:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:18:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:18:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:18:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:18:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:18:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:18:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:18:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:18:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:18:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:18:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:18:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:18:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:18:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:18:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:18:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:18:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:18:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:18:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:18:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:18:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:18:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:18:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:18:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:18:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:18:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:18:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:18:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:18:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:18:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:18:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:18:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:18:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:18:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:18:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:18:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:18:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:18:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:18:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:18:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:18:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:18:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:18:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:18:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:18:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:18:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:18:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:18:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:18:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:18:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:18:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:18:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:18:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:18:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:18:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:18:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:18:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:18:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:18:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:18:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:18:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:18:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:18:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:18:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:18:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:18:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:18:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:18:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:18:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:18:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:18:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:18:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:18:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:18:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:18:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:18:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:18:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:18:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:18:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:18:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:18:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:18:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:18:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:18:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:18:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:18:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:18:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:18:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:18:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:18:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:18:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:18:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:18:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:18:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:18:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:18:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:18:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:18:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:18:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:18:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:18:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:18:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:18:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:18:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:18:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:18:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:18:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:18:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:18:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:18:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:18:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:18:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:18:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:18:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:18:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:18:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:18:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:18:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:18:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:18:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:18:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:18:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:18:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:18:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:18:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:18:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:18:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:18:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:18:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:18:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:18:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:18:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:18:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:18:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:18:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:18:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:18:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:18:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:18:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:18:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:18:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:18:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:18:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:18:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:18:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:19:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:19:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:19:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:19:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:19:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:19:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:19:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:19:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:19:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:19:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:19:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:19:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:19:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:19:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:19:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:19:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:19:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:19:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:19:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:19:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:19:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:19:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:19:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:19:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:19:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:19:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:19:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:19:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:19:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:19:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:19:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:19:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:19:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:19:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:19:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:19:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:19:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:19:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:19:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:19:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:19:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:19:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:19:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:19:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:19:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:19:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:19:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:19:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:19:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:19:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:19:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:19:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:19:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:19:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:19:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:19:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:19:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:19:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:19:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:19:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:19:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:19:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:19:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:19:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:19:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:19:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:19:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:19:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:19:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:19:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:19:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:19:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:19:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:19:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:19:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:19:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:19:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:19:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:19:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:19:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:19:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:19:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:19:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:19:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:19:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:19:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:19:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:19:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:19:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:19:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:19:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:19:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:19:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:19:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:19:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:19:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:19:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:19:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:19:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:19:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:19:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:19:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:19:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:19:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:19:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:19:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:19:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:19:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:19:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:19:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:19:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:19:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:19:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:19:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:19:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:19:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:19:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:19:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:19:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:19:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:19:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:19:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:19:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:19:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:19:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:19:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:19:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:19:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:19:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:19:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:19:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:19:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:19:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:19:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:19:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:19:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:19:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:19:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:19:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:19:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:19:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:19:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:19:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:19:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:19:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:19:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:19:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:19:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:19:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:19:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:19:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:19:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:19:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:19:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:19:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:19:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:19:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:19:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:19:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:19:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:19:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:19:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:19:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:19:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:19:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:19:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:19:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:19:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:19:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:19:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:19:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:19:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:19:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:19:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:19:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:19:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:19:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:19:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:19:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:19:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:19:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:19:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:19:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:19:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:19:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:19:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:19:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:19:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:19:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:19:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:19:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:19:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:19:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:19:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:19:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:19:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:19:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:19:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:19:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:19:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:19:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:19:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:19:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:19:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:19:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:19:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:19:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:19:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:19:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:19:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:19:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:19:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:19:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:19:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:19:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:19:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:20:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:20:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:20:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:20:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:20:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:20:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:20:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:20:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:20:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:20:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:20:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:20:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:20:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:20:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:20:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:20:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:20:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:20:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:20:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:20:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:20:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:20:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:20:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:20:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:20:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:20:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:20:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:20:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:20:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:20:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:20:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:20:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:20:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:20:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:20:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:20:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:20:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:20:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:20:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:20:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:20:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:20:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:20:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:20:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:20:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:20:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:20:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:20:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:20:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:20:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:20:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:20:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:20:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:20:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:20:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:20:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:20:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:20:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:20:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:20:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:20:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:20:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:20:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:20:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:20:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:20:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:20:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:20:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:20:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:20:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:20:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:20:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:20:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:20:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:20:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:20:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:20:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:20:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:20:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:20:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:20:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:20:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:20:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:20:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:20:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:20:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:20:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:20:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:20:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:20:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:20:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:20:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:20:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:20:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:20:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:20:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:20:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:20:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:20:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:20:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:20:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:20:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:20:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:20:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:20:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:20:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:20:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:20:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:20:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:20:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:20:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:20:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:20:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:20:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:20:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:20:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:20:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:20:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:20:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:20:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:20:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:20:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:20:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:20:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:20:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:20:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:20:32 rdma06 NetworkManager[1025]: <info>  [1681762832.9518] dhcp4 (eno1): state changed new lease, address=10.73.131.45
Apr 17 16:20:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:20:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:20:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:20:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:20:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:20:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:20:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:20:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:20:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:20:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:20:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:20:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:20:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:20:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:20:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:20:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:20:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:20:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:20:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:20:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:20:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:20:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:20:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:20:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:20:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:20:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:20:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:20:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:20:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:20:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:20:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:20:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:20:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:20:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:20:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:20:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:20:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:20:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:20:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:20:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:20:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:20:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:20:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:20:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:20:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:20:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:20:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:20:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:20:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:20:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:20:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:20:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:20:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:20:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:20:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:20:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:20:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:20:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:20:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:20:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:20:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:20:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:20:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:20:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:20:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:20:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:20:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:20:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:20:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:20:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:20:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:20:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:20:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:20:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:20:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:20:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:20:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:20:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:20:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:20:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:20:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:20:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:20:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:20:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:20:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:20:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:20:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:20:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:20:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:20:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:21:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:21:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:21:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:21:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:21:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:21:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:21:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:21:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:21:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:21:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:21:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:21:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:21:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:21:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:21:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:21:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:21:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:21:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:21:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:21:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:21:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:21:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:21:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:21:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:21:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:21:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:21:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:21:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:21:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:21:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:21:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:21:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:21:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:21:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:21:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:21:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:21:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:21:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:21:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:21:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:21:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:21:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:21:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:21:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:21:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:21:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:21:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:21:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:21:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:21:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:21:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:21:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:21:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:21:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:21:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:21:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:21:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:21:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:21:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:21:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:21:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:21:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:21:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:21:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:21:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:21:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:21:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:21:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:21:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:21:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:21:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:21:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:21:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:21:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:21:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:21:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:21:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:21:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:21:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:21:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:21:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:21:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:21:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:21:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:21:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:21:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:21:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:21:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:21:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:21:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:21:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:21:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:21:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:21:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:21:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:21:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:21:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:21:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:21:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:21:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:21:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:21:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:21:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:21:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:21:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:21:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:21:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:21:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:21:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:21:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:21:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:21:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:21:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:21:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:21:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:21:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:21:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:21:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:21:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:21:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:21:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:21:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:21:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:21:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:21:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:21:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:21:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:21:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:21:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:21:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:21:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:21:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:21:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:21:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:21:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:21:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:21:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:21:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:21:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:21:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:21:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:21:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:21:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:21:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:21:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:21:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:21:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:21:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:21:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:21:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:21:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:21:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:21:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:21:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:21:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:21:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:21:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:21:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:21:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:21:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:21:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:21:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:21:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:21:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:21:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:21:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:21:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:21:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:21:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:21:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:21:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:21:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:21:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:21:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:21:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:21:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:21:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:21:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:21:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:21:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:21:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:21:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:21:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:21:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:21:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:21:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:21:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:21:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:21:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:21:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:21:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:21:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:21:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:21:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:21:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:21:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:21:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:21:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:21:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:21:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:21:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:21:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:21:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:21:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:21:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:21:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:22:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:22:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:22:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:22:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:22:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:22:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:22:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:22:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:22:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:22:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:22:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:22:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:22:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:22:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:22:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:22:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:22:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:22:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:22:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:22:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:22:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:22:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:22:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:22:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:22:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:22:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:22:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:22:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:22:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:22:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:22:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:22:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:22:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:22:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:22:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:22:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:22:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:22:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:22:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:22:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:22:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:22:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:22:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:22:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:22:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:22:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:22:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:22:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:22:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:22:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:22:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:22:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:22:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:22:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:22:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:22:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:22:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:22:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:22:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:22:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:22:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:22:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:22:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:22:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:22:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:22:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:22:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:22:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:22:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:22:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:22:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:22:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:22:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:22:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:22:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:22:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:22:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:22:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:22:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:22:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:22:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:22:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:22:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:22:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:22:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:22:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:22:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:22:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:22:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:22:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:22:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:22:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:22:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:22:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:22:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:22:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:22:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:22:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:22:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:22:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:22:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:22:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:22:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:22:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:22:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:22:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:22:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:22:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:22:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:22:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:22:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:22:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:22:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:22:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:22:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:22:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:22:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:22:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:22:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:22:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:22:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:22:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:22:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:22:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:22:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:22:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:22:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:22:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:22:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:22:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:22:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:22:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:22:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:22:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:22:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:22:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:22:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:22:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:22:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:22:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:22:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:22:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:22:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:22:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:22:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:22:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:22:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:22:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:22:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:22:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:22:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:22:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:22:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:22:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:22:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:22:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:22:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:22:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:22:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:22:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:22:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:22:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:22:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:22:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:22:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:22:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:22:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:22:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:22:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:22:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:22:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:22:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:22:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:22:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:22:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:22:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:22:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:22:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:22:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:22:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:22:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:22:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:22:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:22:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:22:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:22:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:22:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:22:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:22:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:22:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:22:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:22:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:22:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:22:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:22:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:22:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:23:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:23:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:23:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:23:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:23:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:23:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:23:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:23:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:23:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:23:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:23:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:23:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:23:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:23:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:23:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:23:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:23:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:23:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:23:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:23:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:23:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:23:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:23:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:23:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:23:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:23:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:23:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:23:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:23:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:23:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:23:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:23:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:23:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:23:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:23:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:23:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:23:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:23:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:23:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:23:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:23:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:23:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:23:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:23:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:23:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:23:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:23:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:23:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:23:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:23:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:23:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:23:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:23:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:23:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:23:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:23:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:23:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:23:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:23:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:23:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:23:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:23:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:23:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:23:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:23:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:23:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:23:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:23:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:23:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:23:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:23:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:23:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:23:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:23:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:23:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:23:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:23:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:23:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:23:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:23:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:23:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:23:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:23:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:23:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:23:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:23:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:23:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:23:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:23:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:23:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:23:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:23:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:23:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:23:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:23:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:23:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:23:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:23:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:23:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:23:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:23:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:23:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:23:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:23:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:23:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:23:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:23:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:23:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:23:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:23:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:23:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:23:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:23:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:23:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:23:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:23:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:23:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:23:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:23:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:23:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:23:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:23:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:23:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:23:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:23:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:23:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:23:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:23:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:23:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:23:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:23:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:23:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:23:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:23:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:23:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:23:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:23:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:23:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:23:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:23:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:23:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:23:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:23:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:23:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:23:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:23:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:23:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:23:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:23:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:23:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:23:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:23:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:23:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:23:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:23:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:23:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:23:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:23:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:23:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:23:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:23:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:23:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:23:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:23:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:23:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:23:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:24:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:24:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:24:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:24:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:24:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:24:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:24:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:24:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:24:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:24:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:24:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:24:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:24:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:24:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:24:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:24:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:24:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:24:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:24:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:24:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:24:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:24:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:24:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:24:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:24:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:24:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:24:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:24:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:24:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:24:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:24:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:24:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:24:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:24:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:24:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:24:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:24:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:24:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:24:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:24:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:24:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:24:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:24:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:24:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:24:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:24:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:24:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:24:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:24:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:24:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:24:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:24:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:24:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:24:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:24:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:24:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:24:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:24:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:24:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:24:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:24:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:24:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:24:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:24:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:24:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:24:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:24:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:24:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:24:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:24:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:24:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:24:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:24:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:24:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:24:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:24:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:24:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:24:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:24:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:24:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:24:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:24:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:24:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:24:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:24:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:24:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:24:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:24:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:24:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:24:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:24:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:24:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:24:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:24:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:24:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:24:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:24:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:24:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:24:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:24:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:24:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:24:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:24:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:24:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:24:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:24:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:24:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:24:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:24:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:24:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:24:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:24:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:24:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:24:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:24:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:24:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:24:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:24:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:24:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:24:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:24:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:24:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:24:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:24:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:24:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:24:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:24:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:24:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:24:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:24:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:24:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:24:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:24:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:24:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:24:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:24:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:24:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:24:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:24:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:24:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:24:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:24:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:24:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:24:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:24:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:24:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:24:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:24:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:24:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:24:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:24:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:24:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:24:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:24:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:24:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:24:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:24:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:24:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:24:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:24:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:24:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:24:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:24:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:24:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:24:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:24:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:24:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:24:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:24:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:24:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:24:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:24:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:24:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:24:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:24:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:24:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:24:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:24:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:24:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:24:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:24:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:24:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:24:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:24:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:24:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:24:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:24:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:24:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:24:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:24:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:24:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:24:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:24:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:24:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:24:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:24:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:25:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:25:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:25:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:25:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:25:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:25:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:25:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:25:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:25:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:25:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:25:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:25:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:25:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:25:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:25:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:25:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:25:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:25:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:25:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:25:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:25:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:25:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:25:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:25:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:25:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:25:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:25:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:25:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:25:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:25:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:25:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:25:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:25:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:25:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:25:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:25:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:25:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:25:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:25:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:25:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:25:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:25:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:25:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:25:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:25:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:25:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:25:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:25:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:25:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:25:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:25:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:25:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:25:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:25:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:25:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:25:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:25:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:25:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:25:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:25:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:25:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:25:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:25:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:25:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:25:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:25:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:25:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:25:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:25:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:25:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:25:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:25:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:25:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:25:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:25:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:25:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:25:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:25:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:25:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:25:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:25:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:25:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:25:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:25:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:25:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:25:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:25:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:25:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:25:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:25:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:25:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:25:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:25:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:25:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:25:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:25:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:25:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:25:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:25:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:25:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:25:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:25:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:25:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:25:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:25:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:25:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:25:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:25:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:25:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:25:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:25:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:25:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:25:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:25:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:25:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:25:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:25:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:25:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:25:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:25:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:25:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:25:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:25:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:25:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:25:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:25:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:25:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:25:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:25:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:25:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:25:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:25:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:25:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:25:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:25:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:25:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:25:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:25:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:25:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:25:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:25:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:25:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:25:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:25:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:25:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:25:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:25:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:25:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:28:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:28:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:28:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:28:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:28:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:28:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:28:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:28:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:28:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:28:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:28:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:28:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:28:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:28:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:28:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:28:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:28:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:28:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:28:03 rdma06 rsyslogd[1041]: imjournal: 5292 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 16:28:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:28:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:28:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:28:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:28:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:28:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:28:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:28:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:28:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:28:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:28:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:28:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:28:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:28:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:28:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:28:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:28:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:28:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:28:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:28:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:28:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:28:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:28:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:28:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:28:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:28:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:28:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:28:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:28:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:28:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:28:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:28:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:28:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:28:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:28:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:28:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:28:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:28:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:28:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:28:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:28:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:28:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:28:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:28:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:28:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:28:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:28:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:28:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:28:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:28:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:28:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:28:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:28:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:28:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:28:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:28:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:28:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:28:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:28:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:28:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:28:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:28:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:28:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:28:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:28:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:28:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:28:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:28:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:28:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:28:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:28:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:28:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:28:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:28:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:28:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:28:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:28:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:28:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:28:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:28:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:28:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:28:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:28:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:28:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:28:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:28:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:28:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:28:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:28:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:28:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:28:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:28:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:28:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:28:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:28:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:28:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:28:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:28:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:28:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:28:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:28:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:28:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:28:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:28:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:28:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:28:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:28:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:28:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:28:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:28:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:28:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:28:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:28:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:28:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:28:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:28:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:28:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:28:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:28:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:28:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:28:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:28:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:28:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:28:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:28:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:28:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:28:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:28:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:28:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:28:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:28:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:28:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:28:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:28:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:28:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:28:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:28:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:28:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:28:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:28:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:28:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:28:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:28:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:28:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:28:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:28:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:28:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:28:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:28:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:28:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:28:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:28:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:28:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:28:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:28:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:28:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:28:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:28:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:28:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:28:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:28:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:28:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:28:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:28:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:28:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:28:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:28:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:28:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:28:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:28:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:28:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:28:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:28:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:28:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:28:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:28:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:28:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:28:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:28:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:28:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:28:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:28:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:28:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:28:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:28:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:28:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:28:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:28:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:28:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:28:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:28:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:28:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:28:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:28:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:28:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:28:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:28:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:28:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:29:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:29:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:29:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:29:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:29:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:29:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:29:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:29:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:29:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:29:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:29:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:29:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:29:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:29:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:29:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:29:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:29:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:29:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:29:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:29:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:29:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:29:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:29:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:29:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:29:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:29:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:29:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:29:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:29:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:29:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:29:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:29:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:29:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:29:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:29:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:29:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:29:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:29:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:29:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:29:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:29:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:29:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:29:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:29:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:29:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:29:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:29:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:29:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:29:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:29:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:29:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:29:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:29:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:29:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:29:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:29:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:29:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:29:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:29:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:29:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:29:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:29:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:29:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:29:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:29:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:29:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:29:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:29:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:29:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:29:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:29:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:29:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:29:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:29:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:29:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:29:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:29:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:29:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:29:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:29:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:29:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:29:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:29:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:29:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:29:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:29:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:29:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:29:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:29:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:29:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:29:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:29:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:29:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:29:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:29:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:29:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:29:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:29:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:29:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:29:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:29:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:29:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:29:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:29:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:29:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:29:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:29:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:29:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:29:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:29:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:29:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:29:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:29:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:29:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:29:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:29:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:29:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:29:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:29:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:29:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:29:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:29:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:29:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:29:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:29:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:29:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:29:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:29:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:29:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:29:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:29:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:29:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:29:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:29:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:29:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:29:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:29:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:29:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:29:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:29:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:29:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:29:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:29:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:29:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:29:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:29:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:29:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:29:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:29:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:29:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:29:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:29:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:29:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:29:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:29:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:29:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:29:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:29:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:29:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:29:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:29:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:29:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:29:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:29:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:29:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:29:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:29:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:29:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:29:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:29:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:29:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:29:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:29:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:29:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:29:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:29:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:29:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:29:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:29:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:29:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:29:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:29:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:29:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:29:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:29:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:29:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:29:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:29:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:29:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:29:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:29:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:29:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:29:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:29:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:29:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:29:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:29:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:29:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:29:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:29:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:29:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:29:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:29:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:29:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:29:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:29:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:29:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:29:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:29:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:29:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:29:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:29:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:29:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:29:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:29:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:29:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:30:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:30:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:30:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:30:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:30:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:30:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:30:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:30:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:30:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:30:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:30:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:30:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:30:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:30:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:30:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:30:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:30:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:30:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:30:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:30:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:30:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:30:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:30:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:30:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:30:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:30:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:30:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:30:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:30:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:30:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:30:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:30:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:30:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:30:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:30:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:30:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:30:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:30:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:30:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:30:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:30:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:30:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:30:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:30:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:30:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:30:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:30:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:30:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:30:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:30:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:30:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:30:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:30:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:30:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:30:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:30:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:30:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:30:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:30:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:30:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:30:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:30:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:30:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:30:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:30:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:30:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:30:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:30:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:30:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:30:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:30:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:30:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:30:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:30:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:30:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:30:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:30:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:30:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:30:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:30:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:30:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:30:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:30:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:30:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:30:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:30:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:30:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:30:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:30:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:30:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:30:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:30:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:30:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:30:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:30:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:30:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:30:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:30:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:30:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:30:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:30:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:30:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:30:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:30:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:30:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:30:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:30:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:30:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:30:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:30:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:30:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:30:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:30:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:30:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:30:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:30:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:31:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:31:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:31:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:31:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:31:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:31:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:31:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:31:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:31:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:31:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:31:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:31:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:31:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:31:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:31:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:31:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:31:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:31:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:31:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:31:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:31:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:31:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:31:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:31:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:31:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:31:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:31:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:31:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:31:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:31:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:31:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:31:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:31:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:31:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:31:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:31:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:31:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:31:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:31:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:31:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:31:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:31:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:31:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:31:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:31:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:31:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:31:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:31:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:31:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:31:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:31:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:31:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:31:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:31:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:31:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:31:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:31:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:31:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:31:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:31:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:31:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:31:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:31:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:31:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:31:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:31:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:31:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:31:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:31:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:31:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:31:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:31:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:31:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:31:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:31:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:31:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:31:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:31:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:31:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:31:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:31:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:31:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:31:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:31:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:31:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:31:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:31:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:31:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:31:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:31:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:31:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:31:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:31:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:31:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:31:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:31:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:31:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:31:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:31:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:31:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:31:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:31:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:31:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:31:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:31:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:31:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:31:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:31:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:31:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:31:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:31:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:31:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:31:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:31:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:31:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:31:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:31:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:31:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:31:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:31:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:31:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:31:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:31:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:31:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:31:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:31:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:31:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:31:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:31:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:31:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:31:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:31:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:31:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:31:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:31:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:31:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:31:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:31:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:31:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:31:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:31:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:31:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:31:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:31:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:31:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:31:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:31:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:31:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:31:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:31:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:31:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:31:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:31:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:31:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:31:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:31:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:31:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:31:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:31:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:31:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:31:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:31:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:31:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:31:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:31:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:31:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:31:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:31:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:31:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:31:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:31:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:31:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:31:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:31:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:31:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:31:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:31:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:31:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:31:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:31:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:31:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:31:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:31:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:31:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:31:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:31:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:31:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:31:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:31:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:31:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:31:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:31:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:31:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:31:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:31:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:31:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:31:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:31:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:31:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:31:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:31:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:31:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:31:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:31:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:31:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:31:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:31:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:31:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:31:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:31:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:31:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:31:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:31:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:31:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:31:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:31:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:32:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:32:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:32:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:32:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:32:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:32:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:32:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:32:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:32:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:32:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:32:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:32:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:32:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:32:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:32:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:32:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:32:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:32:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:32:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:32:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:32:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:32:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:32:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:32:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:32:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:32:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:32:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:32:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:32:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:32:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:32:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:32:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:32:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:32:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:32:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:32:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:32:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:32:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:32:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:32:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:32:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:32:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:32:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:32:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:32:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:32:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:32:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:32:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:32:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:32:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:32:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:32:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:32:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:32:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:32:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:32:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:32:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:32:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:32:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:32:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:32:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:32:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:32:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:32:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:32:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:32:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:32:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:32:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:32:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:32:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:32:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:32:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:32:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:32:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:32:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:32:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:32:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:32:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:32:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:32:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:32:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:32:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:32:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:32:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:32:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:32:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:32:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:32:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:32:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:32:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:32:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:32:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:32:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:32:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:32:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:32:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:32:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:32:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:32:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:32:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:32:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:32:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:32:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:32:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:32:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:32:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:32:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:32:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:32:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:32:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:32:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:32:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:32:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:32:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:32:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:32:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:32:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:32:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:32:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:32:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:32:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:32:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:32:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:32:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:32:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:32:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:32:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:32:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:32:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:32:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:32:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:32:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:32:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:32:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:32:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:32:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:32:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:32:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:32:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:32:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:32:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:32:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:32:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:32:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:32:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:32:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:32:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:32:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:32:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:32:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:32:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:32:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:32:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:32:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:32:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:32:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:32:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:32:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:32:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:32:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:32:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:32:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:32:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:32:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:32:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:32:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:32:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:32:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:32:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:32:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:32:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:32:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:32:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:32:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:32:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:32:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:32:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:32:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:32:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:32:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:32:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:32:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:32:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:32:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:32:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:32:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:32:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:32:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:32:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:32:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:32:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:32:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:32:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:32:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:32:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:32:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:32:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:32:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:32:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:32:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:32:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:32:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:32:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:32:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:32:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:32:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:32:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:32:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:32:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:32:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:32:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:32:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:32:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:32:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:32:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:32:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:33:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:33:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:33:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:33:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:33:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:33:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:33:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:33:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:33:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:33:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:33:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:33:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:33:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:33:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:33:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:33:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:33:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:33:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:33:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:33:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:33:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:33:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:33:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:33:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:33:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:33:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:33:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:33:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:33:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:33:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:33:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:33:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:33:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:33:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:33:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:33:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:33:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:33:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:33:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:33:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:33:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:33:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:33:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:33:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:33:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:33:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:33:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:33:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:33:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:33:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:33:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:33:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:33:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:33:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:33:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:33:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:33:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:33:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:33:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:33:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:33:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:33:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:33:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:33:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:33:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:33:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:33:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:33:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:33:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:33:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:33:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:33:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:33:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:33:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:33:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:33:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:33:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:33:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:33:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:33:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:33:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:33:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:33:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:33:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:33:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:33:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:33:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:33:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:33:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:33:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:33:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:33:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:33:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:33:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:33:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:33:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:33:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:33:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:33:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:33:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:33:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:33:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:33:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:33:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:33:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:33:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:33:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:33:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:33:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:33:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:33:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:33:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:33:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:33:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:33:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:33:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:33:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:33:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:33:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:33:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:33:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:33:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:33:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:33:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:33:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:33:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:33:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:33:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:33:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:33:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:33:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:33:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:33:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:33:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:33:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:33:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:33:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:33:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:33:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:33:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:33:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:33:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:33:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:33:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:33:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:33:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:33:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:33:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:33:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:33:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:33:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:33:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:33:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:33:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:33:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:33:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:33:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:33:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:33:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:33:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:33:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:33:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:33:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:33:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:33:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:33:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:33:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:33:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:33:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:33:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:33:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:33:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:33:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:33:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:33:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:33:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:33:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:33:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:33:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:33:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:33:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:33:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:33:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:33:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:33:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:33:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:33:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:33:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:33:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:33:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:33:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:33:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:33:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:33:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:33:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:33:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:33:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:33:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:33:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:33:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:33:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:33:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:33:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:33:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:33:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:33:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:33:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:33:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:33:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:33:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:33:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:33:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:33:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:33:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:33:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:33:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:34:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:34:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:34:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:34:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:34:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:34:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:34:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:34:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:34:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:34:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:34:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:34:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:34:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:34:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:34:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:34:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:34:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:34:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:34:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:34:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:34:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:34:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:34:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:34:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:34:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:34:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:34:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:34:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:34:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:34:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:34:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:34:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:34:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:34:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:34:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:34:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:34:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:34:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:34:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:34:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:34:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:34:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:34:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:34:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:34:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:34:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:34:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:34:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:34:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:34:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:34:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:34:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:34:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:34:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:34:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:34:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:34:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:34:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:34:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:34:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:34:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:34:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:34:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:34:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:34:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:34:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:34:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:34:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:34:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:34:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:34:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:34:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:34:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:34:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:34:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:34:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:34:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:34:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:34:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:34:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:34:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:34:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:34:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:34:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:34:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:34:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:34:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:34:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:34:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:34:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:34:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:34:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:34:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:34:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:34:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:34:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:34:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:34:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:34:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:34:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:34:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:34:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:34:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:34:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:34:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:34:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:34:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:34:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:34:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:34:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:34:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:34:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:34:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:34:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:34:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:34:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:34:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:34:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:34:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:34:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:34:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:34:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:34:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:34:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:34:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:34:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:34:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:34:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:34:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:34:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:34:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:34:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:34:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:34:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:34:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:34:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:34:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:34:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:34:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:34:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:34:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:34:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:34:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:34:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:34:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:34:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:34:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:34:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:34:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:34:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:34:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:34:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:34:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:34:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:34:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:34:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:34:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:34:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:34:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:34:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:34:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:34:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:34:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:34:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:34:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:34:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:34:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:34:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:34:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:34:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:34:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:34:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:34:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:34:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:34:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:34:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:34:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:34:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:34:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:34:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:34:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:34:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:34:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:34:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:34:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:34:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:34:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:34:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:34:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:34:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:34:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:34:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:34:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:34:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:34:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:34:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:34:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:34:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:34:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:34:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:34:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:34:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:34:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:34:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:34:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:34:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:34:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:34:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:34:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:34:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:34:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:34:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:34:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:34:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:34:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:34:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:35:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:35:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:35:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:35:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:35:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:35:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:35:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:35:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:35:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:35:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:35:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:35:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:35:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:35:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:35:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:35:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:35:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:35:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:35:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:35:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:35:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:35:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:35:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:35:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:35:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:35:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:35:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:35:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:35:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:35:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:35:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:35:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:35:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:35:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:35:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:35:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:35:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:35:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:35:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:35:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:35:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:35:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:35:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:35:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:35:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:35:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:35:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:35:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:35:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:35:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:35:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:35:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:35:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:35:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:35:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:35:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:35:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:35:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:35:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:35:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:35:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:35:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:35:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:35:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:35:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:35:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:35:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:35:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:35:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:35:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:35:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:35:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:35:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:35:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:35:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:35:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:35:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:35:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:35:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:35:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:35:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:35:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:35:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:35:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:35:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:35:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:35:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:35:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:35:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:35:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:35:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:35:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:35:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:35:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:35:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:35:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:35:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:35:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:35:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:35:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:35:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:35:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:35:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:35:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:35:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:35:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:35:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:35:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:35:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:35:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:35:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:35:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:35:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:35:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:35:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:35:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:35:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:35:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:35:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:35:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:35:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:35:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:35:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:35:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:35:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:35:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:35:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:35:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:35:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:35:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:35:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:35:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:35:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:35:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:35:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:35:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:35:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:35:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:35:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:35:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:35:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:35:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:35:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:35:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:35:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:35:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:35:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:35:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:35:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:35:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:35:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:35:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:35:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:35:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:35:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:35:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:35:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:35:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:35:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:35:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:35:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:35:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:35:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:35:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:35:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:35:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:35:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:35:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:35:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:35:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:35:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:35:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:35:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:35:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:35:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:35:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:35:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:35:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:35:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:35:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:35:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:35:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:35:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:35:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:35:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:35:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:35:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:35:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:35:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:35:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:35:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:35:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:35:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:35:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:35:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:35:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:35:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:35:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:35:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:35:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:35:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:35:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:35:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:35:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:35:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:35:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:35:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:35:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:35:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:35:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:35:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:35:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:35:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:35:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:35:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:35:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:38:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:38:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:38:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:38:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:38:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:38:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:38:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:38:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:38:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:38:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:38:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:38:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:38:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:38:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:38:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:38:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:38:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:38:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:38:08 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 16:38:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:38:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:38:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:38:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:38:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:38:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:38:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:38:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:38:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:38:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:38:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:38:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:38:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:38:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:38:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:38:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:38:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:38:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:38:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:38:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:38:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:38:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:38:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:38:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:38:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:38:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:38:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:38:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:38:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:38:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:38:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:38:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:38:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:38:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:38:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:38:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:38:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:38:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:38:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:38:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:38:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:38:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:38:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:38:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:38:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:38:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:38:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:38:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:38:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:38:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:38:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:38:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:38:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:38:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:38:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:38:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:38:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:38:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:38:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:38:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:38:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:38:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:38:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:38:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:38:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:38:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:38:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:38:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:38:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:38:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:38:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:38:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:38:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:38:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:38:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:38:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:38:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:38:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:38:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:38:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:38:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:38:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:38:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:38:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:38:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:38:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:38:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:38:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:38:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:38:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:38:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:38:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:38:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:38:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:38:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:38:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:38:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:38:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:38:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:38:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:38:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:38:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:38:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:38:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:38:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:38:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:38:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:38:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:38:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:38:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:38:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:38:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:38:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:38:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:38:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:38:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:38:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:38:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:38:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:38:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:38:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:38:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:38:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:38:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:38:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:38:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:38:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:38:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:38:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:38:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:38:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:38:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:38:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:38:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:38:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:38:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:38:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:38:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:38:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:38:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:38:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:38:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:38:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:38:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:38:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:38:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:38:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:38:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:38:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:38:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:38:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:38:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:38:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:38:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:38:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:38:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:38:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:38:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:38:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:38:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:38:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:38:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:38:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:38:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:38:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:38:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:38:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:38:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:38:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:38:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:38:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:38:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:38:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:38:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:38:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:38:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:38:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:38:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:38:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:38:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:39:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:39:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:39:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:39:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:39:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:39:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:39:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:39:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:39:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:39:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:39:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:39:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:39:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:39:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:39:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:39:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:39:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:39:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:39:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:39:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:39:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:39:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:39:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:39:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:39:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:39:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:39:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:39:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:39:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:39:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:39:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:39:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:39:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:39:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:39:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:39:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:39:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:39:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:39:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:39:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:39:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:39:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:39:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:39:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:39:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:39:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:39:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:39:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:39:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:39:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:39:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:39:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:39:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:39:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:39:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:39:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:39:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:39:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:39:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:39:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:39:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:39:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:39:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:39:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:39:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:39:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:39:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:39:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:39:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:39:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:39:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:39:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:39:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:39:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:39:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:39:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:39:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:39:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:39:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:39:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:39:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:39:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:39:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:39:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:39:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:39:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:39:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:39:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:39:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:39:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:39:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:39:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:39:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:39:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:39:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:39:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:39:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:39:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:39:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:39:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:39:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:39:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:39:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:39:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:39:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:39:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:39:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:39:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:39:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:39:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:39:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:39:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:39:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:39:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:39:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:39:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:39:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:39:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:39:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:39:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:39:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:39:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:39:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:39:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:39:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:39:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:39:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:39:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:39:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:39:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:39:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:39:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:39:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:39:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:39:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:39:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:39:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:39:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:39:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:39:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:39:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:39:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:39:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:39:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:39:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:39:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:39:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:39:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:39:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:39:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:39:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:39:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:39:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:39:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:39:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:39:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:39:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:39:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:39:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:39:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:39:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:39:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:39:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:39:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:39:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:39:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:39:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:39:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:39:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:39:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:39:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:39:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:39:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:39:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:39:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:39:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:39:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:39:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:39:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:39:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:39:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:39:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:39:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:39:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:39:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:39:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:39:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:39:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:39:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:39:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:39:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:39:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:39:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:39:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:39:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:39:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:39:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:39:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:39:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:39:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:39:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:39:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:39:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:39:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:39:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:39:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:39:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:39:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:39:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:39:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:39:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:39:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:39:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:39:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:39:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:39:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:40:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:40:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:40:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:40:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:40:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:40:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:40:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:40:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:40:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:40:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:40:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:40:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:40:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:40:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:40:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:40:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:40:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:40:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:40:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:40:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:40:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:40:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:40:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:40:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:40:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:40:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:40:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:40:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:40:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:40:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:40:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:40:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:40:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:40:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:40:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:40:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:40:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:40:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:40:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:40:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:40:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:40:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:40:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:40:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:40:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:40:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:40:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:40:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:40:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:40:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:40:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:40:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:40:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:40:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:40:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:40:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:40:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:40:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:40:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:40:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:40:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:40:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:40:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:40:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:40:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:40:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:40:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:40:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:40:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:40:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:40:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:40:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:40:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:40:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:40:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:40:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:40:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:40:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:40:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:40:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:40:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:40:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:40:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:40:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:40:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:40:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:40:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:40:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:40:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:40:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:40:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:40:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:40:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:40:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:40:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:40:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:40:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:40:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:40:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:40:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:40:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:40:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:40:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:40:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:40:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:40:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:40:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:40:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:40:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:40:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:40:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:40:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:40:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:40:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:40:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:40:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:40:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:40:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:40:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:40:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:40:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:40:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:40:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:40:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:40:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:40:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:40:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:40:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:40:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:40:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:40:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:40:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:40:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:40:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:40:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:40:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:40:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:40:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:40:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:40:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:40:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:40:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:40:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:40:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:40:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:40:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:40:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:40:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:40:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:40:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:40:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:40:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:40:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:40:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:40:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:40:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:40:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:40:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:40:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:40:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:40:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:40:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:40:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:40:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:40:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:40:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:40:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:40:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:40:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:40:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:40:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:40:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:40:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:40:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:40:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:40:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:40:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:40:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:40:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:40:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:40:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:40:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:40:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:40:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:40:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:40:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:40:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:40:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:40:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:40:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:40:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:40:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:40:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:40:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:40:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:40:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:40:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:40:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:40:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:40:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:40:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:40:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:40:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:40:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:40:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:40:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:40:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:40:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:40:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:40:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:40:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:40:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:40:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:40:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:40:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:40:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:41:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:41:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:41:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:41:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:41:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:41:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:41:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:41:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:41:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:41:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:41:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:41:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:41:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:41:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:41:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:41:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:41:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:41:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:41:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:41:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:41:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:41:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:41:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:41:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:41:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:41:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:41:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:41:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:41:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:41:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:41:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:41:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:41:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:41:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:41:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:41:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:41:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:41:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:41:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:41:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:41:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:41:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:41:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:41:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:41:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:41:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:41:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:41:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:41:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:41:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:41:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:41:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:41:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:41:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:41:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:41:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:41:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:41:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:41:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:41:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:41:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:41:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:41:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:41:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:41:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:41:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:41:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:41:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:41:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:41:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:41:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:41:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:41:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:41:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:41:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:41:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:41:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:41:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:41:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:41:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:41:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:41:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:41:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:41:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:41:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:41:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:41:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:41:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:41:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:41:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:41:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:41:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:41:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:41:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:41:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:41:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:41:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:41:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:41:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:41:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:41:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:41:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:41:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:41:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:41:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:41:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:41:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:41:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:41:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:41:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:41:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:41:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:41:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:41:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:41:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:41:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:41:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:41:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:41:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:41:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:41:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:41:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:41:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:41:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:41:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:41:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:41:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:41:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:41:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:41:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:41:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:41:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:41:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:41:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:41:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:41:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:41:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:41:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:41:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:41:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:41:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:41:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:41:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:41:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:41:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:41:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:41:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:41:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:41:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:41:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:41:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:41:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:41:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:41:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:41:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:41:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:41:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:41:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:41:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:41:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:41:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:41:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:41:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:41:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:41:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:41:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:41:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:41:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:41:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:41:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:41:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:41:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:41:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:41:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:41:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:41:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:41:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:41:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:41:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:41:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:41:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:41:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:41:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:41:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:41:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:41:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:41:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:41:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:41:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:41:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:41:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:41:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:41:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:41:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:41:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:41:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:41:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:41:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:41:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:41:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:41:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:41:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:41:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:41:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:41:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:41:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:41:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:41:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:41:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:41:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:41:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:41:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:41:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:41:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:41:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:41:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:42:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:42:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:42:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:42:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:42:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:42:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:42:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:42:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:42:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:42:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:42:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:42:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:42:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:42:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:42:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:42:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:42:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:42:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:42:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:42:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:42:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:42:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:42:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:42:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:42:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:42:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:42:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:42:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:42:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:42:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:42:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:42:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:42:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:42:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:42:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:42:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:42:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:42:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:42:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:42:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:42:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:42:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:42:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:42:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:42:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:42:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:42:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:42:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:42:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:42:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:42:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:42:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:42:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:42:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:42:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:42:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:42:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:42:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:42:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:42:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:42:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:42:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:42:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:42:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:42:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:42:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:42:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:42:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:42:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:42:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:42:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:42:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:42:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:42:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:42:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:42:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:42:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:42:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:42:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:42:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:42:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:42:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:42:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:42:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:42:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:42:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:42:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:42:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:42:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:42:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:42:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:42:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:42:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:42:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:42:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:42:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:42:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:42:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:42:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:42:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:42:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:42:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:42:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:42:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:42:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:42:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:42:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:42:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:42:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:42:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:42:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:42:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:42:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:42:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:42:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:42:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:42:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:42:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:42:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:42:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:42:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:42:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:42:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:42:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:42:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:42:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:42:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:42:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:42:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:42:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:42:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:42:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:42:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:42:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:42:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:42:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:42:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:42:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:42:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:42:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:42:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:42:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:42:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:42:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:42:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:42:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:42:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:42:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:42:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:42:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:42:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:42:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:42:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:42:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:42:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:42:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:42:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:42:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:42:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:42:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:42:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:42:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:42:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:42:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:42:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:42:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:42:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:42:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:42:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:42:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:42:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:42:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:42:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:42:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:42:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:42:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:42:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:42:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:42:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:42:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:42:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:42:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:42:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:42:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:42:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:42:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:42:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:42:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:42:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:42:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:42:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:42:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:42:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:42:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:42:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:42:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:42:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:42:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:42:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:42:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:42:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:42:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:42:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:42:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:42:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:42:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:42:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:42:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:42:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:42:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:42:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:42:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:42:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:42:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:42:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:42:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:43:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:43:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:43:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:43:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:43:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:43:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:43:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:43:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:43:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:43:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:43:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:43:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:43:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:43:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:43:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:43:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:43:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:43:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:43:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:43:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:43:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:43:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:43:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:43:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:43:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:43:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:43:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:43:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:43:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:43:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:43:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:43:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:43:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:43:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:43:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:43:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:43:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:43:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:43:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:43:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:43:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:43:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:43:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:43:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:43:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:43:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:43:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:43:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:43:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:43:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:43:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:43:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:43:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:43:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:43:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:43:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:43:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:43:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:43:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:43:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:43:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:43:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:43:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:43:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:43:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:43:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:43:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:43:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:43:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:43:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:43:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:43:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:43:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:43:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:43:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:43:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:43:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:43:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:43:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:43:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:43:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:43:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:43:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:43:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:43:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:43:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:43:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:43:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:43:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:43:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:43:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:43:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:43:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:43:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:43:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:43:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:43:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:43:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:43:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:43:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:43:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:43:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:43:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:43:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:43:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:43:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:43:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:43:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:43:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:43:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:43:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:43:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:43:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:43:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:43:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:43:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:43:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:43:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:43:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:43:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:43:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:43:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:43:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:43:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:43:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:43:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:43:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:43:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:43:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:43:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:43:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:43:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:43:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:43:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:43:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:43:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:43:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:43:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:43:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:43:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:43:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:43:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:43:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:43:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:43:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:43:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:43:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:43:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:43:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:43:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:43:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:43:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:43:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:43:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:43:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:43:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:43:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:43:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:43:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:43:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:43:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:43:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:43:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:43:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:43:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:43:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:43:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:43:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:43:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:43:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:43:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:43:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:43:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:43:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:43:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:43:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:43:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:43:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:43:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:43:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:43:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:43:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:43:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:43:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:43:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:43:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:43:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:43:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:43:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:43:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:43:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:43:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:43:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:43:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:43:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:43:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:43:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:43:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:43:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:43:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:43:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:43:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:43:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:43:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:43:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:43:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:43:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:43:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:43:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:43:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:43:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:43:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:43:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:43:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:43:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:43:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:44:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:44:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:44:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:44:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:44:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:44:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:44:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:44:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:44:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:44:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:44:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:44:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:44:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:44:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:44:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:44:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:44:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:44:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:44:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:44:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:44:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:44:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:44:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:44:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:44:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:44:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:44:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:44:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:44:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:44:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:44:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:44:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:44:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:44:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:44:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:44:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:44:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:44:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:44:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:44:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:44:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:44:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:44:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:44:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:44:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:44:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:44:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:44:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:44:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:44:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:44:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:44:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:44:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:44:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:44:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:44:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:44:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:44:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:44:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:44:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:44:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:44:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:44:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:44:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:44:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:44:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:44:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:44:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:44:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:44:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:44:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:44:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:44:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:44:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:44:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:44:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:44:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:44:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:44:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:44:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:44:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:44:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:44:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:44:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:44:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:44:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:44:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:44:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:44:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:44:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:44:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:44:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:44:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:44:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:44:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:44:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:44:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:44:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:44:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:44:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:44:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:44:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:44:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:44:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:44:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:44:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:44:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:44:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:44:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:44:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:44:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:44:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:44:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:44:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:44:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:44:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:44:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:44:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:44:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:44:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:44:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:44:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:44:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:44:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:44:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:44:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:44:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:44:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:44:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:44:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:44:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:44:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:44:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:44:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:44:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:44:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:44:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:44:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:44:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:44:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:44:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:44:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:44:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:44:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:44:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:44:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:44:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:44:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:44:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:44:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:44:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:44:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:44:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:44:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:44:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:44:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:44:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:44:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:44:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:44:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:44:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:44:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:44:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:44:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:44:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:44:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:44:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:44:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:44:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:44:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:44:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:44:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:44:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:44:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:44:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:44:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:44:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:44:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:44:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:44:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:44:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:44:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:44:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:44:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:44:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:44:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:44:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:44:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:44:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:44:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:44:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:44:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:44:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:44:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:44:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:44:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:44:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:44:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:44:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:44:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:44:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:44:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:44:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:44:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:44:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:44:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:44:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:44:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:44:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:44:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:44:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:44:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:44:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:44:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:44:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:44:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:45:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:45:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:45:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:45:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:45:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:45:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:45:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:45:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:45:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:45:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:45:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:45:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:45:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:45:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:45:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:45:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:45:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:45:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:45:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:45:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:45:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:45:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:45:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:45:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:45:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:45:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:45:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:45:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:45:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:45:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:45:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:45:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:45:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:45:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:45:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:45:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:45:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:45:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:45:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:45:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:45:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:45:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:45:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:45:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:45:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:45:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:45:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:45:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:45:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:45:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:45:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:45:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:45:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:45:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:45:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:45:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:45:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:45:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:45:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:45:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:45:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:45:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:45:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:45:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:45:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:45:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:45:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:45:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:45:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:45:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:45:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:45:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:45:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:45:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:45:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:45:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:45:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:45:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:45:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:45:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:45:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:45:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:45:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:45:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:45:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:45:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:45:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:45:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:45:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:45:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:45:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:45:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:45:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:45:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:45:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:45:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:45:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:45:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:45:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:45:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:45:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:45:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:45:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:45:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:45:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:45:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:45:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:45:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:45:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:45:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:45:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:45:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:45:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:45:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:45:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:45:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:45:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:45:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:45:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:45:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:45:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:45:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:45:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:45:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:45:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:45:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:45:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:45:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:45:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:45:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:45:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:45:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:45:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:45:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:45:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:45:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:45:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:45:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:45:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:45:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:45:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:45:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:45:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:45:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:45:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:45:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:45:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:45:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:45:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:45:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:45:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:45:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:45:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:45:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:45:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:45:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:45:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:45:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:45:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:45:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:45:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:45:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:45:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:45:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:45:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:45:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:45:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:45:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:45:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:45:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:45:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:45:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:45:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:45:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:45:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:45:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:45:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:45:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:45:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:45:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:45:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:45:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:45:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:45:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:45:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:45:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:45:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:45:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:45:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:45:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:45:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:45:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:45:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:45:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:45:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:45:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:45:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:45:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:45:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:45:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:45:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:45:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:45:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:45:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:45:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:45:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:45:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:45:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:45:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:45:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:45:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:45:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:45:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:45:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:45:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:45:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:46:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:46:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:46:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:46:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:46:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:46:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:46:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:46:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:46:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:46:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:46:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:46:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:46:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:46:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:46:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:46:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:46:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:46:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:48:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:48:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:48:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:48:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:48:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:48:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:48:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:48:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:48:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:48:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:48:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:48:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:48:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:48:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:48:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:48:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:48:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:48:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:48:09 rdma06 rsyslogd[1041]: imjournal: 5082 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 16:48:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:48:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:48:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:48:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:48:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:48:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:48:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:48:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:48:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:48:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:48:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:48:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:48:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:48:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:48:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:48:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:48:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:48:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:48:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:48:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:48:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:48:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:48:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:48:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:48:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:48:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:48:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:48:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:48:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:48:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:48:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:48:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:48:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:48:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:48:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:48:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:48:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:48:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:48:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:48:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:48:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:48:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:48:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:48:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:48:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:48:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:48:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:48:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:48:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:48:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:48:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:48:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:48:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:48:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:48:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:48:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:48:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:48:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:48:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:48:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:48:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:48:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:48:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:48:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:48:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:48:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:48:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:48:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:48:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:48:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:48:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:48:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:48:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:48:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:48:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:48:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:48:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:48:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:48:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:48:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:48:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:48:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:48:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:48:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:48:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:48:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:48:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:48:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:48:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:48:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:48:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:48:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:48:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:48:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:48:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:48:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:48:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:48:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:48:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:48:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:48:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:48:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:48:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:48:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:48:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:48:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:48:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:48:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:48:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:48:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:48:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:48:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:48:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:48:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:48:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:48:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:48:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:48:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:48:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:48:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:48:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:48:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:48:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:48:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:48:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:48:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:48:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:48:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:48:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:48:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:48:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:48:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:48:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:48:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:48:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:48:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:48:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:48:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:48:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:48:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:48:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:48:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:48:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:48:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:48:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:48:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:48:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:48:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:48:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:48:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:48:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:48:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:48:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:48:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:48:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:48:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:48:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:48:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:48:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:48:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:48:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:48:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:48:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:48:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:48:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:48:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:48:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:48:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:48:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:48:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:48:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:48:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:48:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:48:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:48:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:48:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:48:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:48:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:48:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:48:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:49:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:49:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:49:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:49:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:49:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:49:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:49:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:49:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:49:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:49:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:49:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:49:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:49:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:49:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:49:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:49:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:49:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:49:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:49:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:49:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:49:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:49:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:49:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:49:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:49:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:49:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:49:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:49:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:49:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:49:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:49:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:49:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:49:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:49:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:49:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:49:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:49:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:49:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:49:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:49:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:49:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:49:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:49:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:49:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:49:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:49:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:49:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:49:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:49:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:49:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:49:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:49:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:49:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:49:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:49:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:49:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:49:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:49:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:49:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:49:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:49:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:49:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:49:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:49:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:49:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:49:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:49:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:49:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:49:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:49:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:49:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:49:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:49:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:49:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:49:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:49:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:49:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:49:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:49:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:49:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:49:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:49:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:49:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:49:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:49:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:49:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:49:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:49:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:49:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:49:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:49:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:49:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:49:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:49:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:49:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:49:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:49:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:49:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:49:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:49:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:49:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:49:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:49:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:49:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:49:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:49:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:49:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:49:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:49:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:49:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:49:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:49:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:49:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:49:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:49:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:49:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:49:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:49:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:49:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:49:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:49:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:49:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:49:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:49:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:49:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:49:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:49:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:49:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:49:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:49:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:49:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:49:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:49:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:49:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:49:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:49:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:49:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:49:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:49:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:49:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:49:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:49:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:49:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:49:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:49:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:49:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:49:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:49:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:49:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:49:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:49:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:49:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:49:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:49:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:49:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:49:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:49:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:49:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:49:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:49:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:49:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:49:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:49:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:49:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:49:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:49:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:49:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:49:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:49:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:49:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:49:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:49:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:49:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:49:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:49:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:49:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:49:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:49:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:49:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:49:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:49:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:49:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:49:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:49:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:49:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:49:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:49:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:49:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:49:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:49:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:49:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:49:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:49:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:49:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:49:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:49:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:49:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:49:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:49:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:49:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:49:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:49:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:49:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:49:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:49:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:49:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:49:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:49:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:49:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:49:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:49:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:49:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:49:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:49:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:49:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:49:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:50:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:50:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:50:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:50:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:50:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:50:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:50:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:50:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:50:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:50:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:50:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:50:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:50:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:50:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:50:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:50:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:50:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:50:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:50:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:50:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:50:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:50:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:50:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:50:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:50:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:50:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:50:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:50:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:50:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:50:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:50:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:50:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:50:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:50:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:50:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:50:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:50:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:50:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:50:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:50:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:50:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:50:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:50:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:50:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:50:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:50:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:50:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:50:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:50:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:50:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:50:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:50:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:50:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:50:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:50:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:50:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:50:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:50:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:50:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:50:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:50:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:50:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:50:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:50:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:50:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:50:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:50:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:50:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:50:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:50:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:50:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:50:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:50:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:50:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:50:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:50:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:50:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:50:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:50:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:50:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:50:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:50:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:50:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:50:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:50:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:50:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:50:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:50:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:50:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:50:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:50:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:50:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:50:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:50:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:50:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:50:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:50:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:50:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:50:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:50:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:50:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:50:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:50:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:50:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:50:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:50:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:50:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:50:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:50:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:50:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:50:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:50:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:50:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:50:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:50:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:50:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:50:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:50:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:50:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:50:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:50:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:50:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:50:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:50:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:50:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:50:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:50:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:50:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:50:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:50:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:50:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:50:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:50:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:50:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:50:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:50:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:50:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:50:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:50:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:50:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:50:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:50:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:50:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:50:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:50:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:50:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:50:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:50:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:50:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:50:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:50:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:50:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:50:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:50:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:50:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:50:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:50:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:50:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:50:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:50:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:50:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:50:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:50:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:50:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:50:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:50:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:50:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:50:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:50:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:50:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:50:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:50:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:50:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:50:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:50:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:50:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:50:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:50:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:50:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:50:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:50:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:50:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:50:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:50:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:50:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:50:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:50:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:50:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:50:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:50:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:50:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:50:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:50:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:50:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:50:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:50:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:50:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:50:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:50:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:50:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:50:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:50:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:50:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:50:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:50:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:50:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:50:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:50:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:50:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:50:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:50:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:50:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:50:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:50:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:50:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:50:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:51:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:51:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:51:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:51:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:51:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:51:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:51:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:51:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:51:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:51:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:51:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:51:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:51:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:51:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:51:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:51:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:51:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:51:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:51:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:51:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:51:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:51:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:51:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:51:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:51:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:51:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:51:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:51:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:51:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:51:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:51:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:51:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:51:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:51:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:51:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:51:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:51:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:51:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:51:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:51:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:51:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:51:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:51:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:51:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:51:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:51:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:51:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:51:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:51:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:51:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:51:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:51:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:51:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:51:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:51:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:51:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:51:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:51:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:51:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:51:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:51:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:51:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:51:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:51:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:51:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:51:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:51:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:51:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:51:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:51:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:51:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:51:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:51:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:51:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:51:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:51:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:51:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:51:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:51:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:51:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:51:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:51:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:51:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:51:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:51:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:51:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:51:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:51:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:51:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:51:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:51:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:51:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:51:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:51:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:51:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:51:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:51:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:51:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:51:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:51:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:51:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:51:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:51:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:51:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:51:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:51:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:51:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:51:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:51:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:51:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:51:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:51:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:51:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:51:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:51:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:51:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:51:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:51:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:51:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:51:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:51:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:51:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:51:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:51:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:51:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:51:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:51:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:51:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:51:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:51:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:51:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:51:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:51:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:51:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:51:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:51:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:51:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:51:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:51:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:51:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:51:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:51:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:51:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:51:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:51:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:51:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:52:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:52:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:52:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:52:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:52:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:52:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:52:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:52:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:52:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:52:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:52:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:52:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:52:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:52:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:52:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:52:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:52:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:52:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:52:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:52:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:52:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:52:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:52:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:52:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:52:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:52:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:52:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:52:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:52:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:52:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:52:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:52:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:52:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:52:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:52:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:52:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:52:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:52:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:52:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:52:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:52:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:52:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:52:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:52:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:52:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:52:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:52:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:52:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:52:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:52:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:52:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:52:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:52:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:52:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:52:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:52:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:52:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:52:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:52:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:52:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:52:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:52:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:52:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:52:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:52:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:52:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:52:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:52:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:52:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:52:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:52:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:52:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:52:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:52:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:52:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:52:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:52:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:52:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:52:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:52:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:52:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:52:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:52:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:52:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:52:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:52:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:52:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:52:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:52:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:52:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:52:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:52:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:52:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:52:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:52:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:52:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:52:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:52:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:52:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:52:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:52:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:52:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:52:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:52:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:52:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:52:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:52:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:52:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:52:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:52:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:52:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:52:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:52:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:52:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:52:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:52:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:52:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:52:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:52:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:52:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:52:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:52:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:52:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:52:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:52:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:52:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:52:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:52:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:52:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:52:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:52:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:52:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:52:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:52:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:52:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:52:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:52:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:52:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:52:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:52:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:52:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:52:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:52:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:52:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:52:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:52:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:52:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:52:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:52:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:52:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:52:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:52:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:52:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:52:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:52:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:52:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:52:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:52:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:52:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:52:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:52:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:52:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:52:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:52:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:52:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:52:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:52:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:52:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:52:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:52:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:52:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:52:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:52:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:52:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:52:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:52:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:53:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:53:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:53:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:53:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:53:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:53:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:53:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:53:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:53:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:53:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:53:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:53:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:53:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:53:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:53:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:53:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:53:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:53:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:53:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:53:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:53:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:53:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:53:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:53:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:53:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:53:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:53:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:53:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:53:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:53:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:53:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:53:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:53:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:53:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:53:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:53:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:53:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:53:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:53:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:53:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:53:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:53:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:53:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:53:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:53:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:53:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:53:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:53:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:53:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:53:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:53:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:53:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:53:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:53:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:53:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:53:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:53:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:53:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:53:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:53:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:53:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:53:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:53:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:53:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:53:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:53:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:53:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:53:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:53:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:53:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:53:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:53:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:53:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:53:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:53:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:53:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:53:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:53:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:53:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:53:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:53:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:53:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:53:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:53:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:53:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:53:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:53:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:53:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:53:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:53:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:53:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:53:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:53:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:53:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:53:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:53:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:53:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:53:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:53:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:53:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:53:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:53:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:53:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:53:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:53:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:53:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:53:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:53:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:53:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:53:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:53:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:53:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:53:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:53:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:53:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:53:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:53:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:53:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:53:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:53:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:53:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:53:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:53:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:53:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:53:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:53:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:53:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:53:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:53:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:53:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:53:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:53:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:53:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:53:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:53:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:53:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:54:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:54:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:54:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:54:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:54:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:54:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:54:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:54:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:54:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:54:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:54:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:54:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:54:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:54:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:54:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:54:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:54:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:54:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:54:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:54:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:54:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:54:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:54:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:54:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:54:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:54:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:54:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:54:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:54:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:54:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:54:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:54:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:54:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:54:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:54:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:54:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:54:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:54:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:54:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:54:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:54:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:54:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:54:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:54:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:54:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:54:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:54:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:54:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:54:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:54:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:54:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:54:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:54:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:54:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:54:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:54:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:54:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:54:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:54:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:54:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:54:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:54:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:54:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:54:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:54:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:54:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:54:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:54:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:54:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:54:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:54:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:54:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:54:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:54:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:54:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:54:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:54:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:54:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:54:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:54:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:54:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:54:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:54:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:54:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:54:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:54:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:54:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:54:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:54:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:54:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:54:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:54:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:54:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:54:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:54:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:54:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:54:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:54:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:54:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:54:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:54:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:54:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:54:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:54:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:54:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:54:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:54:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:54:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:54:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:54:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:54:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:54:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:54:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:54:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:54:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:54:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:54:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:54:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:54:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:54:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:54:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:54:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:54:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:54:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:54:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:54:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:54:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:54:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:54:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:54:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:54:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:54:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:54:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:54:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:54:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:54:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:54:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:54:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:54:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:54:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:54:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:54:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:54:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:54:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:54:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:54:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:54:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:54:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:54:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:54:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:54:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:54:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:54:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:54:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:54:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:54:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:54:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:54:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:54:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:54:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:54:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:54:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:54:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:54:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:54:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:54:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:54:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:54:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:54:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:54:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:54:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:54:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:54:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:54:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:54:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:54:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:55:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:55:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:55:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:55:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:55:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:55:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:55:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:55:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:55:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:55:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:55:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:55:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:55:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:55:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:55:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:55:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:55:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:55:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:55:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:55:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:55:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:55:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:55:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:55:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:55:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:55:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:55:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:55:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:55:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:55:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:55:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:55:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:55:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:55:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:55:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:55:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:55:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:55:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:55:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:55:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:55:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:55:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:55:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:55:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:55:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:55:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:55:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:55:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:55:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:55:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:55:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:55:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:55:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:55:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:55:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:55:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:55:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:55:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:55:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:55:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:55:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:55:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:55:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:55:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:55:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:55:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:55:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:55:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:55:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:55:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:55:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:55:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:55:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:55:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:55:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:55:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:55:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:55:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:55:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:55:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:55:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:55:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:55:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:55:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:55:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:55:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:55:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:55:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:55:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:55:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:55:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:55:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:55:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:55:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:55:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:55:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:55:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:55:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:55:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:55:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:55:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:55:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:55:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:55:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:55:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:55:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:55:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:55:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:55:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:55:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:55:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:55:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:55:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:55:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:55:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:55:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:55:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:55:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:55:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:55:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:55:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:55:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:55:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:55:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:55:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:55:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:55:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:55:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:55:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:55:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:55:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:55:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:55:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:55:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:55:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:55:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:55:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:55:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:55:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:55:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:55:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:55:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:55:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:55:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:55:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:55:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:55:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:55:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:55:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:55:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:55:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:55:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:55:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:55:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:55:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:55:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:55:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:55:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:55:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:55:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:55:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:55:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:55:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:55:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:55:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:55:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:55:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:55:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:55:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:55:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:55:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:55:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:55:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:55:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:55:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:55:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:55:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:55:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:55:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:55:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:55:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:55:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:55:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:55:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:55:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:55:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:55:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:55:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:55:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:55:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:55:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:55:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:55:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:55:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:55:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:55:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:55:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:55:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:55:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:55:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:55:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:55:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:55:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:55:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:55:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:55:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:55:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:55:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:55:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:55:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:55:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:55:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:55:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:55:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:55:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:55:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:56:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:56:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:56:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:56:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:56:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:56:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:56:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:56:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:56:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:56:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:56:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:56:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:56:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:56:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:56:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:56:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:56:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:56:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:58:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:58:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:58:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:58:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:58:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:58:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:58:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:58:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:58:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:58:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:58:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:58:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:58:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:58:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:58:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:58:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:58:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:58:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:58:14 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 16:58:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:58:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:58:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:58:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:58:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:58:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:58:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:58:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:58:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:58:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:58:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:58:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:58:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:58:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:58:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:58:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:58:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:58:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:58:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:58:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:58:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:58:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:58:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:58:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:58:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:58:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:58:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:58:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:58:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:58:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:58:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:58:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:58:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:58:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:58:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:58:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:58:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:58:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:58:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:58:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:58:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:58:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:58:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:58:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:58:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:58:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:58:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:58:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:58:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:58:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:58:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:58:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:58:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:58:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:58:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:58:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:58:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:58:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:58:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:58:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:58:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:58:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:58:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:58:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:58:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:58:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:58:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:58:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:58:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:58:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:58:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:58:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:58:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:58:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:58:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:58:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:58:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:58:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:58:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:58:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:58:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:58:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:58:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:58:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:58:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:58:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:58:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:58:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:58:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:58:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:58:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:58:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:58:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:58:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:58:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:58:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:58:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:58:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:58:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:58:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:58:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:58:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:58:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:58:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:58:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:58:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:58:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:58:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:58:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:58:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:58:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:58:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:58:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:58:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:58:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:58:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:58:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:58:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:58:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:58:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:58:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:58:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:58:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:58:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:58:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:58:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:58:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:58:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:58:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:58:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:58:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:58:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:58:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:58:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:58:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:58:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:58:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:58:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:58:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:58:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:58:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:58:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:58:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:58:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:58:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:58:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:58:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:58:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:58:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:58:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:58:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:58:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:58:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:58:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:58:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:58:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:58:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:58:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:58:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:58:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:58:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:58:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:59:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:59:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:59:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:59:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:59:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:59:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:59:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:59:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:59:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:59:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:59:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:59:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:59:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:59:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:59:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:59:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:59:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:59:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:59:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:59:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:59:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:59:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:59:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:59:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:59:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:59:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:59:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:59:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:59:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:59:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:59:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:59:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:59:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:59:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:59:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:59:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:59:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:59:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:59:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:59:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:59:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:59:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:59:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:59:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:59:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:59:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:59:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:59:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:59:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:59:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:59:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:59:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:59:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:59:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:59:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:59:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:59:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:59:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:59:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:59:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:59:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:59:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:59:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:59:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:59:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:59:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:59:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:59:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:59:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:59:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:59:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:59:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:59:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:59:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:59:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:59:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:59:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:59:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:59:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:59:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:59:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:59:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:59:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:59:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:59:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:59:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:59:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:59:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:59:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:59:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:59:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:59:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:59:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:59:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:59:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:59:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:59:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:59:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:59:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:59:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:59:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:59:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:59:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:59:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:59:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:59:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:59:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:59:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:59:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:59:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:59:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:59:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:59:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:59:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:59:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:59:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:59:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:59:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:59:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:59:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:59:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:59:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:59:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:59:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:59:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:59:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:59:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:59:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:59:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:59:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:59:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:59:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:59:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:59:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:59:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:59:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:59:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:59:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 16:59:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 16:59:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:59:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:59:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:59:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 16:59:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 16:59:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 16:59:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 16:59:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 16:59:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 16:59:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 16:59:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 16:59:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 16:59:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 16:59:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 16:59:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 16:59:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 16:59:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:00:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:00:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:00:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:00:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:00:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:00:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:00:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:00:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:00:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:00:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:00:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:00:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:00:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:00:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:00:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:00:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:00:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:00:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:00:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:00:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:00:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:00:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:00:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:00:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:00:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:00:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:00:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:00:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:00:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:00:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:00:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:00:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:00:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:00:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:00:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:00:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:00:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:00:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:00:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:00:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:00:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:00:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:00:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:00:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:00:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:00:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:00:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:00:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:00:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:00:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:00:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:00:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:00:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:00:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:00:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:00:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:00:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:00:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:00:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:00:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:00:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:00:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:00:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:00:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:00:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:00:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:00:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:00:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:00:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:00:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:00:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:00:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:00:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:00:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:00:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:00:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:00:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:00:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:00:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:00:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:00:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:00:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:00:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:00:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:00:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:00:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:00:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:00:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:00:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:00:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:00:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:00:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:00:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:00:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:00:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:00:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:00:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:00:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:00:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:00:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:00:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:00:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:00:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:00:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:00:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:00:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:00:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:00:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:00:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:00:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:00:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:00:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:00:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:00:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:00:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:00:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:00:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:00:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:00:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:00:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:00:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:00:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:00:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:00:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:00:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:00:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:00:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:00:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:00:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:00:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:00:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:00:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:00:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:00:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:00:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:00:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:00:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:00:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:00:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:00:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:00:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:00:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:00:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:00:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:00:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:00:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:00:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:00:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:00:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:00:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:00:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:00:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:00:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:00:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:00:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:00:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:00:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:00:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:00:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:00:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:00:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:00:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:00:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:00:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:00:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:00:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:00:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:00:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:00:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:00:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:00:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:00:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:00:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:00:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:00:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:00:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:00:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:00:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:00:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:00:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:00:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:00:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:00:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:00:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:00:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:00:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:01:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:01:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:01:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:01:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:01:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:01:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:01:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:01:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:01:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:01:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:01:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:01:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:01:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:01:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:01:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:01:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:01:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:01:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:01:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:01:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:01:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:01:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:01:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:01:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:01:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:01:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:01:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:01:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:01:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:01:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:01:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:01:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:01:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:01:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:01:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:01:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:01:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:01:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:01:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:01:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:01:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:01:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:01:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:01:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:01:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:01:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:01:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:01:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:01:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:01:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:01:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:01:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:01:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:01:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:01:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:01:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:01:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:01:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:01:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:01:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:01:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:01:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:01:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:01:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:01:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:01:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:01:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:01:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:01:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:01:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:01:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:01:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:01:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:01:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:01:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:01:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:01:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:01:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:01:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:01:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:01:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:01:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:01:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:01:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:01:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:01:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:01:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:01:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:01:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:01:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:01:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:01:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:01:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:01:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:01:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:01:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:01:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:01:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:01:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:01:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:01:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:01:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:01:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:01:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:01:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:01:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:01:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:01:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:01:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:01:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:01:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:01:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:01:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:01:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:01:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:01:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:01:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:01:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:01:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:01:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:01:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:01:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:01:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:01:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:01:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:01:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:01:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:01:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:01:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:01:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:01:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:01:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:01:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:01:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:01:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:01:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:01:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:01:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:01:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:01:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:01:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:01:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:01:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:01:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:01:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:01:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:01:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:01:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:01:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:01:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:01:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:01:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:01:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:01:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:01:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:01:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:01:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:01:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:01:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:01:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:01:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:01:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:01:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:01:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:01:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:01:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:01:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:01:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:01:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:01:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:01:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:01:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:01:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:01:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:01:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:01:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:01:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:01:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:01:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:01:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:01:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:01:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:01:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:01:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:01:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:01:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:01:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:01:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:01:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:01:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:01:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:01:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:01:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:01:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:01:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:01:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:01:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:01:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:01:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:01:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:01:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:01:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:01:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:01:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:01:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:01:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:01:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:01:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:01:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:01:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:01:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:01:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:01:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:01:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:01:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:01:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:02:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:02:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:02:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:02:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:02:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:02:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:02:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:02:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:02:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:02:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:02:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:02:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:02:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:02:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:02:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:02:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:02:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:02:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:02:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:02:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:02:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:02:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:02:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:02:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:02:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:02:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:02:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:02:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:02:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:02:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:02:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:02:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:02:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:02:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:02:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:02:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:02:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:02:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:02:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:02:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:02:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:02:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:02:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:02:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:02:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:02:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:02:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:02:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:02:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:02:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:02:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:02:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:02:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:02:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:02:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:02:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:02:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:02:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:02:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:02:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:02:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:02:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:02:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:02:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:02:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:02:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:02:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:02:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:02:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:02:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:02:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:02:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:02:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:02:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:02:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:02:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:02:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:02:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:02:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:02:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:02:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:02:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:02:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:02:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:02:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:02:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:02:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:02:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:02:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:02:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:02:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:02:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:02:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:02:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:02:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:02:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:02:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:02:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:02:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:02:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:02:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:02:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:02:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:02:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:02:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:02:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:02:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:02:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:02:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:02:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:02:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:02:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:02:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:02:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:02:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:02:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:02:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:02:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:02:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:02:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:02:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:02:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:02:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:02:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:02:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:02:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:02:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:02:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:02:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:02:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:02:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:02:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:02:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:02:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:02:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:02:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:02:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:02:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:02:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:02:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:02:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:02:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:02:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:02:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:02:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:02:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:02:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:02:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:02:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:02:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:02:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:02:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:02:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:02:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:02:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:02:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:02:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:02:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:02:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:02:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:02:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:02:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:02:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:02:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:02:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:02:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:02:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:02:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:02:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:02:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:02:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:02:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:02:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:02:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:02:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:02:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:02:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:02:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:02:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:02:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:02:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:02:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:02:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:02:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:02:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:02:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:02:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:02:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:02:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:02:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:02:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:02:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:02:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:02:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:02:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:02:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:02:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:02:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:02:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:02:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:02:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:02:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:02:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:02:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:02:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:02:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:02:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:02:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:02:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:02:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:02:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:02:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:02:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:02:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:02:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:02:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:03:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:03:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:03:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:03:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:03:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:03:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:03:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:03:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:03:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:03:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:03:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:03:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:03:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:03:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:03:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:03:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:03:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:03:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:03:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:03:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:03:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:03:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:03:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:03:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:03:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:03:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:03:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:03:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:03:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:03:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:03:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:03:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:03:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:03:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:03:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:03:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:03:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:03:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:03:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:03:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:03:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:03:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:03:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:03:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:03:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:03:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:03:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:03:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:03:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:03:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:03:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:03:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:03:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:03:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:03:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:03:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:03:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:03:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:03:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:03:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:03:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:03:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:03:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:03:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:03:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:03:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:03:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:03:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:03:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:03:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:03:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:03:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:03:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:03:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:03:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:03:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:03:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:03:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:03:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:03:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:03:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:03:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:03:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:03:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:03:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:03:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:03:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:03:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:03:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:03:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:03:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:03:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:03:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:03:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:03:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:03:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:03:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:03:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:03:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:03:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:03:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:03:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:03:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:03:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:03:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:03:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:03:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:03:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:03:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:03:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:03:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:03:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:03:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:03:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:03:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:03:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:03:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:03:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:03:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:03:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:03:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:03:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:03:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:03:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:03:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:03:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:03:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:03:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:03:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:03:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:03:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:03:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:03:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:03:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:03:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:03:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:03:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:03:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:03:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:03:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:03:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:03:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:03:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:03:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:03:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:03:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:03:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:03:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:03:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:03:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:03:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:03:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:03:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:03:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:03:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:03:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:03:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:03:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:03:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:03:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:03:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:03:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:03:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:03:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:03:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:03:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:03:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:03:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:03:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:03:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:03:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:03:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:03:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:03:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:03:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:03:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:03:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:03:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:03:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:03:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:03:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:03:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:03:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:03:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:03:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:03:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:03:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:03:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:03:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:03:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:03:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:03:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:03:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:03:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:03:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:03:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:03:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:03:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:03:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:03:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:03:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:03:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:03:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:03:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:03:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:03:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:03:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:03:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:03:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:03:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:03:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:03:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:03:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:03:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:03:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:03:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:04:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:04:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:04:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:04:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:04:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:04:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:04:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:04:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:04:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:04:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:04:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:04:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:04:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:04:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:04:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:04:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:04:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:04:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:04:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:04:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:04:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:04:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:04:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:04:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:04:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:04:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:04:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:04:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:04:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:04:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:04:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:04:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:04:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:04:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:04:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:04:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:04:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:04:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:04:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:04:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:04:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:04:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:04:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:04:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:04:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:04:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:04:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:04:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:04:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:04:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:04:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:04:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:04:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:04:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:04:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:04:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:04:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:04:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:04:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:04:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:04:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:04:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:04:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:04:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:04:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:04:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:04:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:04:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:04:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:04:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:04:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:04:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:04:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:04:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:04:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:04:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:04:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:04:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:04:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:04:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:04:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:04:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:04:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:04:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:04:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:04:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:04:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:04:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:04:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:04:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:04:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:04:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:04:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:04:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:04:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:04:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:04:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:04:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:04:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:04:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:04:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:04:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:04:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:04:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:04:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:04:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:04:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:04:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:04:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:04:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:04:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:04:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:04:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:04:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:04:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:04:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:04:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:04:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:04:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:04:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:04:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:04:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:04:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:04:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:04:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:04:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:04:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:04:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:04:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:04:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:04:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:04:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:04:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:04:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:04:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:04:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:04:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:04:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:04:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:04:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:04:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:04:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:04:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:04:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:04:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:04:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:04:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:04:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:04:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:04:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:04:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:04:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:04:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:04:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:04:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:04:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:04:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:04:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:04:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:04:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:04:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:04:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:04:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:04:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:04:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:04:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:04:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:04:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:04:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:04:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:04:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:04:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:04:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:04:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:04:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:04:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:04:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:04:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:04:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:04:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:04:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:04:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:04:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:04:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:04:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:04:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:04:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:04:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:04:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:04:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:04:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:04:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:04:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:04:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:04:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:04:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:04:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:04:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:04:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:04:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:04:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:04:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:04:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:04:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:04:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:04:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:04:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:04:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:04:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:04:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:04:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:04:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:04:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:04:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:04:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:04:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:05:00 rdma06 systemd[1]: Starting dnf makecache...
Apr 17 17:05:01 rdma06 dnf[1161947]: Updating Subscription Management repositories.
Apr 17 17:05:01 rdma06 dnf[1161947]: Unable to read consumer identity
Apr 17 17:05:01 rdma06 dnf[1161947]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 17 17:05:02 rdma06 dnf[1161947]: Copr repo for fmf owned by psss                 2.1 kB/s | 3.3 kB     00:01
Apr 17 17:05:02 rdma06 dnf[1161947]: beaker-AppStream                                 48 kB/s | 168  B     00:00
Apr 17 17:05:02 rdma06 dnf[1161947]: Errors during downloading metadata for repository 'beaker-AppStream':
Apr 17 17:05:02 rdma06 dnf[1161947]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 17:05:02 rdma06 dnf[1161947]: Error: Failed to download metadata for repo 'beaker-AppStream': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 17:05:02 rdma06 dnf[1161947]: beaker-AppStream-debuginfo                       53 kB/s | 168  B     00:00
Apr 17 17:05:02 rdma06 dnf[1161947]: Errors during downloading metadata for repository 'beaker-AppStream-debuginfo':
Apr 17 17:05:02 rdma06 dnf[1161947]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 17:05:02 rdma06 dnf[1161947]: Error: Failed to download metadata for repo 'beaker-AppStream-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 17:05:02 rdma06 dnf[1161947]: beaker-BaseOS                                    55 kB/s | 168  B     00:00
Apr 17 17:05:02 rdma06 dnf[1161947]: Errors during downloading metadata for repository 'beaker-BaseOS':
Apr 17 17:05:02 rdma06 dnf[1161947]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 17:05:02 rdma06 dnf[1161947]: Error: Failed to download metadata for repo 'beaker-BaseOS': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 17:05:03 rdma06 dnf[1161947]: beaker-BaseOS-debuginfo                          63 kB/s | 168  B     00:00
Apr 17 17:05:03 rdma06 dnf[1161947]: Errors during downloading metadata for repository 'beaker-BaseOS-debuginfo':
Apr 17 17:05:03 rdma06 dnf[1161947]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 17:05:03 rdma06 dnf[1161947]: Error: Failed to download metadata for repo 'beaker-BaseOS-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 17:05:03 rdma06 dnf[1161947]: beaker-CRB                                       61 kB/s | 168  B     00:00
Apr 17 17:05:03 rdma06 dnf[1161947]: Errors during downloading metadata for repository 'beaker-CRB':
Apr 17 17:05:03 rdma06 dnf[1161947]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 17:05:03 rdma06 dnf[1161947]: Error: Failed to download metadata for repo 'beaker-CRB': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 17:05:03 rdma06 dnf[1161947]: beaker-CRB-debuginfo                             61 kB/s | 168  B     00:00
Apr 17 17:05:03 rdma06 dnf[1161947]: Errors during downloading metadata for repository 'beaker-CRB-debuginfo':
Apr 17 17:05:03 rdma06 dnf[1161947]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 17:05:03 rdma06 dnf[1161947]: Error: Failed to download metadata for repo 'beaker-CRB-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 17:05:03 rdma06 dnf[1161947]: beaker-HighAvailability                          56 kB/s | 168  B     00:00
Apr 17 17:05:03 rdma06 dnf[1161947]: Errors during downloading metadata for repository 'beaker-HighAvailability':
Apr 17 17:05:03 rdma06 dnf[1161947]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 17:05:03 rdma06 dnf[1161947]: Error: Failed to download metadata for repo 'beaker-HighAvailability': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 17:05:03 rdma06 dnf[1161947]: beaker-HighAvailability-debuginfo                48 kB/s | 168  B     00:00
Apr 17 17:05:03 rdma06 dnf[1161947]: Errors during downloading metadata for repository 'beaker-HighAvailability-debuginfo':
Apr 17 17:05:03 rdma06 dnf[1161947]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 17:05:03 rdma06 dnf[1161947]: Error: Failed to download metadata for repo 'beaker-HighAvailability-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 17:05:03 rdma06 dnf[1161947]: beaker-NFV                                       65 kB/s | 168  B     00:00
Apr 17 17:05:03 rdma06 dnf[1161947]: Errors during downloading metadata for repository 'beaker-NFV':
Apr 17 17:05:03 rdma06 dnf[1161947]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 17:05:03 rdma06 dnf[1161947]: Error: Failed to download metadata for repo 'beaker-NFV': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 17:05:03 rdma06 dnf[1161947]: beaker-NFV-debuginfo                             64 kB/s | 168  B     00:00
Apr 17 17:05:03 rdma06 dnf[1161947]: Errors during downloading metadata for repository 'beaker-NFV-debuginfo':
Apr 17 17:05:03 rdma06 dnf[1161947]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 17:05:03 rdma06 dnf[1161947]: Error: Failed to download metadata for repo 'beaker-NFV-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 17:05:03 rdma06 dnf[1161947]: beaker-RT                                        58 kB/s | 168  B     00:00
Apr 17 17:05:03 rdma06 dnf[1161947]: Errors during downloading metadata for repository 'beaker-RT':
Apr 17 17:05:03 rdma06 dnf[1161947]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 17:05:03 rdma06 dnf[1161947]: Error: Failed to download metadata for repo 'beaker-RT': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 17:05:03 rdma06 dnf[1161947]: beaker-RT-debuginfo                              61 kB/s | 168  B     00:00
Apr 17 17:05:03 rdma06 dnf[1161947]: Errors during downloading metadata for repository 'beaker-RT-debuginfo':
Apr 17 17:05:03 rdma06 dnf[1161947]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 17:05:03 rdma06 dnf[1161947]: Error: Failed to download metadata for repo 'beaker-RT-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 17:05:03 rdma06 dnf[1161947]: beaker-ResilientStorage                          54 kB/s | 168  B     00:00
Apr 17 17:05:03 rdma06 dnf[1161947]: Errors during downloading metadata for repository 'beaker-ResilientStorage':
Apr 17 17:05:03 rdma06 dnf[1161947]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 17:05:03 rdma06 dnf[1161947]: Error: Failed to download metadata for repo 'beaker-ResilientStorage': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 17:05:03 rdma06 dnf[1161947]: beaker-ResilientStorage-debuginfo                58 kB/s | 168  B     00:00
Apr 17 17:05:03 rdma06 dnf[1161947]: Errors during downloading metadata for repository 'beaker-ResilientStorage-debuginfo':
Apr 17 17:05:03 rdma06 dnf[1161947]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 17:05:03 rdma06 dnf[1161947]: Error: Failed to download metadata for repo 'beaker-ResilientStorage-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 17:05:03 rdma06 dnf[1161947]: beaker-SAP                                       64 kB/s | 168  B     00:00
Apr 17 17:05:03 rdma06 dnf[1161947]: Errors during downloading metadata for repository 'beaker-SAP':
Apr 17 17:05:03 rdma06 dnf[1161947]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 17:05:03 rdma06 dnf[1161947]: Error: Failed to download metadata for repo 'beaker-SAP': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 17:05:03 rdma06 dnf[1161947]: beaker-SAP-debuginfo                             67 kB/s | 168  B     00:00
Apr 17 17:05:03 rdma06 dnf[1161947]: Errors during downloading metadata for repository 'beaker-SAP-debuginfo':
Apr 17 17:05:03 rdma06 dnf[1161947]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 17:05:03 rdma06 dnf[1161947]: Error: Failed to download metadata for repo 'beaker-SAP-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 17:05:03 rdma06 dnf[1161947]: beaker-SAPHANA                                   64 kB/s | 168  B     00:00
Apr 17 17:05:03 rdma06 dnf[1161947]: Errors during downloading metadata for repository 'beaker-SAPHANA':
Apr 17 17:05:03 rdma06 dnf[1161947]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 17:05:03 rdma06 dnf[1161947]: Error: Failed to download metadata for repo 'beaker-SAPHANA': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 17:05:03 rdma06 dnf[1161947]: beaker-SAPHANA-debuginfo                         55 kB/s | 168  B     00:00
Apr 17 17:05:03 rdma06 dnf[1161947]: Errors during downloading metadata for repository 'beaker-SAPHANA-debuginfo':
Apr 17 17:05:03 rdma06 dnf[1161947]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 17:05:03 rdma06 dnf[1161947]: Error: Failed to download metadata for repo 'beaker-SAPHANA-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 17:05:03 rdma06 dnf[1161947]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr 17 17:05:04 rdma06 dnf[1161947]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr 17 17:05:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:05:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:05:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:05:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:05:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:05:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:05:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:05:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:05:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:05:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:05:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:05:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:05:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:05:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:05:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:05:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:05:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:05:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:05:04 rdma06 dnf[1161947]: Ignoring repositories: beaker-AppStream, beaker-AppStream-debuginfo, beaker-BaseOS, beaker-BaseOS-debuginfo, beaker-CRB, beaker-CRB-debuginfo, beaker-HighAvailability, beaker-HighAvailability-debuginfo, beaker-NFV, beaker-NFV-debuginfo, beaker-RT, beaker-RT-debuginfo, beaker-ResilientStorage, beaker-ResilientStorage-debuginfo, beaker-SAP, beaker-SAP-debuginfo, beaker-SAPHANA, beaker-SAPHANA-debuginfo
Apr 17 17:05:04 rdma06 dnf[1161947]: Metadata cache created.
Apr 17 17:05:04 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 17 17:05:04 rdma06 systemd[1]: Finished dnf makecache.
Apr 17 17:05:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:05:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:05:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:05:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:05:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:05:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:05:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:05:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:05:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:05:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:05:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:05:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:05:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:05:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:05:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:05:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:05:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:05:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:05:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:05:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:05:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:05:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:05:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:05:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:05:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:05:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:05:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:05:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:05:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:05:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:05:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:05:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:05:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:05:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:05:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:05:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:05:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:05:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:05:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:05:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:05:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:05:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:05:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:05:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:05:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:05:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:05:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:05:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:05:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:05:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:05:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:05:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:05:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:05:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:05:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:05:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:05:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:05:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:05:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:05:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:05:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:05:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:05:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:05:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:05:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:05:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:05:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:05:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:05:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:05:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:05:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:05:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:05:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:05:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:05:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:05:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:05:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:05:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:05:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:05:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:05:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:05:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:05:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:05:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:05:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:05:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:05:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:05:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:05:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:05:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:05:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:05:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:05:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:05:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:05:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:05:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:05:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:05:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:05:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:05:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:05:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:05:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:05:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:05:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:05:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:05:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:05:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:05:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:05:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:05:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:05:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:05:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:05:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:05:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:05:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:05:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:05:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:05:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:05:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:05:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:05:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:05:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:05:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:05:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:05:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:05:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:05:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:05:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:05:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:05:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:05:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:05:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:05:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:05:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:05:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:05:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:05:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:05:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:05:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:05:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:05:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:05:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:05:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:05:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:05:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:05:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:05:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:05:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:05:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:05:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:05:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:05:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:05:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:05:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:05:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:05:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:05:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:05:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:05:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:05:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:05:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:05:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:05:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:05:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:05:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:05:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:05:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:05:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:05:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:05:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:05:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:05:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:05:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:05:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:05:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:05:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:05:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:05:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:05:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:05:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:05:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:05:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:05:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:05:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:05:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:05:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:05:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:05:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:05:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:05:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:05:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:05:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:05:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:05:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:05:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:05:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:05:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:05:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:06:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:06:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:06:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:06:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:06:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:06:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:06:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:06:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:06:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:06:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:06:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:06:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:06:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:06:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:06:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:06:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:06:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:06:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:06:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:06:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:06:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:06:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:06:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:06:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:06:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:06:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:06:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:06:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:06:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:06:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:06:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:06:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:06:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:08:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:08:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:08:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:08:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:08:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:08:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:08:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:08:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:08:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:08:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:08:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:08:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:08:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:08:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:08:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:08:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:08:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:08:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:08:19 rdma06 rsyslogd[1041]: imjournal: 5378 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 17:08:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:08:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:08:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:08:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:08:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:08:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:08:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:08:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:08:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:08:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:08:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:08:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:08:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:08:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:08:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:08:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:08:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:08:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:08:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:08:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:08:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:08:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:08:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:08:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:08:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:08:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:08:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:08:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:08:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:08:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:08:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:08:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:08:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:08:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:08:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:08:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:08:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:08:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:08:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:08:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:08:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:08:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:08:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:08:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:08:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:08:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:08:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:08:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:08:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:08:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:08:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:08:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:08:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:08:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:08:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:08:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:08:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:08:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:08:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:08:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:08:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:08:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:08:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:08:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:08:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:08:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:08:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:08:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:08:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:08:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:08:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:08:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:08:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:08:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:08:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:08:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:08:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:08:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:08:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:08:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:08:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:08:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:08:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:08:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:08:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:08:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:08:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:08:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:08:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:08:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:08:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:08:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:08:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:08:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:08:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:08:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:08:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:08:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:08:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:08:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:08:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:08:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:08:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:08:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:08:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:08:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:08:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:08:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:08:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:08:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:08:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:08:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:08:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:08:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:08:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:08:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:08:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:08:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:08:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:08:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:08:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:08:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:08:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:08:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:08:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:08:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:08:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:08:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:08:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:08:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:08:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:08:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:08:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:08:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:08:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:08:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:08:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:08:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:08:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:08:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:08:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:08:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:08:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:08:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:09:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:09:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:09:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:09:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:09:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:09:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:09:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:09:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:09:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:09:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:09:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:09:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:09:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:09:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:09:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:09:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:09:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:09:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:09:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:09:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:09:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:09:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:09:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:09:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:09:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:09:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:09:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:09:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:09:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:09:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:09:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:09:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:09:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:09:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:09:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:09:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:09:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:09:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:09:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:09:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:09:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:09:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:09:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:09:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:09:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:09:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:09:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:09:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:09:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:09:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:09:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:09:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:09:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:09:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:09:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:09:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:09:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:09:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:09:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:09:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:09:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:09:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:09:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:09:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:09:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:09:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:09:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:09:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:09:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:09:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:09:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:09:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:09:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:09:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:09:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:09:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:09:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:09:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:09:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:09:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:09:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:09:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:09:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:09:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:09:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:09:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:09:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:09:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:09:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:09:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:09:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:09:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:09:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:09:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:09:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:09:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:09:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:09:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:09:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:09:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:09:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:09:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:09:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:09:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:09:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:09:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:09:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:09:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:09:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:09:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:09:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:09:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:09:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:09:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:09:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:09:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:09:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:09:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:09:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:09:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:09:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:09:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:09:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:09:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:09:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:09:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:09:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:09:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:09:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:09:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:09:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:09:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:09:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:09:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:09:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:09:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:09:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:09:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:09:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:09:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:09:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:09:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:09:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:09:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:09:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:09:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:09:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:09:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:09:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:09:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:09:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:09:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:09:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:09:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:09:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:09:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:09:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:09:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:09:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:09:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:09:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:09:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:09:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:09:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:09:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:09:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:09:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:09:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:09:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:09:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:09:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:09:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:09:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:09:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:09:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:09:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:09:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:09:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:09:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:09:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:09:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:09:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:09:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:09:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:09:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:09:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:09:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:09:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:09:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:09:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:09:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:09:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:09:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:09:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:09:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:09:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:09:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:09:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:09:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:09:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:09:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:09:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:09:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:09:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:09:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:09:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:09:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:09:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:09:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:09:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:09:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:09:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:09:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:09:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:09:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:09:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:10:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:10:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:10:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:10:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:10:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:10:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:10:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:10:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:10:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:10:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:10:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:10:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:10:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:10:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:10:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:10:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:10:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:10:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:10:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:10:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:10:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:10:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:10:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:10:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:10:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:10:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:10:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:10:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:10:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:10:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:10:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:10:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:10:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:10:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:10:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:10:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:10:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:10:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:10:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:10:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:10:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:10:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:10:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:10:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:10:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:10:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:10:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:10:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:10:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:10:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:10:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:10:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:10:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:10:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:10:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:10:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:10:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:10:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:10:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:10:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:10:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:10:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:10:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:10:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:10:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:10:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:10:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:10:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:10:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:10:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:10:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:10:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:10:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:10:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:10:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:10:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:10:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:10:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:10:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:10:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:10:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:10:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:10:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:10:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:10:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:10:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:10:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:10:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:10:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:10:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:10:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:10:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:10:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:10:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:10:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:10:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:10:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:10:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:10:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:10:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:10:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:10:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:10:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:10:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:10:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:10:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:10:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:10:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:10:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:10:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:10:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:10:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:10:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:10:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:10:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:10:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:10:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:10:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:10:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:10:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:10:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:10:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:10:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:10:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:10:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:10:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:10:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:10:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:10:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:10:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:10:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:10:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:10:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:10:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:10:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:10:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:10:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:10:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:10:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:10:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:10:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:10:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:10:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:10:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:10:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:10:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:10:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:10:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:10:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:10:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:10:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:10:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:10:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:10:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:10:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:10:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:10:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:10:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:10:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:10:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:10:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:10:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:10:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:10:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:10:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:10:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:10:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:10:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:10:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:10:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:10:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:10:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:10:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:10:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:10:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:10:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:10:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:10:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:10:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:10:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:10:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:10:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:10:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:10:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:10:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:10:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:10:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:10:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:10:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:10:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:10:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:10:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:10:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:10:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:10:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:10:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:10:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:10:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:10:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:10:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:10:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:10:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:10:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:10:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:10:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:10:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:10:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:10:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:10:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:10:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:10:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:10:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:10:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:10:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:10:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:10:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:11:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:11:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:11:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:11:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:11:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:11:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:11:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:11:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:11:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:11:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:11:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:11:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:11:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:11:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:11:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:11:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:11:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:11:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:11:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:11:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:11:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:11:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:11:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:11:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:11:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:11:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:11:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:11:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:11:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:11:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:11:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:11:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:11:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:11:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:11:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:11:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:11:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:11:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:11:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:11:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:11:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:11:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:11:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:11:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:11:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:11:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:11:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:11:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:11:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:11:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:11:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:11:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:11:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:11:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:11:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:11:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:11:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:11:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:11:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:11:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:11:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:11:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:11:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:11:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:11:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:11:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:11:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:11:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:11:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:11:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:11:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:11:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:11:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:11:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:11:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:11:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:11:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:11:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:11:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:11:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:11:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:11:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:11:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:11:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:11:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:11:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:11:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:11:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:11:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:11:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:11:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:11:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:11:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:11:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:11:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:11:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:11:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:11:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:11:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:11:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:11:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:11:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:11:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:11:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:11:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:11:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:11:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:11:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:11:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:11:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:11:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:11:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:11:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:11:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:11:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:11:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:11:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:11:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:11:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:11:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:11:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:11:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:11:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:11:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:11:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:11:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:11:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:11:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:11:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:11:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:11:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:11:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:11:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:11:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:11:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:11:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:11:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:11:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:11:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:11:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:11:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:11:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:11:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:11:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:11:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:11:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:11:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:11:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:11:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:11:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:11:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:11:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:11:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:11:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:11:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:11:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:11:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:11:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:11:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:11:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:11:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:11:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:11:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:11:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:11:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:11:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:11:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:11:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:11:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:11:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:11:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:11:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:11:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:11:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:11:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:11:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:11:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:11:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:11:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:11:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:11:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:11:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:11:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:11:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:11:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:11:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:11:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:11:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:11:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:11:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:11:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:11:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:11:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:11:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:11:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:11:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:11:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:11:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:11:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:11:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:11:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:11:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:11:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:11:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:11:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:11:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:11:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:11:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:11:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:11:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:11:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:11:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:11:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:11:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:11:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:11:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:12:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:12:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:12:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:12:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:12:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:12:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:12:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:12:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:12:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:12:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:12:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:12:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:12:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:12:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:12:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:12:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:12:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:12:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:12:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:12:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:12:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:12:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:12:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:12:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:12:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:12:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:12:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:12:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:12:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:12:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:12:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:12:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:12:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:12:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:12:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:12:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:12:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:12:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:12:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:12:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:12:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:12:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:12:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:12:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:12:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:12:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:12:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:12:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:12:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:12:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:12:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:12:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:12:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:12:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:12:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:12:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:12:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:12:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:12:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:12:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:12:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:12:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:12:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:12:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:12:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:12:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:12:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:12:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:12:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:12:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:12:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:12:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:12:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:12:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:12:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:12:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:12:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:12:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:12:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:12:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:12:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:12:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:12:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:12:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:12:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:12:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:12:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:12:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:12:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:12:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:12:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:12:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:12:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:12:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:12:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:12:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:12:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:12:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:12:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:12:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:12:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:12:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:12:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:12:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:12:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:12:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:12:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:12:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:12:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:12:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:12:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:12:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:12:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:12:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:12:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:12:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:12:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:12:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:12:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:12:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:12:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:12:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:12:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:12:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:12:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:12:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:12:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:12:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:12:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:12:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:12:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:12:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:12:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:12:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:12:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:12:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:12:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:12:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:12:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:12:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:12:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:12:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:12:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:12:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:12:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:12:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:12:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:12:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:12:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:12:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:12:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:12:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:12:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:12:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:12:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:12:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:12:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:12:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:12:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:12:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:12:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:12:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:12:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:12:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:12:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:12:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:12:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:12:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:12:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:12:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:12:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:12:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:12:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:12:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:12:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:12:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:12:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:12:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:12:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:12:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:12:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:12:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:12:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:12:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:12:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:12:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:12:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:12:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:12:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:12:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:12:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:12:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:12:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:12:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:12:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:12:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:12:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:12:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:12:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:12:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:12:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:12:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:12:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:12:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:12:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:12:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:12:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:12:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:12:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:12:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:12:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:12:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:12:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:12:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:12:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:12:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:13:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:13:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:13:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:13:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:13:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:13:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:13:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:13:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:13:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:13:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:13:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:13:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:13:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:13:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:13:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:13:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:13:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:13:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:13:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:13:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:13:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:13:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:13:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:13:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:13:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:13:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:13:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:13:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:13:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:13:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:13:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:13:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:13:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:13:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:13:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:13:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:13:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:13:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:13:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:13:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:13:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:13:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:13:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:13:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:13:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:13:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:13:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:13:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:13:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:13:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:13:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:13:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:13:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:13:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:13:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:13:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:13:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:13:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:13:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:13:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:13:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:13:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:13:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:13:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:13:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:13:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:13:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:13:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:13:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:13:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:13:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:13:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:13:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:13:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:13:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:13:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:13:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:13:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:13:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:13:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:13:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:13:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:13:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:13:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:13:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:13:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:13:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:13:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:13:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:13:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:13:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:13:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:13:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:13:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:13:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:13:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:13:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:13:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:13:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:13:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:13:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:13:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:13:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:13:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:13:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:13:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:13:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:13:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:13:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:13:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:13:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:13:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:13:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:13:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:13:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:13:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:13:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:13:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:13:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:13:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:13:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:13:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:13:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:13:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:13:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:13:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:13:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:13:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:13:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:13:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:13:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:13:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:13:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:13:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:13:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:13:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:13:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:13:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:13:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:13:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:13:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:13:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:13:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:13:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:13:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:13:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:13:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:13:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:13:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:13:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:13:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:13:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:13:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:13:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:13:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:13:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:13:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:13:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:13:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:13:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:13:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:13:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:13:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:13:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:13:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:13:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:13:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:13:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:13:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:13:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:13:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:13:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:13:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:13:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:13:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:13:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:13:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:13:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:13:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:13:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:13:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:13:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:13:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:13:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:13:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:13:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:13:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:13:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:13:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:13:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:13:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:13:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:13:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:13:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:13:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:13:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:13:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:13:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:13:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:13:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:13:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:13:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:13:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:13:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:13:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:13:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:13:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:13:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:13:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:13:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:13:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:13:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:13:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:13:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:13:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:13:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:14:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:14:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:14:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:14:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:14:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:14:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:14:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:14:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:14:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:14:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:14:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:14:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:14:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:14:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:14:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:14:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:14:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:14:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:14:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:14:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:14:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:14:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:14:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:14:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:14:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:14:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:14:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:14:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:14:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:14:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:14:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:14:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:14:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:14:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:14:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:14:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:14:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:14:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:14:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:14:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:14:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:14:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:14:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:14:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:14:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:14:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:14:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:14:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:14:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:14:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:14:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:14:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:14:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:14:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:14:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:14:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:14:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:14:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:14:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:14:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:14:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:14:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:14:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:14:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:14:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:14:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:14:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:14:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:14:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:14:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:14:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:14:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:14:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:14:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:14:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:14:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:14:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:14:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:14:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:14:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:14:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:14:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:14:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:14:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:14:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:14:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:14:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:14:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:14:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:14:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:14:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:14:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:14:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:14:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:14:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:14:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:14:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:14:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:14:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:14:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:14:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:14:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:14:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:14:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:14:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:14:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:14:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:14:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:14:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:14:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:14:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:14:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:14:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:14:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:14:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:14:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:14:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:14:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:14:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:14:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:14:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:14:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:14:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:14:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:14:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:14:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:14:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:14:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:14:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:14:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:14:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:14:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:14:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:14:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:14:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:14:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:14:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:14:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:14:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:14:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:14:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:14:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:14:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:14:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:14:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:14:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:14:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:14:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:14:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:14:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:14:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:14:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:14:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:14:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:14:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:14:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:14:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:14:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:14:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:14:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:14:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:14:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:14:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:14:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:14:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:14:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:14:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:14:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:14:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:14:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:14:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:14:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:14:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:14:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:14:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:14:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:14:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:14:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:14:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:14:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:14:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:14:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:14:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:14:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:14:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:14:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:14:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:14:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:14:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:14:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:14:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:14:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:14:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:14:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:14:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:14:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:14:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:14:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:14:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:14:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:14:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:14:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:14:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:14:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:14:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:14:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:14:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:14:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:14:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:14:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:14:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:14:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:14:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:14:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:14:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:14:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:15:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:15:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:15:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:15:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:15:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:15:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:15:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:15:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:15:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:15:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:15:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:15:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:15:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:15:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:15:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:15:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:15:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:15:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:15:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:15:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:15:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:15:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:15:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:15:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:15:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:15:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:15:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:15:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:15:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:15:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:15:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:15:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:15:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:15:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:15:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:15:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:15:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:15:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:15:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:15:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:15:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:15:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:15:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:15:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:15:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:15:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:15:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:15:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:15:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:15:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:15:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:15:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:15:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:15:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:15:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:15:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:15:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:15:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:15:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:15:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:15:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:15:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:15:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:15:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:15:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:15:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:15:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:15:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:15:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:15:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:15:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:15:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:15:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:15:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:15:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:15:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:15:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:15:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:15:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:15:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:15:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:15:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:15:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:15:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:15:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:15:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:15:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:15:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:15:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:15:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:15:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:15:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:15:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:15:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:15:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:15:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:15:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:15:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:15:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:15:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:15:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:15:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:15:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:15:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:15:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:15:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:15:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:15:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:15:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:15:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:15:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:15:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:15:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:15:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:15:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:15:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:15:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:15:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:15:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:15:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:15:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:15:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:15:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:15:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:15:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:15:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:15:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:15:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:15:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:15:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:15:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:15:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:15:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:15:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:15:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:15:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:15:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:15:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:15:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:15:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:15:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:15:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:15:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:15:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:15:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:15:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:15:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:15:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:15:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:15:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:15:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:15:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:15:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:15:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:15:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:15:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:15:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:15:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:15:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:15:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:15:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:15:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:15:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:15:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:15:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:15:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:15:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:15:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:15:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:15:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:15:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:15:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:15:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:15:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:15:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:15:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:15:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:15:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:15:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:15:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:15:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:15:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:15:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:15:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:15:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:15:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:15:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:15:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:15:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:15:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:15:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:15:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:15:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:15:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:15:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:15:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:15:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:15:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:15:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:15:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:15:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:15:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:15:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:15:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:15:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:15:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:15:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:15:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:15:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:15:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:15:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:15:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:15:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:15:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:15:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:15:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:16:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:16:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:16:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:16:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:16:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:16:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:16:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:16:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:16:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:16:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:16:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:16:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:16:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:16:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:16:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:16:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:16:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:16:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:16:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:16:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:16:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:16:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:16:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:16:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:16:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:16:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:16:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:16:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:16:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:16:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:16:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:16:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:16:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:16:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:16:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:16:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:16:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:16:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:16:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:16:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:16:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:16:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:16:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:16:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:16:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:16:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:16:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:16:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:16:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:16:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:16:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:16:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:16:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:16:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:18:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:18:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:18:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:18:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:18:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:18:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:18:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:18:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:18:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:18:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:18:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:18:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:18:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:18:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:18:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:18:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:18:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:18:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:18:20 rdma06 rsyslogd[1041]: imjournal: 5087 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 17:18:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:18:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:18:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:18:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:18:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:18:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:18:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:18:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:18:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:18:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:18:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:18:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:18:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:18:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:18:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:18:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:18:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:18:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:18:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:18:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:18:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:18:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:18:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:18:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:18:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:18:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:18:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:18:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:18:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:18:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:18:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:18:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:18:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:18:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:18:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:18:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:18:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:18:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:18:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:18:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:18:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:18:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:18:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:18:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:18:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:18:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:18:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:18:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:18:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:18:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:18:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:18:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:18:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:18:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:18:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:18:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:18:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:18:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:18:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:18:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:18:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:18:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:18:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:18:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:18:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:18:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:18:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:18:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:18:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:18:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:18:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:18:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:18:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:18:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:18:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:18:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:18:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:18:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:18:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:18:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:18:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:18:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:18:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:18:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:18:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:18:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:18:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:18:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:18:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:18:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:18:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:18:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:18:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:18:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:18:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:18:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:18:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:18:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:18:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:18:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:18:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:18:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:18:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:18:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:18:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:18:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:18:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:18:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:18:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:18:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:18:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:18:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:18:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:18:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:18:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:18:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:18:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:18:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:18:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:18:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:18:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:18:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:18:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:18:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:18:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:18:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:18:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:18:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:18:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:18:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:18:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:18:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:18:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:18:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:18:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:18:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:18:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:18:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:18:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:18:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:18:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:18:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:18:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:18:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:19:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:19:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:19:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:19:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:19:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:19:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:19:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:19:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:19:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:19:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:19:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:19:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:19:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:19:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:19:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:19:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:19:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:19:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:19:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:19:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:19:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:19:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:19:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:19:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:19:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:19:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:19:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:19:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:19:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:19:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:19:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:19:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:19:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:19:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:19:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:19:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:19:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:19:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:19:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:19:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:19:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:19:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:19:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:19:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:19:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:19:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:19:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:19:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:19:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:19:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:19:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:19:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:19:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:19:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:19:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:19:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:19:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:19:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:19:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:19:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:19:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:19:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:19:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:19:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:19:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:19:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:19:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:19:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:19:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:19:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:19:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:19:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:19:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:19:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:19:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:19:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:19:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:19:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:19:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:19:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:19:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:19:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:19:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:19:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:19:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:19:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:19:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:19:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:19:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:19:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:19:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:19:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:19:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:19:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:19:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:19:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:19:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:19:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:19:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:19:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:19:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:19:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:19:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:19:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:19:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:19:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:19:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:19:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:19:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:19:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:19:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:19:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:19:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:19:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:19:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:19:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:19:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:19:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:19:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:19:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:19:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:19:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:19:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:19:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:19:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:19:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:19:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:19:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:19:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:19:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:19:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:19:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:19:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:19:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:19:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:19:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:19:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:19:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:19:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:19:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:19:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:19:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:19:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:19:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:19:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:19:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:19:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:19:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:19:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:19:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:19:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:19:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:19:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:19:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:19:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:19:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:19:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:19:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:19:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:19:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:19:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:19:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:19:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:19:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:19:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:19:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:19:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:19:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:19:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:19:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:19:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:19:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:19:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:19:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:19:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:19:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:19:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:19:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:19:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:19:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:19:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:19:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:19:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:19:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:19:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:19:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:19:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:19:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:19:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:19:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:19:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:19:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:19:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:19:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:19:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:19:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:19:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:19:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:19:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:19:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:19:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:19:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:19:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:19:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:19:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:19:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:19:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:19:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:19:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:19:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:19:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:19:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:19:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:19:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:19:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:19:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:20:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:20:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:20:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:20:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:20:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:20:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:20:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:20:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:20:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:20:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:20:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:20:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:20:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:20:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:20:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:20:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:20:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:20:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:20:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:20:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:20:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:20:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:20:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:20:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:20:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:20:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:20:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:20:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:20:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:20:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:20:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:20:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:20:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:20:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:20:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:20:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:20:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:20:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:20:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:20:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:20:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:20:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:20:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:20:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:20:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:20:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:20:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:20:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:20:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:20:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:20:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:20:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:20:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:20:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:20:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:20:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:20:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:20:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:20:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:20:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:20:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:20:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:20:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:20:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:20:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:20:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:20:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:20:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:20:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:20:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:20:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:20:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:20:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:20:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:20:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:20:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:20:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:20:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:20:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:20:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:20:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:20:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:20:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:20:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:20:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:20:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:20:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:20:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:20:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:20:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:20:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:20:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:20:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:20:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:20:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:20:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:20:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:20:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:20:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:20:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:20:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:20:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:20:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:20:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:20:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:20:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:20:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:20:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:20:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:20:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:20:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:20:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:20:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:20:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:20:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:20:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:20:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:20:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:20:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:20:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:20:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:20:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:20:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:20:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:20:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:20:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:20:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:20:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:20:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:20:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:20:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:20:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:20:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:20:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:20:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:20:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:20:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:20:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:20:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:20:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:20:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:20:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:20:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:20:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:20:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:20:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:20:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:20:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:20:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:20:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:20:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:20:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:20:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:20:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:20:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:20:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:20:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:20:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:20:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:20:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:20:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:20:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:20:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:20:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:20:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:20:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:20:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:20:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:20:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:20:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:20:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:20:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:20:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:20:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:20:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:20:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:20:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:20:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:20:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:20:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:20:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:20:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:20:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:20:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:20:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:20:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:20:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:20:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:20:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:20:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:20:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:20:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:20:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:20:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:20:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:20:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:20:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:20:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:20:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:20:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:20:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:20:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:20:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:20:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:20:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:20:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:20:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:20:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:20:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:20:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:20:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:20:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:20:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:20:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:20:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:20:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:21:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:21:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:21:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:21:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:21:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:21:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:21:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:21:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:21:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:21:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:21:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:21:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:21:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:21:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:21:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:21:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:21:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:21:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:21:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:21:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:21:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:21:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:21:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:21:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:21:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:21:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:21:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:21:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:21:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:21:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:21:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:21:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:21:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:21:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:21:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:21:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:21:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:21:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:21:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:21:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:21:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:21:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:21:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:21:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:21:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:21:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:21:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:21:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:21:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:21:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:21:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:21:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:21:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:21:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:21:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:21:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:21:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:21:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:21:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:21:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:21:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:21:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:21:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:21:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:21:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:21:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:21:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:21:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:21:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:21:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:21:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:21:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:21:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:21:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:21:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:21:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:21:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:21:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:21:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:21:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:21:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:21:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:21:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:21:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:21:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:21:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:21:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:21:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:21:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:21:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:21:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:21:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:21:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:21:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:21:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:21:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:21:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:21:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:21:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:21:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:21:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:21:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:21:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:21:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:21:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:22:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:22:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:22:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:22:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:22:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:22:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:22:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:22:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:22:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:22:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:22:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:22:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:22:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:22:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:22:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:22:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:22:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:22:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:22:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:22:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:22:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:22:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:22:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:22:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:22:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:22:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:22:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:22:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:22:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:22:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:22:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:22:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:22:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:22:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:22:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:22:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:22:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:22:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:22:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:22:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:22:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:22:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:22:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:22:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:22:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:22:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:22:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:22:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:22:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:22:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:22:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:22:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:22:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:22:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:22:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:22:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:22:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:22:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:22:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:22:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:22:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:22:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:22:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:22:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:22:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:22:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:22:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:22:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:22:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:22:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:22:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:22:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:22:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:22:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:22:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:22:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:22:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:22:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:22:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:22:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:22:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:22:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:22:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:22:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:22:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:22:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:22:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:22:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:22:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:22:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:22:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:22:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:22:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:22:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:22:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:22:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:22:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:22:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:22:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:22:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:22:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:22:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:22:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:22:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:22:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:22:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:22:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:22:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:22:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:22:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:22:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:22:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:22:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:22:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:22:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:22:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:22:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:22:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:22:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:22:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:22:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:22:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:22:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:22:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:22:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:22:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:22:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:22:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:22:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:22:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:22:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:22:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:22:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:22:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:22:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:22:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:22:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:22:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:22:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:22:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:22:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:22:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:22:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:22:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:22:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:22:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:22:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:22:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:22:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:22:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:22:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:22:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:22:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:22:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:22:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:22:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:22:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:22:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:22:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:22:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:22:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:22:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:22:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:22:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:22:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:22:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:22:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:22:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:22:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:22:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:22:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:22:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:22:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:22:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:22:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:22:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:22:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:22:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:22:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:22:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:22:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:22:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:22:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:22:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:22:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:22:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:22:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:22:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:22:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:22:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:22:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:22:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:22:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:22:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:22:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:22:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:22:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:22:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:22:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:22:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:22:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:22:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:22:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:22:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:22:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:22:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:22:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:22:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:22:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:22:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:22:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:22:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:23:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:23:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:23:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:23:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:23:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:23:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:23:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:23:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:23:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:23:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:23:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:23:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:23:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:23:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:23:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:23:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:23:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:23:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:23:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:23:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:23:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:23:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:23:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:23:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:23:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:23:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:23:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:23:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:23:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:23:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:23:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:23:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:23:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:23:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:23:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:23:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:23:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:23:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:23:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:23:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:23:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:23:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:23:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:23:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:23:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:23:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:23:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:23:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:23:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:23:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:23:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:23:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:23:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:23:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:23:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:23:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:23:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:23:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:23:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:23:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:23:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:23:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:23:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:23:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:23:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:23:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:23:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:23:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:23:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:23:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:23:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:23:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:23:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:23:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:23:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:23:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:23:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:23:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:23:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:23:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:23:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:23:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:23:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:23:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:23:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:23:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:23:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:23:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:23:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:23:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:23:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:23:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:23:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:23:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:23:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:23:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:23:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:23:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:23:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:23:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:23:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:23:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:23:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:23:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:23:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:23:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:23:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:23:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:23:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:23:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:23:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:23:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:23:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:23:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:23:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:23:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:23:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:23:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:23:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:23:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:23:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:23:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:23:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:23:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:23:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:23:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:23:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:23:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:23:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:23:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:23:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:23:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:23:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:23:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:23:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:23:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:23:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:23:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:23:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:23:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:23:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:23:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:23:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:23:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:23:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:23:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:23:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:23:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:23:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:23:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:23:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:23:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:23:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:23:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:23:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:23:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:23:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:23:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:23:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:23:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:23:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:23:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:23:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:23:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:23:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:23:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:23:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:23:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:23:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:23:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:23:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:23:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:23:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:23:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:23:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:23:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:23:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:23:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:23:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:23:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:23:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:23:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:23:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:23:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:23:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:23:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:23:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:23:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:23:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:23:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:23:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:23:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:23:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:23:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:23:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:23:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:23:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:23:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:23:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:23:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:23:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:23:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:23:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:23:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:23:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:23:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:23:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:23:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:23:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:23:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:23:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:23:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:23:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:23:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:23:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:23:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:24:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:24:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:24:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:24:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:24:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:24:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:24:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:24:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:24:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:24:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:24:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:24:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:24:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:24:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:24:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:24:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:24:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:24:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:24:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:24:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:24:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:24:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:24:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:24:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:24:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:24:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:24:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:24:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:24:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:24:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:24:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:24:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:24:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:24:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:24:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:24:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:24:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:24:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:24:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:24:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:24:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:24:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:24:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:24:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:24:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:24:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:24:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:24:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:24:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:24:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:24:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:24:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:24:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:24:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:24:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:24:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:24:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:24:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:24:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:24:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:24:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:24:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:24:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:24:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:24:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:24:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:24:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:24:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:24:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:24:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:24:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:24:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:24:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:24:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:24:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:24:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:24:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:24:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:24:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:24:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:24:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:24:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:24:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:24:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:24:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:24:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:24:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:24:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:24:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:24:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:24:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:24:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:24:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:24:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:24:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:24:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:24:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:24:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:24:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:24:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:24:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:24:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:24:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:24:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:24:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:24:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:24:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:24:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:24:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:24:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:24:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:24:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:24:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:24:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:24:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:24:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:24:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:24:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:24:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:24:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:24:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:24:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:24:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:24:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:24:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:24:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:24:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:24:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:24:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:24:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:24:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:24:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:24:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:24:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:24:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:24:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:24:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:24:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:24:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:24:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:24:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:24:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:24:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:24:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:24:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:24:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:24:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:24:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:24:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:24:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:24:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:24:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:24:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:24:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:24:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:24:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:24:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:24:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:24:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:24:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:24:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:24:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:24:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:24:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:24:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:24:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:24:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:24:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:24:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:24:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:24:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:24:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:24:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:24:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:24:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:24:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:24:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:24:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:24:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:24:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:24:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:24:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:24:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:24:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:24:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:24:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:24:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:24:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:24:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:24:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:24:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:24:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:24:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:24:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:24:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:24:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:24:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:24:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:24:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:24:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:24:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:24:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:24:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:24:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:24:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:24:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:24:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:24:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:24:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:24:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:24:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:24:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:24:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:24:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:24:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:24:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:25:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:25:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:25:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:25:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:25:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:25:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:25:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:25:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:25:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:25:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:25:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:25:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:25:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:25:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:25:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:25:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:25:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:25:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:25:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:25:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:25:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:25:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:25:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:25:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:25:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:25:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:25:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:25:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:25:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:25:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:25:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:25:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:25:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:25:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:25:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:25:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:25:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:25:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:25:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:25:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:25:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:25:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:25:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:25:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:25:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:25:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:25:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:25:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:25:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:25:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:25:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:25:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:25:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:25:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:25:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:25:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:25:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:25:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:25:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:25:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:25:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:25:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:25:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:25:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:25:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:25:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:25:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:25:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:25:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:25:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:25:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:25:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:25:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:25:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:25:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:25:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:25:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:25:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:25:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:25:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:25:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:25:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:25:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:25:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:25:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:25:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:25:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:25:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:25:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:25:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:25:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:25:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:25:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:25:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:25:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:25:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:25:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:25:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:25:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:25:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:25:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:25:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:25:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:25:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:25:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:25:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:25:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:25:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:25:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:25:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:25:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:25:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:25:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:25:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:25:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:25:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:25:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:25:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:25:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:25:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:25:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:25:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:25:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:25:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:25:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:25:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:25:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:25:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:25:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:25:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:25:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:25:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:25:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:25:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:25:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:25:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:25:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:25:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:25:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:25:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:25:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:25:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:25:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:25:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:25:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:25:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:25:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:25:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:25:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:25:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:25:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:25:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:25:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:25:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:25:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:25:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:25:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:25:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:25:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:25:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:25:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:25:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:25:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:25:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:25:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:25:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:25:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:25:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:25:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:25:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:25:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:25:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:25:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:25:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:25:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:25:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:25:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:25:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:25:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:25:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:25:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:25:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:25:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:25:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:25:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:25:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:25:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:25:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:25:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:25:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:25:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:25:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:25:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:25:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:25:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:25:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:25:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:25:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:25:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:25:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:25:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:25:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:25:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:25:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:25:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:25:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:25:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:25:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:25:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:25:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:25:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:25:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:25:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:25:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:25:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:25:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:26:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:26:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:26:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:26:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:26:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:26:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:26:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:26:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:26:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:26:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:26:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:26:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:26:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:26:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:26:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:26:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:26:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:26:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:26:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:26:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:26:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:26:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:26:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:26:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:26:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:26:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:26:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:26:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:26:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:26:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:26:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:26:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:26:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:26:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:26:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:26:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:26:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:26:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:26:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:26:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:26:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:26:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:26:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:26:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:26:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:26:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:26:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:26:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:26:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:26:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:26:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:26:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:26:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:26:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:26:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:26:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:26:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:26:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:26:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:26:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:26:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:26:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:26:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:26:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:26:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:26:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:26:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:26:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:26:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:26:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:26:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:26:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:28:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:28:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:28:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:28:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:28:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:28:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:28:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:28:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:28:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:28:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:28:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:28:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:28:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:28:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:28:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:28:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:28:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:28:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:28:25 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 17:28:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:28:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:28:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:28:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:28:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:28:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:28:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:28:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:28:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:28:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:28:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:28:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:28:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:28:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:28:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:28:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:28:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:28:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:28:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:28:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:28:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:28:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:28:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:28:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:28:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:28:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:28:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:28:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:28:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:28:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:28:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:28:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:28:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:28:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:28:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:28:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:28:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:28:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:28:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:28:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:28:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:28:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:28:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:28:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:28:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:28:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:28:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:28:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:28:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:28:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:28:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:28:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:28:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:28:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:28:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:28:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:28:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:28:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:28:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:28:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:28:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:28:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:28:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:28:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:28:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:28:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:28:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:28:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:28:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:28:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:28:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:28:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:28:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:28:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:28:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:28:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:28:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:28:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:28:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:28:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:28:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:28:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:28:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:28:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:28:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:28:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:28:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:28:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:28:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:28:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:28:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:28:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:28:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:28:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:28:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:28:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:28:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:28:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:28:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:28:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:28:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:28:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:28:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:28:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:28:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:28:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:28:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:28:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:29:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:29:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:29:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:29:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:29:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:29:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:29:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:29:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:29:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:29:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:29:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:29:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:29:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:29:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:29:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:29:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:29:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:29:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:29:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:29:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:29:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:29:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:29:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:29:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:29:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:29:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:29:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:29:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:29:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:29:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:29:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:29:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:29:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:29:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:29:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:29:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:29:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:29:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:29:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:29:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:29:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:29:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:29:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:29:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:29:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:29:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:29:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:29:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:29:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:29:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:29:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:29:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:29:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:29:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:29:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:29:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:29:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:29:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:29:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:29:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:29:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:29:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:29:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:29:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:29:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:29:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:29:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:29:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:29:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:29:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:29:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:29:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:29:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:29:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:29:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:29:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:29:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:29:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:29:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:29:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:29:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:29:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:29:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:29:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:29:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:29:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:29:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:29:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:29:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:29:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:29:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:29:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:29:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:29:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:29:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:29:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:29:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:29:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:29:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:29:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:29:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:29:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:29:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:29:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:29:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:29:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:29:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:29:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:29:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:29:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:29:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:29:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:29:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:29:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:29:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:29:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:29:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:29:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:29:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:29:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:29:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:29:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:29:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:29:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:29:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:29:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:29:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:29:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:29:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:29:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:29:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:29:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:29:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:29:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:29:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:29:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:29:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:29:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:29:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:29:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:29:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:29:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:29:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:29:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:29:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:29:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:29:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:29:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:29:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:29:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:29:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:29:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:29:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:29:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:29:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:29:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:29:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:29:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:29:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:29:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:29:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:29:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:29:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:29:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:29:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:29:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:29:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:29:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:29:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:29:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:29:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:29:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:29:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:29:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:29:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:29:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:29:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:29:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:29:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:29:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:29:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:29:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:29:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:29:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:29:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:29:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:29:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:29:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:29:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:29:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:29:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:29:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:29:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:29:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:29:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:29:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:29:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:29:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:29:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:29:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:29:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:29:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:29:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:29:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:29:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:29:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:29:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:29:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:29:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:29:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:29:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:29:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:29:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:29:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:29:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:29:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:30:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:30:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:30:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:30:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:30:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:30:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:30:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:30:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:30:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:30:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:30:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:30:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:30:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:30:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:30:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:30:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:30:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:30:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:30:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:30:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:30:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:30:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:30:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:30:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:30:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:30:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:30:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:30:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:30:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:30:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:30:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:30:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:30:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:30:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:30:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:30:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:30:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:30:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:30:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:30:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:30:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:30:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:30:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:30:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:30:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:30:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:30:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:30:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:30:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:30:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:30:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:30:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:30:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:30:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:30:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:30:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:30:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:30:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:30:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:30:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:30:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:30:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:30:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:30:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:30:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:30:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:30:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:30:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:30:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:30:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:30:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:30:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:30:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:30:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:30:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:30:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:30:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:30:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:30:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:30:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:30:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:30:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:30:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:30:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:30:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:30:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:30:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:30:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:30:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:30:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:30:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:30:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:30:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:30:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:30:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:30:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:30:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:30:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:30:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:30:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:30:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:30:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:30:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:30:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:30:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:30:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:30:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:30:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:30:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:30:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:30:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:30:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:30:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:30:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:30:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:30:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:30:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:30:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:30:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:30:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:30:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:30:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:30:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:30:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:30:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:30:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:30:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:30:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:30:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:30:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:30:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:30:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:30:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:30:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:30:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:30:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:30:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:30:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:30:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:30:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:30:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:30:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:30:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:30:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:30:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:30:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:30:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:30:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:30:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:30:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:30:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:30:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:30:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:30:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:30:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:30:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:30:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:30:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:30:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:30:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:30:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:30:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:30:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:30:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:30:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:30:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:30:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:30:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:30:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:30:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:30:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:30:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:30:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:30:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:30:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:30:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:30:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:30:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:30:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:30:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:30:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:30:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:30:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:30:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:30:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:30:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:30:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:30:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:30:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:30:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:30:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:30:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:30:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:30:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:30:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:30:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:30:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:30:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:30:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:30:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:30:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:30:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:30:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:30:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:30:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:30:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:30:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:30:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:30:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:30:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:30:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:30:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:30:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:30:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:30:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:30:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:31:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:31:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:31:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:31:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:31:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:31:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:31:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:31:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:31:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:31:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:31:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:31:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:31:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:31:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:31:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:31:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:31:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:31:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:31:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:31:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:31:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:31:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:31:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:31:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:31:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:31:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:31:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:31:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:31:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:31:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:31:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:31:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:31:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:31:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:31:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:31:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:31:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:31:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:31:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:31:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:31:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:31:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:31:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:31:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:31:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:31:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:31:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:31:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:31:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:31:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:31:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:31:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:31:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:31:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:31:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:31:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:31:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:31:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:31:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:31:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:31:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:31:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:31:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:31:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:31:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:31:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:31:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:31:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:31:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:31:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:31:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:31:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:31:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:31:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:31:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:31:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:31:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:31:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:31:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:31:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:31:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:31:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:31:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:31:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:31:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:31:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:31:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:31:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:31:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:31:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:31:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:31:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:31:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:31:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:31:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:31:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:31:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:31:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:31:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:31:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:31:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:31:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:31:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:31:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:31:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:31:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:31:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:31:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:31:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:31:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:31:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:31:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:31:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:31:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:31:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:31:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:31:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:31:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:31:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:31:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:31:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:31:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:31:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:31:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:31:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:31:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:31:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:31:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:31:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:31:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:31:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:31:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:31:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:31:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:31:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:31:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:31:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:31:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:31:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:31:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:31:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:31:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:31:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:31:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:31:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:31:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:31:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:31:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:31:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:31:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:31:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:31:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:31:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:31:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:31:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:31:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:31:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:31:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:31:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:31:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:31:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:31:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:31:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:31:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:31:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:31:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:31:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:31:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:31:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:31:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:31:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:31:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:31:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:31:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:31:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:31:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:31:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:31:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:31:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:31:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:31:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:31:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:31:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:31:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:31:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:31:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:31:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:31:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:31:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:31:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:31:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:31:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:31:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:31:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:31:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:31:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:31:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:31:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:31:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:31:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:31:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:31:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:31:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:31:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:31:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:31:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:31:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:31:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:31:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:31:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:31:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:31:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:31:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:31:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:31:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:31:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:32:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:32:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:32:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:32:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:32:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:32:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:32:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:32:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:32:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:32:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:32:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:32:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:32:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:32:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:32:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:32:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:32:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:32:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:32:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:32:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:32:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:32:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:32:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:32:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:32:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:32:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:32:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:32:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:32:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:32:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:32:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:32:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:32:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:32:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:32:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:32:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:32:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:32:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:32:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:32:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:32:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:32:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:32:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:32:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:32:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:32:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:32:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:32:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:32:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:32:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:32:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:32:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:32:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:32:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:32:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:32:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:32:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:32:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:32:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:32:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:32:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:32:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:32:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:32:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:32:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:32:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:32:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:32:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:32:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:32:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:32:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:32:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:32:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:32:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:32:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:32:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:32:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:32:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:32:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:32:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:32:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:32:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:32:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:32:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:32:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:32:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:32:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:32:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:32:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:32:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:32:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:32:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:32:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:32:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:32:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:32:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:32:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:32:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:32:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:32:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:32:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:32:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:32:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:32:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:32:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:32:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:32:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:32:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:32:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:32:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:32:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:32:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:32:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:32:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:32:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:32:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:32:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:32:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:32:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:32:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:32:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:32:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:32:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:32:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:32:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:32:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:32:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:32:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:32:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:32:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:32:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:32:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:32:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:32:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:32:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:32:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:32:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:32:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:32:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:32:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:32:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:32:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:32:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:32:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:32:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:32:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:32:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:32:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:32:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:32:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:32:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:32:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:32:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:32:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:32:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:32:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:32:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:32:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:32:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:32:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:32:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:32:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:32:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:32:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:32:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:32:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:32:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:32:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:32:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:32:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:32:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:32:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:32:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:32:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:32:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:32:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:32:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:32:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:32:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:32:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:32:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:32:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:32:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:32:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:32:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:32:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:32:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:32:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:32:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:32:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:32:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:32:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:32:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:32:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:32:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:32:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:32:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:32:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:32:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:32:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:32:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:32:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:32:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:32:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:32:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:32:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:32:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:32:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:32:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:32:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:32:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:32:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:32:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:32:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:32:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:32:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:33:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:33:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:33:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:33:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:33:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:33:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:33:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:33:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:33:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:33:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:33:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:33:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:33:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:33:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:33:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:33:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:33:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:33:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:33:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:33:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:33:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:33:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:33:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:33:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:33:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:33:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:33:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:33:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:33:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:33:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:33:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:33:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:33:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:33:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:33:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:33:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:33:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:33:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:33:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:33:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:33:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:33:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:33:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:33:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:33:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:33:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:33:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:33:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:33:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:33:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:33:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:33:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:33:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:33:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:33:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:33:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:33:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:33:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:33:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:33:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:33:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:33:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:33:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:33:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:33:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:33:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:33:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:33:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:33:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:33:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:33:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:33:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:33:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:33:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:33:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:33:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:33:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:33:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:33:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:33:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:33:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:33:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:33:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:33:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:33:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:33:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:33:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:33:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:33:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:33:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:33:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:33:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:33:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:33:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:33:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:33:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:33:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:33:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:33:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:33:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:33:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:33:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:33:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:33:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:33:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:33:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:33:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:33:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:33:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:33:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:33:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:33:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:33:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:33:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:33:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:33:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:33:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:33:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:33:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:33:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:33:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:33:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:33:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:33:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:33:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:33:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:33:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:33:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:33:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:33:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:33:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:33:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:33:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:33:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:33:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:33:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:33:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:33:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:33:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:33:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:33:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:33:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:33:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:33:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:33:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:33:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:33:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:33:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:33:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:33:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:33:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:33:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:33:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:33:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:33:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:33:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:33:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:33:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:33:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:33:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:33:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:33:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:33:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:33:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:33:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:33:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:33:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:33:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:33:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:33:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:33:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:33:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:33:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:33:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:33:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:33:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:33:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:33:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:33:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:33:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:33:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:33:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:33:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:33:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:33:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:33:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:33:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:33:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:33:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:33:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:33:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:33:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:33:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:33:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:33:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:33:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:33:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:33:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:33:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:33:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:33:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:33:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:33:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:33:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:33:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:33:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:33:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:33:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:33:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:33:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:33:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:33:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:33:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:33:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:33:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:33:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:34:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:34:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:34:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:34:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:34:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:34:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:34:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:34:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:34:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:34:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:34:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:34:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:34:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:34:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:34:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:34:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:34:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:34:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:34:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:34:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:34:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:34:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:34:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:34:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:34:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:34:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:34:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:34:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:34:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:34:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:34:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:34:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:34:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:34:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:34:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:34:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:34:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:34:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:34:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:34:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:34:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:34:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:34:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:34:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:34:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:34:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:34:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:34:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:34:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:34:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:34:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:34:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:34:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:34:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:34:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:34:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:34:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:34:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:34:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:34:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:34:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:34:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:34:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:34:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:34:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:34:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:34:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:34:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:34:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:34:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:34:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:34:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:34:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:34:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:34:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:34:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:34:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:34:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:34:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:34:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:34:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:34:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:34:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:34:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:34:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:34:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:34:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:34:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:34:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:34:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:34:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:34:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:34:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:34:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:34:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:34:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:34:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:34:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:34:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:34:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:34:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:34:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:34:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:34:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:34:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:34:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:34:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:34:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:34:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:34:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:34:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:34:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:34:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:34:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:34:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:34:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:34:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:34:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:34:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:34:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:34:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:34:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:34:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:34:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:34:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:34:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:34:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:34:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:34:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:34:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:34:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:34:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:34:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:34:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:34:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:34:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:34:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:34:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:34:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:34:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:34:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:34:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:34:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:34:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:34:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:34:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:34:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:34:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:34:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:34:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:34:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:34:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:34:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:34:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:34:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:34:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:34:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:34:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:34:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:34:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:34:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:34:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:34:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:34:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:34:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:34:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:34:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:34:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:34:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:34:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:34:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:34:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:34:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:34:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:34:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:34:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:34:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:34:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:34:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:34:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:34:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:34:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:34:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:34:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:34:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:34:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:34:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:34:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:34:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:34:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:34:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:34:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:34:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:34:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:34:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:34:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:34:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:34:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:34:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:34:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:34:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:34:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:34:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:34:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:34:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:34:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:34:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:34:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:34:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:34:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:34:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:34:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:34:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:34:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:34:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:34:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:35:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:35:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:35:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:35:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:35:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:35:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:35:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:35:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:35:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:35:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:35:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:35:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:35:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:35:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:35:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:35:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:35:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:35:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:35:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:35:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:35:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:35:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:35:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:35:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:35:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:35:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:35:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:35:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:35:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:35:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:35:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:35:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:35:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:35:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:35:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:35:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:35:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:35:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:35:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:35:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:35:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:35:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:35:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:35:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:35:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:35:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:35:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:35:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:35:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:35:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:35:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:35:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:35:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:35:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:35:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:35:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:35:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:35:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:35:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:35:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:35:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:35:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:35:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:35:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:35:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:35:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:35:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:35:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:35:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:35:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:35:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:35:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:35:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:35:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:35:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:35:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:35:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:35:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:35:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:35:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:35:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:35:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:35:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:35:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:35:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:35:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:35:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:35:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:35:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:35:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:35:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:35:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:35:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:35:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:35:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:35:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:35:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:35:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:35:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:35:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:35:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:35:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:35:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:35:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:35:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:35:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:35:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:35:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:35:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:35:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:35:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:35:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:35:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:35:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:35:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:35:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:35:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:35:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:35:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:35:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:35:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:35:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:35:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:35:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:35:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:35:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:35:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:35:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:35:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:35:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:35:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:35:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:35:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:35:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:35:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:35:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:35:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:35:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:35:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:35:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:35:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:35:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:35:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:35:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:35:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:35:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:35:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:35:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:35:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:35:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:35:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:35:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:35:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:35:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:35:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:35:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:35:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:35:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:35:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:35:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:35:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:35:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:35:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:35:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:35:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:35:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:35:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:35:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:35:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:35:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:35:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:35:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:35:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:35:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:35:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:35:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:35:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:35:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:35:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:35:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:35:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:35:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:35:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:35:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:35:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:35:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:35:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:35:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:35:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:35:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:35:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:35:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:35:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:35:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:35:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:35:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:35:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:35:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:35:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:35:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:35:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:35:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:35:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:35:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:35:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:35:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:35:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:35:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:35:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:35:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:35:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:35:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:35:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:35:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:35:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:35:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:36:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:36:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:36:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:36:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:36:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:36:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:36:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:36:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:36:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:36:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:36:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:36:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:36:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:36:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:36:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:36:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:36:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:36:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:36:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:36:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:36:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:36:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:36:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:36:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:36:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:36:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:36:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:36:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:36:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:36:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:36:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:36:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:36:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:36:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:36:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:36:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:36:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:36:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:36:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:36:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:36:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:36:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:36:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:36:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:36:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:36:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:36:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:36:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:36:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:36:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:36:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:36:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:36:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:36:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:36:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:36:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:36:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:36:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:36:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:36:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:36:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:36:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:36:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:36:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:36:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:36:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:36:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:36:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:36:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:36:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:36:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:36:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:36:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:36:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:36:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:36:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:36:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:36:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:36:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:36:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:36:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:36:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:36:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:36:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:36:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:36:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:36:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:36:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:36:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:36:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:38:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:38:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:38:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:38:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:38:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:38:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:38:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:38:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:38:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:38:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:38:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:38:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:38:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:38:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:38:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:38:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:38:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:38:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:38:26 rdma06 rsyslogd[1041]: imjournal: 5082 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 17:38:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:38:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:38:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:38:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:38:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:38:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:38:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:38:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:38:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:38:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:38:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:38:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:38:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:38:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:38:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:38:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:38:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:38:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:38:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:38:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:38:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:38:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:38:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:38:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:38:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:38:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:38:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:38:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:38:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:38:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:38:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:38:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:38:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:38:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:38:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:38:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:38:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:38:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:38:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:38:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:38:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:38:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:38:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:38:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:38:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:38:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:38:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:38:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:38:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:38:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:38:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:38:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:38:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:38:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:38:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:38:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:38:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:38:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:38:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:38:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:38:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:38:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:38:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:38:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:38:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:38:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:38:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:38:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:38:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:38:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:38:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:38:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:38:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:38:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:38:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:38:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:38:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:38:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:38:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:38:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:38:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:38:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:38:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:38:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:38:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:38:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:38:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:38:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:38:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:38:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:38:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:38:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:38:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:38:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:38:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:38:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:38:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:38:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:38:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:38:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:38:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:38:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:38:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:38:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:38:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:38:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:38:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:38:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:39:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:39:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:39:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:39:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:39:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:39:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:39:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:39:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:39:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:39:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:39:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:39:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:39:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:39:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:39:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:39:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:39:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:39:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:39:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:39:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:39:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:39:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:39:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:39:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:39:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:39:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:39:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:39:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:39:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:39:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:39:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:39:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:39:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:39:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:39:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:39:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:39:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:39:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:39:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:39:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:39:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:39:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:39:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:39:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:39:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:39:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:39:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:39:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:39:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:39:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:39:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:39:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:39:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:39:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:39:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:39:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:39:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:39:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:39:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:39:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:39:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:39:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:39:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:39:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:39:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:39:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:39:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:39:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:39:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:39:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:39:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:39:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:39:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:39:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:39:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:39:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:39:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:39:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:39:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:39:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:39:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:39:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:39:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:39:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:39:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:39:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:39:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:39:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:39:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:39:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:39:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:39:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:39:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:39:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:39:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:39:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:39:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:39:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:39:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:39:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:39:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:39:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:39:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:39:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:39:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:39:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:39:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:39:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:39:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:39:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:39:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:39:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:39:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:39:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:39:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:39:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:39:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:39:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:39:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:39:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:39:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:39:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:39:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:39:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:39:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:39:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:39:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:39:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:39:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:39:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:39:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:39:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:39:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:39:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:39:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:39:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:39:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:39:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:39:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:39:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:39:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:39:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:39:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:39:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:39:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:39:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:39:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:39:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:39:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:39:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:39:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:39:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:39:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:39:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:39:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:39:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:39:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:39:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:39:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:39:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:39:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:39:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:39:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:39:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:39:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:39:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:39:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:39:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:39:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:39:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:39:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:39:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:39:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:39:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:39:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:39:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:39:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:39:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:39:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:39:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:39:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:39:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:39:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:39:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:39:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:39:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:39:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:39:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:39:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:39:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:39:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:39:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:39:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:39:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:39:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:39:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:39:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:39:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:39:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:39:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:39:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:39:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:39:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:39:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:39:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:39:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:39:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:39:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:39:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:39:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:39:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:39:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:39:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:39:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:39:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:39:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:40:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:40:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:40:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:40:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:40:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:40:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:40:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:40:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:40:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:40:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:40:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:40:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:40:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:40:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:40:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:40:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:40:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:40:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:40:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:40:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:40:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:40:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:40:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:40:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:40:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:40:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:40:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:40:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:40:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:40:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:40:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:40:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:40:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:40:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:40:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:40:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:40:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:40:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:40:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:40:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:40:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:40:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:40:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:40:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:40:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:40:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:40:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:40:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:40:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:40:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:40:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:40:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:40:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:40:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:40:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:40:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:40:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:40:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:40:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:40:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:40:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:40:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:40:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:40:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:40:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:40:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:40:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:40:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:40:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:40:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:40:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:40:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:40:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:40:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:40:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:40:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:40:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:40:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:40:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:40:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:40:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:40:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:40:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:40:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:40:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:40:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:40:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:40:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:40:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:40:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:40:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:40:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:40:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:40:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:40:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:40:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:40:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:40:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:40:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:40:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:40:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:40:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:40:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:40:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:40:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:40:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:40:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:40:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:40:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:40:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:40:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:40:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:40:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:40:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:40:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:40:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:40:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:40:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:40:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:40:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:40:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:40:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:40:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:40:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:40:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:40:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:40:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:40:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:40:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:40:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:40:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:40:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:40:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:40:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:40:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:40:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:40:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:40:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:40:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:40:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:40:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:40:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:40:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:40:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:40:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:40:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:40:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:40:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:40:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:40:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:40:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:40:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:40:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:40:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:40:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:40:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:40:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:40:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:40:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:40:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:40:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:40:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:40:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:40:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:40:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:40:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:40:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:40:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:40:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:40:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:40:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:40:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:40:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:40:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:40:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:40:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:40:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:40:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:40:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:40:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:40:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:40:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:40:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:40:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:40:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:40:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:40:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:40:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:40:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:40:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:40:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:40:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:40:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:40:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:40:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:40:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:40:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:40:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:40:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:40:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:40:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:40:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:40:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:40:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:40:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:40:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:40:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:40:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:40:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:40:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:40:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:40:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:40:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:40:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:40:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:40:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:41:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:41:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:41:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:41:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:41:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:41:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:41:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:41:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:41:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:41:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:41:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:41:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:41:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:41:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:41:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:41:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:41:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:41:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:41:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:41:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:41:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:41:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:41:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:41:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:41:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:41:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:41:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:41:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:41:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:41:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:41:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:41:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:41:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:41:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:41:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:41:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:41:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:41:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:41:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:41:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:41:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:41:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:41:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:41:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:41:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:41:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:41:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:41:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:41:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:41:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:41:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:41:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:41:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:41:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:41:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:41:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:41:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:41:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:41:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:41:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:41:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:41:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:41:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:41:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:41:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:41:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:41:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:41:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:41:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:41:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:41:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:41:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:41:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:41:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:41:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:41:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:41:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:41:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:41:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:41:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:41:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:41:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:41:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:41:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:41:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:41:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:41:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:41:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:41:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:41:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:41:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:41:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:41:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:41:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:41:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:41:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:41:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:41:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:41:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:41:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:41:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:41:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:41:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:41:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:41:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:41:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:41:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:41:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:41:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:41:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:41:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:41:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:41:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:41:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:41:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:41:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:41:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:41:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:41:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:41:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:41:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:41:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:41:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:41:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:41:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:41:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:41:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:41:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:41:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:41:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:41:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:41:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:41:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:41:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:41:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:41:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:41:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:41:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:41:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:41:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:41:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:41:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:41:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:41:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:41:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:41:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:41:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:41:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:41:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:41:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:41:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:41:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:41:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:41:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:41:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:41:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:41:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:41:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:41:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:41:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:41:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:41:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:41:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:41:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:41:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:41:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:41:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:41:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:41:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:41:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:41:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:41:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:41:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:41:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:41:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:41:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:41:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:41:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:41:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:41:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:41:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:41:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:41:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:41:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:41:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:41:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:41:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:41:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:41:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:41:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:41:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:41:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:41:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:41:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:41:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:41:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:41:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:41:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:41:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:41:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:41:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:41:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:41:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:41:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:41:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:41:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:41:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:41:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:41:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:41:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:41:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:41:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:41:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:41:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:41:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:41:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:42:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:42:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:42:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:42:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:42:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:42:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:42:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:42:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:42:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:42:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:42:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:42:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:42:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:42:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:42:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:42:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:42:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:42:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:42:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:42:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:42:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:42:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:42:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:42:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:42:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:42:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:42:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:42:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:42:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:42:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:42:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:42:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:42:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:42:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:42:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:42:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:42:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:42:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:42:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:42:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:42:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:42:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:42:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:42:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:42:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:42:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:42:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:42:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:42:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:42:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:42:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:42:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:42:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:42:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:42:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:42:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:42:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:42:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:42:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:42:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:42:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:42:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:42:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:42:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:42:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:42:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:42:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:42:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:42:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:42:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:42:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:42:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:42:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:42:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:42:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:42:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:42:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:42:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:42:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:42:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:42:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:42:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:42:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:42:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:42:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:42:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:42:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:42:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:42:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:42:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:42:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:42:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:42:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:42:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:42:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:42:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:42:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:42:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:42:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:42:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:42:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:42:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:42:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:42:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:42:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:42:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:42:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:42:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:42:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:42:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:42:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:42:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:42:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:42:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:42:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:42:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:42:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:42:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:42:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:42:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:42:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:42:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:42:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:42:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:42:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:42:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:42:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:42:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:42:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:42:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:42:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:42:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:42:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:42:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:42:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:42:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:42:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:42:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:42:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:42:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:42:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:42:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:42:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:42:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:42:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:42:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:42:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:42:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:42:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:42:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:42:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:42:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:42:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:42:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:42:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:42:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:42:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:42:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:42:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:42:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:42:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:42:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:42:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:42:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:42:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:42:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:42:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:42:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:42:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:42:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:42:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:42:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:42:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:42:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:42:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:42:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:42:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:42:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:42:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:42:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:42:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:42:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:42:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:42:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:42:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:42:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:42:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:42:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:42:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:42:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:42:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:42:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:42:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:42:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:42:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:42:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:42:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:42:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:42:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:42:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:42:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:42:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:42:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:42:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:42:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:42:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:42:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:42:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:42:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:42:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:42:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:42:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:42:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:42:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:42:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:42:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:43:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:43:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:43:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:43:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:43:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:43:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:43:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:43:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:43:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:43:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:43:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:43:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:43:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:43:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:43:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:43:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:43:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:43:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:43:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:43:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:43:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:43:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:43:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:43:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:43:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:43:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:43:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:43:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:43:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:43:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:43:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:43:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:43:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:43:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:43:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:43:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:43:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:43:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:43:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:43:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:43:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:43:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:43:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:43:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:43:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:43:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:43:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:43:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:43:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:43:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:43:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:43:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:43:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:43:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:43:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:43:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:43:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:43:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:43:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:43:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:43:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:43:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:43:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:43:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:43:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:43:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:43:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:43:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:43:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:43:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:43:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:43:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:43:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:43:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:43:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:43:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:43:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:43:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:43:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:43:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:43:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:43:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:43:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:43:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:43:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:43:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:43:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:43:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:43:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:43:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:43:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:43:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:43:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:43:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:43:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:43:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:43:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:43:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:43:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:43:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:43:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:43:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:43:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:43:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:43:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:43:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:43:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:43:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:43:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:43:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:43:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:43:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:43:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:43:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:43:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:43:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:43:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:43:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:43:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:43:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:43:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:43:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:43:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:43:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:43:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:43:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:43:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:43:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:43:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:43:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:43:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:43:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:43:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:43:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:43:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:43:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:43:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:43:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:43:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:43:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:43:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:43:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:43:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:43:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:43:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:43:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:43:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:43:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:43:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:43:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:43:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:43:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:43:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:43:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:43:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:43:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:43:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:43:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:43:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:43:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:43:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:43:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:43:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:43:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:43:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:43:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:43:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:43:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:43:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:43:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:43:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:43:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:43:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:43:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:43:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:43:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:43:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:43:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:43:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:43:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:43:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:43:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:43:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:43:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:43:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:43:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:43:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:43:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:43:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:43:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:43:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:43:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:43:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:43:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:43:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:43:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:43:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:43:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:43:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:43:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:43:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:43:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:43:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:43:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:43:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:43:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:43:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:43:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:43:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:43:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:43:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:43:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:43:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:43:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:43:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:43:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:44:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:44:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:44:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:44:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:44:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:44:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:44:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:44:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:44:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:44:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:44:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:44:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:44:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:44:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:44:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:44:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:44:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:44:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:44:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:44:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:44:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:44:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:44:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:44:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:44:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:44:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:44:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:44:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:44:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:44:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:44:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:44:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:44:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:44:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:44:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:44:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:44:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:44:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:44:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:44:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:44:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:44:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:44:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:44:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:44:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:44:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:44:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:44:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:44:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:44:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:44:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:44:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:44:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:44:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:44:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:44:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:44:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:44:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:44:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:44:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:44:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:44:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:44:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:44:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:44:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:44:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:44:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:44:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:44:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:44:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:44:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:44:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:44:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:44:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:44:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:44:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:44:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:44:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:44:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:44:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:44:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:44:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:44:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:44:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:44:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:44:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:44:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:44:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:44:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:44:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:44:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:44:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:44:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:44:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:44:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:44:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:44:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:44:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:44:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:44:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:44:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:44:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:44:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:44:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:44:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:44:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:44:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:44:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:44:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:44:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:44:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:44:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:44:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:44:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:44:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:44:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:44:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:44:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:44:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:44:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:44:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:44:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:44:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:44:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:44:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:44:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:44:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:44:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:44:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:44:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:44:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:44:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:44:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:44:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:44:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:44:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:44:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:44:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:44:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:44:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:44:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:44:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:44:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:44:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:44:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:44:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:44:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:44:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:44:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:44:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:44:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:44:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:44:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:44:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:44:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:44:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:44:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:44:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:44:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:44:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:44:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:44:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:44:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:44:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:44:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:44:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:44:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:44:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:44:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:44:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:44:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:44:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:44:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:44:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:44:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:44:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:44:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:44:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:44:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:44:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:44:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:44:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:44:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:44:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:44:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:44:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:44:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:44:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:44:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:44:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:44:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:44:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:44:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:44:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:44:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:44:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:44:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:44:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:44:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:44:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:44:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:44:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:44:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:44:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:44:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:44:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:44:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:44:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:44:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:44:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:44:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:44:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:44:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:44:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:44:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:44:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:45:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:45:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:45:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:45:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:45:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:45:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:45:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:45:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:45:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:45:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:45:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:45:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:45:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:45:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:45:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:45:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:45:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:45:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:45:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:45:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:45:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:45:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:45:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:45:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:45:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:45:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:45:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:45:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:45:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:45:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:45:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:45:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:45:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:45:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:45:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:45:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:45:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:45:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:45:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:45:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:45:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:45:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:45:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:45:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:45:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:45:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:45:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:45:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:45:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:45:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:45:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:45:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:45:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:45:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:45:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:45:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:45:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:45:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:45:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:45:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:45:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:45:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:45:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:45:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:45:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:45:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:45:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:45:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:45:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:45:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:45:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:45:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:45:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:45:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:45:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:45:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:45:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:45:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:45:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:45:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:45:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:45:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:45:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:45:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:45:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:45:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:45:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:45:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:45:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:45:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:45:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:45:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:45:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:45:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:45:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:45:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:45:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:45:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:45:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:45:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:45:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:45:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:45:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:45:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:45:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:45:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:45:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:45:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:45:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:45:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:45:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:45:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:45:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:45:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:45:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:45:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:45:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:45:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:45:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:45:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:45:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:45:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:45:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:45:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:45:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:45:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:45:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:45:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:45:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:45:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:45:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:45:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:45:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:45:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:45:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:45:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:45:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:45:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:45:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:45:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:45:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:45:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:45:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:45:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:45:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:45:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:45:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:45:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:45:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:45:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:45:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:45:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:45:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:45:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:45:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:45:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:45:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:45:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:45:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:45:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:45:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:45:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:45:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:45:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:45:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:45:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:45:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:45:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:45:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:45:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:45:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:45:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:45:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:45:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:45:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:45:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:45:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:45:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:45:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:45:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:45:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:45:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:45:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:45:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:45:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:45:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:45:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:45:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:45:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:45:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:45:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:45:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:45:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:45:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:45:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:45:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:45:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:45:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:45:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:45:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:45:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:45:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:45:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:45:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:45:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:45:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:45:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:45:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:45:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:45:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:45:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:45:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:45:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:45:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:45:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:45:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:46:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:46:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:46:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:46:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:46:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:46:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:46:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:46:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:46:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:46:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:46:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:46:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:46:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:46:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:46:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:46:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:46:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:46:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:46:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:46:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:46:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:46:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:46:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:46:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:46:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:46:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:46:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:46:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:46:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:46:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:46:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:46:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:46:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:46:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:46:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:46:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:46:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:46:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:46:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:46:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:46:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:46:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:46:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:46:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:46:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:46:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:46:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:46:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:46:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:46:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:46:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:46:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:46:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:46:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:46:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:46:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:46:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:46:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:46:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:46:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:46:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:46:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:46:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:46:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:46:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:46:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:46:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:46:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:46:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:46:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:46:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:46:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:46:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:46:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:46:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:46:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:46:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:46:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:46:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:46:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:46:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:46:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:46:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:46:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:46:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:46:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:46:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:46:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:46:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:46:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:48:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:48:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:48:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:48:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:48:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:48:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:48:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:48:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:48:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:48:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:48:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:48:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:48:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:48:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:48:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:48:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:48:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:48:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:48:31 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 17:48:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:48:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:48:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:48:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:48:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:48:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:48:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:48:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:48:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:48:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:48:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:48:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:48:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:48:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:48:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:48:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:48:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:48:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:48:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:48:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:48:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:48:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:48:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:48:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:48:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:48:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:48:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:48:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:48:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:48:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:48:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:48:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:48:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:48:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:48:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:48:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:48:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:48:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:48:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:48:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:48:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:48:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:48:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:48:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:48:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:48:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:48:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:48:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:48:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:48:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:48:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:48:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:48:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:48:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:48:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:48:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:48:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:48:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:48:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:48:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:48:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:48:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:48:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:48:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:48:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:48:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:48:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:48:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:48:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:48:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:48:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:48:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:48:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:48:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:48:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:48:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:48:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:48:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:48:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:48:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:48:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:48:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:48:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:48:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:48:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:48:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:48:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:48:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:48:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:48:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:49:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:49:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:49:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:49:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:49:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:49:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:49:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:49:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:49:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:49:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:49:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:49:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:49:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:49:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:49:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:49:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:49:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:49:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:49:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:49:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:49:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:49:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:49:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:49:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:49:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:49:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:49:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:49:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:49:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:49:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:49:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:49:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:49:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:49:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:49:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:49:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:49:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:49:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:49:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:49:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:49:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:49:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:49:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:49:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:49:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:49:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:49:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:49:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:49:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:49:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:49:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:49:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:49:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:49:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:49:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:49:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:49:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:49:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:49:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:49:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:49:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:49:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:49:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:49:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:49:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:49:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:49:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:49:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:49:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:49:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:49:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:49:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:49:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:49:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:49:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:49:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:49:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:49:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:49:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:49:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:49:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:49:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:49:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:49:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:49:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:49:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:49:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:49:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:49:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:49:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:49:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:49:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:49:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:49:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:49:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:49:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:49:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:49:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:49:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:49:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:49:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:49:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:49:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:49:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:49:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:49:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:49:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:49:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:49:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:49:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:49:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:49:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:49:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:49:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:49:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:49:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:49:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:49:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:49:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:49:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:49:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:49:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:49:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:49:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:49:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:49:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:49:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:49:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:49:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:49:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:49:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:49:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:49:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:49:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:49:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:49:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:49:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:49:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:49:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:49:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:49:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:49:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:49:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:49:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:49:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:49:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:49:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:49:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:49:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:49:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:49:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:49:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:49:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:49:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:49:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:49:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:49:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:49:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:49:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:49:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:49:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:49:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:49:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:49:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:49:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:49:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:49:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:49:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:49:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:49:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:49:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:49:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:49:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:49:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:49:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:49:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:49:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:49:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:49:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:49:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:49:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:49:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:49:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:49:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:49:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:49:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:49:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:49:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:49:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:49:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:49:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:49:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:49:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:49:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:49:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:49:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:49:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:49:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:49:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:49:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:49:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:49:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:49:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:49:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:49:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:49:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:49:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:49:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:49:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:49:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:49:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:49:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:49:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:49:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:49:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:49:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:50:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:50:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:50:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:50:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:50:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:50:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:50:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:50:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:50:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:50:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:50:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:50:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:50:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:50:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:50:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:50:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:50:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:50:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:50:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:50:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:50:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:50:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:50:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:50:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:50:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:50:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:50:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:50:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:50:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:50:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:50:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:50:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:50:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:50:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:50:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:50:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:50:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:50:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:50:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:50:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:50:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:50:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:50:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:50:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:50:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:50:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:50:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:50:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:50:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:50:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:50:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:50:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:50:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:50:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:50:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:50:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:50:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:50:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:50:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:50:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:50:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:50:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:50:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:50:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:50:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:50:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:50:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:50:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:50:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:50:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:50:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:50:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:50:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:50:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:50:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:50:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:50:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:50:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:50:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:50:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:50:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:50:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:50:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:50:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:50:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:50:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:50:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:50:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:50:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:50:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:50:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:50:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:50:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:50:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:50:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:50:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:50:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:50:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:50:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:50:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:50:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:50:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:50:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:50:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:50:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:50:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:50:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:50:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:50:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:50:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:50:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:50:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:50:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:50:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:50:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:50:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:50:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:50:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:50:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:50:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:50:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:50:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:50:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:50:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:50:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:50:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:50:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:50:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:50:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:50:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:50:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:50:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:50:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:50:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:50:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:50:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:50:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:50:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:50:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:50:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:50:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:50:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:50:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:50:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:50:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:50:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:50:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:50:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:50:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:50:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:50:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:50:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:50:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:50:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:50:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:50:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:50:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:50:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:50:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:50:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:50:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:50:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:50:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:50:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:50:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:50:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:50:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:50:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:50:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:50:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:50:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:50:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:50:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:50:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:50:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:50:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:50:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:50:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:50:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:50:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:50:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:50:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:50:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:50:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:50:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:50:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:50:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:50:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:50:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:50:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:50:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:50:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:50:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:50:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:50:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:50:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:50:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:50:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:50:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:50:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:50:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:50:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:50:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:50:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:50:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:50:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:50:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:50:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:50:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:50:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:50:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:50:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:50:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:50:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:50:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:50:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:51:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:51:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:51:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:51:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:51:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:51:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:51:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:51:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:51:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:51:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:51:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:51:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:51:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:51:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:51:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:51:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:51:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:51:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:51:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:51:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:51:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:51:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:51:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:51:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:51:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:51:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:51:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:51:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:51:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:51:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:51:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:51:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:51:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:51:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:51:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:51:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:51:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:51:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:51:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:51:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:51:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:51:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:51:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:51:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:51:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:51:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:51:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:51:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:51:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:51:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:51:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:51:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:51:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:51:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:51:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:51:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:51:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:51:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:51:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:51:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:51:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:51:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:51:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:51:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:51:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:51:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:51:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:51:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:51:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:51:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:51:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:51:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:51:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:51:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:51:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:51:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:51:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:51:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:51:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:51:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:51:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:51:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:51:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:51:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:51:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:51:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:51:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:51:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:51:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:51:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:51:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:51:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:51:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:51:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:51:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:51:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:51:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:51:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:51:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:51:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:51:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:51:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:51:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:51:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:51:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:51:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:51:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:51:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:51:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:51:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:51:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:51:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:51:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:51:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:51:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:51:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:51:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:51:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:51:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:51:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:51:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:51:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:51:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:51:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:51:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:51:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:51:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:51:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:51:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:51:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:51:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:51:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:51:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:51:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:51:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:51:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:51:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:51:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:51:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:51:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:51:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:51:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:51:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:51:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:51:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:51:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:51:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:51:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:51:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:51:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:51:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:51:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:51:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:51:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:51:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:51:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:51:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:51:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:51:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:51:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:51:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:51:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:51:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:51:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:51:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:51:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:51:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:51:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:51:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:51:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:51:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:51:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:51:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:51:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:51:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:51:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:51:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:51:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:51:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:51:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:51:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:51:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:51:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:51:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:51:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:51:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:51:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:51:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:51:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:52:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:52:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:52:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:52:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:52:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:52:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:52:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:52:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:52:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:52:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:52:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:52:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:52:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:52:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:52:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:52:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:52:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:52:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:52:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:52:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:52:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:52:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:52:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:52:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:52:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:52:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:52:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:52:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:52:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:52:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:52:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:52:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:52:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:52:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:52:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:52:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:52:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:52:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:52:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:52:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:52:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:52:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:52:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:52:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:52:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:52:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:52:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:52:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:52:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:52:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:52:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:52:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:52:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:52:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:52:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:52:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:52:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:52:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:52:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:52:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:52:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:52:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:52:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:52:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:52:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:52:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:52:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:52:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:52:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:52:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:52:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:52:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:52:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:52:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:52:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:52:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:52:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:52:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:52:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:52:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:52:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:52:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:52:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:52:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:52:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:52:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:52:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:52:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:52:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:52:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:52:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:52:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:52:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:52:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:52:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:52:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:52:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:52:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:52:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:52:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:52:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:52:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:52:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:52:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:52:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:52:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:52:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:52:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:52:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:52:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:52:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:52:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:52:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:52:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:52:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:52:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:52:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:52:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:52:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:52:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:52:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:52:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:52:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:52:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:52:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:52:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:52:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:52:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:52:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:52:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:52:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:52:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:52:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:52:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:52:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:52:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:52:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:52:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:52:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:52:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:52:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:52:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:52:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:52:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:52:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:52:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:52:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:52:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:52:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:52:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:52:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:52:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:52:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:52:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:52:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:52:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:52:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:52:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:52:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:52:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:52:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:52:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:52:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:52:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:52:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:52:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:52:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:52:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:52:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:52:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:52:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:52:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:52:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:52:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:52:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:52:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:52:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:52:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:52:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:52:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:52:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:52:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:52:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:52:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:52:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:52:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:52:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:52:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:52:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:52:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:52:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:52:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:52:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:52:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:52:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:52:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:52:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:52:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:52:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:52:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:52:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:52:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:52:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:52:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:52:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:52:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:52:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:52:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:52:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:52:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:52:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:52:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:52:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:52:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:52:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:52:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:53:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:53:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:53:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:53:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:53:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:53:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:53:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:53:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:53:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:53:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:53:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:53:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:53:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:53:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:53:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:53:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:53:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:53:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:53:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:53:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:53:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:53:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:53:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:53:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:53:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:53:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:53:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:53:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:53:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:53:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:53:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:53:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:53:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:53:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:53:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:53:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:53:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:53:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:53:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:53:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:53:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:53:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:53:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:53:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:53:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:53:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:53:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:53:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:53:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:53:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:53:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:53:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:53:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:53:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:53:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:53:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:53:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:53:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:53:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:53:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:53:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:53:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:53:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:53:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:53:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:53:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:53:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:53:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:53:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:53:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:53:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:53:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:53:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:53:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:53:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:53:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:53:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:53:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:53:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:53:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:53:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:53:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:53:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:53:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:53:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:53:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:53:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:53:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:53:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:53:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:53:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:53:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:53:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:53:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:53:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:53:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:53:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:53:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:53:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:53:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:53:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:53:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:53:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:53:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:53:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:53:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:53:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:53:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:53:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:53:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:53:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:53:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:53:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:53:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:53:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:53:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:53:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:53:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:53:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:53:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:53:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:53:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:53:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:53:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:53:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:53:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:53:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:53:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:53:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:53:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:53:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:53:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:53:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:53:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:53:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:53:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:53:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:53:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:53:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:53:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:53:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:53:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:53:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:53:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:53:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:53:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:53:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:53:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:53:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:53:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:53:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:53:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:53:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:53:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:53:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:53:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:53:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:53:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:53:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:53:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:53:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:53:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:53:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:53:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:53:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:53:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:53:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:53:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:53:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:53:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:53:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:53:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:53:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:53:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:53:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:53:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:53:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:53:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:53:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:53:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:53:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:53:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:53:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:53:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:53:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:53:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:53:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:53:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:53:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:53:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:53:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:53:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:53:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:53:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:53:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:53:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:53:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:53:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:53:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:53:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:53:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:53:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:53:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:53:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:53:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:53:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:53:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:53:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:53:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:53:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:53:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:53:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:53:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:53:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:53:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:53:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:54:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:54:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:54:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:54:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:54:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:54:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:54:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:54:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:54:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:54:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:54:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:54:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:54:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:54:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:54:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:54:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:54:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:54:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:54:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:54:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:54:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:54:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:54:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:54:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:54:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:54:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:54:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:54:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:54:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:54:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:54:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:54:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:54:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:54:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:54:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:54:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:54:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:54:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:54:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:54:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:54:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:54:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:54:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:54:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:54:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:54:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:54:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:54:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:54:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:54:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:54:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:54:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:54:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:54:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:54:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:54:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:54:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:54:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:54:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:54:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:54:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:54:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:54:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:54:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:54:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:54:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:54:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:54:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:54:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:54:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:54:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:54:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:54:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:54:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:54:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:54:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:54:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:54:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:54:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:54:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:54:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:54:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:54:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:54:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:54:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:54:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:54:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:54:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:54:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:54:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:54:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:54:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:54:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:54:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:54:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:54:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:54:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:54:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:54:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:54:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:54:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:54:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:54:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:54:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:54:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:54:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:54:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:54:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:54:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:54:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:54:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:54:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:54:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:54:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:54:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:54:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:54:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:54:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:54:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:54:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:54:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:54:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:54:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:54:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:54:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:54:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:54:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:54:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:54:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:54:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:54:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:54:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:54:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:54:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:54:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:54:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:54:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:54:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:54:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:54:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:54:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:54:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:54:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:54:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:54:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:54:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:54:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:54:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:54:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:54:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:54:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:54:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:54:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:54:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:54:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:54:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:54:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:54:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:54:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:54:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:54:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:54:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:54:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:54:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:54:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:54:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:54:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:54:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:54:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:54:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:54:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:54:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:54:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:54:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:54:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:54:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:54:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:54:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:54:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:54:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:54:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:54:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:54:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:54:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:54:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:54:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:54:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:54:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:54:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:54:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:54:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:54:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:54:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:54:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:54:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:54:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:54:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:54:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:54:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:54:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:54:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:54:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:54:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:54:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:54:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:54:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:54:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:54:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:54:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:54:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:54:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:54:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:54:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:54:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:54:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:54:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:55:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:55:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:55:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:55:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:55:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:55:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:55:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:55:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:55:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:55:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:55:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:55:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:55:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:55:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:55:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:55:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:55:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:55:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:55:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:55:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:55:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:55:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:55:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:55:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:55:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:55:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:55:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:55:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:55:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:55:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:55:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:55:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:55:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:55:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:55:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:55:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:55:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:55:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:55:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:55:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:55:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:55:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:55:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:55:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:55:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:55:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:55:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:55:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:55:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:55:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:55:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:55:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:55:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:55:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:55:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:55:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:55:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:55:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:55:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:55:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:55:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:55:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:55:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:55:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:55:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:55:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:55:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:55:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:55:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:55:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:55:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:55:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:55:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:55:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:55:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:55:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:55:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:55:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:55:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:55:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:55:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:55:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:55:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:55:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:55:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:55:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:55:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:55:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:55:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:55:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:55:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:55:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:55:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:55:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:55:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:55:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:55:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:55:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:55:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:55:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:55:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:55:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:55:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:55:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:55:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:55:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:55:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:55:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:55:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:55:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:55:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:55:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:55:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:55:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:55:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:55:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:55:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:55:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:55:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:55:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:55:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:55:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:55:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:55:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:55:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:55:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:55:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:55:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:55:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:55:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:55:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:55:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:55:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:55:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:55:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:55:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:55:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:55:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:55:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:55:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:55:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:55:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:55:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:55:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:55:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:55:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:55:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:55:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:55:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:55:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:55:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:55:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:55:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:55:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:55:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:55:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:55:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:55:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:55:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:55:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:55:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:55:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:55:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:55:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:55:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:55:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:55:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:55:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:55:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:55:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:55:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:55:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:55:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:55:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:55:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:55:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:55:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:55:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:55:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:55:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:55:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:55:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:55:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:55:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:55:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:55:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:55:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:55:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:55:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:55:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:55:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:55:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:55:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:55:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:55:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:55:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:55:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:55:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:55:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:55:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:55:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:55:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:55:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:55:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:55:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:55:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:55:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:55:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:55:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:55:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:55:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:55:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:55:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:55:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:55:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:55:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:56:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:56:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:56:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:56:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:56:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:56:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:56:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:56:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:56:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:56:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:56:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:56:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:56:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:56:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:56:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:56:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:56:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:56:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:56:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:56:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:56:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:56:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:56:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:56:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:56:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:56:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:56:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:56:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:56:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:56:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:56:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:56:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:56:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:56:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:56:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:56:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:56:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:56:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:56:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:56:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:56:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:56:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:56:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:56:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:56:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:56:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:56:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:56:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:56:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:56:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:56:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:56:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:56:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:56:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:56:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:56:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:56:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:56:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:56:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:56:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:56:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:56:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:56:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:56:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:56:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:56:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:56:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:56:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:56:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:56:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:56:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:56:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:56:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:56:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:56:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:56:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:56:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:56:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:56:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:56:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:56:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:56:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:56:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:56:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:56:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:56:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:56:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:56:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:56:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:56:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:56:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:56:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:56:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:56:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:56:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:56:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:56:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:56:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:56:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:56:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:56:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:56:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:56:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:56:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:56:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:56:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:56:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:56:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:58:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:58:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:58:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:58:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:58:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:58:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:58:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:58:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:58:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:58:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:58:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:58:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:58:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:58:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:58:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:58:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:58:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:58:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:58:36 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 17:58:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:58:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:58:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:58:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:58:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:58:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:58:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:58:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:58:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:58:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:58:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:58:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:58:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:58:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:58:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:58:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:58:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:58:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:58:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:58:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:58:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:58:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:58:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:58:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:58:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:58:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:58:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:58:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:58:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:58:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:58:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:58:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:58:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:58:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:58:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:58:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:58:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:58:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:58:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:58:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:58:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:58:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:58:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:58:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:58:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:58:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:58:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:58:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:58:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:58:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:58:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:58:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:58:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:58:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:58:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:58:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:58:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:58:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:58:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:58:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:58:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:58:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:58:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:58:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:58:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:58:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:58:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:58:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:58:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:58:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:58:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:58:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:59:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:59:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:59:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:59:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:59:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:59:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:59:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:59:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:59:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:59:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:59:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:59:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:59:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:59:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:59:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:59:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:59:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:59:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:59:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:59:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:59:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:59:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:59:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:59:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:59:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:59:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:59:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:59:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:59:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:59:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:59:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:59:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:59:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:59:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:59:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:59:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:59:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:59:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:59:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:59:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:59:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:59:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:59:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:59:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:59:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:59:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:59:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:59:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:59:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:59:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:59:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:59:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:59:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:59:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:59:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:59:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:59:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:59:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:59:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:59:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:59:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:59:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:59:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:59:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:59:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:59:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:59:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:59:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:59:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:59:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:59:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:59:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:59:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:59:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:59:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:59:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:59:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:59:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:59:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:59:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:59:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:59:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:59:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:59:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:59:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:59:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:59:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:59:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:59:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:59:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:59:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:59:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:59:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:59:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:59:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:59:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:59:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:59:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:59:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:59:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:59:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:59:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:59:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:59:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:59:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:59:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:59:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:59:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:59:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:59:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:59:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:59:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:59:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:59:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:59:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:59:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:59:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:59:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:59:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:59:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:59:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:59:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:59:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:59:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:59:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:59:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:59:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:59:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:59:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:59:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:59:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:59:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:59:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:59:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:59:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:59:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:59:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:59:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:59:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:59:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:59:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:59:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:59:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:59:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:59:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:59:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:59:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:59:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:59:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:59:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:59:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:59:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:59:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:59:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:59:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:59:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:59:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:59:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:59:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:59:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:59:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:59:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:59:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:59:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:59:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:59:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:59:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:59:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:59:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:59:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:59:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:59:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:59:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:59:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:59:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:59:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:59:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:59:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:59:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:59:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:59:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:59:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:59:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:59:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:59:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:59:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:59:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:59:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:59:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:59:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:59:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:59:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:59:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:59:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:59:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:59:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:59:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:59:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 17:59:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 17:59:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:59:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:59:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:59:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 17:59:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 17:59:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 17:59:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 17:59:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 17:59:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 17:59:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 17:59:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 17:59:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 17:59:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 17:59:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 17:59:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 17:59:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 17:59:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:00:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:00:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:00:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:00:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:00:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:00:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:00:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:00:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:00:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:00:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:00:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:00:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:00:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:00:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:00:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:00:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:00:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:00:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:00:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:00:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:00:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:00:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:00:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:00:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:00:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:00:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:00:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:00:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:00:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:00:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:00:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:00:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:00:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:00:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:00:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:00:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:00:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:00:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:00:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:00:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:00:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:00:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:00:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:00:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:00:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:00:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:00:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:00:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:00:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:00:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:00:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:00:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:00:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:00:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:00:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:00:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:00:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:00:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:00:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:00:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:00:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:00:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:00:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:00:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:00:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:00:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:00:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:00:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:00:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:00:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:00:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:00:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:00:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:00:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:00:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:00:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:00:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:00:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:00:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:00:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:00:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:00:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:00:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:00:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:00:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:00:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:00:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:00:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:00:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:00:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:00:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:00:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:00:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:00:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:00:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:00:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:00:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:00:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:00:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:00:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:00:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:00:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:00:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:00:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:00:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:00:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:00:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:00:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:00:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:00:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:00:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:00:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:00:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:00:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:00:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:00:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:00:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:00:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:00:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:00:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:00:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:00:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:00:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:00:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:00:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:00:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:00:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:00:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:00:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:00:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:00:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:00:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:00:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:00:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:00:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:00:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:00:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:00:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:00:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:00:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:00:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:00:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:00:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:00:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:00:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:00:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:00:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:00:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:00:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:00:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:00:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:00:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:00:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:00:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:00:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:00:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:00:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:00:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:00:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:00:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:00:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:00:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:00:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:00:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:00:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:00:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:00:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:00:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:00:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:00:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:00:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:00:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:00:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:00:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:00:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:00:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:00:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:00:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:00:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:00:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:00:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:00:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:00:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:00:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:00:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:00:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:00:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:00:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:00:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:00:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:00:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:00:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:00:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:00:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:00:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:00:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:00:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:00:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:00:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:00:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:00:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:00:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:00:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:00:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:00:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:00:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:00:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:00:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:00:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:00:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:00:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:00:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:00:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:00:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:00:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:00:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:01:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:01:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:01:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:01:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:01:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:01:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:01:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:01:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:01:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:01:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:01:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:01:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:01:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:01:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:01:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:01:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:01:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:01:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:01:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:01:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:01:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:01:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:01:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:01:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:01:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:01:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:01:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:01:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:01:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:01:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:01:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:01:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:01:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:01:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:01:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:01:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:01:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:01:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:01:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:01:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:01:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:01:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:01:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:01:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:01:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:01:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:01:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:01:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:01:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:01:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:01:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:01:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:01:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:01:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:01:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:01:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:01:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:01:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:01:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:01:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:01:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:01:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:01:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:01:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:01:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:01:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:01:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:01:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:01:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:01:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:01:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:01:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:01:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:01:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:01:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:01:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:01:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:01:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:01:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:01:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:01:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:01:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:01:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:01:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:01:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:01:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:01:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:01:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:01:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:01:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:01:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:01:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:01:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:01:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:01:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:01:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:01:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:01:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:01:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:01:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:01:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:01:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:01:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:01:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:01:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:01:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:01:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:01:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:01:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:01:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:01:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:01:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:01:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:01:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:01:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:01:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:01:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:01:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:01:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:01:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:01:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:01:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:01:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:01:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:01:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:01:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:01:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:01:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:01:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:01:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:01:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:01:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:01:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:01:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:01:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:01:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:01:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:01:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:01:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:01:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:01:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:01:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:01:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:01:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:01:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:01:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:01:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:01:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:01:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:01:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:01:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:01:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:01:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:01:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:01:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:01:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:01:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:01:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:01:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:01:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:01:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:01:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:01:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:01:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:01:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:01:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:01:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:01:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:01:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:01:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:01:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:01:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:01:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:01:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:01:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:01:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:01:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:01:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:01:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:01:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:01:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:01:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:01:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:01:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:01:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:01:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:01:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:01:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:01:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:01:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:01:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:01:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:01:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:01:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:01:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:01:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:01:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:01:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:01:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:01:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:01:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:01:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:01:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:01:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:01:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:01:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:01:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:01:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:01:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:01:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:01:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:01:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:01:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:01:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:01:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:01:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:02:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:02:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:02:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:02:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:02:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:02:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:02:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:02:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:02:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:02:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:02:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:02:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:02:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:02:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:02:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:02:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:02:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:02:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:02:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:02:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:02:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:02:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:02:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:02:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:02:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:02:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:02:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:02:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:02:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:02:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:02:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:02:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:02:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:02:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:02:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:02:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:02:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:02:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:02:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:02:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:02:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:02:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:02:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:02:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:02:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:02:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:02:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:02:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:02:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:02:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:02:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:02:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:02:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:02:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:02:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:02:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:02:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:02:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:02:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:02:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:02:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:02:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:02:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:02:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:02:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:02:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:02:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:02:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:02:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:02:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:02:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:02:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:02:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:02:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:02:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:02:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:02:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:02:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:02:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:02:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:02:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:02:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:02:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:02:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:02:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:02:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:02:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:02:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:02:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:02:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:02:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:02:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:02:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:02:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:02:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:02:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:02:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:02:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:02:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:02:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:02:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:02:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:02:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:02:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:02:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:02:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:02:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:02:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:02:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:02:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:02:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:02:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:02:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:02:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:02:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:02:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:02:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:02:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:02:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:02:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:02:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:02:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:02:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:02:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:02:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:02:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:02:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:02:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:02:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:02:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:02:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:02:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:02:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:02:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:02:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:02:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:02:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:02:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:02:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:02:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:02:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:02:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:02:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:02:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:02:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:02:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:02:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:02:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:02:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:02:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:02:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:02:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:02:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:02:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:02:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:02:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:02:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:02:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:02:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:02:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:02:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:02:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:02:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:02:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:02:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:02:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:02:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:02:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:02:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:02:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:02:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:02:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:02:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:02:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:02:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:02:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:02:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:02:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:02:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:02:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:02:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:02:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:02:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:02:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:02:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:02:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:02:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:02:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:02:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:02:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:02:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:02:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:02:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:02:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:02:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:02:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:02:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:02:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:02:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:02:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:02:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:02:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:02:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:02:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:02:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:02:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:02:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:02:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:02:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:02:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:02:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:02:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:02:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:02:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:02:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:02:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:03:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:03:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:03:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:03:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:03:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:03:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:03:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:03:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:03:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:03:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:03:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:03:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:03:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:03:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:03:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:03:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:03:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:03:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:03:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:03:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:03:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:03:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:03:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:03:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:03:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:03:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:03:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:03:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:03:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:03:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:03:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:03:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:03:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:03:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:03:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:03:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:03:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:03:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:03:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:03:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:03:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:03:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:03:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:03:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:03:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:03:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:03:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:03:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:03:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:03:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:03:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:03:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:03:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:03:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:03:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:03:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:03:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:03:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:03:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:03:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:03:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:03:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:03:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:03:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:03:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:03:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:03:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:03:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:03:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:03:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:03:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:03:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:03:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:03:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:03:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:03:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:03:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:03:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:03:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:03:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:03:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:03:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:03:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:03:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:03:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:03:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:03:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:03:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:03:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:03:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:03:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:03:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:03:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:03:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:03:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:03:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:03:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:03:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:03:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:03:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:03:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:03:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:03:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:03:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:03:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:03:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:03:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:03:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:03:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:03:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:03:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:03:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:03:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:03:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:03:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:03:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:03:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:03:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:03:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:03:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:03:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:03:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:03:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:03:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:03:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:03:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:03:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:03:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:03:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:03:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:03:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:03:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:03:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:03:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:03:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:03:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:03:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:03:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:03:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:03:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:03:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:03:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:03:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:03:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:03:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:03:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:03:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:03:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:03:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:03:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:03:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:03:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:03:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:03:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:03:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:03:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:03:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:03:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:03:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:03:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:03:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:03:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:03:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:03:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:03:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:03:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:03:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:03:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:03:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:03:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:03:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:03:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:03:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:03:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:03:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:03:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:03:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:03:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:03:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:03:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:03:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:03:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:03:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:03:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:03:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:03:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:03:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:03:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:03:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:03:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:03:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:03:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:03:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:03:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:03:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:03:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:03:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:03:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:03:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:03:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:03:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:03:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:03:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:03:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:03:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:03:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:03:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:03:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:03:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:03:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:03:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:03:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:03:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:03:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:03:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:03:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:04:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:04:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:04:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:04:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:04:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:04:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:04:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:04:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:04:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:04:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:04:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:04:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:04:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:04:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:04:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:04:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:04:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:04:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:04:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:04:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:04:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:04:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:04:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:04:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:04:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:04:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:04:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:04:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:04:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:04:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:04:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:04:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:04:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:04:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:04:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:04:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:04:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:04:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:04:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:04:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:04:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:04:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:04:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:04:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:04:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:04:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:04:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:04:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:04:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:04:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:04:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:04:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:04:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:04:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:04:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:04:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:04:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:04:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:04:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:04:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:04:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:04:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:04:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:04:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:04:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:04:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:04:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:04:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:04:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:04:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:04:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:04:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:04:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:04:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:04:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:04:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:04:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:04:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:04:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:04:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:04:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:04:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:04:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:04:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:04:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:04:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:04:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:04:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:04:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:04:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:04:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:04:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:04:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:04:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:04:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:04:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:04:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:04:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:04:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:04:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:04:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:04:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:04:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:04:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:04:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:04:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:04:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:04:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:04:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:04:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:04:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:04:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:04:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:04:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:04:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:04:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:04:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:04:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:04:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:04:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:04:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:04:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:04:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:04:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:04:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:04:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:04:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:04:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:04:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:04:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:04:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:04:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:04:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:04:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:04:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:04:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:04:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:04:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:04:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:04:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:04:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:04:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:04:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:04:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:04:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:04:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:04:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:04:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:04:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:04:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:04:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:04:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:04:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:04:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:04:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:04:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:04:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:04:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:04:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:04:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:04:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:04:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:04:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:04:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:04:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:04:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:04:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:04:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:04:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:04:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:04:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:04:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:04:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:04:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:04:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:04:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:04:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:04:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:04:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:04:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:04:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:04:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:04:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:04:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:04:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:04:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:04:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:04:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:04:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:04:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:04:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:04:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:04:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:04:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:04:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:04:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:04:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:04:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:04:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:04:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:04:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:04:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:04:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:04:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:04:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:04:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:04:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:04:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:04:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:04:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:04:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:04:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:04:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:04:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:04:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:04:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:05:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:05:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:05:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:05:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:05:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:05:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:05:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:05:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:05:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:05:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:05:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:05:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:05:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:05:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:05:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:05:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:05:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:05:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:05:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:05:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:05:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:05:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:05:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:05:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:05:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:05:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:05:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:05:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:05:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:05:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:05:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:05:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:05:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:05:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:05:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:05:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:05:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:05:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:05:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:05:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:05:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:05:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:05:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:05:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:05:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:05:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:05:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:05:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:05:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:05:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:05:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:05:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:05:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:05:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:05:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:05:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:05:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:05:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:05:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:05:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:05:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:05:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:05:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:05:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:05:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:05:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:05:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:05:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:05:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:05:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:05:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:05:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:05:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:05:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:05:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:05:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:05:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:05:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:05:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:05:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:05:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:05:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:05:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:05:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:05:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:05:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:05:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:05:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:05:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:05:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:05:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:05:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:05:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:05:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:05:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:05:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:05:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:05:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:05:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:05:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:05:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:05:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:05:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:05:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:05:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:05:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:05:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:05:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:05:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:05:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:05:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:05:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:05:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:05:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:05:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:05:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:05:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:05:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:05:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:05:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:05:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:05:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:05:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:05:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:05:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:05:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:05:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:05:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:05:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:05:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:05:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:05:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:05:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:05:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:05:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:05:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:05:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:05:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:05:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:05:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:05:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:05:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:05:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:05:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:05:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:05:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:05:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:05:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:05:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:05:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:05:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:05:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:05:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:05:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:05:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:05:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:05:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:05:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:05:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:05:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:05:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:05:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:05:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:05:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:05:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:05:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:05:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:05:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:05:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:05:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:05:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:05:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:05:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:05:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:05:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:05:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:05:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:05:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:05:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:05:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:05:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:05:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:05:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:05:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:05:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:05:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:05:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:05:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:05:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:05:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:05:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:05:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:05:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:05:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:05:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:05:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:05:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:05:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:05:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:05:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:05:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:05:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:05:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:05:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:05:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:05:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:05:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:05:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:05:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:05:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:05:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:05:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:05:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:05:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:05:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:05:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:06:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:06:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:06:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:06:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:06:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:06:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:06:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:06:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:06:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:06:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:06:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:06:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:06:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:06:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:06:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:06:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:06:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:06:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:06:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:06:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:06:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:06:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:06:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:06:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:06:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:06:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:06:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:06:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:06:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:06:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:06:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:06:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:06:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:06:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:06:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:06:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:06:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:06:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:06:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:06:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:06:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:06:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:06:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:06:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:06:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:06:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:06:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:06:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:06:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:06:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:06:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:06:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:06:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:06:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:06:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:06:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:06:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:06:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:06:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:06:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:06:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:06:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:06:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:06:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:06:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:06:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:06:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:06:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:06:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:06:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:06:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:06:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:06:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:06:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:06:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:06:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:06:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:06:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:06:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:06:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:06:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:06:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:06:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:06:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:06:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:06:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:06:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:06:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:06:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:06:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:06:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:06:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:06:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:06:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:06:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:06:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:06:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:06:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:06:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:06:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:06:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:06:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:06:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:06:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:06:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:06:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:06:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:06:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:06:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:06:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:06:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:06:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:06:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:06:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:06:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:06:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:06:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:06:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:06:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:06:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:06:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:06:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:06:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:06:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:06:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:06:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:08:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:08:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:08:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:08:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:08:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:08:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:08:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:08:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:08:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:08:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:08:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:08:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:08:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:08:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:08:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:08:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:08:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:08:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:08:37 rdma06 rsyslogd[1041]: imjournal: 5086 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 18:08:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:08:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:08:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:08:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:08:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:08:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:08:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:08:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:08:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:08:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:08:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:08:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:08:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:08:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:08:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:08:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:08:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:08:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:08:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:08:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:08:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:08:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:08:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:08:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:08:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:08:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:08:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:08:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:08:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:08:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:08:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:08:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:08:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:08:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:08:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:08:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:08:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:08:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:08:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:08:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:08:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:08:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:08:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:08:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:08:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:08:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:08:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:08:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:08:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:08:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:08:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:08:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:08:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:08:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:08:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:08:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:08:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:08:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:08:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:08:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:08:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:08:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:08:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:08:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:08:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:08:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:08:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:08:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:08:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:08:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:08:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:08:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:09:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:09:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:09:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:09:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:09:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:09:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:09:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:09:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:09:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:09:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:09:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:09:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:09:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:09:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:09:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:09:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:09:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:09:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:09:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:09:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:09:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:09:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:09:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:09:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:09:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:09:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:09:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:09:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:09:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:09:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:09:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:09:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:09:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:09:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:09:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:09:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:09:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:09:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:09:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:09:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:09:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:09:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:09:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:09:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:09:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:09:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:09:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:09:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:09:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:09:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:09:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:09:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:09:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:09:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:09:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:09:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:09:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:09:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:09:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:09:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:09:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:09:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:09:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:09:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:09:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:09:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:09:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:09:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:09:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:09:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:09:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:09:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:09:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:09:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:09:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:09:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:09:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:09:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:09:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:09:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:09:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:09:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:09:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:09:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:09:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:09:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:09:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:09:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:09:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:09:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:09:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:09:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:09:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:09:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:09:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:09:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:09:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:09:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:09:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:09:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:09:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:09:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:09:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:09:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:09:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:09:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:09:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:09:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:09:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:09:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:09:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:09:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:09:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:09:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:09:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:09:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:09:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:09:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:09:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:09:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:09:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:09:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:09:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:09:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:09:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:09:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:09:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:09:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:09:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:09:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:09:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:09:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:09:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:09:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:09:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:09:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:09:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:09:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:09:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:09:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:09:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:09:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:09:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:09:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:09:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:09:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:09:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:09:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:09:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:09:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:09:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:09:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:09:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:09:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:09:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:09:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:09:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:09:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:09:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:09:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:09:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:09:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:09:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:09:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:09:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:09:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:09:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:09:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:09:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:09:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:09:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:09:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:09:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:09:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:09:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:09:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:09:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:09:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:09:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:09:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:09:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:09:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:09:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:09:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:09:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:09:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:09:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:09:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:09:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:09:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:09:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:09:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:09:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:09:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:09:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:09:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:09:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:09:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:09:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:09:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:09:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:09:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:09:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:09:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:09:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:09:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:09:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:09:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:09:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:09:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:09:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:09:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:09:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:09:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:09:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:09:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:10:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:10:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:10:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:10:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:10:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:10:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:10:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:10:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:10:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:10:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:10:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:10:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:10:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:10:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:10:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:10:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:10:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:10:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:10:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:10:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:10:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:10:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:10:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:10:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:10:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:10:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:10:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:10:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:10:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:10:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:10:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:10:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:10:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:10:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:10:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:10:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:10:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:10:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:10:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:10:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:10:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:10:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:10:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:10:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:10:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:10:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:10:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:10:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:10:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:10:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:10:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:10:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:10:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:10:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:10:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:10:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:10:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:10:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:10:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:10:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:10:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:10:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:10:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:10:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:10:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:10:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:10:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:10:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:10:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:10:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:10:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:10:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:10:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:10:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:10:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:10:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:10:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:10:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:10:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:10:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:10:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:10:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:10:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:10:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:10:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:10:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:10:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:10:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:10:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:10:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:10:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:10:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:10:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:10:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:10:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:10:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:10:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:10:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:10:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:10:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:10:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:10:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:10:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:10:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:10:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:10:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:10:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:10:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:10:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:10:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:10:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:10:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:10:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:10:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:10:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:10:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:10:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:10:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:10:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:10:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:10:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:10:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:10:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:10:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:10:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:10:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:10:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:10:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:10:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:10:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:10:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:10:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:10:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:10:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:10:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:10:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:10:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:10:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:10:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:10:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:10:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:10:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:10:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:10:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:10:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:10:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:10:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:10:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:10:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:10:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:10:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:10:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:10:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:10:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:10:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:10:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:10:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:10:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:10:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:10:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:10:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:10:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:10:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:10:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:10:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:10:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:10:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:10:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:10:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:10:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:10:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:10:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:10:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:10:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:10:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:10:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:10:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:10:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:10:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:10:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:10:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:10:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:10:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:10:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:10:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:10:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:10:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:10:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:10:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:10:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:10:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:10:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:10:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:10:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:10:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:10:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:10:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:10:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:10:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:10:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:10:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:10:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:10:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:10:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:10:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:10:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:10:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:10:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:10:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:10:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:10:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:10:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:10:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:10:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:10:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:10:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:11:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:11:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:11:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:11:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:11:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:11:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:11:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:11:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:11:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:11:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:11:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:11:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:11:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:11:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:11:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:11:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:11:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:11:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:11:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:11:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:11:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:11:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:11:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:11:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:11:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:11:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:11:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:11:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:11:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:11:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:11:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:11:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:11:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:11:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:11:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:11:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:11:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:11:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:11:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:11:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:11:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:11:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:11:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:11:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:11:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:11:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:11:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:11:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:11:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:11:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:11:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:11:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:11:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:11:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:11:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:11:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:11:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:11:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:11:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:11:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:11:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:11:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:11:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:11:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:11:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:11:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:11:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:11:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:11:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:11:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:11:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:11:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:11:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:11:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:11:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:11:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:11:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:11:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:11:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:11:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:11:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:11:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:11:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:11:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:11:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:11:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:11:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:11:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:11:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:11:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:11:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:11:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:11:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:11:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:11:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:11:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:11:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:11:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:11:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:11:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:11:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:11:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:11:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:11:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:11:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:11:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:11:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:11:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:11:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:11:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:11:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:11:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:11:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:11:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:11:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:11:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:11:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:11:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:11:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:11:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:11:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:11:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:11:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:11:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:11:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:11:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:11:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:11:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:11:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:11:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:11:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:11:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:11:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:11:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:11:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:11:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:11:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:11:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:11:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:11:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:11:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:11:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:11:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:11:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:11:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:11:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:11:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:11:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:11:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:11:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:11:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:11:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:11:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:11:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:11:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:11:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:11:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:11:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:11:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:11:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:11:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:11:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:11:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:11:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:11:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:11:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:11:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:11:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:11:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:11:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:11:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:11:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:11:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:11:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:11:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:11:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:11:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:11:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:11:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:11:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:11:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:11:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:11:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:11:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:11:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:11:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:11:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:11:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:11:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:11:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:11:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:11:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:11:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:11:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:11:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:11:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:11:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:11:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:11:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:11:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:11:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:11:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:11:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:11:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:11:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:11:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:11:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:11:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:11:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:11:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:11:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:11:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:11:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:11:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:11:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:11:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:12:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:12:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:12:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:12:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:12:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:12:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:12:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:12:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:12:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:12:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:12:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:12:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:12:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:12:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:12:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:12:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:12:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:12:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:12:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:12:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:12:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:12:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:12:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:12:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:12:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:12:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:12:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:12:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:12:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:12:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:12:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:12:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:12:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:12:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:12:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:12:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:12:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:12:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:12:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:12:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:12:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:12:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:12:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:12:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:12:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:12:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:12:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:12:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:12:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:12:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:12:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:12:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:12:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:12:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:12:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:12:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:12:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:12:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:12:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:12:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:12:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:12:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:12:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:12:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:12:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:12:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:12:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:12:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:12:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:12:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:12:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:12:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:12:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:12:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:12:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:12:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:12:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:12:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:12:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:12:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:12:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:12:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:12:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:12:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:12:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:12:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:12:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:12:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:12:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:12:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:12:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:12:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:12:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:12:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:12:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:12:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:12:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:12:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:12:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:12:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:12:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:12:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:12:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:12:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:12:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:12:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:12:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:12:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:12:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:12:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:12:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:12:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:12:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:12:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:12:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:12:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:12:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:12:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:12:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:12:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:12:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:12:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:12:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:12:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:12:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:12:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:12:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:12:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:12:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:12:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:12:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:12:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:12:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:12:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:12:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:12:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:12:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:12:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:12:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:12:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:12:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:12:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:12:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:12:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:12:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:12:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:12:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:12:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:12:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:12:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:12:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:12:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:12:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:12:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:12:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:12:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:12:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:12:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:12:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:12:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:12:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:12:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:12:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:12:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:12:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:12:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:12:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:12:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:12:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:12:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:12:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:12:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:12:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:12:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:12:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:12:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:12:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:12:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:12:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:12:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:12:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:12:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:12:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:12:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:12:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:12:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:12:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:12:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:12:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:12:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:12:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:12:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:12:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:12:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:12:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:12:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:12:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:12:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:12:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:12:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:12:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:12:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:12:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:12:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:12:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:12:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:12:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:12:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:12:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:12:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:12:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:12:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:12:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:12:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:12:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:12:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:13:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:13:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:13:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:13:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:13:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:13:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:13:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:13:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:13:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:13:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:13:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:13:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:13:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:13:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:13:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:13:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:13:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:13:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:13:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:13:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:13:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:13:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:13:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:13:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:13:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:13:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:13:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:13:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:13:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:13:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:13:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:13:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:13:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:13:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:13:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:13:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:13:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:13:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:13:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:13:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:13:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:13:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:13:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:13:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:13:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:13:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:13:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:13:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:13:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:13:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:13:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:13:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:13:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:13:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:13:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:13:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:13:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:13:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:13:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:13:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:13:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:13:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:13:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:13:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:13:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:13:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:13:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:13:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:13:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:13:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:13:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:13:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:13:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:13:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:13:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:13:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:13:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:13:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:13:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:13:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:13:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:13:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:13:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:13:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:13:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:13:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:13:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:13:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:13:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:13:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:13:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:13:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:13:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:13:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:13:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:13:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:13:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:13:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:13:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:13:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:13:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:13:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:13:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:13:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:13:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:13:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:13:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:13:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:13:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:13:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:13:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:13:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:13:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:13:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:13:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:13:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:13:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:13:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:13:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:13:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:13:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:13:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:13:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:13:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:13:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:13:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:13:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:13:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:13:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:13:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:13:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:13:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:13:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:13:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:13:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:13:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:13:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:13:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:13:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:13:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:13:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:13:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:13:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:13:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:13:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:13:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:13:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:13:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:13:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:13:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:13:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:13:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:13:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:13:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:13:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:13:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:13:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:13:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:13:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:13:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:13:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:13:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:13:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:13:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:13:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:13:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:13:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:13:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:13:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:13:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:13:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:13:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:13:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:13:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:13:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:13:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:13:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:13:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:13:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:13:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:13:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:13:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:13:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:13:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:13:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:13:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:13:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:13:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:13:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:13:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:13:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:13:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:13:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:13:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:13:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:13:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:13:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:13:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:13:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:13:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:13:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:13:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:13:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:13:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:13:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:13:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:13:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:13:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:13:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:13:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:13:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:13:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:13:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:13:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:13:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:13:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:14:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:14:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:14:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:14:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:14:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:14:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:14:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:14:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:14:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:14:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:14:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:14:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:14:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:14:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:14:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:14:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:14:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:14:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:14:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:14:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:14:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:14:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:14:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:14:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:14:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:14:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:14:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:14:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:14:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:14:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:14:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:14:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:14:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:14:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:14:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:14:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:14:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:14:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:14:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:14:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:14:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:14:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:14:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:14:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:14:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:14:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:14:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:14:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:14:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:14:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:14:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:14:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:14:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:14:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:14:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:14:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:14:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:14:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:14:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:14:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:14:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:14:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:14:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:14:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:14:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:14:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:14:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:14:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:14:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:14:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:14:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:14:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:14:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:14:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:14:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:14:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:14:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:14:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:14:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:14:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:14:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:14:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:14:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:14:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:14:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:14:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:14:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:14:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:14:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:14:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:14:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:14:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:14:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:14:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:14:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:14:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:14:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:14:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:14:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:14:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:14:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:14:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:14:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:14:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:14:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:14:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:14:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:14:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:14:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:14:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:14:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:14:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:14:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:14:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:14:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:14:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:14:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:14:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:14:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:14:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:14:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:14:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:14:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:14:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:14:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:14:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:14:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:14:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:14:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:14:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:14:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:14:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:14:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:14:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:14:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:14:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:14:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:14:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:14:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:14:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:14:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:14:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:14:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:14:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:14:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:14:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:14:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:14:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:14:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:14:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:14:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:14:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:14:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:14:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:14:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:14:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:14:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:14:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:14:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:14:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:14:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:14:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:14:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:14:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:14:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:14:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:14:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:14:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:14:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:14:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:14:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:14:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:14:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:14:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:14:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:14:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:14:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:14:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:14:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:14:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:14:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:14:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:14:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:14:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:14:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:14:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:14:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:14:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:14:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:14:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:14:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:14:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:14:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:14:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:14:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:14:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:14:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:14:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:14:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:14:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:14:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:14:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:14:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:14:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:14:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:14:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:14:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:14:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:14:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:14:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:14:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:14:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:14:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:14:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:14:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:14:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:15:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:15:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:15:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:15:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:15:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:15:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:15:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:15:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:15:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:15:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:15:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:15:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:15:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:15:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:15:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:15:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:15:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:15:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:15:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:15:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:15:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:15:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:15:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:15:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:15:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:15:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:15:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:15:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:15:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:15:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:15:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:15:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:15:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:15:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:15:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:15:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:15:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:15:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:15:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:15:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:15:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:15:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:15:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:15:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:15:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:15:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:15:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:15:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:15:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:15:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:15:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:15:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:15:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:15:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:15:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:15:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:15:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:15:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:15:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:15:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:15:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:15:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:15:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:15:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:15:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:15:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:15:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:15:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:15:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:15:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:15:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:15:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:15:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:15:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:15:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:15:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:15:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:15:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:15:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:15:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:15:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:15:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:15:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:15:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:15:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:15:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:15:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:15:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:15:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:15:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:15:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:15:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:15:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:15:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:15:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:15:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:15:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:15:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:15:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:15:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:15:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:15:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:15:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:15:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:15:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:15:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:15:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:15:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:15:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:15:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:15:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:15:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:15:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:15:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:15:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:15:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:15:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:15:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:15:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:15:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:15:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:15:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:15:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:15:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:15:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:15:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:15:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:15:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:15:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:15:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:15:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:15:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:15:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:15:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:15:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:15:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:15:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:15:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:15:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:15:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:15:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:15:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:15:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:15:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:15:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:15:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:15:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:15:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:15:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:15:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:15:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:15:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:15:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:15:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:15:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:15:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:15:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:15:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:15:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:15:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:15:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:15:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:15:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:15:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:15:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:15:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:15:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:15:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:15:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:15:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:15:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:15:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:15:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:15:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:15:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:15:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:15:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:15:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:15:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:15:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:15:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:15:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:15:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:15:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:15:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:15:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:15:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:15:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:15:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:15:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:15:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:15:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:15:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:15:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:15:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:15:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:15:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:15:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:15:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:15:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:15:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:15:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:15:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:15:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:15:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:15:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:15:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:15:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:15:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:15:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:15:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:15:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:15:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:15:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:15:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:15:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:16:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:16:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:16:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:16:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:16:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:16:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:16:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:16:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:16:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:16:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:16:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:16:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:16:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:16:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:16:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:16:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:16:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:16:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:16:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:16:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:16:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:16:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:16:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:16:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:16:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:16:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:16:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:16:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:16:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:16:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:16:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:16:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:16:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:16:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:16:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:16:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:16:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:16:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:16:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:16:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:16:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:16:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:16:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:16:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:16:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:16:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:16:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:16:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:16:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:16:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:16:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:16:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:16:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:16:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:16:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:16:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:16:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:16:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:16:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:16:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:16:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:16:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:16:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:16:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:16:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:16:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:16:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:16:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:16:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:16:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:16:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:16:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:16:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:16:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:16:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:16:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:16:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:16:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:16:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:16:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:16:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:16:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:16:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:16:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:16:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:16:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:16:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:16:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:16:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:16:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:16:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:16:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:16:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:16:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:16:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:16:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:16:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:16:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:16:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:16:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:16:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:16:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:16:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:16:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:16:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:16:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:16:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:16:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:16:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:16:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:16:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:16:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:16:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:16:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:16:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:16:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:16:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:16:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:16:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:16:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:16:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:16:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:16:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:16:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:16:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:16:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:18:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:18:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:18:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:18:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:18:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:18:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:18:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:18:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:18:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:18:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:18:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:18:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:18:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:18:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:18:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:18:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:18:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:18:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:18:42 rdma06 rsyslogd[1041]: imjournal: 5295 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 18:18:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:18:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:18:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:18:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:18:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:18:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:18:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:18:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:18:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:18:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:18:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:18:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:18:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:18:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:18:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:18:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:18:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:18:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:18:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:18:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:18:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:18:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:18:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:18:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:18:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:18:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:18:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:18:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:18:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:18:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:18:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:18:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:18:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:18:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:18:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:18:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:18:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:18:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:18:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:18:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:18:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:18:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:18:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:18:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:18:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:18:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:18:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:18:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:18:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:18:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:18:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:18:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:18:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:18:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:19:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:19:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:19:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:19:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:19:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:19:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:19:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:19:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:19:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:19:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:19:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:19:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:19:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:19:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:19:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:19:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:19:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:19:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:19:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:19:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:19:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:19:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:19:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:19:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:19:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:19:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:19:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:19:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:19:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:19:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:19:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:19:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:19:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:19:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:19:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:19:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:19:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:19:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:19:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:19:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:19:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:19:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:19:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:19:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:19:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:19:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:19:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:19:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:19:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:19:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:19:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:19:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:19:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:19:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:19:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:19:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:19:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:19:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:19:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:19:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:19:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:19:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:19:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:19:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:19:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:19:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:19:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:19:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:19:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:19:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:19:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:19:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:19:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:19:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:19:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:19:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:19:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:19:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:19:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:19:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:19:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:19:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:19:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:19:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:19:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:19:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:19:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:19:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:19:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:19:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:19:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:19:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:19:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:19:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:19:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:19:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:19:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:19:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:19:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:19:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:19:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:19:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:19:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:19:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:19:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:19:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:19:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:19:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:19:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:19:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:19:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:19:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:19:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:19:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:19:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:19:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:19:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:19:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:19:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:19:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:19:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:19:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:19:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:19:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:19:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:19:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:19:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:19:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:19:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:19:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:19:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:19:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:19:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:19:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:19:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:19:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:19:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:19:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:19:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:19:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:19:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:19:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:19:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:19:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:19:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:19:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:19:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:19:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:19:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:19:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:19:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:19:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:19:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:19:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:19:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:19:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:19:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:19:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:19:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:19:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:19:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:19:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:19:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:19:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:19:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:19:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:19:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:19:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:19:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:19:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:19:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:19:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:19:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:19:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:19:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:19:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:19:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:19:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:19:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:19:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:19:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:19:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:19:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:19:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:19:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:19:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:19:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:19:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:19:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:19:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:19:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:19:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:19:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:19:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:19:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:19:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:19:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:19:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:19:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:19:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:19:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:19:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:19:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:19:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:19:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:19:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:19:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:19:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:19:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:19:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:19:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:19:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:19:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:19:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:19:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:19:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:20:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:20:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:20:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:20:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:20:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:20:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:20:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:20:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:20:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:20:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:20:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:20:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:20:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:20:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:20:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:20:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:20:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:20:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:20:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:20:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:20:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:20:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:20:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:20:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:20:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:20:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:20:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:20:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:20:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:20:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:20:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:20:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:20:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:20:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:20:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:20:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:20:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:20:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:20:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:20:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:20:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:20:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:20:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:20:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:20:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:20:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:20:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:20:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:20:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:20:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:20:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:20:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:20:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:20:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:20:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:20:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:20:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:20:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:20:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:20:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:20:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:20:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:20:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:20:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:20:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:20:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:20:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:20:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:20:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:20:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:20:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:20:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:20:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:20:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:20:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:20:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:20:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:20:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:20:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:20:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:20:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:20:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:20:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:20:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:20:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:20:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:20:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:20:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:20:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:20:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:20:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:20:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:20:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:20:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:20:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:20:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:20:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:20:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:20:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:20:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:20:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:20:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:20:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:20:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:20:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:20:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:20:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:20:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:20:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:20:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:20:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:20:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:20:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:20:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:20:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:20:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:20:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:20:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:20:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:20:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:20:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:20:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:20:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:20:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:20:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:20:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:20:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:20:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:20:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:20:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:20:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:20:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:20:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:20:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:20:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:20:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:20:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:20:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:20:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:20:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:20:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:20:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:20:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:20:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:20:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:20:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:20:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:20:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:20:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:20:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:20:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:20:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:20:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:20:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:20:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:20:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:20:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:20:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:20:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:20:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:20:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:20:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:20:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:20:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:20:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:20:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:20:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:20:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:20:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:20:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:20:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:20:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:20:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:20:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:20:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:20:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:20:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:20:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:20:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:20:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:20:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:20:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:20:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:20:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:20:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:20:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:20:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:20:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:20:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:20:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:20:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:20:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:20:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:20:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:20:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:20:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:20:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:20:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:20:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:20:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:20:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:20:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:20:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:20:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:20:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:20:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:20:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:20:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:20:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:20:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:20:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:20:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:20:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:20:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:20:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:20:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:21:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:21:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:21:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:21:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:21:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:21:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:21:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:21:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:21:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:21:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:21:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:21:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:21:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:21:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:21:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:21:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:21:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:21:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:21:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:21:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:21:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:21:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:21:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:21:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:21:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:21:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:21:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:21:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:21:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:21:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:21:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:21:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:21:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:21:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:21:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:21:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:21:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:21:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:21:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:21:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:21:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:21:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:21:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:21:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:21:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:21:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:21:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:21:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:21:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:21:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:21:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:21:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:21:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:21:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:21:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:21:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:21:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:21:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:21:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:21:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:21:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:21:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:21:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:21:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:21:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:21:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:21:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:21:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:21:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:21:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:21:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:21:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:21:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:21:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:21:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:21:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:21:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:21:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:21:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:21:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:21:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:21:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:21:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:21:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:21:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:21:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:21:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:21:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:21:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:21:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:21:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:21:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:21:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:21:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:21:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:21:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:21:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:21:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:21:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:21:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:21:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:21:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:21:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:21:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:21:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:21:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:21:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:21:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:21:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:21:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:21:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:21:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:21:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:21:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:21:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:21:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:21:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:21:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:21:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:21:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:21:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:21:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:21:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:21:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:21:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:21:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:21:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:21:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:21:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:21:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:21:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:21:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:21:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:21:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:21:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:21:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:21:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:21:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:21:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:21:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:21:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:21:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:21:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:21:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:21:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:21:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:21:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:21:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:21:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:21:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:21:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:21:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:21:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:21:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:21:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:21:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:21:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:21:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:21:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:21:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:21:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:21:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:21:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:21:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:21:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:21:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:21:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:21:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:21:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:21:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:21:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:21:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:21:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:21:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:21:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:21:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:21:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:21:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:21:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:21:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:21:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:21:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:21:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:21:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:21:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:21:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:21:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:21:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:21:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:21:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:21:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:21:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:21:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:21:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:21:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:21:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:21:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:21:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:21:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:21:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:21:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:21:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:21:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:21:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:21:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:21:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:22:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:22:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:22:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:22:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:22:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:22:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:22:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:22:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:22:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:22:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:22:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:22:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:22:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:22:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:22:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:22:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:22:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:22:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:22:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:22:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:22:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:22:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:22:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:22:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:22:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:22:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:22:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:22:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:22:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:22:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:22:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:22:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:22:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:22:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:22:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:22:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:22:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:22:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:22:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:22:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:22:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:22:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:22:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:22:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:22:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:22:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:22:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:22:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:22:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:22:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:22:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:22:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:22:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:22:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:22:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:22:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:22:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:22:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:22:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:22:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:22:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:22:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:22:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:22:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:22:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:22:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:22:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:22:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:22:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:22:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:22:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:22:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:22:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:22:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:22:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:22:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:22:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:22:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:22:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:22:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:22:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:22:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:22:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:22:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:22:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:22:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:22:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:22:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:22:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:22:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:22:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:22:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:22:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:22:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:22:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:22:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:22:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:22:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:22:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:22:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:22:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:22:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:22:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:22:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:22:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:22:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:22:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:22:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:22:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:22:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:22:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:22:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:22:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:22:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:22:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:22:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:22:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:22:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:22:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:22:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:22:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:22:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:22:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:22:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:22:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:22:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:22:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:22:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:22:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:22:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:22:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:22:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:22:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:22:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:22:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:22:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:22:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:22:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:22:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:22:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:22:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:22:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:22:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:22:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:22:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:22:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:22:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:22:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:22:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:22:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:22:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:22:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:22:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:22:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:22:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:22:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:22:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:22:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:22:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:22:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:22:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:22:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:22:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:22:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:22:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:22:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:22:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:22:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:22:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:22:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:22:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:22:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:22:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:22:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:22:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:22:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:22:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:22:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:22:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:22:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:22:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:22:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:22:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:22:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:22:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:22:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:22:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:22:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:22:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:22:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:22:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:22:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:22:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:22:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:22:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:22:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:23:00 rdma06 systemd[1]: Starting dnf makecache...
Apr 17 18:23:01 rdma06 dnf[1163882]: Updating Subscription Management repositories.
Apr 17 18:23:01 rdma06 dnf[1163882]: Unable to read consumer identity
Apr 17 18:23:01 rdma06 dnf[1163882]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 17 18:23:01 rdma06 dnf[1163882]: Metadata cache refreshed recently.
Apr 17 18:23:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 17 18:23:01 rdma06 systemd[1]: Finished dnf makecache.
Apr 17 18:23:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:23:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:23:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:23:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:23:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:23:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:23:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:23:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:23:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:23:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:23:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:23:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:23:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:23:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:23:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:23:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:23:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:23:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:23:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:23:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:23:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:23:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:23:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:23:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:23:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:23:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:23:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:23:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:23:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:23:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:23:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:23:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:23:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:23:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:23:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:23:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:23:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:23:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:23:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:23:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:23:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:23:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:23:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:23:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:23:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:23:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:23:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:23:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:23:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:23:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:23:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:23:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:23:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:23:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:23:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:23:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:23:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:23:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:23:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:23:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:23:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:23:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:23:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:23:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:23:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:23:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:23:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:23:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:23:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:23:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:23:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:23:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:23:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:23:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:23:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:23:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:23:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:23:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:23:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:23:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:23:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:23:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:23:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:23:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:23:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:23:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:23:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:23:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:23:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:23:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:23:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:23:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:23:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:23:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:23:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:23:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:23:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:23:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:23:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:23:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:23:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:23:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:23:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:23:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:23:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:23:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:23:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:23:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:23:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:23:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:23:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:23:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:23:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:23:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:23:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:23:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:23:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:23:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:23:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:23:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:23:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:23:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:23:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:23:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:23:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:23:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:23:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:23:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:23:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:23:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:23:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:23:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:23:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:23:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:23:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:23:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:23:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:23:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:23:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:23:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:23:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:23:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:23:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:23:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:23:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:23:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:23:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:23:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:23:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:23:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:23:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:23:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:23:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:23:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:23:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:23:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:23:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:23:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:23:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:23:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:23:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:23:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:23:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:23:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:23:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:23:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:23:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:23:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:23:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:23:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:23:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:23:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:23:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:23:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:23:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:23:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:23:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:23:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:23:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:23:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:23:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:23:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:23:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:23:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:23:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:23:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:23:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:23:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:23:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:23:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:23:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:23:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:23:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:23:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:23:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:23:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:23:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:23:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:23:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:23:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:23:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:23:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:23:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:23:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:23:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:23:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:23:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:23:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:23:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:23:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:23:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:23:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:23:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:23:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:23:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:23:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:24:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:24:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:24:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:24:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:24:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:24:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:24:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:24:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:24:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:24:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:24:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:24:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:24:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:24:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:24:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:24:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:24:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:24:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:24:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:24:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:24:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:24:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:24:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:24:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:24:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:24:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:24:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:24:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:24:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:24:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:24:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:24:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:24:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:24:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:24:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:24:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:24:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:24:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:24:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:24:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:24:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:24:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:24:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:24:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:24:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:24:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:24:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:24:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:24:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:24:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:24:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:24:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:24:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:24:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:24:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:24:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:24:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:24:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:24:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:24:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:24:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:24:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:24:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:24:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:24:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:24:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:24:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:24:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:24:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:24:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:24:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:24:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:24:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:24:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:24:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:24:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:24:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:24:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:24:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:24:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:24:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:24:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:24:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:24:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:24:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:24:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:24:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:24:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:24:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:24:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:24:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:24:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:24:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:24:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:24:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:24:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:24:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:24:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:24:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:24:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:24:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:24:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:24:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:24:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:24:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:24:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:24:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:24:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:24:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:24:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:24:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:24:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:24:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:24:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:24:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:24:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:24:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:24:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:24:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:24:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:24:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:24:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:24:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:24:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:24:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:24:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:24:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:24:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:24:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:24:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:24:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:24:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:24:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:24:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:24:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:24:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:24:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:24:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:24:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:24:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:24:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:24:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:24:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:24:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:24:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:24:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:24:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:24:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:24:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:24:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:24:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:24:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:24:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:24:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:24:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:24:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:24:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:24:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:24:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:24:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:24:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:24:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:24:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:24:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:24:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:24:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:24:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:24:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:24:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:24:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:24:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:24:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:24:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:24:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:24:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:24:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:24:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:24:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:24:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:24:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:24:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:24:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:24:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:24:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:24:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:24:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:24:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:24:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:24:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:24:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:24:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:24:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:24:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:24:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:24:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:24:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:24:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:24:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:24:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:24:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:24:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:24:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:24:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:24:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:24:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:24:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:24:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:24:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:24:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:24:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:24:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:24:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:24:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:24:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:24:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:24:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:25:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:25:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:25:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:25:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:25:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:25:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:25:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:25:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:25:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:25:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:25:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:25:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:25:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:25:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:25:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:25:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:25:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:25:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:25:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:25:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:25:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:25:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:25:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:25:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:25:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:25:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:25:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:25:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:25:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:25:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:25:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:25:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:25:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:25:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:25:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:25:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:25:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:25:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:25:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:25:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:25:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:25:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:25:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:25:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:25:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:25:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:25:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:25:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:25:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:25:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:25:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:25:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:25:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:25:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:25:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:25:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:25:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:25:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:25:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:25:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:25:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:25:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:25:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:25:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:25:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:25:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:25:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:25:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:25:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:25:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:25:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:25:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:25:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:25:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:25:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:25:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:25:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:25:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:25:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:25:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:25:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:25:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:25:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:25:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:25:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:25:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:25:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:25:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:25:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:25:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:25:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:25:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:25:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:25:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:25:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:25:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:25:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:25:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:25:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:25:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:25:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:25:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:25:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:25:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:25:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:25:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:25:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:25:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:25:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:25:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:25:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:25:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:25:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:25:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:25:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:25:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:25:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:25:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:25:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:25:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:25:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:25:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:25:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:25:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:25:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:25:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:25:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:25:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:25:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:25:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:25:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:25:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:25:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:25:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:25:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:25:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:25:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:25:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:25:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:25:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:25:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:25:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:25:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:25:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:25:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:25:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:25:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:25:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:25:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:25:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:25:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:25:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:25:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:25:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:25:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:25:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:25:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:25:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:25:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:25:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:25:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:25:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:25:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:25:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:25:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:25:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:25:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:25:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:25:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:25:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:25:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:25:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:25:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:25:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:25:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:25:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:25:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:25:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:25:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:25:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:25:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:25:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:25:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:25:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:25:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:25:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:25:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:25:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:25:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:25:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:25:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:25:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:25:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:25:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:25:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:25:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:25:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:25:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:25:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:25:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:25:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:25:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:25:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:25:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:25:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:25:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:25:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:25:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:25:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:25:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:25:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:25:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:25:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:25:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:25:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:25:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:26:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:26:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:26:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:26:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:26:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:26:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:26:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:26:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:26:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:26:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:26:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:26:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:26:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:26:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:26:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:26:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:26:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:26:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:26:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:26:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:26:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:26:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:26:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:26:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:26:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:26:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:26:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:26:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:26:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:26:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:26:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:26:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:26:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:26:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:26:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:26:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:26:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:26:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:26:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:26:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:26:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:26:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:26:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:26:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:26:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:26:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:26:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:26:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:26:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:26:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:26:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:26:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:26:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:26:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:26:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:26:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:26:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:26:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:26:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:26:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:26:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:26:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:26:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:26:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:26:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:26:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:26:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:26:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:26:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:26:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:26:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:26:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:26:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:26:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:26:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:26:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:26:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:26:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:26:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:26:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:26:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:26:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:26:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:26:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:26:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:26:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:26:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:26:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:26:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:26:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:26:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:26:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:26:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:26:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:26:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:26:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:26:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:26:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:26:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:26:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:26:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:26:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:26:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:26:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:26:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:26:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:26:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:26:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:26:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:26:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:26:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:26:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:26:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:26:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:26:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:26:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:26:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:26:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:26:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:26:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:26:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:26:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:26:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:26:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:26:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:26:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:26:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:26:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:26:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:26:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:26:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:26:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:26:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:26:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:26:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:26:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:26:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:26:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:26:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:26:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:26:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:26:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:26:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:26:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:28:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:28:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:28:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:28:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:28:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:28:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:28:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:28:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:28:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:28:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:28:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:28:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:28:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:28:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:28:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:28:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:28:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:28:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:28:47 rdma06 rsyslogd[1041]: imjournal: 5298 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 18:28:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:28:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:28:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:28:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:28:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:28:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:28:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:28:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:28:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:28:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:28:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:28:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:28:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:28:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:28:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:28:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:28:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:28:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:28:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:28:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:28:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:28:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:28:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:28:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:28:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:28:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:28:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:28:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:28:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:28:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:28:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:28:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:28:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:28:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:28:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:28:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:29:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:29:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:29:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:29:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:29:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:29:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:29:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:29:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:29:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:29:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:29:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:29:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:29:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:29:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:29:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:29:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:29:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:29:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:29:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:29:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:29:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:29:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:29:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:29:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:29:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:29:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:29:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:29:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:29:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:29:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:29:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:29:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:29:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:29:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:29:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:29:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:29:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:29:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:29:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:29:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:29:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:29:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:29:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:29:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:29:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:29:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:29:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:29:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:29:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:29:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:29:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:29:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:29:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:29:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:29:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:29:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:29:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:29:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:29:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:29:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:29:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:29:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:29:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:29:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:29:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:29:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:29:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:29:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:29:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:29:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:29:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:29:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:29:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:29:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:29:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:29:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:29:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:29:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:29:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:29:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:29:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:29:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:29:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:29:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:29:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:29:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:29:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:29:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:29:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:29:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:29:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:29:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:29:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:29:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:29:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:29:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:29:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:29:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:29:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:29:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:29:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:29:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:29:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:29:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:29:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:29:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:29:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:29:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:29:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:29:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:29:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:29:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:29:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:29:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:29:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:29:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:29:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:29:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:29:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:29:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:29:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:29:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:29:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:29:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:29:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:29:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:29:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:29:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:29:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:29:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:29:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:29:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:29:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:29:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:29:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:29:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:29:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:29:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:29:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:29:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:29:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:29:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:29:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:29:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:29:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:29:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:29:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:29:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:29:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:29:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:29:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:29:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:29:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:29:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:29:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:29:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:29:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:29:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:29:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:29:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:29:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:29:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:29:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:29:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:29:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:29:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:29:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:29:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:29:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:29:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:29:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:29:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:29:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:29:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:29:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:29:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:29:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:29:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:29:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:29:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:29:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:29:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:29:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:29:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:29:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:29:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:29:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:29:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:29:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:29:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:29:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:29:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:29:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:29:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:29:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:29:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:29:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:29:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:29:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:29:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:29:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:29:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:29:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:29:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:29:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:29:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:29:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:29:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:29:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:29:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:29:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:29:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:29:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:29:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:29:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:29:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:30:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:30:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:30:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:30:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:30:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:30:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:30:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:30:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:30:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:30:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:30:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:30:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:30:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:30:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:30:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:30:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:30:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:30:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:30:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:30:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:30:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:30:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:30:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:30:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:30:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:30:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:30:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:30:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:30:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:30:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:30:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:30:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:30:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:30:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:30:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:30:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:30:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:30:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:30:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:30:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:30:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:30:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:30:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:30:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:30:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:30:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:30:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:30:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:30:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:30:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:30:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:30:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:30:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:30:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:30:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:30:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:30:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:30:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:30:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:30:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:30:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:30:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:30:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:30:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:30:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:30:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:30:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:30:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:30:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:30:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:30:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:30:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:30:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:30:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:30:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:30:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:30:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:30:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:30:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:30:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:30:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:30:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:30:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:30:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:30:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:30:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:30:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:30:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:30:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:30:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:30:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:30:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:30:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:30:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:30:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:30:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:30:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:30:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:30:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:30:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:30:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:30:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:30:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:30:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:30:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:30:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:30:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:30:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:30:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:30:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:30:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:30:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:30:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:30:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:30:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:30:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:30:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:30:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:30:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:30:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:30:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:30:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:30:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:30:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:30:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:30:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:30:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:30:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:30:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:30:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:30:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:30:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:30:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:30:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:30:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:30:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:30:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:30:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:30:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:30:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:30:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:30:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:30:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:30:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:30:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:30:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:30:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:30:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:30:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:30:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:30:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:30:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:30:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:30:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:30:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:30:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:30:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:30:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:30:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:30:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:30:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:30:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:30:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:30:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:30:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:30:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:30:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:30:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:30:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:30:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:30:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:30:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:30:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:30:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:30:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:30:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:30:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:30:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:30:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:30:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:30:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:30:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:30:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:30:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:30:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:30:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:30:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:30:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:30:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:30:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:30:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:30:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:30:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:30:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:30:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:30:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:30:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:30:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:30:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:30:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:30:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:30:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:30:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:30:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:30:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:30:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:30:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:30:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:30:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:30:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:30:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:30:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:30:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:30:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:30:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:30:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:31:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:31:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:31:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:31:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:31:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:31:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:31:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:31:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:31:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:31:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:31:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:31:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:31:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:31:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:31:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:31:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:31:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:31:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:31:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:31:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:31:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:31:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:31:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:31:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:31:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:31:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:31:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:31:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:31:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:31:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:31:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:31:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:31:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:31:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:31:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:31:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:31:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:31:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:31:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:31:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:31:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:31:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:31:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:31:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:31:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:31:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:31:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:31:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:31:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:31:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:31:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:31:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:31:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:31:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:31:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:31:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:31:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:31:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:31:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:31:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:31:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:31:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:31:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:31:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:31:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:31:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:31:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:31:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:31:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:31:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:31:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:31:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:31:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:31:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:31:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:31:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:31:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:31:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:31:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:31:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:31:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:31:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:31:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:31:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:31:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:31:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:31:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:31:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:31:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:31:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:31:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:31:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:31:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:31:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:31:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:31:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:31:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:31:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:31:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:31:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:31:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:31:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:31:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:31:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:31:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:31:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:31:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:31:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:31:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:31:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:31:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:31:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:31:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:31:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:31:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:31:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:31:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:31:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:31:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:31:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:31:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:31:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:31:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:31:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:31:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:31:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:31:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:31:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:31:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:31:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:31:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:31:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:31:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:31:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:31:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:31:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:31:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:31:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:31:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:31:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:31:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:31:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:31:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:31:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:31:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:31:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:31:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:31:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:31:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:31:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:31:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:31:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:31:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:31:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:31:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:31:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:31:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:31:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:31:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:31:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:31:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:31:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:31:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:31:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:31:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:31:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:31:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:31:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:31:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:31:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:31:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:31:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:31:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:31:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:31:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:31:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:31:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:31:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:31:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:31:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:31:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:31:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:31:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:31:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:31:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:31:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:31:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:31:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:31:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:31:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:31:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:31:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:31:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:31:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:31:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:31:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:31:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:31:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:31:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:31:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:31:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:31:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:31:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:31:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:31:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:31:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:31:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:31:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:31:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:31:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:31:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:31:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:31:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:31:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:31:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:31:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:32:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:32:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:32:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:32:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:32:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:32:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:32:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:32:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:32:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:32:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:32:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:32:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:32:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:32:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:32:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:32:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:32:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:32:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:32:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:32:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:32:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:32:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:32:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:32:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:32:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:32:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:32:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:32:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:32:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:32:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:32:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:32:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:32:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:32:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:32:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:32:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:32:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:32:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:32:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:32:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:32:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:32:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:32:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:32:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:32:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:32:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:32:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:32:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:32:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:32:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:32:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:32:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:32:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:32:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:32:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:32:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:32:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:32:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:32:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:32:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:32:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:32:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:32:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:32:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:32:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:32:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:32:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:32:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:32:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:32:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:32:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:32:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:32:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:32:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:32:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:32:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:32:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:32:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:32:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:32:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:32:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:32:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:32:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:32:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:32:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:32:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:32:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:32:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:32:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:32:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:32:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:32:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:32:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:32:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:32:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:32:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:32:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:32:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:32:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:32:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:32:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:32:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:32:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:32:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:32:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:32:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:32:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:32:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:32:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:32:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:32:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:32:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:32:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:32:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:32:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:32:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:32:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:32:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:32:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:32:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:32:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:32:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:32:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:32:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:32:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:32:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:32:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:32:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:32:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:32:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:32:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:32:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:32:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:32:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:32:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:32:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:32:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:32:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:32:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:32:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:32:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:32:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:32:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:32:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:32:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:32:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:32:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:32:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:32:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:32:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:32:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:32:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:32:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:32:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:32:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:32:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:32:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:32:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:32:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:32:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:32:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:32:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:32:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:32:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:32:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:32:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:32:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:32:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:32:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:32:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:32:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:32:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:32:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:32:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:32:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:32:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:32:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:32:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:32:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:32:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:32:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:32:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:32:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:32:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:32:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:32:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:32:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:32:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:32:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:32:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:32:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:32:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:32:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:32:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:32:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:32:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:32:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:32:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:32:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:32:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:32:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:32:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:32:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:32:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:32:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:32:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:32:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:32:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:32:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:32:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:32:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:32:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:32:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:32:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:32:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:32:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:33:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:33:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:33:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:33:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:33:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:33:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:33:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:33:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:33:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:33:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:33:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:33:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:33:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:33:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:33:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:33:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:33:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:33:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:33:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:33:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:33:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:33:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:33:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:33:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:33:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:33:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:33:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:33:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:33:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:33:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:33:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:33:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:33:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:33:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:33:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:33:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:33:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:33:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:33:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:33:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:33:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:33:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:33:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:33:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:33:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:33:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:33:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:33:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:33:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:33:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:33:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:33:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:33:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:33:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:33:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:33:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:33:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:33:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:33:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:33:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:33:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:33:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:33:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:33:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:33:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:33:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:33:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:33:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:33:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:33:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:33:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:33:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:33:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:33:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:33:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:33:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:33:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:33:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:33:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:33:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:33:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:33:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:33:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:33:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:33:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:33:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:33:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:33:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:33:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:33:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:33:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:33:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:33:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:33:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:33:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:33:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:33:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:33:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:33:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:33:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:33:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:33:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:33:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:33:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:33:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:33:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:33:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:33:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:33:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:33:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:33:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:33:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:33:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:33:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:33:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:33:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:33:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:33:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:33:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:33:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:33:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:33:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:33:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:33:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:33:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:33:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:33:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:33:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:33:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:33:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:33:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:33:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:33:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:33:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:33:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:33:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:33:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:33:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:33:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:33:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:33:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:33:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:33:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:33:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:33:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:33:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:33:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:33:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:33:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:33:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:33:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:33:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:33:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:33:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:33:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:33:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:33:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:33:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:33:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:33:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:33:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:33:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:33:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:33:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:33:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:33:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:33:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:33:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:33:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:33:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:33:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:33:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:33:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:33:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:33:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:33:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:33:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:33:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:33:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:33:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:33:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:33:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:33:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:33:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:33:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:33:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:33:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:33:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:33:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:33:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:33:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:33:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:33:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:33:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:33:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:33:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:33:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:33:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:33:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:33:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:33:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:33:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:33:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:33:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:33:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:33:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:33:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:33:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:33:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:33:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:33:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:33:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:33:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:33:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:33:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:33:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:34:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:34:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:34:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:34:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:34:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:34:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:34:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:34:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:34:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:34:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:34:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:34:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:34:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:34:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:34:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:34:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:34:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:34:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:34:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:34:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:34:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:34:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:34:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:34:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:34:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:34:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:34:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:34:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:34:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:34:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:34:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:34:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:34:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:34:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:34:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:34:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:34:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:34:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:34:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:34:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:34:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:34:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:34:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:34:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:34:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:34:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:34:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:34:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:34:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:34:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:34:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:34:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:34:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:34:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:34:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:34:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:34:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:34:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:34:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:34:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:34:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:34:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:34:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:34:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:34:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:34:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:34:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:34:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:34:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:34:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:34:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:34:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:34:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:34:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:34:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:34:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:34:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:34:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:34:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:34:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:34:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:34:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:34:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:34:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:34:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:34:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:34:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:34:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:34:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:34:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:34:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:34:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:34:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:34:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:34:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:34:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:34:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:34:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:34:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:34:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:34:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:34:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:34:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:34:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:34:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:34:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:34:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:34:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:34:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:34:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:34:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:34:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:34:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:34:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:34:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:34:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:34:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:34:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:34:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:34:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:34:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:34:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:34:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:34:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:34:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:34:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:34:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:34:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:34:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:34:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:34:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:34:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:34:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:34:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:34:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:34:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:34:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:34:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:34:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:34:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:34:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:34:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:34:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:34:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:34:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:34:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:34:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:34:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:34:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:34:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:34:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:34:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:34:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:34:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:34:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:34:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:34:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:34:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:34:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:34:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:34:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:34:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:34:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:34:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:34:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:34:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:34:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:34:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:34:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:34:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:34:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:34:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:34:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:34:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:34:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:34:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:34:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:34:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:34:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:34:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:34:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:34:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:34:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:34:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:34:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:34:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:34:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:34:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:34:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:34:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:34:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:34:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:34:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:34:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:34:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:34:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:34:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:34:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:34:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:34:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:34:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:34:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:34:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:34:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:34:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:34:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:34:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:34:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:34:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:34:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:34:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:34:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:34:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:34:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:34:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:34:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:35:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:35:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:35:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:35:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:35:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:35:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:35:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:35:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:35:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:35:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:35:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:35:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:35:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:35:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:35:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:35:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:35:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:35:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:35:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:35:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:35:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:35:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:35:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:35:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:35:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:35:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:35:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:35:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:35:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:35:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:35:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:35:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:35:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:35:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:35:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:35:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:35:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:35:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:35:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:35:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:35:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:35:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:35:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:35:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:35:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:35:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:35:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:35:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:35:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:35:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:35:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:35:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:35:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:35:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:35:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:35:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:35:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:35:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:35:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:35:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:35:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:35:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:35:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:35:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:35:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:35:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:35:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:35:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:35:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:35:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:35:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:35:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:35:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:35:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:35:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:35:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:35:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:35:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:35:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:35:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:35:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:35:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:35:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:35:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:35:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:35:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:35:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:35:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:35:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:35:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:35:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:35:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:35:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:35:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:35:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:35:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:35:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:35:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:35:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:35:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:35:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:35:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:35:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:35:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:35:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:35:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:35:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:35:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:35:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:35:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:35:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:35:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:35:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:35:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:35:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:35:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:35:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:35:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:35:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:35:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:35:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:35:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:35:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:35:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:35:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:35:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:35:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:35:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:35:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:35:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:35:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:35:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:35:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:35:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:35:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:35:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:35:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:35:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:35:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:35:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:35:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:35:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:35:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:35:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:35:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:35:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:35:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:35:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:35:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:35:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:35:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:35:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:35:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:35:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:35:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:35:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:35:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:35:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:35:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:35:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:35:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:35:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:35:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:35:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:35:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:35:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:35:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:35:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:35:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:35:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:35:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:35:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:35:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:35:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:35:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:35:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:35:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:35:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:35:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:35:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:35:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:35:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:35:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:35:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:35:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:35:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:35:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:35:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:35:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:35:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:35:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:35:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:35:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:35:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:35:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:35:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:35:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:35:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:35:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:35:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:35:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:35:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:35:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:35:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:35:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:35:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:35:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:35:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:35:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:35:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:35:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:35:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:35:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:35:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:35:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:35:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:36:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:36:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:36:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:36:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:36:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:36:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:36:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:36:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:36:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:36:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:36:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:36:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:36:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:36:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:36:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:36:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:36:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:36:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:36:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:36:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:36:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:36:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:36:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:36:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:36:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:36:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:36:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:36:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:36:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:36:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:36:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:36:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:36:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:36:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:36:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:36:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:36:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:36:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:36:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:36:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:36:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:36:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:36:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:36:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:36:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:36:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:36:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:36:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:36:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:36:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:36:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:36:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:36:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:36:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:36:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:36:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:36:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:36:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:36:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:36:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:36:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:36:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:36:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:36:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:36:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:36:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:36:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:36:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:36:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:36:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:36:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:36:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:36:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:36:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:36:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:36:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:36:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:36:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:36:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:36:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:36:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:36:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:36:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:36:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:36:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:36:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:36:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:36:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:36:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:36:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:36:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:36:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:36:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:36:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:36:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:36:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:36:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:36:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:36:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:36:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:36:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:36:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:36:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:36:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:36:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:36:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:36:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:36:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:36:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:36:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:36:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:36:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:36:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:36:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:36:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:36:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:36:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:36:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:36:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:36:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:36:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:36:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:36:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:36:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:36:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:36:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:36:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:36:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:36:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:36:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:36:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:36:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:36:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:36:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:36:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:36:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:36:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:36:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:36:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:36:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:36:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:36:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:36:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:36:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:36:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:36:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:36:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:36:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:36:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:36:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:36:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:36:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:36:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:36:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:36:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:36:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:36:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:36:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:36:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:36:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:36:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:36:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:38:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:38:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:38:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:38:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:38:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:38:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:38:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:38:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:38:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:38:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:38:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:38:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:38:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:38:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:38:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:38:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:38:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:38:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:38:48 rdma06 rsyslogd[1041]: imjournal: 5082 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 18:38:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:38:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:38:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:38:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:38:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:38:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:38:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:38:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:38:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:38:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:38:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:38:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:38:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:38:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:38:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:38:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:38:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:38:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:38:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:38:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:38:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:38:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:38:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:38:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:38:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:38:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:38:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:38:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:38:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:38:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:38:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:38:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:38:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:38:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:38:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:38:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:39:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:39:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:39:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:39:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:39:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:39:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:39:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:39:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:39:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:39:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:39:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:39:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:39:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:39:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:39:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:39:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:39:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:39:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:39:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:39:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:39:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:39:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:39:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:39:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:39:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:39:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:39:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:39:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:39:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:39:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:39:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:39:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:39:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:39:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:39:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:39:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:39:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:39:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:39:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:39:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:39:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:39:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:39:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:39:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:39:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:39:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:39:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:39:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:39:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:39:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:39:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:39:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:39:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:39:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:39:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:39:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:39:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:39:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:39:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:39:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:39:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:39:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:39:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:39:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:39:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:39:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:39:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:39:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:39:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:39:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:39:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:39:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:39:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:39:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:39:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:39:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:39:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:39:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:39:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:39:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:39:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:39:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:39:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:39:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:39:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:39:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:39:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:39:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:39:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:39:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:39:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:39:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:39:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:39:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:39:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:39:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:39:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:39:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:39:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:39:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:39:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:39:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:39:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:39:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:39:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:39:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:39:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:39:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:39:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:39:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:39:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:39:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:39:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:39:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:39:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:39:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:39:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:39:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:39:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:39:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:39:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:39:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:39:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:39:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:39:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:39:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:39:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:39:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:39:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:39:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:39:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:39:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:39:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:39:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:39:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:39:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:39:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:39:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:39:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:39:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:39:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:39:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:39:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:39:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:39:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:39:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:39:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:39:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:39:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:39:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:39:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:39:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:39:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:39:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:39:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:39:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:39:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:39:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:39:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:39:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:39:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:39:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:39:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:39:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:39:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:39:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:39:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:39:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:39:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:39:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:39:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:39:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:39:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:39:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:39:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:39:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:39:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:39:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:39:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:39:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:39:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:39:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:39:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:39:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:39:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:39:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:39:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:39:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:39:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:39:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:39:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:39:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:39:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:39:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:39:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:39:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:39:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:39:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:39:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:39:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:39:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:39:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:39:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:39:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:39:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:39:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:39:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:39:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:39:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:39:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:39:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:39:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:39:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:39:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:39:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:39:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:40:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:40:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:40:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:40:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:40:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:40:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:40:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:40:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:40:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:40:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:40:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:40:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:40:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:40:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:40:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:40:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:40:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:40:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:40:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:40:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:40:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:40:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:40:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:40:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:40:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:40:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:40:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:40:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:40:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:40:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:40:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:40:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:40:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:40:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:40:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:40:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:40:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:40:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:40:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:40:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:40:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:40:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:40:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:40:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:40:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:40:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:40:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:40:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:40:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:40:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:40:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:40:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:40:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:40:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:40:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:40:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:40:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:40:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:40:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:40:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:40:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:40:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:40:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:40:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:40:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:40:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:40:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:40:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:40:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:40:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:40:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:40:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:40:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:40:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:40:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:40:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:40:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:40:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:40:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:40:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:40:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:40:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:40:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:40:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:40:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:40:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:40:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:40:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:40:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:40:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:40:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:40:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:40:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:40:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:40:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:40:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:40:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:40:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:40:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:40:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:40:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:40:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:40:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:40:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:40:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:40:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:40:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:40:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:40:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:40:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:40:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:40:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:40:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:40:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:40:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:40:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:40:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:40:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:40:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:40:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:40:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:40:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:40:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:40:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:40:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:40:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:40:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:40:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:40:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:40:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:40:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:40:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:40:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:40:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:40:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:40:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:40:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:40:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:40:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:40:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:40:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:40:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:40:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:40:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:40:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:40:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:40:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:40:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:40:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:40:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:40:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:40:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:40:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:40:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:40:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:40:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:40:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:40:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:40:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:40:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:40:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:40:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:40:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:40:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:40:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:40:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:40:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:40:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:40:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:40:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:40:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:40:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:40:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:40:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:40:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:40:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:40:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:40:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:40:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:40:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:40:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:40:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:40:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:40:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:40:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:40:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:40:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:40:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:40:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:40:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:40:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:40:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:40:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:40:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:40:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:40:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:40:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:40:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:40:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:40:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:40:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:40:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:40:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:40:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:40:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:40:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:40:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:40:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:40:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:40:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:40:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:40:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:40:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:40:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:40:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:40:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:41:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:41:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:41:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:41:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:41:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:41:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:41:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:41:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:41:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:41:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:41:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:41:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:41:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:41:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:41:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:41:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:41:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:41:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:41:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:41:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:41:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:41:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:41:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:41:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:41:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:41:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:41:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:41:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:41:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:41:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:41:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:41:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:41:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:41:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:41:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:41:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:41:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:41:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:41:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:41:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:41:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:41:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:41:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:41:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:41:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:41:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:41:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:41:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:41:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:41:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:41:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:41:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:41:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:41:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:41:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:41:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:41:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:41:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:41:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:41:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:41:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:41:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:41:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:41:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:41:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:41:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:41:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:41:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:41:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:41:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:41:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:41:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:41:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:41:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:41:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:41:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:41:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:41:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:41:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:41:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:41:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:41:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:41:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:41:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:41:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:41:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:41:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:41:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:41:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:41:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:41:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:41:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:41:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:41:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:41:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:41:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:41:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:41:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:41:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:41:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:41:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:41:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:41:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:41:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:41:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:41:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:41:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:41:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:41:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:41:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:41:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:41:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:41:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:41:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:41:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:41:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:41:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:41:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:41:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:41:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:41:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:41:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:41:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:41:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:41:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:41:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:41:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:41:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:41:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:41:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:41:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:41:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:41:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:41:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:41:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:41:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:41:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:41:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:41:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:41:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:41:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:41:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:41:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:41:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:41:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:41:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:41:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:41:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:41:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:41:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:41:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:41:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:41:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:41:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:41:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:41:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:41:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:41:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:41:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:41:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:41:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:41:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:41:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:41:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:41:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:41:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:41:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:41:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:41:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:41:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:41:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:41:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:41:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:41:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:41:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:41:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:41:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:41:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:41:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:41:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:41:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:41:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:41:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:41:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:41:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:41:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:41:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:41:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:41:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:41:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:41:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:41:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:41:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:41:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:41:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:41:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:41:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:41:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:41:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:41:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:41:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:41:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:41:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:41:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:41:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:41:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:41:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:41:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:41:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:41:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:41:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:41:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:41:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:41:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:41:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:41:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:42:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:42:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:42:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:42:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:42:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:42:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:42:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:42:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:42:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:42:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:42:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:42:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:42:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:42:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:42:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:42:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:42:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:42:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:42:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:42:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:42:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:42:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:42:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:42:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:42:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:42:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:42:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:42:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:42:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:42:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:42:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:42:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:42:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:42:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:42:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:42:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:42:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:42:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:42:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:42:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:42:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:42:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:42:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:42:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:42:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:42:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:42:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:42:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:42:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:42:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:42:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:42:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:42:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:42:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:42:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:42:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:42:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:42:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:42:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:42:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:42:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:42:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:42:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:42:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:42:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:42:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:42:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:42:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:42:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:42:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:42:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:42:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:42:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:42:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:42:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:42:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:42:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:42:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:42:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:42:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:42:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:42:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:42:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:42:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:42:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:42:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:42:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:42:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:42:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:42:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:42:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:42:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:42:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:42:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:42:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:42:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:42:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:42:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:42:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:42:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:42:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:42:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:42:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:42:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:42:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:42:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:42:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:42:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:42:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:42:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:42:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:42:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:42:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:42:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:42:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:42:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:42:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:42:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:42:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:42:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:42:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:42:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:42:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:42:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:42:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:42:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:42:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:42:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:42:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:42:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:42:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:42:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:42:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:42:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:42:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:42:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:42:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:42:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:42:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:42:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:42:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:42:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:42:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:42:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:42:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:42:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:42:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:42:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:42:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:42:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:42:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:42:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:42:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:42:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:42:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:42:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:42:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:42:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:42:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:42:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:42:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:42:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:42:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:42:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:42:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:42:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:42:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:42:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:42:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:42:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:42:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:42:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:42:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:42:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:42:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:42:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:42:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:42:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:42:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:42:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:42:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:42:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:42:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:42:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:42:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:42:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:42:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:42:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:42:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:42:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:42:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:42:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:42:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:42:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:42:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:42:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:42:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:42:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:42:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:42:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:42:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:42:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:42:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:42:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:42:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:42:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:42:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:42:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:42:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:42:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:42:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:42:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:42:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:42:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:42:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:42:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:43:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:43:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:43:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:43:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:43:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:43:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:43:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:43:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:43:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:43:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:43:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:43:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:43:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:43:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:43:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:43:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:43:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:43:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:43:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:43:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:43:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:43:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:43:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:43:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:43:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:43:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:43:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:43:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:43:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:43:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:43:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:43:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:43:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:43:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:43:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:43:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:43:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:43:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:43:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:43:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:43:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:43:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:43:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:43:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:43:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:43:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:43:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:43:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:43:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:43:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:43:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:43:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:43:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:43:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:43:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:43:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:43:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:43:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:43:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:43:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:43:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:43:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:43:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:43:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:43:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:43:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:43:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:43:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:43:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:43:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:43:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:43:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:43:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:43:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:43:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:43:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:43:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:43:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:43:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:43:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:43:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:43:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:43:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:43:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:43:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:43:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:43:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:43:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:43:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:43:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:43:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:43:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:43:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:43:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:43:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:43:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:43:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:43:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:43:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:43:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:43:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:43:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:43:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:43:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:43:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:43:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:43:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:43:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:43:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:43:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:43:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:43:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:43:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:43:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:43:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:43:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:43:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:43:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:43:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:43:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:43:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:43:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:43:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:43:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:43:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:43:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:43:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:43:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:43:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:43:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:43:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:43:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:43:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:43:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:43:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:43:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:43:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:43:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:43:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:43:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:43:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:43:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:43:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:43:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:43:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:43:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:43:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:43:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:43:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:43:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:43:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:43:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:43:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:43:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:43:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:43:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:43:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:43:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:43:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:43:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:43:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:43:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:43:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:43:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:43:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:43:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:43:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:43:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:43:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:43:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:43:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:43:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:43:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:43:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:43:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:43:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:43:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:43:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:43:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:43:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:43:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:43:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:43:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:43:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:43:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:43:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:43:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:43:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:43:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:43:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:43:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:43:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:43:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:43:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:43:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:43:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:43:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:43:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:43:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:43:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:43:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:43:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:43:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:43:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:43:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:43:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:43:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:43:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:43:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:43:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:43:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:43:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:43:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:43:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:43:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:43:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:44:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:44:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:44:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:44:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:44:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:44:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:44:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:44:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:44:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:44:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:44:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:44:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:44:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:44:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:44:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:44:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:44:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:44:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:44:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:44:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:44:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:44:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:44:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:44:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:44:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:44:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:44:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:44:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:44:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:44:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:44:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:44:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:44:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:44:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:44:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:44:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:44:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:44:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:44:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:44:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:44:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:44:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:44:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:44:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:44:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:44:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:44:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:44:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:44:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:44:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:44:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:44:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:44:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:44:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:44:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:44:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:44:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:44:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:44:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:44:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:44:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:44:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:44:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:44:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:44:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:44:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:44:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:44:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:44:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:44:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:44:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:44:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:44:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:44:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:44:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:44:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:44:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:44:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:44:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:44:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:44:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:44:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:44:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:44:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:44:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:44:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:44:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:44:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:44:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:44:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:44:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:44:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:44:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:44:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:44:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:44:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:44:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:44:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:44:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:44:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:44:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:44:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:44:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:44:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:44:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:44:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:44:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:44:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:44:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:44:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:44:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:44:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:44:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:44:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:44:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:44:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:44:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:44:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:44:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:44:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:44:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:44:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:44:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:44:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:44:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:44:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:44:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:44:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:44:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:44:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:44:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:44:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:44:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:44:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:44:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:44:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:44:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:44:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:44:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:44:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:44:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:44:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:44:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:44:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:44:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:44:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:44:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:44:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:44:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:44:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:44:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:44:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:44:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:44:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:44:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:44:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:44:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:44:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:44:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:44:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:44:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:44:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:44:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:44:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:44:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:44:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:44:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:44:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:44:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:44:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:44:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:44:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:44:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:44:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:44:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:44:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:44:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:44:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:44:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:44:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:44:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:44:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:44:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:44:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:44:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:44:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:44:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:44:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:44:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:44:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:44:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:44:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:44:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:44:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:44:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:44:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:44:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:44:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:44:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:44:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:44:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:44:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:44:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:44:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:44:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:44:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:44:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:44:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:44:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:44:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:44:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:44:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:44:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:44:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:44:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:44:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:45:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:45:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:45:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:45:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:45:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:45:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:45:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:45:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:45:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:45:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:45:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:45:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:45:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:45:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:45:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:45:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:45:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:45:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:45:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:45:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:45:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:45:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:45:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:45:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:45:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:45:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:45:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:45:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:45:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:45:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:45:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:45:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:45:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:45:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:45:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:45:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:45:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:45:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:45:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:45:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:45:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:45:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:45:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:45:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:45:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:45:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:45:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:45:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:45:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:45:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:45:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:45:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:45:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:45:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:45:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:45:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:45:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:45:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:45:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:45:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:45:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:45:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:45:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:45:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:45:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:45:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:45:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:45:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:45:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:45:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:45:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:45:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:45:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:45:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:45:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:45:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:45:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:45:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:45:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:45:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:45:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:45:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:45:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:45:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:45:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:45:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:45:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:45:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:45:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:45:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:45:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:45:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:45:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:45:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:45:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:45:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:45:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:45:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:45:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:45:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:45:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:45:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:45:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:45:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:45:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:45:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:45:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:45:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:45:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:45:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:45:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:45:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:45:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:45:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:45:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:45:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:45:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:45:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:45:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:45:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:45:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:45:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:45:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:45:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:45:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:45:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:45:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:45:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:45:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:45:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:45:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:45:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:45:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:45:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:45:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:45:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:45:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:45:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:45:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:45:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:45:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:45:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:45:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:45:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:45:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:45:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:45:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:45:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:45:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:45:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:45:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:45:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:45:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:45:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:45:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:45:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:45:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:45:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:45:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:45:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:45:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:45:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:45:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:45:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:45:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:45:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:45:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:45:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:45:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:45:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:45:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:45:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:45:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:45:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:45:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:45:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:45:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:45:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:45:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:45:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:45:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:45:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:45:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:45:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:45:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:45:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:45:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:45:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:45:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:45:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:45:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:45:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:45:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:45:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:45:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:45:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:45:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:45:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:45:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:45:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:45:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:45:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:45:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:45:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:45:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:45:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:45:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:45:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:45:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:45:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:45:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:45:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:45:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:45:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:45:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:45:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:46:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:46:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:46:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:46:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:46:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:46:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:46:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:46:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:46:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:46:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:46:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:46:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:46:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:46:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:46:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:46:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:46:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:46:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:46:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:46:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:46:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:46:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:46:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:46:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:46:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:46:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:46:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:46:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:46:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:46:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:46:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:46:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:46:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:46:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:46:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:46:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:46:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:46:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:46:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:46:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:46:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:46:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:46:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:46:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:46:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:46:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:46:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:46:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:46:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:46:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:46:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:46:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:46:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:46:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:46:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:46:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:46:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:46:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:46:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:46:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:46:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:46:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:46:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:46:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:46:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:46:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:46:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:46:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:46:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:46:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:46:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:46:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:46:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:46:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:46:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:46:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:46:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:46:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:46:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:46:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:46:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:46:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:46:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:46:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:46:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:46:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:46:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:46:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:46:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:46:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:46:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:46:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:46:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:46:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:46:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:46:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:46:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:46:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:46:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:46:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:46:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:46:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:46:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:46:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:46:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:46:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:46:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:46:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:46:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:46:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:46:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:46:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:46:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:46:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:46:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:46:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:46:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:46:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:46:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:46:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:46:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:46:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:46:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:46:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:46:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:46:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:46:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:46:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:46:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:46:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:46:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:46:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:46:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:46:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:46:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:46:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:46:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:46:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:46:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:46:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:46:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:46:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:46:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:46:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:46:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:46:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:46:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:46:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:46:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:46:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:46:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:46:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:46:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:46:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:46:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:46:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:46:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:46:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:46:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:46:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:46:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:46:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:48:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:48:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:48:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:48:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:48:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:48:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:48:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:48:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:48:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:48:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:48:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:48:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:48:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:48:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:48:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:48:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:48:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:48:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:48:53 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 18:48:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:48:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:48:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:48:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:48:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:48:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:48:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:48:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:48:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:48:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:48:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:48:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:48:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:48:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:48:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:48:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:48:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:48:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:49:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:49:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:49:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:49:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:49:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:49:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:49:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:49:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:49:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:49:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:49:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:49:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:49:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:49:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:49:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:49:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:49:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:49:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:49:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:49:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:49:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:49:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:49:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:49:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:49:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:49:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:49:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:49:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:49:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:49:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:49:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:49:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:49:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:49:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:49:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:49:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:49:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:49:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:49:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:49:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:49:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:49:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:49:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:49:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:49:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:49:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:49:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:49:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:49:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:49:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:49:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:49:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:49:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:49:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:49:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:49:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:49:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:49:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:49:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:49:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:49:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:49:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:49:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:49:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:49:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:49:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:49:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:49:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:49:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:49:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:49:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:49:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:49:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:49:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:49:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:49:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:49:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:49:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:49:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:49:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:49:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:49:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:49:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:49:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:49:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:49:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:49:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:49:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:49:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:49:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:49:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:49:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:49:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:49:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:49:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:49:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:49:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:49:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:49:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:49:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:49:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:49:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:49:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:49:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:49:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:49:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:49:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:49:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:49:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:49:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:49:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:49:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:49:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:49:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:49:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:49:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:49:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:49:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:49:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:49:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:49:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:49:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:49:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:49:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:49:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:49:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:49:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:49:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:49:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:49:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:49:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:49:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:49:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:49:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:49:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:49:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:49:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:49:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:49:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:49:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:49:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:49:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:49:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:49:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:49:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:49:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:49:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:49:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:49:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:49:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:49:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:49:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:49:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:49:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:49:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:49:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:49:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:49:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:49:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:49:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:49:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:49:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:49:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:49:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:49:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:49:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:49:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:49:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:49:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:49:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:49:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:49:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:49:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:49:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:49:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:49:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:49:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:49:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:49:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:49:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:49:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:49:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:49:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:49:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:49:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:49:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:49:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:49:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:49:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:49:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:49:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:49:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:49:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:49:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:49:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:49:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:49:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:49:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:49:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:49:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:49:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:49:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:49:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:49:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:49:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:49:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:49:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:49:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:49:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:49:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:49:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:49:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:49:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:49:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:49:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:49:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:50:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:50:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:50:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:50:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:50:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:50:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:50:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:50:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:50:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:50:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:50:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:50:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:50:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:50:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:50:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:50:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:50:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:50:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:50:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:50:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:50:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:50:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:50:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:50:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:50:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:50:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:50:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:50:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:50:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:50:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:50:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:50:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:50:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:50:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:50:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:50:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:50:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:50:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:50:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:50:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:50:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:50:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:50:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:50:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:50:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:50:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:50:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:50:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:50:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:50:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:50:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:50:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:50:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:50:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:50:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:50:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:50:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:50:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:50:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:50:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:50:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:50:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:50:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:50:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:50:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:50:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:50:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:50:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:50:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:50:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:50:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:50:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:50:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:50:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:50:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:50:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:50:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:50:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:50:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:50:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:50:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:50:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:50:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:50:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:50:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:50:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:50:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:50:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:50:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:50:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:50:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:50:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:50:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:50:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:50:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:50:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:50:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:50:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:50:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:50:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:50:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:50:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:50:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:50:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:50:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:50:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:50:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:50:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:50:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:50:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:50:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:50:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:50:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:50:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:50:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:50:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:50:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:50:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:50:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:50:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:50:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:50:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:50:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:50:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:50:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:50:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:50:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:50:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:50:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:50:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:50:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:50:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:50:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:50:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:50:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:50:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:50:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:50:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:50:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:50:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:50:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:50:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:50:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:50:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:50:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:50:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:50:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:50:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:50:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:50:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:50:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:50:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:50:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:50:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:50:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:50:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:50:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:50:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:50:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:50:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:50:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:50:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:50:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:50:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:50:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:50:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:50:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:50:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:50:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:50:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:50:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:50:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:50:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:50:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:50:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:50:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:50:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:50:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:50:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:50:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:50:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:50:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:50:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:50:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:50:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:50:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:50:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:50:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:50:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:50:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:50:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:50:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:50:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:50:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:50:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:50:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:50:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:50:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:50:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:50:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:50:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:50:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:50:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:50:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:50:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:50:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:50:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:50:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:50:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:50:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:50:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:50:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:50:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:50:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:50:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:50:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:51:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:51:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:51:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:51:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:51:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:51:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:51:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:51:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:51:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:51:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:51:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:51:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:51:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:51:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:51:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:51:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:51:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:51:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:51:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:51:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:51:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:51:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:51:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:51:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:51:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:51:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:51:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:51:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:51:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:51:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:51:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:51:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:51:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:51:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:51:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:51:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:51:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:51:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:51:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:51:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:51:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:51:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:51:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:51:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:51:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:51:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:51:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:51:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:51:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:51:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:51:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:51:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:51:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:51:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:51:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:51:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:51:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:51:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:51:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:51:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:51:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:51:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:51:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:51:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:51:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:51:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:51:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:51:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:51:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:51:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:51:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:51:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:51:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:51:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:51:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:51:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:51:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:51:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:51:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:51:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:51:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:51:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:51:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:51:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:51:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:51:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:51:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:51:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:51:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:51:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:51:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:51:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:51:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:51:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:51:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:51:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:51:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:51:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:51:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:51:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:51:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:51:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:51:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:51:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:51:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:51:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:51:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:51:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:51:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:51:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:51:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:51:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:51:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:51:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:51:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:51:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:51:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:51:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:51:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:51:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:51:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:51:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:51:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:51:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:51:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:51:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:51:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:51:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:51:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:51:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:51:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:51:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:51:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:51:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:51:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:51:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:51:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:51:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:51:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:51:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:51:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:51:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:51:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:51:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:51:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:51:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:52:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:52:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:52:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:52:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:52:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:52:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:52:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:52:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:52:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:52:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:52:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:52:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:52:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:52:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:52:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:52:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:52:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:52:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:52:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:52:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:52:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:52:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:52:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:52:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:52:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:52:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:52:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:52:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:52:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:52:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:52:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:52:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:52:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:52:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:52:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:52:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:52:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:52:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:52:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:52:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:52:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:52:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:52:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:52:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:52:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:52:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:52:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:52:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:52:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:52:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:52:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:52:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:52:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:52:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:52:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:52:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:52:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:52:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:52:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:52:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:52:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:52:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:52:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:52:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:52:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:52:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:52:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:52:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:52:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:52:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:52:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:52:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:52:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:52:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:52:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:52:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:52:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:52:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:52:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:52:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:52:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:52:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:52:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:52:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:52:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:52:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:52:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:52:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:52:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:52:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:52:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:52:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:52:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:52:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:52:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:52:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:52:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:52:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:52:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:52:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:52:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:52:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:52:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:52:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:52:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:52:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:52:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:52:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:52:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:52:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:52:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:52:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:52:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:52:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:52:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:52:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:52:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:52:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:52:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:52:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:52:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:52:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:52:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:52:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:52:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:52:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:52:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:52:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:52:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:52:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:52:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:52:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:52:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:52:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:52:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:52:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:52:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:52:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:52:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:52:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:52:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:52:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:52:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:52:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:52:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:52:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:52:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:52:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:52:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:52:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:52:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:52:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:52:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:52:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:52:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:52:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:52:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:52:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:52:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:52:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:52:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:52:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:52:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:52:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:52:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:52:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:52:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:52:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:52:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:52:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:52:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:52:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:52:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:52:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:52:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:52:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:53:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:53:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:53:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:53:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:53:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:53:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:53:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:53:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:53:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:53:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:53:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:53:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:53:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:53:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:53:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:53:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:53:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:53:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:53:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:53:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:53:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:53:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:53:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:53:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:53:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:53:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:53:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:53:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:53:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:53:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:53:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:53:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:53:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:53:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:53:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:53:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:53:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:53:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:53:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:53:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:53:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:53:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:53:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:53:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:53:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:53:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:53:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:53:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:53:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:53:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:53:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:53:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:53:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:53:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:53:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:53:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:53:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:53:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:53:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:53:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:53:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:53:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:53:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:53:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:53:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:53:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:53:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:53:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:53:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:53:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:53:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:53:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:53:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:53:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:53:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:53:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:53:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:53:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:53:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:53:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:53:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:53:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:53:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:53:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:53:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:53:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:53:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:53:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:53:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:53:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:53:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:53:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:53:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:53:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:53:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:53:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:53:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:53:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:53:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:53:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:53:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:53:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:53:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:53:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:53:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:53:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:53:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:53:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:53:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:53:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:53:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:53:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:53:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:53:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:53:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:53:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:53:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:53:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:53:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:53:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:53:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:53:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:53:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:53:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:53:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:53:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:53:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:53:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:53:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:53:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:53:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:53:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:53:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:53:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:53:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:53:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:53:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:53:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:53:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:53:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:53:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:53:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:53:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:53:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:53:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:53:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:53:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:53:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:53:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:53:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:53:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:53:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:53:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:53:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:53:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:53:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:53:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:53:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:53:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:53:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:53:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:53:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:53:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:53:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:53:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:53:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:53:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:53:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:53:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:53:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:53:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:53:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:53:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:53:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:53:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:53:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:53:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:53:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:53:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:53:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:53:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:53:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:53:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:53:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:53:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:53:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:53:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:53:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:53:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:53:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:53:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:53:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:53:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:53:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:53:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:53:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:53:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:53:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:53:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:53:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:53:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:53:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:53:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:53:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:53:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:53:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:53:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:53:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:53:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:53:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:53:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:53:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:53:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:53:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:53:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:53:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:54:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:54:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:54:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:54:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:54:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:54:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:54:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:54:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:54:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:54:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:54:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:54:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:54:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:54:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:54:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:54:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:54:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:54:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:54:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:54:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:54:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:54:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:54:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:54:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:54:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:54:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:54:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:54:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:54:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:54:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:54:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:54:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:54:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:54:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:54:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:54:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:54:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:54:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:54:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:54:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:54:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:54:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:54:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:54:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:54:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:54:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:54:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:54:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:54:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:54:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:54:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:54:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:54:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:54:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:54:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:54:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:54:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:54:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:54:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:54:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:54:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:54:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:54:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:54:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:54:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:54:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:54:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:54:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:54:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:54:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:54:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:54:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:54:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:54:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:54:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:54:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:54:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:54:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:54:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:54:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:54:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:54:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:54:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:54:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:54:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:54:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:54:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:54:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:54:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:54:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:54:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:54:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:54:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:54:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:54:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:54:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:54:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:54:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:54:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:54:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:54:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:54:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:54:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:54:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:54:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:54:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:54:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:54:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:54:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:54:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:54:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:54:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:54:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:54:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:54:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:54:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:54:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:54:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:54:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:54:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:54:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:54:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:54:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:54:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:54:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:54:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:54:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:54:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:54:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:54:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:54:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:54:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:54:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:54:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:54:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:54:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:54:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:54:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:54:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:54:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:54:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:54:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:54:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:54:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:54:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:54:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:54:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:54:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:54:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:54:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:54:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:54:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:54:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:54:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:54:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:54:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:54:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:54:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:54:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:54:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:54:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:54:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:54:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:54:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:54:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:54:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:54:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:54:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:54:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:54:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:54:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:54:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:54:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:54:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:54:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:54:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:54:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:54:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:54:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:54:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:54:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:54:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:54:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:54:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:54:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:54:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:54:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:54:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:54:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:54:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:54:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:54:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:54:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:54:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:54:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:54:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:54:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:54:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:54:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:54:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:54:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:54:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:54:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:54:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:54:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:54:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:54:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:54:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:54:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:54:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:54:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:54:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:54:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:54:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:54:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:54:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:55:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:55:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:55:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:55:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:55:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:55:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:55:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:55:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:55:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:55:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:55:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:55:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:55:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:55:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:55:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:55:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:55:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:55:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:55:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:55:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:55:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:55:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:55:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:55:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:55:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:55:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:55:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:55:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:55:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:55:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:55:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:55:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:55:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:55:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:55:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:55:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:55:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:55:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:55:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:55:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:55:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:55:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:55:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:55:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:55:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:55:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:55:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:55:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:55:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:55:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:55:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:55:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:55:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:55:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:55:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:55:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:55:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:55:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:55:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:55:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:55:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:55:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:55:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:55:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:55:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:55:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:55:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:55:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:55:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:55:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:55:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:55:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:55:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:55:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:55:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:55:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:55:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:55:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:55:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:55:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:55:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:55:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:55:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:55:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:55:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:55:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:55:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:55:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:55:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:55:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:55:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:55:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:55:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:55:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:55:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:55:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:55:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:55:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:55:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:55:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:55:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:55:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:55:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:55:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:55:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:55:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:55:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:55:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:55:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:55:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:55:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:55:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:55:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:55:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:55:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:55:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:55:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:55:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:55:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:55:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:55:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:55:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:55:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:55:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:55:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:55:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:55:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:55:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:55:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:55:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:55:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:55:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:55:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:55:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:55:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:55:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:55:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:55:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:55:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:55:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:55:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:55:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:55:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:55:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:55:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:55:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:55:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:55:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:55:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:55:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:55:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:55:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:55:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:55:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:55:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:55:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:55:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:55:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:55:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:55:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:55:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:55:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:55:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:55:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:55:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:55:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:55:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:55:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:55:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:55:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:55:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:55:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:55:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:55:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:55:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:55:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:55:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:55:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:55:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:55:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:55:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:55:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:55:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:55:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:55:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:55:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:55:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:55:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:55:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:55:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:55:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:55:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:55:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:55:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:55:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:55:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:55:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:55:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:55:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:55:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:55:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:55:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:55:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:55:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:55:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:55:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:55:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:55:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:55:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:55:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:55:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:55:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:55:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:55:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:55:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:55:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:56:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:56:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:56:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:56:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:56:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:56:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:56:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:56:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:56:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:56:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:56:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:56:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:56:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:56:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:56:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:56:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:56:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:56:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:56:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:56:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:56:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:56:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:56:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:56:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:56:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:56:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:56:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:56:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:56:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:56:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:56:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:56:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:56:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:56:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:56:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:56:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:56:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:56:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:56:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:56:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:56:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:56:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:56:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:56:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:56:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:56:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:56:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:56:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:56:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:56:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:56:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:56:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:56:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:56:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:56:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:56:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:56:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:56:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:56:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:56:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:56:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:56:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:56:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:56:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:56:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:56:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:56:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:56:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:56:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:56:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:56:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:56:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:56:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:56:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:56:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:56:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:56:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:56:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:56:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:56:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:56:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:56:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:56:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:56:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:56:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:56:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:56:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:56:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:56:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:56:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:56:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:56:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:56:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:56:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:56:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:56:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:56:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:56:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:56:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:56:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:56:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:56:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:56:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:56:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:56:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:56:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:56:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:56:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:56:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:56:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:56:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:56:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:56:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:56:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:56:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:56:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:56:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:56:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:56:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:56:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:56:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:56:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:56:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:56:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:56:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:56:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:56:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:56:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:56:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:56:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:56:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:56:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:56:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:56:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:56:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:56:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:56:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:56:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:56:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:56:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:56:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:56:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:56:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:56:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:56:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:56:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:56:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:56:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:56:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:56:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:56:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:56:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:56:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:56:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:56:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:56:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:56:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:56:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:56:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:56:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:56:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:56:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:56:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:56:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:56:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:56:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:56:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:56:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:56:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:56:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:56:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:56:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:56:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:56:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:56:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:56:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:56:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:56:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:56:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:56:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:58:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:58:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:58:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:58:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:58:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:58:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:58:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:58:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:58:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:58:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:58:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:58:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:58:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:58:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:58:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:58:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:58:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:58:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:58:58 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 18:59:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:59:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:59:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:59:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:59:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:59:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:59:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:59:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:59:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:59:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:59:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:59:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:59:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:59:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:59:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:59:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:59:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:59:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:59:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:59:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:59:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:59:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:59:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:59:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:59:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:59:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:59:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:59:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:59:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:59:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:59:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:59:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:59:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:59:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:59:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:59:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:59:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:59:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:59:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:59:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:59:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:59:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:59:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:59:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:59:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:59:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:59:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:59:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:59:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:59:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:59:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:59:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:59:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:59:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:59:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:59:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:59:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:59:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:59:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:59:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:59:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:59:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:59:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:59:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:59:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:59:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:59:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:59:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:59:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:59:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:59:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:59:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:59:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:59:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:59:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:59:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:59:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:59:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:59:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:59:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:59:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:59:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:59:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:59:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:59:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:59:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:59:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:59:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:59:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:59:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:59:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:59:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:59:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:59:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:59:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:59:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:59:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:59:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:59:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:59:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:59:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:59:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:59:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:59:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:59:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:59:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:59:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:59:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:59:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:59:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:59:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:59:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:59:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:59:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:59:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:59:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:59:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:59:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:59:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:59:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:59:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:59:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:59:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:59:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:59:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:59:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:59:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:59:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:59:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:59:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:59:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:59:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:59:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:59:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:59:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:59:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:59:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:59:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:59:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:59:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:59:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:59:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:59:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:59:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:59:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:59:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:59:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:59:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:59:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:59:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:59:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:59:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:59:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:59:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:59:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:59:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:59:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:59:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:59:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:59:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:59:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:59:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:59:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:59:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:59:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:59:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:59:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:59:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:59:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:59:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:59:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:59:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:59:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:59:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:59:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:59:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:59:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:59:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:59:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:59:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:59:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:59:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:59:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:59:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:59:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:59:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:59:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:59:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:59:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:59:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:59:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:59:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:59:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:59:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:59:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:59:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:59:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:59:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 18:59:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 18:59:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:59:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:59:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:59:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 18:59:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 18:59:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 18:59:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 18:59:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 18:59:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 18:59:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 18:59:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 18:59:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 18:59:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 18:59:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 18:59:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 18:59:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 18:59:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:00:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:00:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:00:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:00:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:00:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:00:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:00:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:00:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:00:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:00:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:00:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:00:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:00:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:00:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:00:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:00:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:00:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:00:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:00:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:00:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:00:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:00:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:00:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:00:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:00:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:00:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:00:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:00:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:00:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:00:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:00:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:00:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:00:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:00:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:00:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:00:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:00:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:00:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:00:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:00:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:00:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:00:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:00:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:00:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:00:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:00:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:00:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:00:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:00:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:00:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:00:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:00:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:00:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:00:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:00:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:00:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:00:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:00:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:00:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:00:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:00:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:00:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:00:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:00:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:00:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:00:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:00:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:00:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:00:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:00:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:00:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:00:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:00:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:00:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:00:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:00:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:00:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:00:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:00:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:00:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:00:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:00:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:00:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:00:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:00:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:00:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:00:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:00:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:00:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:00:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:00:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:00:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:00:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:00:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:00:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:00:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:00:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:00:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:00:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:00:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:00:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:00:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:00:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:00:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:00:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:00:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:00:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:00:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:00:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:00:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:00:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:00:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:00:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:00:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:00:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:00:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:00:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:00:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:00:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:00:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:00:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:00:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:00:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:00:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:00:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:00:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:00:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:00:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:00:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:00:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:00:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:00:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:00:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:00:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:00:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:00:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:00:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:00:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:00:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:00:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:00:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:00:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:00:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:00:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:00:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:00:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:00:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:00:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:00:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:00:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:00:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:00:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:00:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:00:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:00:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:00:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:00:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:00:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:00:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:00:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:00:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:00:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:00:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:00:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:00:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:00:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:00:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:00:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:00:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:00:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:00:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:00:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:00:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:00:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:00:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:00:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:00:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:00:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:00:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:00:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:00:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:00:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:00:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:00:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:00:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:00:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:00:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:00:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:00:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:00:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:00:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:00:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:00:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:00:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:00:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:00:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:00:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:00:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:00:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:00:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:00:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:00:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:00:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:00:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:00:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:00:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:00:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:00:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:00:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:00:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:00:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:00:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:00:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:00:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:00:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:00:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:01:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:01:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:01:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:01:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:01:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:01:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:01:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:01:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:01:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:01:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:01:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:01:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:01:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:01:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:01:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:01:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:01:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:01:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:01:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:01:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:01:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:01:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:01:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:01:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:01:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:01:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:01:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:01:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:01:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:01:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:01:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:01:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:01:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:01:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:01:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:01:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:01:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:01:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:01:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:01:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:01:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:01:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:01:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:01:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:01:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:01:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:01:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:01:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:01:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:01:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:01:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:01:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:01:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:01:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:01:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:01:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:01:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:01:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:01:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:01:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:01:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:01:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:01:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:01:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:01:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:01:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:01:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:01:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:01:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:01:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:01:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:01:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:01:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:01:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:01:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:01:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:01:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:01:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:01:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:01:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:01:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:01:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:01:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:01:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:01:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:01:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:01:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:01:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:01:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:01:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:01:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:01:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:01:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:01:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:01:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:01:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:01:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:01:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:01:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:01:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:01:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:01:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:01:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:01:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:01:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:01:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:01:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:01:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:01:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:01:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:01:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:01:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:01:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:01:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:01:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:01:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:01:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:01:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:01:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:01:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:01:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:01:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:01:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:01:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:01:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:01:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:01:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:01:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:01:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:01:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:01:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:01:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:01:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:01:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:01:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:01:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:01:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:01:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:01:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:01:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:01:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:01:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:01:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:01:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:01:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:01:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:01:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:01:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:01:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:01:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:01:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:01:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:01:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:01:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:01:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:01:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:01:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:01:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:01:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:01:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:01:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:01:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:01:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:01:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:01:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:01:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:01:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:01:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:01:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:01:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:01:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:01:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:01:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:01:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:01:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:01:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:01:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:01:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:01:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:01:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:01:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:01:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:01:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:01:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:01:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:01:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:01:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:01:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:01:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:01:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:01:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:01:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:01:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:01:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:01:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:01:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:01:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:01:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:01:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:01:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:01:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:01:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:01:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:01:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:01:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:01:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:01:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:01:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:01:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:01:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:01:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:01:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:01:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:01:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:01:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:01:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:02:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:02:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:02:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:02:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:02:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:02:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:02:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:02:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:02:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:02:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:02:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:02:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:02:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:02:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:02:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:02:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:02:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:02:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:02:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:02:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:02:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:02:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:02:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:02:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:02:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:02:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:02:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:02:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:02:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:02:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:02:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:02:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:02:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:02:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:02:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:02:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:02:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:02:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:02:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:02:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:02:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:02:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:02:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:02:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:02:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:02:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:02:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:02:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:02:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:02:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:02:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:02:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:02:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:02:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:02:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:02:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:02:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:02:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:02:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:02:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:02:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:02:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:02:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:02:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:02:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:02:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:02:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:02:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:02:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:02:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:02:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:02:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:02:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:02:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:02:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:02:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:02:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:02:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:02:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:02:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:02:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:02:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:02:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:02:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:02:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:02:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:02:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:02:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:02:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:02:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:02:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:02:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:02:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:02:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:02:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:02:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:02:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:02:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:02:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:02:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:02:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:02:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:02:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:02:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:02:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:02:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:02:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:02:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:02:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:02:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:02:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:02:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:02:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:02:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:02:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:02:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:02:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:02:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:02:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:02:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:02:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:02:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:02:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:02:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:02:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:02:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:02:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:02:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:02:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:02:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:02:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:02:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:02:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:02:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:02:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:02:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:02:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:02:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:02:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:02:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:02:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:02:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:02:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:02:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:02:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:02:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:02:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:02:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:02:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:02:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:02:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:02:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:02:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:02:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:02:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:02:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:02:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:02:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:02:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:02:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:02:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:02:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:02:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:02:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:02:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:02:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:02:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:02:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:02:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:02:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:02:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:02:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:02:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:02:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:02:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:02:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:02:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:02:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:02:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:02:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:02:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:02:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:02:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:02:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:02:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:02:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:02:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:02:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:02:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:02:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:02:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:02:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:02:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:02:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:02:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:02:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:02:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:02:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:02:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:02:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:02:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:02:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:02:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:02:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:02:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:02:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:02:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:02:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:02:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:02:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:02:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:02:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:02:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:02:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:02:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:02:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:03:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:03:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:03:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:03:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:03:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:03:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:03:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:03:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:03:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:03:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:03:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:03:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:03:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:03:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:03:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:03:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:03:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:03:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:03:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:03:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:03:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:03:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:03:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:03:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:03:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:03:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:03:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:03:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:03:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:03:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:03:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:03:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:03:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:03:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:03:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:03:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:03:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:03:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:03:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:03:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:03:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:03:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:03:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:03:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:03:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:03:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:03:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:03:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:03:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:03:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:03:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:03:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:03:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:03:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:03:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:03:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:03:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:03:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:03:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:03:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:03:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:03:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:03:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:03:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:03:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:03:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:03:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:03:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:03:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:03:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:03:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:03:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:03:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:03:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:03:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:03:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:03:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:03:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:03:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:03:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:03:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:03:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:03:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:03:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:03:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:03:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:03:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:03:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:03:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:03:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:03:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:03:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:03:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:03:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:03:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:03:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:03:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:03:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:03:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:03:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:03:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:03:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:03:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:03:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:03:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:03:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:03:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:03:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:03:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:03:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:03:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:03:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:03:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:03:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:03:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:03:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:03:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:03:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:03:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:03:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:03:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:03:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:03:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:03:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:03:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:03:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:03:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:03:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:03:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:03:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:03:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:03:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:03:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:03:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:03:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:03:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:03:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:03:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:03:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:03:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:03:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:03:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:03:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:03:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:03:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:03:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:03:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:03:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:03:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:03:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:03:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:03:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:03:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:03:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:03:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:03:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:03:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:03:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:03:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:03:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:03:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:03:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:03:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:03:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:03:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:03:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:03:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:03:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:03:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:03:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:03:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:03:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:03:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:03:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:03:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:03:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:03:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:03:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:03:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:03:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:03:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:03:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:03:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:03:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:03:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:03:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:03:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:03:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:03:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:03:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:03:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:03:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:03:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:03:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:03:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:03:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:03:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:03:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:03:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:03:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:03:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:03:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:03:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:03:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:03:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:03:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:03:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:03:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:03:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:03:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:03:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:03:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:03:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:03:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:03:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:03:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:04:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:04:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:04:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:04:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:04:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:04:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:04:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:04:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:04:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:04:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:04:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:04:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:04:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:04:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:04:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:04:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:04:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:04:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:04:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:04:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:04:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:04:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:04:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:04:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:04:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:04:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:04:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:04:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:04:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:04:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:04:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:04:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:04:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:04:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:04:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:04:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:04:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:04:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:04:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:04:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:04:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:04:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:04:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:04:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:04:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:04:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:04:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:04:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:04:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:04:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:04:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:04:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:04:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:04:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:04:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:04:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:04:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:04:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:04:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:04:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:04:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:04:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:04:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:04:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:04:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:04:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:04:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:04:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:04:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:04:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:04:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:04:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:04:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:04:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:04:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:04:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:04:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:04:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:04:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:04:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:04:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:04:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:04:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:04:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:04:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:04:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:04:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:04:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:04:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:04:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:04:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:04:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:04:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:04:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:04:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:04:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:04:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:04:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:04:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:04:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:04:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:04:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:04:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:04:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:04:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:04:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:04:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:04:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:04:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:04:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:04:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:04:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:04:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:04:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:04:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:04:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:04:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:04:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:04:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:04:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:04:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:04:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:04:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:04:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:04:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:04:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:04:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:04:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:04:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:04:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:04:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:04:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:04:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:04:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:04:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:04:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:04:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:04:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:04:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:04:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:04:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:04:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:04:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:04:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:04:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:04:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:04:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:04:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:04:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:04:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:04:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:04:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:04:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:04:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:04:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:04:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:04:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:04:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:04:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:04:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:04:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:04:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:04:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:04:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:04:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:04:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:04:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:04:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:04:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:04:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:04:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:04:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:04:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:04:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:04:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:04:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:04:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:04:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:04:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:04:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:04:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:04:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:04:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:04:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:04:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:04:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:04:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:04:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:04:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:04:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:04:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:04:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:04:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:04:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:04:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:04:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:04:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:04:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:04:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:04:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:04:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:04:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:04:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:04:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:04:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:04:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:04:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:04:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:04:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:04:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:04:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:04:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:04:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:04:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:04:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:04:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:05:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:05:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:05:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:05:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:05:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:05:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:05:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:05:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:05:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:05:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:05:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:05:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:05:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:05:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:05:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:05:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:05:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:05:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:05:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:05:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:05:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:05:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:05:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:05:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:05:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:05:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:05:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:05:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:05:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:05:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:05:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:05:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:05:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:05:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:05:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:05:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:05:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:05:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:05:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:05:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:05:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:05:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:05:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:05:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:05:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:05:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:05:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:05:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:05:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:05:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:05:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:05:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:05:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:05:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:05:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:05:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:05:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:05:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:05:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:05:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:05:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:05:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:05:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:05:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:05:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:05:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:05:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:05:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:05:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:05:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:05:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:05:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:05:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:05:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:05:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:05:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:05:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:05:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:05:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:05:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:05:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:05:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:05:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:05:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:05:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:05:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:05:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:05:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:05:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:05:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:05:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:05:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:05:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:05:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:05:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:05:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:05:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:05:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:05:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:05:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:05:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:05:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:05:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:05:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:05:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:05:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:05:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:05:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:05:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:05:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:05:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:05:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:05:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:05:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:05:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:05:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:05:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:05:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:05:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:05:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:05:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:05:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:05:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:05:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:05:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:05:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:05:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:05:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:05:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:05:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:05:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:05:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:05:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:05:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:05:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:05:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:05:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:05:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:05:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:05:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:05:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:05:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:05:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:05:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:05:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:05:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:05:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:05:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:05:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:05:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:05:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:05:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:05:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:05:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:05:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:05:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:05:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:05:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:05:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:05:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:05:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:05:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:05:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:05:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:05:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:05:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:05:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:05:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:05:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:05:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:05:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:05:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:05:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:05:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:05:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:05:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:05:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:05:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:05:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:05:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:05:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:05:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:05:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:05:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:05:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:05:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:05:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:05:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:05:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:05:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:05:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:05:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:05:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:05:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:05:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:05:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:05:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:05:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:05:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:05:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:05:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:05:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:05:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:05:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:05:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:05:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:05:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:05:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:05:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:05:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:05:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:05:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:05:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:05:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:05:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:05:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:06:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:06:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:06:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:06:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:06:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:06:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:06:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:06:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:06:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:06:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:06:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:06:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:06:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:06:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:06:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:06:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:06:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:06:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:06:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:06:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:06:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:06:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:06:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:06:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:06:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:06:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:06:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:06:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:06:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:06:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:06:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:06:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:06:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:06:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:06:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:06:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:06:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:06:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:06:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:06:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:06:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:06:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:06:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:06:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:06:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:06:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:06:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:06:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:06:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:06:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:06:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:06:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:06:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:06:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:06:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:06:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:06:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:06:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:06:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:06:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:06:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:06:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:06:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:06:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:06:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:06:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:06:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:06:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:06:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:06:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:06:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:06:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:06:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:06:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:06:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:06:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:06:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:06:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:06:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:06:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:06:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:06:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:06:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:06:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:06:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:06:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:06:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:06:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:06:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:06:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:06:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:06:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:06:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:06:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:06:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:06:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:06:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:06:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:06:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:06:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:06:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:06:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:06:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:06:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:06:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:06:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:06:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:06:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:06:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:06:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:06:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:06:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:06:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:06:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:06:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:06:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:06:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:06:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:06:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:06:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:06:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:06:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:06:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:06:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:06:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:06:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:06:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:06:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:06:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:06:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:06:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:06:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:06:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:06:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:06:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:06:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:06:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:06:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:06:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:06:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:06:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:06:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:06:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:06:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:06:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:06:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:06:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:06:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:06:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:06:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:06:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:06:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:06:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:06:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:06:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:06:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:06:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:06:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:06:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:06:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:06:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:06:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:06:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:06:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:06:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:06:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:06:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:06:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:06:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:06:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:06:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:06:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:06:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:06:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:06:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:06:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:06:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:06:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:06:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:06:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:06:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:06:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:06:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:06:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:06:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:06:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:06:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:06:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:06:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:06:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:06:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:06:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:06:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:06:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:06:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:06:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:06:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:06:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:08:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:08:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:08:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:08:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:08:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:08:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:08:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:08:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:08:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:08:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:08:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:08:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:08:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:08:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:08:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:08:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:08:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:08:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:08:59 rdma06 rsyslogd[1041]: imjournal: 5086 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 19:09:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:09:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:09:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:09:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:09:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:09:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:09:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:09:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:09:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:09:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:09:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:09:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:09:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:09:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:09:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:09:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:09:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:09:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:09:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:09:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:09:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:09:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:09:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:09:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:09:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:09:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:09:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:09:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:09:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:09:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:09:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:09:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:09:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:09:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:09:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:09:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:09:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:09:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:09:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:09:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:09:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:09:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:09:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:09:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:09:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:09:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:09:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:09:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:09:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:09:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:09:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:09:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:09:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:09:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:09:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:09:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:09:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:09:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:09:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:09:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:09:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:09:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:09:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:09:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:09:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:09:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:09:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:09:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:09:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:09:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:09:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:09:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:09:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:09:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:09:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:09:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:09:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:09:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:09:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:09:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:09:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:09:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:09:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:09:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:09:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:09:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:09:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:09:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:09:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:09:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:09:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:09:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:09:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:09:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:09:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:09:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:09:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:09:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:09:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:09:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:09:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:09:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:09:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:09:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:09:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:09:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:09:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:09:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:09:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:09:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:09:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:09:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:09:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:09:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:09:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:09:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:09:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:09:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:09:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:09:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:09:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:09:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:09:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:09:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:09:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:09:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:09:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:09:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:09:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:09:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:09:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:09:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:09:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:09:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:09:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:09:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:09:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:09:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:09:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:09:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:09:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:09:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:09:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:09:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:09:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:09:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:09:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:09:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:09:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:09:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:09:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:09:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:09:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:09:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:09:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:09:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:09:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:09:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:09:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:09:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:09:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:09:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:09:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:09:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:09:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:09:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:09:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:09:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:09:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:09:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:09:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:09:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:09:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:09:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:09:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:09:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:09:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:09:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:09:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:09:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:09:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:09:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:09:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:09:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:09:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:09:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:09:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:09:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:09:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:09:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:09:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:09:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:09:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:09:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:09:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:09:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:09:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:09:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:09:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:09:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:09:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:09:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:09:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:09:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:09:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:09:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:09:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:09:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:09:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:09:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:09:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:09:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:09:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:09:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:09:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:09:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:10:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:10:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:10:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:10:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:10:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:10:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:10:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:10:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:10:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:10:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:10:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:10:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:10:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:10:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:10:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:10:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:10:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:10:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:10:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:10:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:10:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:10:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:10:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:10:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:10:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:10:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:10:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:10:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:10:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:10:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:10:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:10:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:10:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:10:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:10:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:10:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:10:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:10:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:10:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:10:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:10:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:10:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:10:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:10:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:10:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:10:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:10:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:10:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:10:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:10:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:10:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:10:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:10:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:10:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:10:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:10:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:10:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:10:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:10:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:10:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:10:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:10:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:10:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:10:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:10:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:10:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:10:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:10:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:10:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:10:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:10:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:10:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:10:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:10:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:10:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:10:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:10:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:10:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:10:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:10:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:10:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:10:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:10:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:10:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:10:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:10:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:10:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:10:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:10:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:10:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:10:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:10:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:10:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:10:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:10:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:10:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:10:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:10:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:10:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:10:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:10:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:10:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:10:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:10:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:10:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:10:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:10:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:10:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:10:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:10:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:10:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:10:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:10:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:10:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:10:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:10:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:10:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:10:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:10:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:10:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:10:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:10:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:10:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:10:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:10:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:10:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:10:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:10:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:10:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:10:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:10:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:10:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:10:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:10:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:10:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:10:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:10:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:10:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:10:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:10:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:10:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:10:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:10:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:10:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:10:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:10:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:10:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:10:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:10:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:10:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:10:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:10:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:10:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:10:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:10:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:10:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:10:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:10:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:10:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:10:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:10:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:10:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:10:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:10:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:10:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:10:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:10:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:10:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:10:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:10:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:10:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:10:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:10:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:10:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:10:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:10:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:10:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:10:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:10:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:10:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:10:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:10:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:10:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:10:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:10:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:10:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:10:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:10:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:10:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:10:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:10:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:10:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:10:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:10:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:10:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:10:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:10:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:10:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:10:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:10:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:10:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:10:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:10:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:10:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:10:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:10:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:10:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:10:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:10:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:10:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:10:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:10:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:10:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:10:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:10:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:10:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:11:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:11:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:11:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:11:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:11:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:11:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:11:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:11:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:11:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:11:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:11:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:11:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:11:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:11:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:11:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:11:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:11:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:11:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:11:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:11:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:11:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:11:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:11:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:11:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:11:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:11:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:11:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:11:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:11:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:11:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:11:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:11:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:11:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:11:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:11:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:11:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:11:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:11:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:11:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:11:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:11:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:11:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:11:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:11:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:11:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:11:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:11:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:11:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:11:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:11:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:11:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:11:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:11:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:11:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:11:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:11:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:11:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:11:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:11:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:11:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:11:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:11:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:11:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:11:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:11:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:11:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:11:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:11:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:11:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:11:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:11:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:11:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:11:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:11:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:11:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:11:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:11:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:11:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:11:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:11:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:11:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:11:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:11:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:11:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:11:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:11:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:11:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:11:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:11:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:11:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:11:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:11:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:11:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:11:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:11:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:11:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:11:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:11:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:11:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:11:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:11:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:11:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:11:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:11:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:11:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:11:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:11:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:11:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:11:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:11:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:11:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:11:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:11:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:11:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:11:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:11:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:11:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:11:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:11:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:11:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:11:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:11:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:11:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:11:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:11:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:11:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:11:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:11:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:11:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:11:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:11:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:11:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:11:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:11:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:11:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:11:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:11:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:11:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:11:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:11:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:11:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:11:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:11:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:11:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:11:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:11:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:11:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:11:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:11:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:11:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:11:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:11:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:11:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:11:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:11:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:11:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:11:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:11:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:11:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:11:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:11:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:11:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:11:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:11:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:11:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:11:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:11:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:11:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:11:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:11:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:11:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:11:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:11:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:11:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:11:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:11:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:11:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:11:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:11:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:11:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:11:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:11:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:11:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:11:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:11:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:11:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:11:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:11:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:11:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:11:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:11:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:11:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:11:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:11:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:11:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:11:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:11:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:11:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:11:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:11:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:11:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:11:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:11:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:11:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:11:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:11:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:11:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:11:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:11:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:11:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:11:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:11:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:11:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:11:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:11:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:11:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:12:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:12:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:12:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:12:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:12:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:12:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:12:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:12:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:12:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:12:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:12:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:12:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:12:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:12:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:12:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:12:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:12:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:12:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:12:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:12:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:12:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:12:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:12:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:12:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:12:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:12:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:12:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:12:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:12:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:12:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:12:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:12:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:12:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:12:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:12:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:12:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:12:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:12:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:12:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:12:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:12:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:12:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:12:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:12:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:12:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:12:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:12:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:12:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:12:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:12:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:12:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:12:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:12:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:12:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:12:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:12:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:12:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:12:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:12:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:12:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:12:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:12:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:12:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:12:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:12:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:12:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:12:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:12:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:12:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:12:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:12:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:12:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:12:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:12:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:12:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:12:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:12:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:12:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:12:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:12:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:12:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:12:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:12:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:12:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:12:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:12:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:12:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:12:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:12:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:12:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:12:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:12:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:12:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:12:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:12:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:12:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:12:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:12:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:12:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:12:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:12:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:12:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:12:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:12:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:12:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:12:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:12:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:12:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:12:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:12:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:12:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:12:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:12:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:12:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:12:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:12:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:12:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:12:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:12:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:12:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:12:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:12:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:12:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:12:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:12:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:12:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:12:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:12:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:12:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:12:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:12:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:12:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:12:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:12:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:12:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:12:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:12:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:12:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:12:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:12:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:12:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:12:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:12:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:12:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:12:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:12:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:12:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:12:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:12:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:12:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:12:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:12:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:12:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:12:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:12:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:12:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:12:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:12:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:12:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:12:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:12:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:12:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:12:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:12:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:12:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:12:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:12:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:12:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:12:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:12:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:12:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:12:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:12:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:12:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:12:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:12:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:12:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:12:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:12:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:12:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:12:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:12:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:12:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:12:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:12:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:12:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:12:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:12:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:12:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:12:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:12:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:12:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:12:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:12:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:12:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:12:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:12:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:12:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:12:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:12:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:12:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:12:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:12:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:12:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:12:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:12:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:12:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:12:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:12:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:12:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:12:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:12:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:12:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:12:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:12:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:12:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:13:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:13:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:13:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:13:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:13:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:13:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:13:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:13:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:13:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:13:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:13:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:13:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:13:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:13:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:13:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:13:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:13:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:13:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:13:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:13:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:13:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:13:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:13:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:13:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:13:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:13:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:13:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:13:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:13:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:13:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:13:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:13:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:13:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:13:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:13:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:13:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:13:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:13:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:13:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:13:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:13:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:13:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:13:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:13:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:13:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:13:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:13:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:13:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:13:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:13:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:13:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:13:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:13:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:13:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:13:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:13:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:13:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:13:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:13:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:13:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:13:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:13:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:13:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:13:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:13:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:13:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:13:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:13:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:13:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:13:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:13:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:13:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:13:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:13:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:13:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:13:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:13:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:13:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:13:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:13:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:13:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:13:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:13:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:13:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:13:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:13:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:13:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:13:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:13:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:13:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:13:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:13:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:13:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:13:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:13:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:13:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:13:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:13:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:13:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:13:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:13:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:13:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:13:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:13:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:13:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:13:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:13:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:13:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:13:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:13:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:13:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:13:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:13:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:13:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:13:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:13:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:13:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:13:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:13:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:13:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:13:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:13:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:13:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:13:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:13:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:13:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:13:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:13:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:13:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:13:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:13:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:13:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:13:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:13:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:13:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:13:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:13:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:13:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:13:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:13:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:13:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:13:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:13:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:13:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:13:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:13:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:13:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:13:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:13:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:13:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:13:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:13:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:13:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:13:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:13:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:13:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:13:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:13:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:13:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:13:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:13:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:13:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:13:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:13:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:13:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:13:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:13:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:13:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:13:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:13:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:13:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:13:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:13:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:13:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:13:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:13:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:13:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:13:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:13:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:13:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:13:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:13:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:13:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:13:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:13:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:13:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:13:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:13:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:13:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:13:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:13:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:13:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:13:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:13:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:13:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:13:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:13:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:13:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:13:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:13:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:13:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:13:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:13:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:13:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:13:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:13:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:13:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:13:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:13:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:13:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:13:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:13:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:13:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:13:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:13:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:13:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:14:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:14:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:14:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:14:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:14:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:14:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:14:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:14:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:14:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:14:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:14:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:14:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:14:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:14:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:14:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:14:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:14:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:14:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:14:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:14:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:14:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:14:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:14:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:14:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:14:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:14:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:14:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:14:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:14:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:14:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:14:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:14:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:14:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:14:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:14:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:14:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:14:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:14:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:14:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:14:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:14:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:14:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:14:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:14:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:14:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:14:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:14:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:14:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:14:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:14:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:14:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:14:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:14:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:14:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:14:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:14:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:14:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:14:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:14:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:14:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:14:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:14:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:14:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:14:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:14:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:14:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:14:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:14:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:14:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:14:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:14:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:14:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:14:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:14:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:14:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:14:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:14:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:14:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:14:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:14:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:14:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:14:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:14:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:14:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:14:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:14:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:14:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:14:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:14:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:14:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:14:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:14:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:14:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:14:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:14:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:14:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:14:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:14:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:14:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:14:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:14:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:14:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:14:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:14:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:14:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:14:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:14:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:14:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:14:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:14:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:14:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:14:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:14:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:14:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:14:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:14:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:14:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:14:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:14:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:14:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:14:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:14:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:14:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:14:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:14:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:14:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:14:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:14:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:14:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:14:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:14:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:14:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:14:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:14:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:14:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:14:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:14:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:14:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:14:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:14:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:14:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:14:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:14:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:14:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:14:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:14:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:14:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:14:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:14:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:14:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:14:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:14:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:14:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:14:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:14:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:14:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:14:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:14:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:14:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:14:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:14:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:14:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:14:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:14:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:14:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:14:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:14:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:14:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:14:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:14:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:14:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:14:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:14:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:14:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:14:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:14:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:14:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:14:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:14:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:14:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:14:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:14:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:14:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:14:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:14:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:14:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:14:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:14:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:14:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:14:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:14:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:14:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:14:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:14:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:14:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:14:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:14:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:14:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:14:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:14:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:14:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:14:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:14:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:14:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:14:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:14:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:14:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:14:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:14:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:14:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:14:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:14:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:14:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:14:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:14:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:14:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:15:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:15:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:15:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:15:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:15:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:15:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:15:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:15:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:15:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:15:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:15:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:15:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:15:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:15:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:15:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:15:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:15:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:15:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:15:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:15:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:15:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:15:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:15:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:15:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:15:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:15:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:15:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:15:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:15:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:15:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:15:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:15:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:15:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:15:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:15:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:15:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:15:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:15:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:15:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:15:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:15:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:15:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:15:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:15:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:15:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:15:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:15:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:15:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:15:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:15:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:15:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:15:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:15:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:15:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:15:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:15:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:15:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:15:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:15:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:15:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:15:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:15:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:15:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:15:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:15:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:15:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:15:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:15:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:15:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:15:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:15:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:15:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:15:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:15:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:15:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:15:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:15:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:15:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:15:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:15:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:15:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:15:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:15:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:15:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:15:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:15:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:15:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:15:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:15:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:15:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:15:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:15:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:15:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:15:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:15:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:15:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:15:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:15:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:15:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:15:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:15:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:15:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:15:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:15:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:15:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:15:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:15:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:15:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:15:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:15:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:15:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:15:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:15:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:15:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:15:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:15:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:15:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:15:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:15:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:15:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:15:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:15:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:15:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:15:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:15:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:15:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:15:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:15:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:15:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:15:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:15:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:15:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:15:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:15:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:15:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:15:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:15:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:15:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:15:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:15:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:15:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:15:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:15:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:15:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:15:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:15:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:15:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:15:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:15:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:15:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:15:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:15:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:15:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:15:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:15:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:15:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:15:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:15:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:15:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:15:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:15:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:15:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:15:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:15:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:15:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:15:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:15:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:15:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:15:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:15:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:15:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:15:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:15:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:15:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:15:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:15:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:15:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:15:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:15:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:15:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:15:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:15:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:15:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:15:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:15:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:15:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:15:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:15:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:15:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:15:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:15:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:15:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:15:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:15:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:15:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:15:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:15:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:15:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:15:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:15:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:15:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:15:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:15:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:15:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:15:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:15:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:15:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:15:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:15:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:15:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:15:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:15:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:15:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:15:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:15:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:15:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:16:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:16:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:16:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:16:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:16:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:16:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:16:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:16:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:16:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:16:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:16:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:16:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:16:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:16:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:16:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:16:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:16:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:16:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:16:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:16:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:16:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:16:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:16:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:16:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:16:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:16:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:16:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:16:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:16:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:16:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:16:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:16:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:16:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:16:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:16:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:16:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:16:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:16:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:16:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:16:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:16:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:16:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:16:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:16:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:16:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:16:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:16:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:16:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:16:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:16:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:16:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:16:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:16:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:16:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:16:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:16:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:16:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:16:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:16:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:16:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:16:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:16:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:16:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:16:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:16:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:16:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:16:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:16:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:16:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:16:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:16:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:16:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:16:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:16:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:16:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:16:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:16:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:16:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:16:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:16:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:16:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:16:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:16:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:16:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:16:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:16:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:16:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:16:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:16:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:16:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:16:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:16:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:16:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:16:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:16:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:16:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:16:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:16:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:16:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:16:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:16:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:16:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:16:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:16:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:16:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:16:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:16:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:16:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:16:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:16:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:16:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:16:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:16:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:16:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:16:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:16:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:16:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:16:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:16:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:16:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:16:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:16:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:16:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:16:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:16:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:16:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:16:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:16:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:16:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:16:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:16:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:16:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:16:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:16:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:16:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:16:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:16:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:16:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:16:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:16:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:16:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:16:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:16:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:16:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:16:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:16:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:16:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:16:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:16:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:16:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:16:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:16:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:16:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:16:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:16:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:16:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:16:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:16:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:16:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:16:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:16:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:16:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:16:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:16:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:16:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:16:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:16:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:16:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:16:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:16:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:16:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:16:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:16:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:16:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:16:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:16:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:16:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:16:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:16:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:16:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:16:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:16:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:16:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:16:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:16:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:16:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:16:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:16:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:16:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:16:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:16:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:16:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:16:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:16:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:16:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:16:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:16:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:16:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:19:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:19:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:19:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:19:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:19:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:19:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:19:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:19:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:19:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:19:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:19:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:19:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:19:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:19:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:19:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:19:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:19:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:19:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:19:04 rdma06 rsyslogd[1041]: imjournal: 5296 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 19:19:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:19:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:19:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:19:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:19:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:19:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:19:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:19:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:19:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:19:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:19:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:19:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:19:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:19:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:19:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:19:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:19:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:19:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:19:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:19:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:19:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:19:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:19:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:19:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:19:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:19:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:19:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:19:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:19:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:19:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:19:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:19:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:19:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:19:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:19:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:19:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:19:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:19:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:19:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:19:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:19:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:19:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:19:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:19:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:19:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:19:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:19:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:19:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:19:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:19:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:19:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:19:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:19:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:19:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:19:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:19:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:19:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:19:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:19:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:19:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:19:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:19:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:19:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:19:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:19:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:19:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:19:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:19:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:19:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:19:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:19:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:19:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:19:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:19:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:19:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:19:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:19:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:19:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:19:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:19:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:19:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:19:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:19:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:19:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:19:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:19:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:19:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:19:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:19:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:19:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:19:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:19:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:19:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:19:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:19:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:19:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:19:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:19:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:19:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:19:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:19:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:19:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:19:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:19:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:19:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:19:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:19:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:19:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:19:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:19:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:19:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:19:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:19:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:19:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:19:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:19:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:19:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:19:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:19:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:19:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:19:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:19:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:19:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:19:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:19:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:19:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:19:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:19:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:19:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:19:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:19:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:19:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:19:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:19:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:19:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:19:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:19:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:19:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:19:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:19:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:19:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:19:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:19:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:19:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:19:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:19:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:19:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:19:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:19:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:19:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:19:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:19:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:19:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:19:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:19:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:19:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:19:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:19:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:19:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:19:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:19:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:19:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:19:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:19:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:19:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:19:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:19:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:19:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:19:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:19:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:19:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:19:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:19:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:19:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:19:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:19:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:19:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:19:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:19:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:19:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:19:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:19:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:19:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:19:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:19:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:19:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:19:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:19:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:19:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:19:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:19:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:19:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:19:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:19:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:19:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:19:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:19:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:19:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:20:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:20:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:20:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:20:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:20:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:20:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:20:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:20:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:20:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:20:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:20:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:20:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:20:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:20:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:20:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:20:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:20:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:20:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:20:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:20:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:20:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:20:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:20:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:20:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:20:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:20:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:20:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:20:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:20:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:20:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:20:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:20:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:20:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:20:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:20:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:20:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:20:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:20:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:20:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:20:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:20:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:20:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:20:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:20:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:20:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:20:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:20:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:20:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:20:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:20:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:20:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:20:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:20:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:20:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:20:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:20:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:20:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:20:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:20:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:20:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:20:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:20:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:20:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:20:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:20:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:20:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:20:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:20:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:20:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:20:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:20:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:20:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:20:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:20:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:20:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:20:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:20:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:20:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:20:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:20:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:20:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:20:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:20:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:20:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:20:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:20:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:20:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:20:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:20:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:20:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:20:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:20:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:20:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:20:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:20:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:20:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:20:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:20:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:20:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:20:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:20:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:20:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:20:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:20:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:20:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:20:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:20:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:20:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:20:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:20:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:20:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:20:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:20:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:20:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:20:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:20:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:20:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:20:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:20:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:20:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:20:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:20:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:20:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:20:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:20:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:20:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:20:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:20:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:20:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:20:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:20:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:20:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:20:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:20:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:20:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:20:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:20:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:20:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:20:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:20:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:20:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:20:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:20:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:20:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:20:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:20:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:20:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:20:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:20:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:20:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:20:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:20:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:20:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:20:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:20:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:20:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:20:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:20:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:20:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:20:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:20:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:20:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:20:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:20:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:20:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:20:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:20:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:20:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:20:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:20:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:20:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:20:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:20:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:20:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:20:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:20:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:20:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:20:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:20:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:20:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:20:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:20:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:20:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:20:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:20:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:20:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:20:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:20:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:20:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:20:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:20:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:20:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:20:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:20:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:20:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:20:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:20:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:20:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:20:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:20:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:20:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:20:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:20:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:20:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:20:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:20:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:20:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:20:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:20:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:20:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:20:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:20:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:20:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:20:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:20:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:20:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:21:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:21:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:21:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:21:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:21:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:21:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:21:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:21:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:21:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:21:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:21:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:21:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:21:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:21:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:21:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:21:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:21:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:21:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:21:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:21:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:21:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:21:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:21:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:21:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:21:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:21:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:21:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:21:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:21:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:21:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:21:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:21:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:21:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:21:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:21:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:21:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:21:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:21:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:21:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:21:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:21:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:21:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:21:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:21:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:21:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:21:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:21:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:21:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:21:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:21:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:21:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:21:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:21:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:21:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:21:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:21:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:21:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:21:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:21:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:21:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:21:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:21:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:21:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:21:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:21:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:21:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:21:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:21:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:21:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:21:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:21:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:21:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:21:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:21:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:21:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:21:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:21:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:21:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:21:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:21:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:21:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:21:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:21:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:21:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:21:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:21:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:21:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:21:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:21:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:21:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:21:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:21:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:21:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:21:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:21:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:21:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:21:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:21:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:21:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:21:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:21:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:21:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:21:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:21:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:21:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:21:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:21:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:21:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:21:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:21:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:21:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:21:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:21:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:21:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:21:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:21:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:22:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:22:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:22:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:22:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:22:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:22:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:22:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:22:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:22:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:22:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:22:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:22:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:22:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:22:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:22:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:22:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:22:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:22:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:22:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:22:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:22:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:22:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:22:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:22:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:22:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:22:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:22:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:22:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:22:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:22:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:22:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:22:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:22:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:22:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:22:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:22:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:22:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:22:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:22:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:22:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:22:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:22:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:22:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:22:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:22:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:22:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:22:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:22:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:22:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:22:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:22:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:22:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:22:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:22:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:22:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:22:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:22:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:22:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:22:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:22:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:22:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:22:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:22:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:22:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:22:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:22:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:22:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:22:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:22:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:22:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:22:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:22:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:22:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:22:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:22:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:22:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:22:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:22:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:22:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:22:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:22:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:22:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:22:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:22:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:22:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:22:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:22:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:22:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:22:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:22:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:22:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:22:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:22:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:22:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:22:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:22:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:22:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:22:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:22:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:22:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:22:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:22:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:22:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:22:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:22:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:22:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:22:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:22:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:22:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:22:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:22:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:22:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:22:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:22:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:22:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:22:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:22:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:22:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:22:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:22:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:22:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:22:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:22:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:22:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:22:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:22:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:22:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:22:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:22:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:22:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:22:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:22:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:22:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:22:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:22:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:22:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:22:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:22:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:22:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:22:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:22:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:22:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:22:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:22:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:22:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:22:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:22:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:22:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:22:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:22:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:22:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:22:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:22:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:22:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:22:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:22:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:22:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:22:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:22:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:22:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:22:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:22:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:22:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:22:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:22:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:22:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:22:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:22:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:22:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:22:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:22:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:22:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:22:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:22:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:22:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:22:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:22:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:22:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:22:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:22:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:22:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:22:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:22:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:22:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:22:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:22:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:22:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:22:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:22:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:22:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:22:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:22:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:22:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:22:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:22:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:22:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:22:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:22:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:22:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:22:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:22:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:22:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:22:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:22:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:22:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:22:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:22:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:22:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:22:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:22:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:22:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:22:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:22:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:22:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:22:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:22:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:23:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:23:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:23:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:23:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:23:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:23:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:23:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:23:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:23:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:23:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:23:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:23:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:23:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:23:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:23:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:23:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:23:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:23:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:23:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:23:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:23:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:23:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:23:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:23:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:23:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:23:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:23:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:23:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:23:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:23:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:23:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:23:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:23:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:23:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:23:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:23:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:23:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:23:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:23:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:23:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:23:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:23:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:23:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:23:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:23:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:23:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:23:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:23:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:23:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:23:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:23:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:23:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:23:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:23:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:23:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:23:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:23:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:23:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:23:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:23:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:23:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:23:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:23:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:23:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:23:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:23:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:23:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:23:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:23:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:23:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:23:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:23:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:23:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:23:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:23:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:23:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:23:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:23:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:23:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:23:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:23:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:23:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:23:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:23:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:23:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:23:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:23:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:23:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:23:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:23:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:23:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:23:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:23:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:23:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:23:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:23:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:23:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:23:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:23:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:23:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:23:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:23:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:23:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:23:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:23:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:23:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:23:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:23:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:23:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:23:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:23:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:23:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:23:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:23:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:23:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:23:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:23:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:23:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:23:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:23:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:23:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:23:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:23:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:23:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:23:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:23:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:23:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:23:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:23:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:23:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:23:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:23:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:23:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:23:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:23:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:23:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:23:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:23:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:23:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:23:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:23:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:23:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:23:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:23:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:23:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:23:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:23:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:23:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:23:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:23:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:23:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:23:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:23:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:23:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:23:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:23:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:23:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:23:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:23:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:23:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:23:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:23:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:23:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:23:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:23:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:23:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:23:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:23:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:23:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:23:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:23:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:23:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:23:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:23:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:23:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:23:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:23:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:23:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:23:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:23:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:23:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:23:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:23:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:23:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:23:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:23:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:23:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:23:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:23:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:23:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:23:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:23:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:23:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:23:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:23:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:23:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:23:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:23:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:23:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:23:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:23:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:23:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:23:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:23:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:23:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:23:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:23:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:23:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:23:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:23:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:23:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:23:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:23:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:23:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:23:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:23:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:24:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:24:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:24:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:24:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:24:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:24:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:24:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:24:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:24:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:24:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:24:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:24:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:24:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:24:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:24:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:24:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:24:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:24:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:24:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:24:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:24:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:24:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:24:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:24:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:24:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:24:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:24:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:24:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:24:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:24:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:24:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:24:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:24:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:24:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:24:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:24:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:24:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:24:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:24:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:24:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:24:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:24:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:24:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:24:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:24:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:24:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:24:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:24:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:24:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:24:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:24:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:24:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:24:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:24:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:24:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:24:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:24:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:24:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:24:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:24:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:24:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:24:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:24:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:24:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:24:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:24:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:24:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:24:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:24:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:24:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:24:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:24:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:24:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:24:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:24:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:24:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:24:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:24:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:24:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:24:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:24:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:24:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:24:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:24:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:24:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:24:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:24:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:24:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:24:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:24:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:24:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:24:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:24:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:24:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:24:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:24:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:24:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:24:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:24:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:24:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:24:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:24:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:24:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:24:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:24:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:24:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:24:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:24:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:24:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:24:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:24:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:24:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:24:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:24:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:24:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:24:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:24:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:24:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:24:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:24:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:24:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:24:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:24:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:24:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:24:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:24:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:24:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:24:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:24:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:24:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:24:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:24:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:24:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:24:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:24:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:24:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:24:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:24:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:24:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:24:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:24:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:24:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:24:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:24:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:24:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:24:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:24:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:24:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:24:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:24:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:24:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:24:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:24:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:24:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:24:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:24:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:24:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:24:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:24:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:24:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:24:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:24:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:24:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:24:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:24:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:24:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:24:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:24:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:24:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:24:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:24:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:24:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:24:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:24:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:24:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:24:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:24:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:24:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:24:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:24:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:24:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:24:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:24:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:24:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:24:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:24:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:24:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:24:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:24:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:24:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:24:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:24:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:24:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:24:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:24:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:24:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:24:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:24:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:24:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:24:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:24:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:24:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:24:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:24:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:24:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:24:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:24:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:24:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:24:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:24:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:24:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:24:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:24:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:24:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:24:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:24:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:25:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:25:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:25:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:25:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:25:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:25:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:25:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:25:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:25:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:25:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:25:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:25:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:25:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:25:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:25:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:25:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:25:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:25:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:25:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:25:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:25:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:25:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:25:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:25:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:25:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:25:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:25:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:25:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:25:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:25:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:25:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:25:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:25:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:25:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:25:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:25:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:25:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:25:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:25:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:25:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:25:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:25:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:25:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:25:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:25:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:25:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:25:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:25:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:25:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:25:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:25:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:25:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:25:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:25:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:25:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:25:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:25:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:25:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:25:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:25:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:25:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:25:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:25:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:25:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:25:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:25:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:25:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:25:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:25:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:25:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:25:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:25:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:25:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:25:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:25:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:25:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:25:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:25:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:25:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:25:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:25:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:25:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:25:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:25:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:25:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:25:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:25:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:25:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:25:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:25:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:25:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:25:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:25:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:25:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:25:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:25:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:25:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:25:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:25:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:25:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:25:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:25:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:25:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:25:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:25:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:25:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:25:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:25:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:25:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:25:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:25:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:25:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:25:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:25:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:25:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:25:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:25:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:25:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:25:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:25:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:25:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:25:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:25:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:25:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:25:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:25:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:25:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:25:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:25:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:25:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:25:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:25:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:25:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:25:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:25:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:25:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:25:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:25:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:25:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:25:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:25:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:25:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:25:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:25:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:25:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:25:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:25:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:25:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:25:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:25:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:25:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:25:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:25:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:25:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:25:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:25:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:25:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:25:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:25:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:25:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:25:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:25:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:25:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:25:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:25:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:25:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:25:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:25:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:25:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:25:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:25:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:25:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:25:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:25:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:25:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:25:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:25:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:25:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:25:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:25:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:25:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:25:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:25:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:25:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:25:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:25:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:25:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:25:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:25:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:25:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:25:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:25:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:25:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:25:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:25:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:25:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:25:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:25:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:25:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:25:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:25:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:25:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:25:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:25:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:25:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:25:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:25:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:25:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:25:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:25:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:25:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:25:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:25:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:25:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:25:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:25:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:26:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:26:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:26:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:26:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:26:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:26:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:26:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:26:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:26:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:26:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:26:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:26:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:26:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:26:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:26:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:26:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:26:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:26:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:26:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:26:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:26:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:26:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:26:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:26:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:26:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:26:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:26:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:26:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:26:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:26:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:26:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:26:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:26:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:26:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:26:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:26:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:26:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:26:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:26:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:26:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:26:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:26:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:26:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:26:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:26:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:26:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:26:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:26:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:26:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:26:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:26:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:26:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:26:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:26:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:26:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:26:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:26:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:26:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:26:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:26:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:26:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:26:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:26:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:26:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:26:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:26:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:26:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:26:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:26:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:26:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:26:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:26:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:26:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:26:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:26:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:26:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:26:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:26:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:26:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:26:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:26:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:26:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:26:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:26:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:26:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:26:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:26:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:26:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:26:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:26:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:26:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:26:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:26:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:26:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:26:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:26:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:26:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:26:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:26:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:26:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:26:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:26:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:26:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:26:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:26:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:26:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:26:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:26:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:26:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:26:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:26:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:26:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:26:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:26:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:26:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:26:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:26:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:26:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:26:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:26:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:26:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:26:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:26:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:26:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:26:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:26:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:26:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:26:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:26:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:26:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:26:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:26:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:26:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:26:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:26:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:26:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:26:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:26:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:26:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:26:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:26:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:26:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:26:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:26:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:26:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:26:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:26:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:26:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:26:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:26:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:26:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:26:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:26:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:26:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:26:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:26:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:26:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:26:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:26:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:26:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:26:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:26:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:26:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:26:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:26:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:26:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:26:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:26:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:26:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:26:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:26:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:26:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:26:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:26:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:26:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:26:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:26:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:26:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:26:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:26:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:26:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:26:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:26:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:26:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:26:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:26:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:26:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:26:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:26:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:26:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:26:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:26:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:26:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:26:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:26:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:26:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:26:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:26:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:26:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:26:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:26:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:26:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:26:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:26:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:26:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:26:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:26:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:26:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:26:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:26:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:26:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:26:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:26:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:26:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:26:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:26:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:29:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:29:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:29:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:29:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:29:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:29:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:29:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:29:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:29:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:29:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:29:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:29:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:29:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:29:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:29:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:29:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:29:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:29:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:29:09 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 19:29:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:29:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:29:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:29:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:29:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:29:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:29:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:29:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:29:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:29:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:29:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:29:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:29:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:29:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:29:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:29:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:29:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:29:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:29:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:29:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:29:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:29:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:29:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:29:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:29:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:29:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:29:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:29:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:29:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:29:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:29:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:29:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:29:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:29:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:29:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:29:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:29:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:29:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:29:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:29:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:29:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:29:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:29:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:29:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:29:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:29:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:29:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:29:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:29:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:29:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:29:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:29:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:29:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:29:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:29:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:29:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:29:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:29:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:29:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:29:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:29:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:29:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:29:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:29:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:29:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:29:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:29:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:29:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:29:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:29:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:29:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:29:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:29:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:29:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:29:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:29:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:29:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:29:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:29:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:29:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:29:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:29:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:29:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:29:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:29:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:29:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:29:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:29:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:29:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:29:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:29:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:29:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:29:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:29:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:29:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:29:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:29:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:29:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:29:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:29:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:29:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:29:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:29:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:29:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:29:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:29:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:29:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:29:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:29:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:29:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:29:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:29:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:29:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:29:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:29:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:29:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:29:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:29:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:29:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:29:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:29:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:29:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:29:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:29:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:29:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:29:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:29:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:29:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:29:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:29:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:29:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:29:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:29:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:29:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:29:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:29:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:29:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:29:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:29:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:29:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:29:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:29:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:29:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:29:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:29:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:29:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:29:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:29:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:29:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:29:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:29:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:29:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:29:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:29:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:29:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:29:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:29:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:29:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:29:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:29:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:29:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:29:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:29:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:29:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:29:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:29:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:29:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:29:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:29:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:29:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:29:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:29:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:29:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:29:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:29:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:29:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:29:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:29:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:29:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:29:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:30:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:30:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:30:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:30:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:30:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:30:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:30:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:30:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:30:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:30:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:30:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:30:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:30:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:30:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:30:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:30:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:30:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:30:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:30:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:30:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:30:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:30:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:30:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:30:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:30:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:30:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:30:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:30:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:30:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:30:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:30:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:30:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:30:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:30:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:30:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:30:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:30:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:30:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:30:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:30:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:30:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:30:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:30:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:30:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:30:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:30:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:30:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:30:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:30:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:30:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:30:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:30:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:30:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:30:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:30:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:30:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:30:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:30:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:30:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:30:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:30:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:30:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:30:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:30:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:30:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:30:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:30:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:30:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:30:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:30:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:30:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:30:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:30:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:30:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:30:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:30:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:30:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:30:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:30:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:30:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:30:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:30:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:30:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:30:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:30:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:30:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:30:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:30:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:30:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:30:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:30:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:30:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:30:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:30:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:30:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:30:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:30:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:30:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:30:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:30:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:30:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:30:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:30:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:30:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:30:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:30:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:30:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:30:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:30:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:30:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:30:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:30:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:30:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:30:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:30:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:30:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:30:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:30:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:30:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:30:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:30:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:30:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:30:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:30:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:30:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:30:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:30:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:30:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:30:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:30:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:30:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:30:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:30:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:30:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:30:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:30:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:30:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:30:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:30:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:30:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:30:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:30:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:30:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:30:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:30:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:30:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:30:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:30:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:30:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:30:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:30:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:30:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:30:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:30:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:30:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:30:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:30:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:30:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:30:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:30:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:30:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:30:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:30:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:30:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:30:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:30:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:30:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:30:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:30:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:30:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:30:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:30:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:30:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:30:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:30:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:30:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:30:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:30:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:30:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:30:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:30:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:30:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:30:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:30:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:30:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:30:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:30:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:30:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:30:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:30:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:30:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:30:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:30:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:30:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:30:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:30:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:30:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:30:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:30:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:30:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:30:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:30:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:30:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:30:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:30:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:30:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:30:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:30:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:30:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:30:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:30:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:30:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:30:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:30:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:30:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:30:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:31:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:31:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:31:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:31:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:31:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:31:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:31:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:31:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:31:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:31:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:31:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:31:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:31:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:31:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:31:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:31:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:31:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:31:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:31:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:31:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:31:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:31:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:31:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:31:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:31:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:31:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:31:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:31:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:31:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:31:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:31:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:31:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:31:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:31:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:31:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:31:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:31:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:31:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:31:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:31:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:31:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:31:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:31:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:31:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:31:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:31:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:31:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:31:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:31:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:31:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:31:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:31:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:31:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:31:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:31:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:31:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:31:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:31:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:31:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:31:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:31:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:31:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:31:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:31:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:31:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:31:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:31:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:31:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:31:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:31:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:31:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:31:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:31:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:31:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:31:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:31:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:31:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:31:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:31:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:31:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:31:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:31:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:31:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:31:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:31:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:31:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:31:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:31:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:31:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:31:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:31:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:31:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:31:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:31:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:31:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:31:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:31:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:31:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:31:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:31:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:31:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:31:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:31:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:31:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:31:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:31:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:31:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:31:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:31:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:31:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:31:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:31:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:31:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:31:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:31:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:31:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:31:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:31:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:31:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:31:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:31:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:31:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:31:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:31:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:31:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:31:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:31:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:31:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:31:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:31:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:31:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:31:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:31:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:31:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:31:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:31:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:31:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:31:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:31:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:31:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:31:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:31:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:31:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:31:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:31:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:31:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:31:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:31:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:31:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:31:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:31:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:31:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:31:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:31:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:31:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:31:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:31:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:31:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:31:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:31:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:31:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:31:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:31:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:31:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:31:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:31:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:31:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:31:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:31:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:31:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:31:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:31:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:31:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:31:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:31:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:31:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:31:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:31:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:31:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:31:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:31:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:31:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:31:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:31:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:31:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:31:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:31:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:31:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:31:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:31:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:31:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:31:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:31:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:31:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:31:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:31:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:31:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:31:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:31:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:31:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:31:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:31:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:31:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:31:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:31:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:31:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:32:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:32:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:32:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:32:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:32:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:32:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:32:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:32:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:32:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:32:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:32:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:32:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:32:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:32:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:32:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:32:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:32:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:32:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:32:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:32:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:32:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:32:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:32:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:32:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:32:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:32:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:32:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:32:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:32:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:32:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:32:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:32:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:32:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:32:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:32:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:32:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:32:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:32:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:32:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:32:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:32:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:32:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:32:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:32:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:32:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:32:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:32:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:32:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:32:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:32:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:32:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:32:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:32:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:32:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:32:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:32:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:32:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:32:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:32:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:32:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:32:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:32:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:32:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:32:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:32:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:32:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:32:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:32:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:32:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:32:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:32:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:32:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:32:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:32:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:32:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:32:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:32:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:32:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:32:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:32:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:32:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:32:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:32:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:32:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:32:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:32:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:32:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:32:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:32:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:32:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:32:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:32:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:32:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:32:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:32:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:32:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:32:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:32:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:32:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:32:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:32:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:32:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:32:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:32:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:32:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:32:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:32:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:32:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:32:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:32:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:32:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:32:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:32:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:32:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:32:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:32:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:32:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:32:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:32:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:32:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:32:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:32:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:32:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:32:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:32:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:32:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:32:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:32:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:32:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:32:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:32:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:32:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:32:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:32:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:32:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:32:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:32:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:32:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:32:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:32:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:32:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:32:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:32:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:32:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:32:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:32:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:32:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:32:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:32:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:32:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:32:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:32:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:32:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:32:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:32:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:32:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:32:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:32:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:32:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:32:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:32:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:32:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:32:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:32:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:32:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:32:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:32:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:32:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:32:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:32:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:32:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:32:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:32:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:32:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:32:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:32:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:32:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:32:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:32:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:32:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:32:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:32:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:32:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:32:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:32:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:32:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:32:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:32:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:32:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:32:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:32:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:32:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:32:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:32:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:32:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:32:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:32:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:32:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:32:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:32:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:32:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:32:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:32:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:32:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:32:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:32:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:32:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:32:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:32:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:32:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:32:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:32:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:32:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:32:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:32:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:32:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:33:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:33:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:33:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:33:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:33:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:33:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:33:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:33:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:33:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:33:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:33:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:33:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:33:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:33:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:33:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:33:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:33:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:33:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:33:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:33:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:33:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:33:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:33:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:33:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:33:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:33:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:33:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:33:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:33:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:33:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:33:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:33:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:33:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:33:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:33:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:33:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:33:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:33:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:33:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:33:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:33:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:33:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:33:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:33:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:33:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:33:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:33:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:33:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:33:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:33:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:33:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:33:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:33:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:33:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:33:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:33:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:33:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:33:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:33:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:33:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:33:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:33:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:33:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:33:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:33:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:33:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:33:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:33:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:33:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:33:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:33:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:33:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:33:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:33:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:33:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:33:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:33:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:33:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:33:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:33:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:33:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:33:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:33:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:33:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:33:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:33:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:33:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:33:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:33:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:33:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:33:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:33:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:33:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:33:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:33:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:33:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:33:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:33:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:33:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:33:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:33:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:33:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:33:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:33:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:33:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:33:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:33:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:33:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:33:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:33:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:33:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:33:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:33:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:33:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:33:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:33:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:33:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:33:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:33:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:33:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:33:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:33:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:33:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:33:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:33:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:33:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:33:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:33:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:33:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:33:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:33:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:33:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:33:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:33:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:33:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:33:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:33:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:33:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:33:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:33:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:33:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:33:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:33:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:33:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:33:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:33:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:33:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:33:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:33:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:33:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:33:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:33:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:33:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:33:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:33:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:33:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:33:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:33:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:33:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:33:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:33:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:33:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:33:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:33:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:33:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:33:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:33:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:33:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:33:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:33:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:33:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:33:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:33:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:33:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:33:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:33:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:33:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:33:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:33:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:33:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:33:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:33:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:33:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:33:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:33:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:33:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:33:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:33:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:33:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:33:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:33:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:33:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:33:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:33:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:33:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:33:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:33:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:33:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:33:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:33:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:33:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:33:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:33:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:33:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:33:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:33:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:33:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:33:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:33:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:33:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:33:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:33:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:33:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:33:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:33:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:33:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:34:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:34:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:34:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:34:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:34:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:34:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:34:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:34:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:34:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:34:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:34:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:34:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:34:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:34:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:34:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:34:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:34:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:34:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:34:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:34:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:34:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:34:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:34:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:34:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:34:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:34:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:34:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:34:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:34:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:34:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:34:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:34:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:34:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:34:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:34:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:34:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:34:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:34:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:34:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:34:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:34:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:34:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:34:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:34:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:34:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:34:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:34:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:34:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:34:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:34:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:34:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:34:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:34:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:34:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:34:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:34:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:34:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:34:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:34:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:34:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:34:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:34:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:34:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:34:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:34:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:34:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:34:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:34:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:34:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:34:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:34:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:34:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:34:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:34:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:34:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:34:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:34:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:34:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:34:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:34:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:34:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:34:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:34:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:34:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:34:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:34:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:34:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:34:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:34:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:34:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:34:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:34:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:34:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:34:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:34:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:34:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:34:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:34:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:34:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:34:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:34:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:34:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:34:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:34:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:34:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:34:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:34:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:34:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:34:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:34:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:34:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:34:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:34:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:34:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:34:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:34:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:34:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:34:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:34:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:34:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:34:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:34:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:34:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:34:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:34:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:34:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:34:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:34:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:34:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:34:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:34:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:34:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:34:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:34:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:34:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:34:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:34:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:34:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:34:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:34:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:34:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:34:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:34:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:34:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:34:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:34:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:34:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:34:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:34:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:34:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:34:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:34:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:34:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:34:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:34:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:34:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:34:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:34:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:34:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:34:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:34:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:34:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:34:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:34:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:34:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:34:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:34:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:34:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:34:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:34:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:34:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:34:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:34:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:34:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:34:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:34:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:34:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:34:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:34:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:34:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:34:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:34:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:34:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:34:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:34:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:34:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:34:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:34:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:34:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:34:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:34:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:34:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:34:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:34:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:34:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:34:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:34:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:34:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:34:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:34:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:34:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:34:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:34:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:34:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:34:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:34:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:34:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:34:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:34:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:34:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:34:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:34:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:34:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:34:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:34:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:34:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:35:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:35:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:35:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:35:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:35:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:35:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:35:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:35:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:35:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:35:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:35:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:35:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:35:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:35:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:35:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:35:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:35:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:35:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:35:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:35:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:35:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:35:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:35:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:35:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:35:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:35:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:35:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:35:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:35:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:35:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:35:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:35:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:35:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:35:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:35:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:35:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:35:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:35:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:35:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:35:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:35:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:35:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:35:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:35:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:35:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:35:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:35:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:35:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:35:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:35:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:35:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:35:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:35:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:35:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:35:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:35:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:35:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:35:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:35:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:35:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:35:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:35:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:35:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:35:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:35:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:35:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:35:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:35:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:35:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:35:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:35:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:35:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:35:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:35:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:35:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:35:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:35:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:35:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:35:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:35:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:35:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:35:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:35:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:35:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:35:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:35:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:35:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:35:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:35:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:35:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:35:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:35:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:35:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:35:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:35:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:35:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:35:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:35:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:35:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:35:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:35:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:35:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:35:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:35:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:35:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:35:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:35:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:35:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:35:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:35:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:35:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:35:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:35:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:35:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:35:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:35:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:35:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:35:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:35:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:35:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:35:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:35:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:35:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:35:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:35:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:35:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:35:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:35:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:35:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:35:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:35:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:35:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:35:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:35:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:35:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:35:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:35:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:35:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:35:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:35:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:35:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:35:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:35:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:35:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:35:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:35:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:35:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:35:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:35:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:35:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:35:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:35:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:35:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:35:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:35:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:35:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:35:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:35:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:35:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:35:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:35:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:35:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:35:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:35:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:35:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:35:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:35:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:35:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:35:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:35:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:35:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:35:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:35:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:35:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:35:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:35:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:35:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:35:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:35:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:35:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:35:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:35:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:35:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:35:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:35:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:35:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:35:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:35:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:35:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:35:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:35:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:35:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:35:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:35:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:35:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:35:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:35:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:35:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:35:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:35:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:35:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:35:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:35:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:35:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:35:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:35:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:35:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:35:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:35:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:35:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:35:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:35:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:35:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:35:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:35:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:35:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:36:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:36:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:36:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:36:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:36:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:36:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:36:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:36:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:36:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:36:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:36:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:36:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:36:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:36:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:36:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:36:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:36:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:36:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:36:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:36:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:36:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:36:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:36:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:36:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:36:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:36:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:36:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:36:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:36:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:36:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:36:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:36:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:36:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:36:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:36:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:36:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:36:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:36:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:36:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:36:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:36:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:36:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:36:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:36:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:36:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:36:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:36:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:36:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:36:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:36:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:36:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:36:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:36:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:36:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:36:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:36:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:36:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:36:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:36:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:36:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:36:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:36:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:36:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:36:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:36:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:36:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:36:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:36:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:36:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:36:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:36:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:36:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:36:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:36:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:36:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:36:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:36:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:36:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:36:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:36:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:36:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:36:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:36:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:36:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:36:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:36:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:36:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:36:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:36:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:36:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:36:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:36:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:36:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:36:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:36:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:36:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:36:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:36:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:36:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:36:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:36:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:36:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:36:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:36:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:36:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:36:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:36:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:36:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:36:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:36:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:36:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:36:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:36:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:36:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:36:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:36:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:36:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:36:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:36:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:36:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:36:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:36:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:36:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:36:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:36:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:36:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:36:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:36:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:36:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:36:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:36:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:36:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:36:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:36:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:36:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:36:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:36:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:36:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:36:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:36:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:36:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:36:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:36:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:36:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:36:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:36:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:36:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:36:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:36:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:36:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:36:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:36:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:36:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:36:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:36:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:36:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:36:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:36:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:36:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:36:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:36:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:36:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:36:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:36:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:36:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:36:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:36:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:36:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:36:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:36:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:36:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:36:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:36:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:36:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:36:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:36:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:36:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:36:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:36:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:36:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:36:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:36:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:36:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:36:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:36:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:36:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:36:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:36:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:36:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:36:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:36:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:36:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:36:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:36:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:36:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:36:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:36:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:36:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:36:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:36:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:36:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:36:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:36:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:36:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:36:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:36:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:36:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:36:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:36:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:36:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:36:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:36:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:36:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:36:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:36:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:36:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:37:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:37:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:37:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:37:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:37:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:37:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:37:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:37:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:37:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:37:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:37:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:37:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:37:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:37:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:37:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:37:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:37:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:37:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:39:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:39:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:39:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:39:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:39:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:39:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:39:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:39:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:39:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:39:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:39:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:39:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:39:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:39:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:39:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:39:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:39:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:39:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:39:10 rdma06 rsyslogd[1041]: imjournal: 5082 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 19:39:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:39:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:39:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:39:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:39:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:39:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:39:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:39:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:39:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:39:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:39:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:39:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:39:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:39:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:39:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:39:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:39:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:39:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:39:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:39:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:39:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:39:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:39:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:39:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:39:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:39:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:39:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:39:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:39:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:39:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:39:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:39:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:39:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:39:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:39:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:39:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:39:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:39:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:39:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:39:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:39:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:39:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:39:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:39:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:39:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:39:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:39:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:39:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:39:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:39:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:39:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:39:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:39:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:39:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:39:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:39:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:39:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:39:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:39:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:39:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:39:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:39:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:39:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:39:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:39:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:39:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:39:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:39:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:39:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:39:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:39:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:39:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:39:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:39:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:39:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:39:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:39:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:39:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:39:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:39:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:39:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:39:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:39:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:39:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:39:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:39:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:39:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:39:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:39:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:39:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:39:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:39:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:39:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:39:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:39:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:39:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:39:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:39:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:39:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:39:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:39:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:39:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:39:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:39:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:39:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:39:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:39:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:39:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:39:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:39:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:39:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:39:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:39:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:39:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:39:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:39:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:39:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:39:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:39:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:39:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:40:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:40:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:40:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:40:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:40:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:40:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:40:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:40:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:40:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:40:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:40:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:40:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:40:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:40:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:40:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:40:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:40:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:40:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:40:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:40:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:40:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:40:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:40:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:40:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:40:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:40:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:40:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:40:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:40:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:40:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:40:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:40:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:40:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:40:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:40:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:40:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:40:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:40:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:40:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:40:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:40:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:40:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:40:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:40:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:40:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:40:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:40:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:40:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:40:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:40:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:40:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:40:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:40:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:40:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:40:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:40:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:40:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:40:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:40:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:40:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:40:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:40:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:40:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:40:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:40:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:40:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:40:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:40:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:40:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:40:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:40:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:40:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:40:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:40:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:40:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:40:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:40:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:40:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:40:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:40:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:40:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:40:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:40:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:40:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:40:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:40:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:40:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:40:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:40:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:40:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:40:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:40:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:40:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:40:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:40:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:40:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:40:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:40:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:40:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:40:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:40:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:40:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:40:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:40:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:40:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:40:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:40:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:40:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:40:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:40:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:40:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:40:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:40:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:40:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:40:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:40:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:40:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:40:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:40:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:40:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:40:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:40:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:40:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:40:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:40:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:40:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:40:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:40:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:40:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:40:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:40:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:40:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:40:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:40:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:40:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:40:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:40:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:40:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:40:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:40:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:40:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:40:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:40:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:40:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:40:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:40:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:40:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:40:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:40:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:40:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:40:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:40:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:40:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:40:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:40:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:40:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:40:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:40:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:40:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:40:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:40:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:40:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:40:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:40:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:40:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:40:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:40:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:40:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:40:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:40:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:40:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:40:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:40:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:40:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:40:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:40:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:40:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:40:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:40:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:40:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:40:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:40:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:40:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:40:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:40:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:40:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:40:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:40:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:40:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:40:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:40:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:40:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:40:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:40:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:40:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:40:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:40:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:40:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:40:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:40:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:40:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:40:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:40:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:40:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:40:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:40:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:40:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:40:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:40:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:40:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:40:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:40:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:40:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:40:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:40:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:40:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:41:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:41:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:41:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:41:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:41:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:41:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:41:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:41:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:41:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:41:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:41:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:41:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:41:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:41:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:41:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:41:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:41:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:41:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:41:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:41:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:41:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:41:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:41:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:41:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:41:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:41:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:41:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:41:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:41:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:41:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:41:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:41:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:41:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:41:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:41:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:41:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:41:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:41:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:41:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:41:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:41:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:41:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:41:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:41:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:41:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:41:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:41:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:41:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:41:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:41:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:41:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:41:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:41:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:41:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:41:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:41:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:41:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:41:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:41:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:41:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:41:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:41:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:41:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:41:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:41:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:41:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:41:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:41:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:41:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:41:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:41:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:41:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:41:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:41:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:41:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:41:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:41:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:41:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:41:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:41:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:41:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:41:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:41:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:41:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:41:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:41:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:41:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:41:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:41:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:41:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:41:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:41:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:41:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:41:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:41:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:41:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:41:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:41:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:41:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:41:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:41:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:41:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:41:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:41:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:41:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:41:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:41:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:41:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:41:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:41:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:41:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:41:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:41:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:41:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:41:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:41:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:41:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:41:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:41:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:41:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:41:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:41:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:41:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:41:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:41:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:41:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:41:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:41:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:41:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:41:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:41:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:41:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:41:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:41:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:41:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:41:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:41:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:41:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:41:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:41:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:41:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:41:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:41:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:41:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:41:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:41:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:41:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:41:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:41:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:41:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:41:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:41:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:41:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:41:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:41:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:41:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:41:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:41:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:41:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:41:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:41:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:41:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:41:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:41:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:41:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:41:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:41:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:41:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:41:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:41:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:41:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:41:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:41:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:41:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:41:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:41:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:41:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:41:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:41:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:41:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:41:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:41:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:41:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:41:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:41:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:41:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:41:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:41:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:41:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:41:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:41:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:41:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:41:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:41:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:41:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:41:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:41:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:41:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:41:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:41:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:41:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:41:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:41:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:41:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:41:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:41:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:41:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:41:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:41:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:41:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:41:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:41:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:41:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:41:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:41:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:41:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:42:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:42:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:42:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:42:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:42:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:42:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:42:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:42:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:42:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:42:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:42:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:42:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:42:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:42:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:42:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:42:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:42:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:42:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:42:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:42:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:42:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:42:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:42:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:42:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:42:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:42:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:42:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:42:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:42:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:42:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:42:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:42:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:42:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:42:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:42:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:42:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:42:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:42:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:42:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:42:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:42:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:42:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:42:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:42:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:42:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:42:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:42:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:42:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:42:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:42:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:42:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:42:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:42:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:42:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:42:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:42:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:42:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:42:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:42:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:42:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:42:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:42:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:42:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:42:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:42:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:42:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:42:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:42:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:42:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:42:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:42:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:42:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:42:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:42:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:42:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:42:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:42:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:42:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:42:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:42:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:42:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:42:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:42:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:42:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:42:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:42:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:42:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:42:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:42:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:42:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:42:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:42:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:42:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:42:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:42:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:42:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:42:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:42:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:42:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:42:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:42:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:42:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:42:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:42:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:42:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:42:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:42:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:42:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:42:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:42:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:42:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:42:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:42:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:42:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:42:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:42:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:42:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:42:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:42:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:42:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:42:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:42:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:42:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:42:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:42:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:42:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:42:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:42:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:42:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:42:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:42:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:42:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:42:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:42:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:42:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:42:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:42:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:42:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:42:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:42:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:42:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:42:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:42:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:42:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:42:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:42:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:42:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:42:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:42:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:42:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:42:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:42:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:42:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:42:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:42:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:42:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:42:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:42:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:42:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:42:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:42:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:42:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:42:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:42:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:42:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:42:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:42:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:42:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:42:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:42:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:42:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:42:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:42:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:42:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:42:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:42:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:42:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:42:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:42:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:42:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:42:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:42:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:42:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:42:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:42:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:42:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:42:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:42:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:42:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:42:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:42:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:42:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:42:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:42:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:42:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:42:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:42:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:42:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:42:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:42:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:42:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:42:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:42:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:42:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:42:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:42:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:42:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:42:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:42:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:42:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:42:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:42:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:42:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:42:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:42:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:42:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:43:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:43:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:43:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:43:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:43:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:43:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:43:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:43:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:43:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:43:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:43:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:43:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:43:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:43:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:43:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:43:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:43:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:43:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:43:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:43:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:43:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:43:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:43:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:43:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:43:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:43:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:43:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:43:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:43:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:43:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:43:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:43:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:43:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:43:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:43:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:43:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:43:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:43:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:43:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:43:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:43:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:43:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:43:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:43:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:43:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:43:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:43:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:43:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:43:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:43:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:43:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:43:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:43:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:43:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:43:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:43:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:43:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:43:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:43:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:43:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:43:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:43:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:43:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:43:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:43:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:43:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:43:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:43:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:43:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:43:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:43:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:43:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:43:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:43:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:43:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:43:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:43:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:43:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:43:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:43:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:43:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:43:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:43:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:43:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:43:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:43:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:43:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:43:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:43:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:43:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:43:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:43:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:43:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:43:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:43:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:43:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:43:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:43:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:43:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:43:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:43:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:43:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:43:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:43:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:43:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:43:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:43:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:43:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:43:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:43:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:43:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:43:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:43:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:43:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:43:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:43:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:43:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:43:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:43:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:43:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:43:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:43:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:43:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:43:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:43:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:43:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:43:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:43:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:43:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:43:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:43:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:43:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:43:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:43:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:43:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:43:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:43:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:43:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:43:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:43:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:43:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:43:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:43:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:43:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:43:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:43:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:43:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:43:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:43:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:43:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:43:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:43:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:43:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:43:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:43:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:43:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:43:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:43:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:43:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:43:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:43:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:43:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:43:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:43:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:43:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:43:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:43:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:43:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:43:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:43:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:43:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:43:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:43:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:43:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:43:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:43:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:43:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:43:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:43:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:43:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:43:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:43:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:43:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:43:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:43:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:43:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:43:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:43:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:43:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:43:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:43:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:43:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:43:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:43:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:43:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:43:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:43:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:43:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:43:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:43:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:43:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:43:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:43:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:43:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:43:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:43:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:43:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:43:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:43:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:43:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:43:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:43:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:43:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:43:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:43:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:43:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:44:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:44:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:44:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:44:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:44:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:44:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:44:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:44:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:44:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:44:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:44:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:44:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:44:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:44:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:44:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:44:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:44:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:44:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:44:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:44:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:44:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:44:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:44:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:44:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:44:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:44:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:44:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:44:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:44:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:44:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:44:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:44:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:44:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:44:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:44:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:44:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:44:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:44:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:44:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:44:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:44:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:44:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:44:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:44:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:44:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:44:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:44:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:44:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:44:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:44:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:44:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:44:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:44:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:44:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:44:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:44:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:44:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:44:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:44:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:44:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:44:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:44:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:44:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:44:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:44:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:44:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:44:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:44:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:44:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:44:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:44:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:44:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:44:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:44:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:44:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:44:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:44:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:44:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:44:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:44:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:44:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:44:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:44:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:44:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:44:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:44:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:44:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:44:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:44:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:44:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:44:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:44:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:44:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:44:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:44:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:44:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:44:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:44:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:44:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:44:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:44:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:44:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:44:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:44:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:44:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:44:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:44:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:44:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:44:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:44:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:44:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:44:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:44:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:44:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:44:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:44:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:44:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:44:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:44:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:44:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:44:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:44:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:44:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:44:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:44:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:44:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:44:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:44:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:44:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:44:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:44:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:44:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:44:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:44:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:44:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:44:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:44:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:44:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:44:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:44:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:44:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:44:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:44:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:44:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:44:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:44:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:44:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:44:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:44:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:44:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:44:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:44:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:44:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:44:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:44:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:44:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:44:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:44:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:44:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:44:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:44:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:44:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:44:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:44:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:44:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:44:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:44:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:44:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:44:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:44:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:44:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:44:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:44:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:44:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:44:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:44:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:44:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:44:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:44:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:44:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:44:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:44:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:44:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:44:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:44:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:44:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:44:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:44:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:44:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:44:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:44:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:44:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:44:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:44:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:44:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:44:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:44:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:44:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:44:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:44:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:44:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:44:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:44:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:44:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:44:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:44:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:44:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:44:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:44:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:44:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:44:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:44:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:44:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:44:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:44:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:44:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:45:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:45:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:45:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:45:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:45:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:45:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:45:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:45:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:45:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:45:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:45:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:45:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:45:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:45:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:45:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:45:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:45:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:45:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:45:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:45:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:45:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:45:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:45:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:45:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:45:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:45:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:45:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:45:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:45:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:45:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:45:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:45:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:45:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:45:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:45:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:45:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:45:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:45:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:45:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:45:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:45:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:45:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:45:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:45:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:45:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:45:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:45:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:45:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:45:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:45:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:45:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:45:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:45:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:45:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:45:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:45:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:45:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:45:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:45:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:45:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:45:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:45:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:45:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:45:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:45:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:45:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:45:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:45:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:45:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:45:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:45:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:45:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:45:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:45:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:45:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:45:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:45:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:45:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:45:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:45:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:45:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:45:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:45:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:45:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:45:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:45:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:45:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:45:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:45:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:45:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:45:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:45:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:45:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:45:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:45:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:45:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:45:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:45:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:45:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:45:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:45:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:45:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:45:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:45:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:45:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:45:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:45:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:45:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:45:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:45:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:45:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:45:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:45:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:45:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:45:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:45:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:45:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:45:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:45:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:45:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:45:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:45:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:45:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:45:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:45:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:45:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:45:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:45:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:45:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:45:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:45:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:45:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:45:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:45:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:45:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:45:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:45:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:45:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:45:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:45:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:45:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:45:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:45:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:45:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:45:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:45:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:45:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:45:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:45:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:45:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:45:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:45:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:45:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:45:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:45:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:45:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:45:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:45:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:45:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:45:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:45:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:45:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:45:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:45:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:45:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:45:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:45:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:45:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:45:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:45:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:45:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:45:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:45:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:45:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:45:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:45:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:45:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:45:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:45:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:45:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:45:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:45:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:45:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:45:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:45:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:45:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:45:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:45:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:45:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:45:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:45:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:45:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:45:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:45:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:45:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:45:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:45:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:45:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:45:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:45:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:45:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:45:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:45:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:45:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:45:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:45:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:45:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:45:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:45:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:45:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:45:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:45:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:45:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:45:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:45:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:45:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:46:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:46:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:46:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:46:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:46:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:46:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:46:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:46:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:46:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:46:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:46:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:46:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:46:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:46:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:46:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:46:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:46:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:46:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:46:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:46:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:46:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:46:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:46:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:46:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:46:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:46:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:46:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:46:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:46:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:46:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:46:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:46:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:46:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:46:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:46:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:46:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:46:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:46:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:46:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:46:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:46:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:46:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:46:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:46:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:46:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:46:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:46:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:46:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:46:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:46:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:46:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:46:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:46:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:46:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:46:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:46:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:46:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:46:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:46:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:46:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:46:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:46:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:46:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:46:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:46:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:46:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:46:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:46:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:46:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:46:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:46:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:46:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:46:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:46:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:46:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:46:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:46:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:46:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:46:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:46:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:46:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:46:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:46:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:46:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:46:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:46:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:46:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:46:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:46:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:46:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:46:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:46:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:46:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:46:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:46:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:46:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:46:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:46:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:46:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:46:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:46:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:46:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:46:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:46:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:46:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:46:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:46:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:46:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:46:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:46:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:46:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:46:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:46:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:46:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:46:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:46:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:46:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:46:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:46:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:46:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:46:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:46:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:46:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:46:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:46:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:46:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:46:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:46:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:46:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:46:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:46:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:46:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:46:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:46:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:46:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:46:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:46:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:46:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:46:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:46:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:46:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:46:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:46:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:46:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:46:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:46:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:46:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:46:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:46:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:46:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:46:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:46:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:46:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:46:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:46:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:46:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:46:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:46:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:46:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:46:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:46:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:46:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:46:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:46:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:46:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:46:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:46:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:46:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:46:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:46:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:46:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:46:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:46:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:46:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:46:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:46:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:46:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:46:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:46:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:46:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:46:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:46:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:46:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:46:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:46:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:46:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:46:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:46:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:46:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:46:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:46:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:46:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:46:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:46:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:46:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:46:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:46:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:46:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:46:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:46:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:46:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:46:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:46:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:46:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:46:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:46:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:46:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:46:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:46:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:46:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:46:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:46:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:46:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:46:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:46:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:46:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:47:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:47:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:47:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:47:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:47:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:47:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:47:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:47:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:47:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:47:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:47:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:47:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:47:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:47:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:47:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:47:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:47:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:47:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:49:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:49:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:49:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:49:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:49:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:49:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:49:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:49:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:49:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:49:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:49:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:49:15 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 19:49:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:49:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:49:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:49:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:49:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:49:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:49:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:49:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:49:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:49:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:49:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:49:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:49:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:49:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:49:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:49:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:49:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:49:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:49:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:49:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:49:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:49:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:49:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:49:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:49:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:49:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:49:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:49:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:49:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:49:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:49:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:49:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:49:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:49:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:49:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:49:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:49:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:49:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:49:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:49:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:49:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:49:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:49:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:49:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:49:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:49:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:49:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:49:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:49:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:49:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:49:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:49:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:49:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:49:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:49:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:49:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:49:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:49:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:49:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:49:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:49:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:49:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:49:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:49:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:49:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:49:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:49:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:49:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:49:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:49:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:49:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:49:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:49:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:49:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:49:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:49:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:49:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:49:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:49:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:49:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:49:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:49:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:49:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:49:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:49:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:49:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:49:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:49:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:49:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:49:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:49:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:49:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:49:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:49:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:49:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:49:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:49:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:49:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:49:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:49:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:49:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:49:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:49:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:49:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:49:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:49:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:49:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:49:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:49:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:49:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:49:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:49:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:49:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:49:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:49:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:49:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:49:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:49:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:49:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:49:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:49:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:49:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:49:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:49:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:49:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:49:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:49:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:49:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:49:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:49:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:49:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:49:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:49:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:49:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:49:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:49:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:49:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:49:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:49:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:49:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:49:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:49:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:49:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:49:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:50:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:50:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:50:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:50:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:50:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:50:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:50:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:50:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:50:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:50:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:50:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:50:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:50:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:50:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:50:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:50:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:50:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:50:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:50:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:50:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:50:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:50:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:50:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:50:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:50:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:50:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:50:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:50:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:50:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:50:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:50:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:50:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:50:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:50:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:50:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:50:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:50:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:50:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:50:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:50:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:50:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:50:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:50:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:50:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:50:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:50:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:50:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:50:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:50:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:50:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:50:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:50:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:50:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:50:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:50:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:50:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:50:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:50:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:50:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:50:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:50:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:50:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:50:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:50:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:50:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:50:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:50:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:50:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:50:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:50:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:50:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:50:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:50:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:50:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:50:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:50:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:50:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:50:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:50:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:50:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:50:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:50:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:50:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:50:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:50:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:50:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:50:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:50:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:50:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:50:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:50:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:50:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:50:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:50:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:50:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:50:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:50:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:50:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:50:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:50:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:50:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:50:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:50:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:50:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:50:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:50:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:50:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:50:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:50:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:50:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:50:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:50:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:50:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:50:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:50:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:50:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:50:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:50:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:50:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:50:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:50:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:50:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:50:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:50:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:50:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:50:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:50:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:50:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:50:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:50:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:50:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:50:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:50:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:50:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:50:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:50:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:50:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:50:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:50:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:50:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:50:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:50:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:50:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:50:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:50:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:50:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:50:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:50:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:50:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:50:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:50:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:50:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:50:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:50:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:50:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:50:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:50:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:50:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:50:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:50:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:50:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:50:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:50:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:50:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:50:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:50:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:50:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:50:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:50:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:50:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:50:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:50:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:50:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:50:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:50:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:50:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:50:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:50:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:50:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:50:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:50:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:50:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:50:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:50:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:50:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:50:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:50:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:50:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:50:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:50:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:50:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:50:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:50:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:50:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:50:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:50:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:50:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:50:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:50:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:50:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:50:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:50:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:50:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:50:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:50:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:50:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:50:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:50:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:50:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:50:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:50:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:50:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:50:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:50:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:50:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:50:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:51:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:51:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:51:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:51:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:51:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:51:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:51:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:51:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:51:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:51:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:51:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:51:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:51:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:51:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:51:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:51:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:51:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:51:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:51:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:51:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:51:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:51:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:51:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:51:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:51:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:51:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:51:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:51:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:51:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:51:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:51:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:51:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:51:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:51:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:51:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:51:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:51:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:51:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:51:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:51:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:51:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:51:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:51:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:51:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:51:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:51:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:51:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:51:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:51:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:51:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:51:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:51:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:51:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:51:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:51:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:51:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:51:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:51:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:51:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:51:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:51:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:51:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:51:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:51:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:51:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:51:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:51:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:51:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:51:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:51:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:51:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:51:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:51:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:51:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:51:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:51:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:51:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:51:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:51:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:51:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:51:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:51:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:51:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:51:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:51:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:51:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:51:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:51:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:51:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:51:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:51:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:51:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:51:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:51:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:51:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:51:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:51:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:51:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:51:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:51:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:51:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:51:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:51:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:51:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:51:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:51:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:51:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:51:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:51:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:51:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:51:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:51:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:51:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:51:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:51:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:51:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:51:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:51:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:51:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:51:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:51:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:51:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:51:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:51:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:51:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:51:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:51:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:51:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:51:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:51:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:51:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:51:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:51:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:51:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:51:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:51:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:51:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:51:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:51:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:51:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:51:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:51:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:51:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:51:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:51:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:51:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:51:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:51:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:51:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:51:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:51:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:51:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:51:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:51:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:51:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:51:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:51:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:51:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:51:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:51:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:51:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:51:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:51:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:51:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:51:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:51:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:51:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:51:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:51:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:51:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:51:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:51:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:51:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:51:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:51:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:51:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:51:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:51:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:51:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:51:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:51:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:51:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:51:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:51:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:51:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:51:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:52:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:52:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:52:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:52:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:52:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:52:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:52:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:52:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:52:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:52:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:52:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:52:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:52:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:52:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:52:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:52:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:52:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:52:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:52:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:52:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:52:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:52:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:52:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:52:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:52:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:52:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:52:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:52:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:52:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:52:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:52:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:52:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:52:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:52:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:52:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:52:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:52:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:52:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:52:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:52:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:52:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:52:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:52:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:52:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:52:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:52:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:52:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:52:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:52:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:52:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:52:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:52:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:52:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:52:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:52:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:52:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:52:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:52:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:52:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:52:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:52:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:52:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:52:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:52:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:52:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:52:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:52:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:52:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:52:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:52:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:52:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:52:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:52:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:52:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:52:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:52:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:52:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:52:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:52:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:52:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:52:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:52:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:52:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:52:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:52:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:52:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:52:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:52:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:52:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:52:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:52:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:52:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:52:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:52:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:52:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:52:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:52:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:52:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:52:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:52:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:52:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:52:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:52:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:52:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:52:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:52:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:52:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:52:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:52:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:52:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:52:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:52:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:52:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:52:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:52:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:52:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:52:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:52:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:52:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:52:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:52:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:52:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:52:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:52:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:52:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:52:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:52:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:52:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:52:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:52:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:52:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:52:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:52:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:52:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:52:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:52:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:52:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:52:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:52:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:52:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:52:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:52:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:52:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:52:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:52:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:52:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:52:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:52:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:52:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:52:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:52:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:52:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:52:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:52:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:52:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:52:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:52:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:52:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:52:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:52:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:52:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:52:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:52:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:52:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:52:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:52:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:52:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:52:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:52:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:52:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:52:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:52:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:52:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:52:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:52:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:52:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:52:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:52:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:52:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:52:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:52:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:52:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:52:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:52:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:52:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:52:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:52:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:52:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:52:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:52:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:52:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:52:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:52:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:52:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:52:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:52:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:52:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:52:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:52:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:52:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:52:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:52:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:52:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:52:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:52:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:52:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:52:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:52:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:52:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:52:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:52:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:52:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:52:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:52:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:52:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:52:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:53:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:53:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:53:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:53:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:53:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:53:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:53:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:53:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:53:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:53:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:53:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:53:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:53:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:53:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:53:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:53:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:53:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:53:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:53:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:53:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:53:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:53:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:53:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:53:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:53:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:53:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:53:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:53:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:53:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:53:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:53:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:53:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:53:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:53:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:53:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:53:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:53:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:53:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:53:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:53:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:53:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:53:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:53:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:53:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:53:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:53:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:53:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:53:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:53:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:53:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:53:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:53:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:53:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:53:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:53:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:53:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:53:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:53:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:53:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:53:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:53:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:53:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:53:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:53:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:53:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:53:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:53:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:53:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:53:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:53:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:53:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:53:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:53:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:53:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:53:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:53:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:53:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:53:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:53:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:53:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:53:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:53:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:53:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:53:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:53:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:53:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:53:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:53:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:53:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:53:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:53:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:53:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:53:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:53:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:53:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:53:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:53:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:53:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:53:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:53:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:53:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:53:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:53:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:53:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:53:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:53:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:53:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:53:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:53:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:53:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:53:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:53:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:53:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:53:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:53:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:53:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:53:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:53:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:53:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:53:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:53:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:53:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:53:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:53:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:53:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:53:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:53:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:53:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:53:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:53:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:53:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:53:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:53:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:53:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:53:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:53:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:53:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:53:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:53:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:53:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:53:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:53:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:53:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:53:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:53:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:53:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:53:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:53:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:53:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:53:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:53:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:53:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:53:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:53:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:53:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:53:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:53:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:53:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:53:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:53:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:53:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:53:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:53:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:53:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:53:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:53:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:53:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:53:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:53:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:53:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:53:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:53:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:53:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:53:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:53:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:53:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:53:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:53:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:53:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:53:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:53:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:53:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:53:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:53:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:53:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:53:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:53:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:53:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:53:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:53:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:53:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:53:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:53:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:53:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:53:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:53:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:53:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:53:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:53:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:53:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:53:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:53:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:53:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:53:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:53:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:53:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:53:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:53:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:53:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:53:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:53:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:53:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:53:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:53:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:53:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:53:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:54:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:54:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:54:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:54:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:54:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:54:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:54:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:54:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:54:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:54:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:54:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:54:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:54:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:54:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:54:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:54:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:54:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:54:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:54:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:54:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:54:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:54:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:54:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:54:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:54:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:54:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:54:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:54:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:54:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:54:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:54:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:54:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:54:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:54:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:54:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:54:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:54:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:54:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:54:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:54:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:54:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:54:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:54:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:54:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:54:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:54:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:54:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:54:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:54:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:54:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:54:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:54:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:54:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:54:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:54:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:54:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:54:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:54:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:54:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:54:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:54:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:54:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:54:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:54:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:54:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:54:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:54:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:54:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:54:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:54:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:54:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:54:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:54:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:54:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:54:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:54:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:54:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:54:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:54:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:54:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:54:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:54:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:54:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:54:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:54:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:54:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:54:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:54:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:54:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:54:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:54:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:54:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:54:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:54:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:54:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:54:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:54:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:54:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:54:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:54:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:54:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:54:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:54:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:54:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:54:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:54:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:54:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:54:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:54:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:54:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:54:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:54:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:54:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:54:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:54:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:54:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:54:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:54:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:54:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:54:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:54:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:54:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:54:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:54:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:54:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:54:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:54:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:54:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:54:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:54:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:54:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:54:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:54:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:54:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:54:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:54:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:54:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:54:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:54:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:54:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:54:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:54:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:54:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:54:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:54:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:54:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:54:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:54:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:54:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:54:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:54:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:54:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:54:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:54:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:54:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:54:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:54:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:54:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:54:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:54:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:54:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:54:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:54:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:54:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:54:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:54:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:54:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:54:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:54:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:54:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:54:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:54:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:54:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:54:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:54:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:54:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:54:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:54:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:54:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:54:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:54:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:54:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:54:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:54:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:54:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:54:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:54:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:54:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:54:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:54:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:54:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:54:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:54:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:54:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:54:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:54:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:54:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:54:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:54:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:54:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:54:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:54:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:54:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:54:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:54:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:54:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:54:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:54:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:54:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:54:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:54:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:54:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:54:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:54:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:54:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:54:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:55:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:55:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:55:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:55:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:55:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:55:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:55:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:55:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:55:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:55:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:55:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:55:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:55:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:55:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:55:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:55:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:55:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:55:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:55:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:55:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:55:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:55:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:55:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:55:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:55:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:55:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:55:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:55:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:55:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:55:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:55:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:55:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:55:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:55:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:55:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:55:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:55:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:55:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:55:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:55:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:55:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:55:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:55:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:55:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:55:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:55:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:55:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:55:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:55:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:55:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:55:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:55:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:55:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:55:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:55:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:55:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:55:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:55:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:55:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:55:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:55:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:55:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:55:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:55:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:55:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:55:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:55:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:55:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:55:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:55:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:55:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:55:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:55:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:55:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:55:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:55:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:55:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:55:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:55:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:55:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:55:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:55:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:55:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:55:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:55:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:55:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:55:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:55:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:55:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:55:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:55:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:55:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:55:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:55:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:55:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:55:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:55:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:55:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:55:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:55:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:55:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:55:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:55:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:55:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:55:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:55:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:55:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:55:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:55:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:55:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:55:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:55:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:55:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:55:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:55:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:55:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:55:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:55:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:55:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:55:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:55:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:55:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:55:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:55:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:55:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:55:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:55:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:55:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:55:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:55:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:55:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:55:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:55:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:55:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:55:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:55:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:55:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:55:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:55:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:55:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:55:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:55:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:55:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:55:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:55:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:55:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:55:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:55:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:55:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:55:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:55:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:55:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:55:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:55:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:55:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:55:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:55:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:55:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:55:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:55:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:55:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:55:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:55:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:55:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:55:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:55:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:55:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:55:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:55:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:55:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:55:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:55:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:55:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:55:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:55:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:55:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:55:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:55:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:55:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:55:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:55:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:55:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:55:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:55:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:55:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:55:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:55:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:55:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:55:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:55:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:55:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:55:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:55:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:55:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:55:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:55:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:55:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:55:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:55:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:55:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:55:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:55:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:55:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:55:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:55:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:55:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:55:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:55:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:55:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:55:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:55:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:55:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:55:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:55:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:55:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:55:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:56:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:56:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:56:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:56:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:56:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:56:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:56:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:56:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:56:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:56:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:56:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:56:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:56:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:56:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:56:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:56:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:56:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:56:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:56:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:56:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:56:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:56:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:56:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:56:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:56:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:56:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:56:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:56:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:56:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:56:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:56:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:56:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:56:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:56:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:56:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:56:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:56:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:56:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:56:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:56:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:56:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:56:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:56:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:56:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:56:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:56:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:56:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:56:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:56:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:56:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:56:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:56:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:56:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:56:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:56:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:56:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:56:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:56:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:56:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:56:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:56:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:56:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:56:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:56:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:56:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:56:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:56:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:56:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:56:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:56:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:56:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:56:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:56:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:56:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:56:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:56:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:56:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:56:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:56:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:56:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:56:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:56:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:56:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:56:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:56:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:56:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:56:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:56:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:56:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:56:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:56:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:56:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:56:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:56:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:56:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:56:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:56:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:56:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:56:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:56:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:56:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:56:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:56:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:56:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:56:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:56:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:56:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:56:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:56:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:56:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:56:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:56:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:56:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:56:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:56:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:56:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:56:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:56:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:56:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:56:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:56:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:56:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:56:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:56:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:56:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:56:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:56:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:56:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:56:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:56:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:56:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:56:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:56:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:56:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:56:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:56:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:56:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:56:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:56:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:56:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:56:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:56:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:56:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:56:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:56:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:56:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:56:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:56:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:56:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:56:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:56:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:56:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:56:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:56:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:56:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:56:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:56:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:56:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:56:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:56:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:56:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:56:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:56:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:56:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:56:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:56:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:56:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:56:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:56:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:56:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:56:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:56:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:56:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:56:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:56:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:56:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:56:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:56:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:56:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:56:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:56:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:56:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:56:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:56:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:56:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:56:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:56:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:56:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:56:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:56:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:56:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:56:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:56:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:56:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:56:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:56:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:56:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:56:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:56:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:56:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:56:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:56:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:56:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:56:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:56:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:56:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:56:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:56:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:56:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:56:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:56:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:56:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:56:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:56:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:56:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:56:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:57:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:57:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:57:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:57:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:57:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:57:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:57:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:57:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:57:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:57:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:57:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:57:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:57:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:57:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:57:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:57:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:57:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:57:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:57:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:57:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:57:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:57:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:57:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:57:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:57:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:57:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:57:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:57:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:57:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:57:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:57:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:57:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:57:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:57:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:57:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:57:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:57:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:57:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:57:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:57:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:57:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:57:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:57:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:57:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:57:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:57:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:57:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:57:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:57:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:57:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:57:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:57:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:57:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:57:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:59:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:59:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:59:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:59:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:59:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:59:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:59:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:59:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:59:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:59:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:59:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:59:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:59:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:59:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:59:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:59:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:59:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:59:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:59:20 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 19:59:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:59:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:59:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:59:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:59:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:59:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:59:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:59:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:59:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:59:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:59:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:59:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:59:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:59:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:59:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:59:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:59:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:59:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:59:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:59:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:59:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:59:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:59:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:59:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:59:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:59:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:59:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:59:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:59:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:59:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:59:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:59:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:59:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:59:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:59:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:59:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:59:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:59:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:59:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:59:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:59:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:59:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:59:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:59:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:59:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:59:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:59:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:59:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:59:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:59:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:59:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:59:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:59:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:59:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:59:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:59:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:59:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:59:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:59:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:59:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:59:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:59:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:59:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:59:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:59:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:59:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:59:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:59:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:59:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:59:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:59:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:59:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:59:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:59:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:59:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:59:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:59:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:59:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:59:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:59:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:59:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:59:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:59:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:59:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:59:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:59:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:59:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:59:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:59:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:59:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:59:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:59:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:59:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:59:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:59:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:59:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:59:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:59:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:59:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:59:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:59:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:59:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:59:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:59:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:59:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:59:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:59:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:59:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 19:59:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 19:59:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:59:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:59:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:59:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 19:59:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 19:59:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 19:59:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 19:59:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 19:59:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 19:59:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 19:59:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 19:59:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 19:59:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 19:59:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 19:59:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 19:59:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 19:59:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:00:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:00:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:00:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:00:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:00:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:00:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:00:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:00:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:00:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:00:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:00:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:00:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:00:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:00:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:00:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:00:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:00:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:00:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:00:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:00:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:00:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:00:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:00:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:00:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:00:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:00:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:00:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:00:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:00:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:00:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:00:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:00:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:00:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:00:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:00:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:00:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:00:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:00:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:00:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:00:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:00:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:00:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:00:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:00:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:00:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:00:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:00:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:00:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:00:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:00:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:00:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:00:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:00:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:00:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:00:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:00:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:00:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:00:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:00:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:00:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:00:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:00:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:00:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:00:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:00:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:00:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:00:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:00:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:00:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:00:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:00:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:00:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:00:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:00:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:00:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:00:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:00:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:00:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:00:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:00:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:00:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:00:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:00:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:00:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:00:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:00:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:00:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:00:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:00:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:00:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:00:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:00:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:00:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:00:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:00:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:00:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:00:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:00:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:00:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:00:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:00:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:00:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:00:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:00:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:00:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:00:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:00:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:00:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:00:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:00:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:00:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:00:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:00:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:00:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:00:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:00:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:00:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:00:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:00:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:00:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:00:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:00:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:00:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:00:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:00:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:00:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:00:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:00:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:00:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:00:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:00:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:00:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:00:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:00:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:00:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:00:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:00:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:00:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:00:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:00:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:00:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:00:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:00:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:00:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:00:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:00:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:00:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:00:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:00:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:00:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:00:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:00:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:00:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:00:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:00:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:00:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:00:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:00:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:00:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:00:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:00:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:00:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:00:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:00:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:00:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:00:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:00:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:00:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:00:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:00:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:00:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:00:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:00:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:00:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:00:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:00:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:00:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:00:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:00:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:00:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:00:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:00:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:00:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:00:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:00:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:00:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:00:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:00:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:00:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:00:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:00:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:00:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:00:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:00:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:00:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:00:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:00:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:00:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:00:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:00:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:00:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:00:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:00:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:00:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:00:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:00:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:00:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:00:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:00:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:00:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:00:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:00:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:00:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:00:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:00:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:00:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:01:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:01:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:01:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:01:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:01:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:01:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:01:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:01:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:01:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:01:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:01:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:01:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:01:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:01:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:01:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:01:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:01:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:01:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:01:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:01:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:01:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:01:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:01:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:01:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:01:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:01:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:01:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:01:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:01:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:01:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:01:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:01:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:01:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:01:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:01:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:01:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:01:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:01:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:01:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:01:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:01:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:01:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:01:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:01:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:01:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:01:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:01:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:01:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:01:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:01:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:01:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:01:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:01:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:01:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:01:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:01:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:01:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:01:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:01:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:01:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:01:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:01:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:01:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:01:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:01:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:01:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:01:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:01:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:01:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:01:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:01:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:01:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:01:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:01:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:01:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:01:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:01:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:01:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:01:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:01:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:01:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:01:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:01:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:01:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:01:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:01:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:01:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:01:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:01:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:01:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:01:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:01:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:01:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:01:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:01:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:01:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:01:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:01:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:01:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:01:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:01:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:01:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:01:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:01:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:01:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:01:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:01:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:01:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:01:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:01:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:01:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:01:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:01:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:01:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:01:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:01:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:01:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:01:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:01:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:01:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:01:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:01:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:01:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:01:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:01:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:01:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:01:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:01:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:01:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:01:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:01:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:01:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:01:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:01:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:01:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:01:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:01:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:01:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:01:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:01:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:01:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:01:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:01:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:01:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:01:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:01:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:01:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:01:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:01:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:01:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:01:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:01:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:01:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:01:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:01:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:01:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:02:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:02:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:02:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:02:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:02:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:02:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:02:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:02:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:02:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:02:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:02:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:02:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:02:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:02:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:02:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:02:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:02:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:02:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:02:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:02:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:02:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:02:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:02:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:02:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:02:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:02:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:02:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:02:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:02:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:02:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:02:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:02:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:02:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:02:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:02:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:02:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:02:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:02:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:02:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:02:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:02:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:02:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:02:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:02:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:02:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:02:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:02:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:02:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:02:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:02:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:02:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:02:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:02:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:02:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:02:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:02:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:02:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:02:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:02:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:02:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:02:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:02:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:02:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:02:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:02:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:02:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:02:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:02:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:02:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:02:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:02:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:02:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:02:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:02:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:02:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:02:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:02:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:02:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:02:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:02:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:02:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:02:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:02:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:02:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:02:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:02:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:02:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:02:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:02:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:02:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:02:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:02:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:02:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:02:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:02:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:02:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:02:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:02:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:02:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:02:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:02:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:02:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:02:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:02:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:02:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:02:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:02:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:02:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:02:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:02:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:02:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:02:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:02:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:02:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:02:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:02:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:02:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:02:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:02:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:02:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:02:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:02:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:02:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:02:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:02:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:02:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:02:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:02:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:02:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:02:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:02:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:02:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:02:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:02:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:02:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:02:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:02:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:02:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:02:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:02:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:02:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:02:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:02:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:02:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:02:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:02:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:02:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:02:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:02:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:02:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:02:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:02:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:02:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:02:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:02:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:02:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:02:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:02:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:02:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:02:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:02:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:02:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:02:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:02:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:02:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:02:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:02:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:02:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:02:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:02:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:02:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:02:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:02:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:02:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:02:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:02:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:02:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:02:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:02:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:02:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:02:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:02:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:02:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:02:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:02:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:02:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:02:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:02:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:02:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:02:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:02:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:02:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:02:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:02:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:02:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:02:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:02:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:02:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:02:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:02:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:02:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:02:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:02:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:02:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:02:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:02:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:02:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:02:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:02:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:02:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:02:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:02:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:02:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:02:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:02:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:02:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:03:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:03:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:03:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:03:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:03:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:03:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:03:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:03:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:03:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:03:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:03:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:03:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:03:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:03:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:03:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:03:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:03:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:03:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:03:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:03:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:03:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:03:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:03:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:03:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:03:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:03:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:03:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:03:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:03:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:03:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:03:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:03:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:03:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:03:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:03:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:03:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:03:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:03:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:03:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:03:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:03:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:03:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:03:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:03:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:03:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:03:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:03:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:03:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:03:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:03:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:03:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:03:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:03:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:03:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:03:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:03:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:03:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:03:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:03:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:03:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:03:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:03:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:03:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:03:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:03:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:03:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:03:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:03:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:03:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:03:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:03:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:03:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:03:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:03:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:03:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:03:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:03:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:03:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:03:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:03:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:03:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:03:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:03:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:03:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:03:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:03:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:03:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:03:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:03:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:03:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:03:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:03:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:03:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:03:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:03:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:03:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:03:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:03:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:03:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:03:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:03:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:03:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:03:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:03:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:03:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:03:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:03:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:03:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:03:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:03:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:03:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:03:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:03:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:03:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:03:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:03:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:03:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:03:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:03:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:03:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:03:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:03:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:03:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:03:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:03:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:03:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:03:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:03:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:03:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:03:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:03:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:03:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:03:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:03:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:03:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:03:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:03:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:03:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:03:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:03:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:03:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:03:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:03:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:03:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:03:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:03:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:03:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:03:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:03:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:03:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:03:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:03:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:03:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:03:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:03:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:03:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:03:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:03:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:03:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:03:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:03:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:03:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:03:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:03:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:03:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:03:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:03:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:03:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:03:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:03:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:03:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:03:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:03:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:03:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:03:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:03:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:03:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:03:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:03:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:03:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:03:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:03:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:03:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:03:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:03:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:03:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:03:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:03:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:03:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:03:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:03:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:03:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:03:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:03:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:03:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:03:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:03:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:03:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:03:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:03:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:03:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:03:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:03:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:03:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:03:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:03:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:03:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:03:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:03:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:03:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:03:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:03:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:03:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:03:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:03:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:03:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:04:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:04:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:04:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:04:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:04:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:04:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:04:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:04:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:04:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:04:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:04:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:04:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:04:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:04:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:04:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:04:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:04:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:04:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:04:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:04:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:04:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:04:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:04:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:04:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:04:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:04:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:04:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:04:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:04:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:04:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:04:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:04:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:04:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:04:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:04:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:04:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:04:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:04:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:04:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:04:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:04:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:04:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:04:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:04:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:04:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:04:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:04:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:04:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:04:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:04:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:04:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:04:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:04:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:04:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:04:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:04:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:04:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:04:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:04:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:04:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:04:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:04:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:04:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:04:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:04:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:04:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:04:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:04:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:04:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:04:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:04:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:04:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:04:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:04:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:04:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:04:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:04:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:04:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:04:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:04:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:04:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:04:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:04:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:04:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:04:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:04:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:04:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:04:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:04:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:04:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:04:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:04:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:04:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:04:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:04:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:04:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:04:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:04:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:04:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:04:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:04:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:04:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:04:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:04:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:04:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:04:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:04:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:04:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:04:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:04:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:04:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:04:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:04:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:04:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:04:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:04:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:04:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:04:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:04:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:04:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:04:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:04:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:04:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:04:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:04:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:04:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:04:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:04:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:04:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:04:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:04:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:04:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:04:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:04:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:04:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:04:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:04:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:04:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:04:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:04:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:04:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:04:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:04:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:04:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:04:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:04:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:04:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:04:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:04:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:04:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:04:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:04:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:04:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:04:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:04:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:04:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:04:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:04:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:04:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:04:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:04:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:04:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:04:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:04:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:04:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:04:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:04:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:04:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:04:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:04:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:04:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:04:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:04:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:04:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:04:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:04:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:04:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:04:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:04:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:04:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:04:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:04:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:04:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:04:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:04:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:04:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:04:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:04:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:04:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:04:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:04:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:04:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:04:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:04:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:04:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:04:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:04:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:04:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:04:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:04:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:04:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:04:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:04:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:04:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:04:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:04:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:04:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:04:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:04:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:04:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:04:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:04:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:04:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:04:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:04:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:04:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:05:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:05:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:05:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:05:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:05:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:05:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:05:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:05:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:05:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:05:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:05:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:05:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:05:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:05:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:05:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:05:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:05:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:05:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:05:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:05:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:05:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:05:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:05:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:05:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:05:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:05:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:05:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:05:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:05:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:05:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:05:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:05:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:05:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:05:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:05:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:05:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:05:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:05:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:05:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:05:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:05:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:05:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:05:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:05:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:05:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:05:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:05:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:05:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:05:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:05:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:05:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:05:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:05:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:05:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:05:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:05:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:05:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:05:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:05:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:05:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:05:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:05:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:05:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:05:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:05:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:05:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:05:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:05:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:05:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:05:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:05:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:05:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:05:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:05:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:05:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:05:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:05:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:05:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:05:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:05:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:05:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:05:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:05:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:05:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:05:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:05:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:05:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:05:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:05:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:05:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:05:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:05:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:05:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:05:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:05:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:05:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:05:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:05:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:05:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:05:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:05:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:05:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:05:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:05:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:05:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:05:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:05:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:05:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:05:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:05:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:05:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:05:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:05:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:05:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:05:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:05:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:05:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:05:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:05:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:05:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:05:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:05:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:05:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:05:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:05:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:05:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:05:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:05:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:05:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:05:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:05:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:05:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:05:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:05:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:05:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:05:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:05:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:05:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:05:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:05:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:05:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:05:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:05:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:05:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:05:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:05:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:05:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:05:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:05:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:05:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:05:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:05:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:05:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:05:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:05:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:05:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:05:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:05:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:05:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:05:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:05:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:05:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:05:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:05:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:05:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:05:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:05:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:05:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:05:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:05:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:05:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:05:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:05:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:05:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:05:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:05:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:05:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:05:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:05:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:05:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:05:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:05:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:05:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:05:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:05:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:05:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:05:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:05:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:05:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:05:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:05:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:05:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:05:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:05:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:05:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:05:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:05:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:05:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:05:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:05:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:05:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:05:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:05:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:05:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:05:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:05:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:05:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:05:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:05:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:05:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:05:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:05:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:05:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:05:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:05:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:05:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:06:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:06:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:06:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:06:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:06:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:06:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:06:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:06:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:06:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:06:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:06:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:06:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:06:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:06:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:06:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:06:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:06:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:06:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:06:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:06:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:06:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:06:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:06:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:06:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:06:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:06:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:06:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:06:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:06:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:06:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:06:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:06:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:06:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:06:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:06:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:06:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:06:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:06:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:06:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:06:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:06:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:06:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:06:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:06:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:06:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:06:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:06:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:06:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:06:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:06:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:06:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:06:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:06:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:06:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:06:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:06:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:06:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:06:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:06:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:06:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:06:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:06:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:06:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:06:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:06:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:06:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:06:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:06:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:06:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:06:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:06:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:06:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:06:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:06:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:06:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:06:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:06:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:06:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:06:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:06:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:06:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:06:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:06:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:06:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:06:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:06:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:06:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:06:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:06:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:06:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:06:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:06:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:06:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:06:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:06:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:06:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:06:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:06:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:06:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:06:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:06:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:06:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:06:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:06:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:06:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:06:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:06:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:06:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:06:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:06:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:06:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:06:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:06:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:06:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:06:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:06:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:06:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:06:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:06:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:06:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:06:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:06:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:06:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:06:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:06:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:06:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:06:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:06:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:06:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:06:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:06:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:06:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:06:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:06:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:06:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:06:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:06:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:06:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:06:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:06:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:06:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:06:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:06:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:06:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:06:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:06:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:06:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:06:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:06:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:06:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:06:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:06:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:06:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:06:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:06:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:06:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:06:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:06:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:06:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:06:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:06:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:06:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:06:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:06:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:06:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:06:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:06:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:06:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:06:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:06:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:06:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:06:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:06:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:06:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:06:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:06:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:06:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:06:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:06:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:06:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:06:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:06:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:06:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:06:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:06:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:06:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:06:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:06:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:06:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:06:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:06:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:06:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:06:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:06:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:06:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:06:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:06:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:06:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:06:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:06:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:06:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:06:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:06:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:06:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:06:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:06:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:06:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:06:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:06:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:06:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:06:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:06:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:06:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:06:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:06:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:06:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:07:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:07:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:07:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:07:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:07:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:07:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:07:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:07:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:07:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:07:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:07:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:07:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:07:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:07:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:07:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:07:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:07:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:07:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:07:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:07:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:07:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:07:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:07:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:07:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:07:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:07:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:07:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:07:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:07:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:07:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:07:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:07:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:07:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:07:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:07:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:07:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:07:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:07:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:07:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:07:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:07:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:07:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:07:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:07:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:07:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:07:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:07:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:07:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:07:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:07:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:07:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:07:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:07:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:07:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:07:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:07:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:07:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:07:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:07:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:07:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:07:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:07:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:07:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:07:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:07:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:07:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:07:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:07:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:07:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:07:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:07:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:07:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:09:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:09:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:09:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:09:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:09:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:09:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:09:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:09:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:09:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:09:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:09:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:09:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:09:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:09:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:09:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:09:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:09:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:09:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:09:21 rdma06 rsyslogd[1041]: imjournal: 5169 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 20:09:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:09:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:09:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:09:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:09:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:09:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:09:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:09:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:09:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:09:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:09:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:09:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:09:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:09:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:09:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:09:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:09:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:09:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:09:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:09:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:09:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:09:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:09:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:09:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:09:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:09:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:09:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:09:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:09:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:09:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:09:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:09:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:09:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:09:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:09:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:09:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:09:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:09:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:09:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:09:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:09:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:09:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:09:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:09:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:09:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:09:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:09:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:09:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:09:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:09:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:09:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:09:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:09:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:09:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:09:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:09:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:09:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:09:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:09:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:09:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:09:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:09:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:09:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:09:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:09:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:09:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:09:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:09:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:09:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:09:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:09:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:09:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:09:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:09:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:09:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:09:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:09:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:09:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:09:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:09:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:09:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:09:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:09:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:09:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:09:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:09:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:09:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:09:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:09:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:09:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:09:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:09:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:09:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:09:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:09:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:09:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:09:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:09:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:09:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:09:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:09:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:09:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:09:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:09:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:09:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:09:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:09:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:09:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:09:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:09:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:09:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:09:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:09:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:09:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:09:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:09:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:09:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:09:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:09:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:09:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:09:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:09:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:09:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:09:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:09:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:09:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:10:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:10:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:10:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:10:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:10:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:10:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:10:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:10:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:10:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:10:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:10:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:10:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:10:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:10:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:10:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:10:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:10:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:10:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:10:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:10:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:10:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:10:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:10:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:10:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:10:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:10:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:10:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:10:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:10:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:10:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:10:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:10:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:10:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:10:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:10:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:10:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:10:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:10:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:10:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:10:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:10:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:10:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:10:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:10:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:10:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:10:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:10:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:10:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:10:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:10:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:10:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:10:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:10:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:10:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:10:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:10:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:10:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:10:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:10:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:10:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:10:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:10:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:10:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:10:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:10:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:10:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:10:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:10:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:10:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:10:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:10:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:10:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:10:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:10:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:10:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:10:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:10:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:10:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:10:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:10:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:10:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:10:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:10:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:10:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:10:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:10:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:10:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:10:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:10:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:10:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:10:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:10:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:10:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:10:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:10:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:10:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:10:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:10:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:10:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:10:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:10:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:10:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:10:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:10:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:10:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:10:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:10:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:10:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:10:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:10:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:10:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:10:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:10:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:10:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:10:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:10:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:10:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:10:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:10:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:10:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:10:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:10:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:10:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:10:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:10:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:10:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:10:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:10:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:10:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:10:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:10:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:10:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:10:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:10:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:10:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:10:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:10:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:10:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:10:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:10:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:10:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:10:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:10:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:10:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:10:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:10:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:10:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:10:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:10:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:10:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:10:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:10:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:10:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:10:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:10:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:10:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:10:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:10:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:10:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:10:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:10:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:10:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:10:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:10:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:10:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:10:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:10:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:10:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:10:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:10:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:10:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:10:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:10:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:10:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:10:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:10:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:10:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:10:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:10:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:10:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:10:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:10:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:10:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:10:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:10:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:10:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:10:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:10:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:10:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:10:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:10:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:10:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:10:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:10:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:10:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:10:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:10:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:10:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:10:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:10:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:10:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:10:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:10:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:10:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:10:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:10:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:10:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:10:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:10:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:10:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:10:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:10:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:10:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:10:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:10:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:10:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:11:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:11:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:11:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:11:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:11:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:11:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:11:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:11:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:11:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:11:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:11:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:11:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:11:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:11:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:11:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:11:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:11:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:11:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:11:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:11:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:11:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:11:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:11:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:11:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:11:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:11:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:11:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:11:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:11:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:11:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:11:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:11:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:11:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:11:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:11:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:11:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:11:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:11:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:11:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:11:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:11:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:11:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:11:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:11:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:11:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:11:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:11:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:11:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:11:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:11:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:11:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:11:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:11:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:11:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:11:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:11:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:11:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:11:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:11:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:11:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:11:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:11:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:11:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:11:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:11:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:11:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:11:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:11:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:11:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:11:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:11:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:11:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:11:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:11:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:11:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:11:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:11:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:11:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:11:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:11:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:11:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:11:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:11:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:11:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:11:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:11:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:11:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:11:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:11:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:11:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:11:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:11:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:11:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:11:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:11:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:11:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:11:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:11:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:11:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:11:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:11:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:11:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:11:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:11:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:11:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:11:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:11:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:11:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:11:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:11:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:11:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:11:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:11:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:11:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:11:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:11:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:11:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:11:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:11:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:11:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:11:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:11:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:11:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:11:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:11:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:11:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:11:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:11:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:11:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:11:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:11:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:11:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:11:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:11:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:11:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:11:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:11:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:11:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:11:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:11:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:11:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:11:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:11:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:11:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:11:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:11:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:11:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:11:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:11:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:11:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:11:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:11:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:11:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:11:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:11:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:11:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:11:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:11:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:11:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:11:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:11:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:11:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:11:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:11:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:11:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:11:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:11:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:11:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:11:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:11:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:11:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:11:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:11:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:11:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:11:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:11:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:11:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:11:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:11:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:11:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:11:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:11:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:11:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:11:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:11:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:11:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:12:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:12:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:12:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:12:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:12:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:12:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:12:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:12:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:12:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:12:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:12:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:12:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:12:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:12:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:12:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:12:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:12:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:12:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:12:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:12:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:12:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:12:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:12:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:12:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:12:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:12:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:12:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:12:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:12:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:12:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:12:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:12:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:12:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:12:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:12:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:12:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:12:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:12:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:12:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:12:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:12:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:12:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:12:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:12:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:12:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:12:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:12:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:12:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:12:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:12:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:12:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:12:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:12:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:12:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:12:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:12:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:12:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:12:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:12:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:12:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:12:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:12:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:12:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:12:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:12:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:12:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:12:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:12:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:12:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:12:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:12:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:12:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:12:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:12:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:12:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:12:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:12:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:12:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:12:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:12:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:12:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:12:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:12:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:12:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:12:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:12:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:12:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:12:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:12:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:12:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:12:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:12:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:12:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:12:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:12:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:12:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:12:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:12:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:12:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:12:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:12:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:12:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:12:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:12:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:12:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:12:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:12:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:12:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:12:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:12:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:12:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:12:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:12:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:12:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:12:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:12:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:12:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:12:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:12:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:12:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:12:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:12:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:12:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:12:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:12:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:12:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:12:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:12:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:12:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:12:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:12:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:12:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:12:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:12:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:12:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:12:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:12:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:12:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:12:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:12:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:12:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:12:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:12:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:12:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:12:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:12:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:12:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:12:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:12:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:12:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:12:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:12:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:12:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:12:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:12:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:12:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:12:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:12:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:12:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:12:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:12:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:12:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:12:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:12:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:12:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:12:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:12:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:12:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:12:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:12:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:12:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:12:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:12:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:12:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:12:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:12:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:12:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:12:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:12:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:12:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:12:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:12:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:12:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:12:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:12:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:12:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:12:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:12:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:12:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:12:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:12:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:12:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:12:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:12:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:12:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:12:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:12:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:12:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:12:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:12:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:12:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:12:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:12:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:12:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:12:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:12:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:12:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:12:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:12:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:12:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:12:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:12:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:12:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:12:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:12:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:12:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:13:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:13:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:13:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:13:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:13:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:13:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:13:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:13:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:13:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:13:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:13:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:13:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:13:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:13:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:13:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:13:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:13:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:13:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:13:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:13:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:13:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:13:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:13:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:13:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:13:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:13:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:13:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:13:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:13:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:13:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:13:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:13:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:13:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:13:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:13:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:13:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:13:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:13:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:13:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:13:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:13:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:13:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:13:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:13:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:13:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:13:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:13:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:13:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:13:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:13:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:13:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:13:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:13:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:13:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:13:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:13:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:13:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:13:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:13:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:13:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:13:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:13:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:13:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:13:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:13:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:13:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:13:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:13:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:13:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:13:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:13:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:13:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:13:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:13:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:13:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:13:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:13:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:13:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:13:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:13:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:13:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:13:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:13:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:13:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:13:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:13:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:13:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:13:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:13:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:13:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:13:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:13:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:13:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:13:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:13:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:13:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:13:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:13:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:13:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:13:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:13:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:13:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:13:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:13:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:13:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:13:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:13:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:13:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:13:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:13:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:13:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:13:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:13:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:13:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:13:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:13:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:13:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:13:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:13:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:13:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:13:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:13:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:13:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:13:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:13:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:13:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:13:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:13:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:13:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:13:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:13:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:13:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:13:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:13:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:13:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:13:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:13:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:13:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:13:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:13:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:13:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:13:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:13:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:13:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:13:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:13:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:13:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:13:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:13:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:13:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:13:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:13:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:13:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:13:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:13:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:13:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:13:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:13:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:13:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:13:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:13:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:13:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:13:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:13:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:13:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:13:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:13:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:13:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:13:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:13:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:13:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:13:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:13:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:13:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:13:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:13:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:13:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:13:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:13:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:13:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:13:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:13:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:13:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:13:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:13:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:13:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:13:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:13:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:13:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:13:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:13:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:13:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:13:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:13:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:13:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:13:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:13:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:13:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:13:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:13:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:13:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:13:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:13:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:13:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:13:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:13:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:13:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:13:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:13:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:13:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:13:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:13:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:13:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:13:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:13:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:13:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:14:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:14:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:14:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:14:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:14:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:14:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:14:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:14:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:14:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:14:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:14:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:14:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:14:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:14:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:14:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:14:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:14:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:14:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:14:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:14:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:14:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:14:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:14:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:14:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:14:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:14:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:14:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:14:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:14:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:14:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:14:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:14:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:14:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:14:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:14:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:14:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:14:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:14:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:14:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:14:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:14:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:14:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:14:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:14:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:14:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:14:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:14:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:14:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:14:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:14:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:14:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:14:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:14:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:14:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:14:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:14:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:14:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:14:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:14:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:14:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:14:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:14:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:14:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:14:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:14:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:14:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:14:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:14:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:14:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:14:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:14:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:14:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:14:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:14:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:14:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:14:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:14:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:14:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:14:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:14:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:14:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:14:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:14:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:14:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:14:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:14:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:14:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:14:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:14:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:14:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:14:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:14:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:14:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:14:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:14:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:14:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:14:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:14:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:14:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:14:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:14:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:14:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:14:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:14:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:14:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:14:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:14:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:14:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:14:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:14:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:14:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:14:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:14:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:14:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:14:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:14:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:14:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:14:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:14:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:14:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:14:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:14:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:14:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:14:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:14:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:14:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:14:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:14:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:14:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:14:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:14:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:14:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:14:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:14:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:14:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:14:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:14:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:14:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:14:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:14:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:14:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:14:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:14:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:14:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:14:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:14:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:14:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:14:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:14:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:14:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:14:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:14:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:14:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:14:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:14:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:14:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:14:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:14:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:14:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:14:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:14:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:14:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:14:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:14:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:14:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:14:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:14:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:14:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:14:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:14:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:14:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:14:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:14:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:14:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:14:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:14:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:14:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:14:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:14:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:14:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:14:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:14:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:14:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:14:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:14:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:14:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:14:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:14:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:14:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:14:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:14:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:14:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:14:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:14:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:14:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:14:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:14:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:14:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:14:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:14:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:14:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:14:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:14:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:14:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:14:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:14:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:14:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:14:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:14:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:14:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:14:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:14:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:14:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:14:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:14:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:14:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:15:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:15:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:15:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:15:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:15:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:15:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:15:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:15:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:15:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:15:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:15:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:15:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:15:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:15:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:15:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:15:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:15:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:15:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:15:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:15:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:15:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:15:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:15:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:15:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:15:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:15:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:15:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:15:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:15:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:15:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:15:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:15:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:15:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:15:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:15:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:15:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:15:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:15:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:15:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:15:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:15:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:15:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:15:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:15:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:15:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:15:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:15:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:15:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:15:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:15:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:15:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:15:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:15:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:15:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:15:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:15:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:15:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:15:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:15:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:15:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:15:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:15:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:15:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:15:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:15:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:15:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:15:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:15:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:15:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:15:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:15:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:15:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:15:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:15:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:15:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:15:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:15:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:15:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:15:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:15:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:15:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:15:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:15:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:15:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:15:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:15:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:15:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:15:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:15:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:15:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:15:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:15:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:15:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:15:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:15:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:15:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:15:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:15:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:15:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:15:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:15:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:15:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:15:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:15:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:15:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:15:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:15:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:15:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:15:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:15:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:15:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:15:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:15:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:15:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:15:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:15:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:15:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:15:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:15:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:15:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:15:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:15:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:15:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:15:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:15:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:15:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:15:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:15:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:15:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:15:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:15:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:15:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:15:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:15:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:15:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:15:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:15:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:15:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:15:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:15:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:15:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:15:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:15:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:15:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:15:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:15:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:15:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:15:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:15:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:15:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:15:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:15:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:15:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:15:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:15:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:15:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:15:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:15:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:15:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:15:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:15:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:15:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:15:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:15:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:15:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:15:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:15:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:15:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:15:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:15:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:15:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:15:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:15:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:15:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:15:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:15:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:15:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:15:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:15:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:15:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:15:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:15:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:15:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:15:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:15:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:15:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:15:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:15:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:15:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:15:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:15:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:15:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:15:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:15:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:15:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:15:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:15:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:15:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:15:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:15:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:15:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:15:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:15:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:15:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:15:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:15:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:15:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:15:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:15:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:15:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:15:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:15:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:15:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:15:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:15:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:15:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:16:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:16:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:16:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:16:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:16:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:16:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:16:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:16:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:16:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:16:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:16:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:16:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:16:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:16:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:16:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:16:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:16:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:16:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:16:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:16:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:16:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:16:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:16:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:16:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:16:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:16:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:16:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:16:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:16:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:16:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:16:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:16:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:16:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:16:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:16:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:16:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:16:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:16:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:16:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:16:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:16:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:16:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:16:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:16:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:16:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:16:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:16:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:16:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:16:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:16:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:16:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:16:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:16:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:16:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:16:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:16:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:16:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:16:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:16:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:16:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:16:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:16:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:16:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:16:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:16:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:16:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:16:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:16:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:16:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:16:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:16:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:16:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:16:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:16:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:16:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:16:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:16:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:16:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:16:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:16:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:16:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:16:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:16:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:16:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:16:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:16:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:16:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:16:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:16:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:16:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:16:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:16:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:16:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:16:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:16:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:16:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:16:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:16:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:16:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:16:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:16:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:16:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:16:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:16:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:16:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:16:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:16:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:16:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:16:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:16:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:16:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:16:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:16:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:16:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:16:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:16:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:16:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:16:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:16:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:16:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:16:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:16:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:16:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:16:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:16:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:16:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:16:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:16:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:16:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:16:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:16:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:16:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:16:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:16:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:16:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:16:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:16:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:16:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:16:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:16:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:16:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:16:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:16:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:16:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:16:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:16:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:16:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:16:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:16:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:16:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:16:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:16:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:16:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:16:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:16:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:16:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:16:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:16:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:16:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:16:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:16:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:16:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:16:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:16:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:16:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:16:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:16:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:16:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:16:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:16:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:16:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:16:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:16:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:16:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:16:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:16:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:16:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:16:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:16:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:16:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:16:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:16:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:16:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:16:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:16:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:16:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:16:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:16:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:16:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:16:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:16:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:16:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:16:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:16:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:16:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:16:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:16:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:16:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:16:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:16:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:16:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:16:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:16:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:16:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:16:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:16:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:16:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:16:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:16:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:16:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:16:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:16:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:16:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:16:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:16:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:16:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:17:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:17:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:17:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:17:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:17:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:17:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:17:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:17:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:17:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:17:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:17:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:17:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:17:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:17:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:17:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:17:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:17:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:17:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:17:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:17:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:17:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:17:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:17:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:17:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:17:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:17:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:17:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:17:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:17:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:17:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:17:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:17:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:17:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:17:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:17:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:17:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:17:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:17:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:17:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:17:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:17:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:17:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:17:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:17:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:17:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:17:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:17:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:17:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:17:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:17:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:17:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:17:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:17:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:17:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:17:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:17:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:17:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:17:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:17:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:17:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:17:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:17:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:17:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:17:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:17:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:17:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:17:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:17:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:17:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:17:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:17:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:17:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:19:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:19:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:19:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:19:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:19:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:19:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:19:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:19:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:19:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:19:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:19:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:19:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:19:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:19:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:19:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:19:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:19:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:19:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:19:26 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 20:19:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:19:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:19:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:19:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:19:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:19:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:19:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:19:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:19:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:19:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:19:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:19:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:19:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:19:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:19:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:19:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:19:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:19:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:19:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:19:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:19:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:19:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:19:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:19:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:19:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:19:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:19:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:19:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:19:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:19:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:19:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:19:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:19:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:19:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:19:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:19:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:19:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:19:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:19:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:19:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:19:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:19:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:19:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:19:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:19:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:19:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:19:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:19:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:19:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:19:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:19:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:19:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:19:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:19:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:19:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:19:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:19:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:19:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:19:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:19:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:19:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:19:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:19:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:19:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:19:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:19:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:19:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:19:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:19:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:19:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:19:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:19:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:19:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:19:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:19:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:19:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:19:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:19:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:19:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:19:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:19:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:19:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:19:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:19:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:19:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:19:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:19:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:19:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:19:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:19:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:19:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:19:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:19:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:19:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:19:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:19:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:19:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:19:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:19:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:19:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:19:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:19:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:19:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:19:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:19:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:19:56 rdma06 systemd-journald[790]: Data hash table of /run/log/journal/95379d2f723847b68d9e976dee8967a9/system.journal has a fill level at 75.0 (163750 of 218332 items, 75497472 file size, 461 bytes per hash table item), suggesting rotation.
Apr 17 20:19:56 rdma06 systemd-journald[790]: /run/log/journal/95379d2f723847b68d9e976dee8967a9/system.journal: Journal header limits reached or header out-of-date, rotating.
Apr 17 20:19:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:19:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:19:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:19:56 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 17 20:19:56 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 17 20:19:56 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 17 20:20:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:20:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:20:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:20:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:20:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:20:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:20:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:20:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:20:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:20:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:20:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:20:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:20:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:20:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:20:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:20:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:20:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:20:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:20:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:20:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:20:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:20:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:20:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:20:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:20:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:20:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:20:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:20:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:20:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:20:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:20:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:20:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:20:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:20:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:20:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:20:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:20:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:20:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:20:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:20:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:20:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:20:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:20:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:20:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:20:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:20:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:20:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:20:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:20:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:20:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:20:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:20:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:20:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:20:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:20:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:20:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:20:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:20:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:20:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:20:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:20:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:20:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:20:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:20:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:20:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:20:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:20:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:20:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:20:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:20:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:20:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:20:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:20:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:20:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:20:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:20:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:20:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:20:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:20:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:20:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:20:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:20:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:20:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:20:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:20:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:20:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:20:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:20:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:20:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:20:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:20:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:20:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:20:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:20:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:20:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:20:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:20:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:20:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:20:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:20:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:20:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:20:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:20:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:20:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:20:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:20:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:20:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:20:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:20:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:20:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:20:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:20:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:20:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:20:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:20:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:20:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:20:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:20:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:20:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:20:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:20:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:20:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:20:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:20:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:20:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:20:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:20:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:20:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:20:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:20:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:20:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:20:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:20:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:20:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:20:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:20:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:20:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:20:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:20:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:20:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:20:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:20:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:20:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:20:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:20:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:20:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:20:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:20:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:20:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:20:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:20:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:20:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:20:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:20:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:20:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:20:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:20:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:20:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:20:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:20:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:20:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:20:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:20:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:20:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:20:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:20:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:20:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:20:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:20:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:20:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:20:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:20:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:20:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:20:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:20:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:20:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:20:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:20:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:20:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:20:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:20:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:20:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:20:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:20:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:20:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:20:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:20:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:20:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:20:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:20:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:20:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:20:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:20:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:20:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:20:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:20:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:20:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:20:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:20:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:20:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:20:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:20:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:20:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:20:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:20:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:20:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:20:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:20:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:20:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:20:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:20:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:20:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:20:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:20:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:20:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:20:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:21:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:21:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:21:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:21:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:21:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:21:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:21:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:21:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:21:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:21:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:21:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:21:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:21:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:21:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:21:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:21:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:21:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:21:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:21:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:21:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:21:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:21:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:21:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:21:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:21:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:21:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:21:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:21:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:21:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:21:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:21:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:21:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:21:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:21:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:21:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:21:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:21:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:21:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:21:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:21:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:21:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:21:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:21:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:21:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:21:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:21:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:21:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:21:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:21:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:21:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:21:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:21:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:21:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:21:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:21:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:21:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:21:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:21:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:21:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:21:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:21:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:21:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:21:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:21:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:21:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:21:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:21:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:21:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:21:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:21:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:21:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:21:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:21:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:21:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:21:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:21:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:21:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:21:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:21:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:21:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:21:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:21:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:21:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:21:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:21:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:21:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:21:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:21:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:21:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:21:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:21:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:21:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:21:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:21:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:21:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:21:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:21:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:21:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:21:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:21:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:21:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:21:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:21:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:21:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:21:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:21:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:21:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:21:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:21:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:21:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:21:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:21:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:21:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:21:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:21:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:21:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:21:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:21:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:21:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:21:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:21:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:21:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:21:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:21:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:21:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:21:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:21:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:21:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:21:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:21:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:21:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:21:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:21:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:21:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:21:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:21:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:21:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:21:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:21:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:21:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:21:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:21:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:21:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:21:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:21:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:21:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:21:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:21:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:21:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:21:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:21:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:21:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:21:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:21:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:21:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:21:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:21:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:21:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:21:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:21:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:21:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:21:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:21:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:21:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:21:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:21:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:21:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:21:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:21:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:21:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:21:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:21:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:21:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:21:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:21:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:21:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:21:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:21:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:21:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:21:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:21:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:21:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:21:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:21:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:21:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:21:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:21:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:21:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:21:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:21:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:21:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:21:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:21:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:21:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:21:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:21:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:21:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:21:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:21:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:21:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:21:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:21:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:21:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:21:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:21:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:21:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:21:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:21:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:21:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:21:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:21:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:21:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:21:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:21:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:21:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:21:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:22:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:22:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:22:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:22:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:22:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:22:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:22:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:22:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:22:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:22:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:22:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:22:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:22:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:22:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:22:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:22:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:22:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:22:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:22:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:22:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:22:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:22:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:22:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:22:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:22:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:22:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:22:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:22:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:22:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:22:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:22:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:22:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:22:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:22:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:22:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:22:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:22:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:22:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:22:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:22:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:22:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:22:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:22:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:22:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:22:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:22:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:22:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:22:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:22:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:22:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:22:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:22:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:22:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:22:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:22:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:22:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:22:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:22:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:22:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:22:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:22:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:22:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:22:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:22:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:22:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:22:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:22:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:22:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:22:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:22:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:22:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:22:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:22:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:22:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:22:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:22:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:22:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:22:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:22:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:22:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:22:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:22:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:22:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:22:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:22:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:22:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:22:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:22:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:22:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:22:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:22:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:22:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:22:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:22:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:22:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:22:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:22:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:22:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:22:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:22:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:22:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:22:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:22:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:22:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:22:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:22:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:22:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:22:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:22:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:22:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:22:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:22:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:22:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:22:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:22:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:22:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:22:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:22:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:22:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:22:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:22:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:22:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:22:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:22:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:22:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:22:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:22:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:22:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:22:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:22:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:22:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:22:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:22:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:22:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:22:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:22:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:22:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:22:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:22:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:22:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:22:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:22:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:22:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:22:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:22:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:22:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:22:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:22:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:22:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:22:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:22:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:22:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:22:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:22:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:22:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:22:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:22:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:22:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:22:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:22:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:22:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:22:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:22:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:22:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:22:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:22:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:22:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:22:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:22:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:22:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:22:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:22:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:22:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:22:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:22:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:22:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:22:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:22:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:22:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:22:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:22:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:22:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:22:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:22:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:22:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:22:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:22:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:22:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:22:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:22:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:22:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:22:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:22:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:22:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:22:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:22:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:22:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:22:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:22:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:22:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:22:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:22:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:22:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:22:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:22:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:22:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:22:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:22:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:22:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:22:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:22:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:22:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:22:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:22:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:22:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:22:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:23:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:23:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:23:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:23:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:23:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:23:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:23:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:23:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:23:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:23:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:23:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:23:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:23:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:23:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:23:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:23:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:23:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:23:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:23:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:23:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:23:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:23:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:23:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:23:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:23:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:23:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:23:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:23:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:23:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:23:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:23:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:23:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:23:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:23:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:23:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:23:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:23:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:23:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:23:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:23:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:23:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:23:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:23:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:23:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:23:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:23:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:23:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:23:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:23:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:23:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:23:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:23:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:23:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:23:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:23:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:23:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:23:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:23:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:23:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:23:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:23:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:23:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:23:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:23:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:23:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:23:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:23:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:23:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:23:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:23:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:23:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:23:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:23:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:23:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:23:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:23:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:23:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:23:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:23:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:23:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:23:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:23:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:23:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:23:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:23:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:23:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:23:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:23:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:23:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:23:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:23:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:23:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:23:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:23:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:23:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:23:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:23:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:23:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:23:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:23:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:23:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:23:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:23:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:23:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:23:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:23:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:23:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:23:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:23:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:23:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:23:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:23:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:23:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:23:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:23:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:23:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:23:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:23:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:23:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:23:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:23:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:23:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:23:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:23:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:23:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:23:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:23:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:23:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:23:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:23:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:23:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:23:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:23:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:23:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:23:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:23:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:23:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:23:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:23:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:23:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:23:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:23:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:23:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:23:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:23:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:23:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:23:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:23:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:23:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:23:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:23:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:23:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:23:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:23:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:23:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:23:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:23:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:23:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:23:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:23:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:23:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:23:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:23:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:23:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:23:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:23:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:23:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:23:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:23:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:23:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:23:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:23:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:23:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:23:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:23:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:23:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:23:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:23:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:23:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:23:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:23:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:23:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:23:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:23:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:23:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:23:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:23:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:23:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:23:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:23:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:23:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:23:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:23:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:23:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:23:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:23:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:23:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:23:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:23:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:23:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:23:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:23:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:23:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:23:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:23:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:23:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:23:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:23:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:23:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:23:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:23:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:23:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:23:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:23:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:23:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:23:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:24:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:24:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:24:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:24:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:24:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:24:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:24:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:24:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:24:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:24:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:24:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:24:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:24:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:24:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:24:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:24:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:24:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:24:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:24:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:24:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:24:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:24:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:24:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:24:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:24:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:24:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:24:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:24:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:24:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:24:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:24:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:24:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:24:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:24:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:24:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:24:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:24:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:24:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:24:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:24:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:24:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:24:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:24:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:24:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:24:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:24:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:24:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:24:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:24:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:24:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:24:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:24:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:24:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:24:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:24:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:24:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:24:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:24:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:24:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:24:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:24:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:24:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:24:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:24:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:24:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:24:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:24:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:24:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:24:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:24:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:24:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:24:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:24:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:24:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:24:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:24:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:24:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:24:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:24:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:24:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:24:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:24:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:24:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:24:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:24:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:24:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:24:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:24:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:24:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:24:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:24:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:24:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:24:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:24:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:24:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:24:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:24:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:24:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:24:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:24:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:24:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:24:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:24:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:24:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:24:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:24:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:24:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:24:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:24:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:24:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:24:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:24:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:24:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:24:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:24:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:24:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:24:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:24:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:24:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:24:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:24:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:24:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:24:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:24:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:24:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:24:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:24:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:24:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:24:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:24:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:24:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:24:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:24:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:24:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:24:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:24:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:24:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:24:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:24:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:24:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:24:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:24:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:24:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:24:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:24:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:24:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:24:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:24:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:24:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:24:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:24:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:24:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:24:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:24:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:24:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:24:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:24:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:24:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:24:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:24:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:24:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:24:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:24:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:24:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:24:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:24:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:24:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:24:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:24:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:24:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:24:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:24:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:24:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:24:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:24:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:24:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:24:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:24:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:24:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:24:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:24:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:24:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:24:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:24:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:24:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:24:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:24:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:24:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:24:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:24:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:24:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:24:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:24:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:24:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:24:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:24:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:24:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:24:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:24:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:24:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:24:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:24:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:24:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:24:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:24:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:24:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:24:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:24:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:24:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:24:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:24:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:24:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:24:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:24:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:24:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:24:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:25:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:25:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:25:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:25:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:25:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:25:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:25:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:25:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:25:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:25:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:25:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:25:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:25:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:25:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:25:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:25:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:25:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:25:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:25:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:25:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:25:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:25:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:25:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:25:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:25:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:25:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:25:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:25:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:25:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:25:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:25:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:25:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:25:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:25:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:25:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:25:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:25:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:25:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:25:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:25:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:25:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:25:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:25:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:25:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:25:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:25:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:25:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:25:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:25:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:25:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:25:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:25:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:25:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:25:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:25:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:25:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:25:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:25:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:25:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:25:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:25:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:25:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:25:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:25:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:25:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:25:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:25:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:25:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:25:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:25:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:25:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:25:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:25:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:25:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:25:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:25:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:25:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:25:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:25:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:25:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:25:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:25:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:25:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:25:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:25:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:25:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:25:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:25:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:25:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:25:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:25:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:25:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:25:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:25:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:25:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:25:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:25:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:25:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:25:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:25:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:25:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:25:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:25:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:25:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:25:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:25:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:25:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:25:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:25:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:25:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:25:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:25:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:25:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:25:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:25:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:25:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:25:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:25:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:25:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:25:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:25:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:25:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:25:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:25:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:25:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:25:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:25:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:25:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:25:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:25:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:25:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:25:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:25:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:25:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:25:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:25:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:25:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:25:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:25:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:25:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:25:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:25:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:25:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:25:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:25:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:25:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:25:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:25:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:25:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:25:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:25:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:25:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:25:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:25:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:25:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:25:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:25:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:25:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:25:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:25:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:25:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:25:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:25:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:25:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:25:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:25:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:25:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:25:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:25:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:25:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:25:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:25:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:25:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:25:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:25:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:25:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:25:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:25:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:25:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:25:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:25:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:25:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:25:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:25:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:25:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:25:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:25:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:25:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:25:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:25:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:25:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:25:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:25:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:25:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:25:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:25:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:25:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:25:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:25:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:25:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:25:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:25:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:25:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:25:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:25:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:25:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:25:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:25:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:25:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:25:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:25:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:25:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:25:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:25:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:25:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:25:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:26:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:26:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:26:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:26:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:26:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:26:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:26:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:26:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:26:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:26:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:26:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:26:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:26:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:26:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:26:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:26:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:26:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:26:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:26:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:26:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:26:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:26:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:26:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:26:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:26:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:26:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:26:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:26:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:26:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:26:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:26:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:26:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:26:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:26:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:26:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:26:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:26:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:26:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:26:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:26:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:26:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:26:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:26:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:26:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:26:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:26:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:26:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:26:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:26:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:26:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:26:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:26:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:26:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:26:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:26:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:26:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:26:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:26:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:26:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:26:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:26:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:26:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:26:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:26:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:26:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:26:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:26:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:26:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:26:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:26:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:26:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:26:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:26:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:26:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:26:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:26:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:26:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:26:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:26:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:26:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:26:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:26:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:26:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:26:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:26:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:26:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:26:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:26:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:26:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:26:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:26:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:26:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:26:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:26:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:26:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:26:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:26:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:26:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:26:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:26:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:26:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:26:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:26:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:26:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:26:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:26:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:26:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:26:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:26:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:26:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:26:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:26:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:26:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:26:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:26:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:26:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:26:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:26:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:26:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:26:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:26:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:26:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:26:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:26:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:26:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:26:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:26:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:26:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:26:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:26:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:26:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:26:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:26:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:26:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:26:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:26:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:26:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:26:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:26:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:26:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:26:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:26:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:26:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:26:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:26:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:26:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:26:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:26:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:26:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:26:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:26:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:26:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:26:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:26:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:26:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:26:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:26:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:26:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:26:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:26:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:26:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:26:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:26:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:26:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:26:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:26:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:26:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:26:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:26:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:26:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:26:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:26:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:26:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:26:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:26:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:26:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:26:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:26:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:26:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:26:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:26:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:26:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:26:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:26:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:26:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:26:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:26:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:26:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:26:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:26:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:26:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:26:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:26:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:26:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:26:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:26:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:26:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:26:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:26:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:26:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:26:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:26:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:26:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:26:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:26:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:26:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:26:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:26:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:26:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:26:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:26:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:26:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:26:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:26:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:26:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:26:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:27:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:27:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:27:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:27:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:27:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:27:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:27:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:27:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:27:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:27:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:27:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:27:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:27:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:27:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:27:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:27:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:27:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:27:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:27:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:27:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:27:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:27:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:27:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:27:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:27:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:27:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:27:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:27:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:27:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:27:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:27:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:27:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:27:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:27:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:27:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:27:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:27:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:27:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:27:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:27:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:27:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:27:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:27:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:27:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:27:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:27:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:27:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:27:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:27:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:27:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:27:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:27:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:27:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:27:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:27:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:27:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:27:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:27:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:27:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:27:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:27:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:27:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:27:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:27:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:27:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:27:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:27:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:27:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:27:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:27:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:27:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:27:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:27:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:27:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:27:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:27:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:27:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:27:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:27:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:27:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:27:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:27:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:27:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:27:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:27:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:27:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:27:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:27:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:27:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:27:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:29:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:29:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:29:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:29:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:29:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:29:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:29:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:29:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:29:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:29:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:29:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:29:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:29:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:29:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:29:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:29:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:29:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:29:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:29:31 rdma06 rsyslogd[1041]: imjournal: 5296 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 20:29:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:29:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:29:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:29:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:29:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:29:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:29:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:29:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:29:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:29:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:29:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:29:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:29:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:29:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:29:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:29:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:29:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:29:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:29:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:29:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:29:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:29:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:29:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:29:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:29:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:29:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:29:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:29:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:29:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:29:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:29:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:29:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:29:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:29:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:29:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:29:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:29:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:29:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:29:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:29:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:29:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:29:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:29:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:29:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:29:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:29:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:29:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:29:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:29:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:29:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:29:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:29:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:29:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:29:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:29:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:29:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:29:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:29:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:29:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:29:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:29:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:29:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:29:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:29:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:29:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:29:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:29:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:29:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:29:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:29:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:29:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:29:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:29:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:29:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:29:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:29:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:29:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:29:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:29:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:29:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:29:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:29:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:29:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:29:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:29:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:29:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:29:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:29:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:29:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:29:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:30:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:30:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:30:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:30:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:30:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:30:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:30:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:30:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:30:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:30:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:30:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:30:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:30:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:30:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:30:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:30:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:30:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:30:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:30:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:30:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:30:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:30:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:30:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:30:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:30:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:30:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:30:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:30:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:30:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:30:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:30:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:30:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:30:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:30:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:30:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:30:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:30:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:30:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:30:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:30:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:30:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:30:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:30:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:30:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:30:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:30:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:30:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:30:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:30:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:30:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:30:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:30:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:30:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:30:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:30:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:30:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:30:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:30:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:30:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:30:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:30:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:30:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:30:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:30:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:30:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:30:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:30:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:30:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:30:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:30:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:30:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:30:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:30:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:30:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:30:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:30:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:30:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:30:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:30:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:30:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:30:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:30:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:30:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:30:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:30:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:30:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:30:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:30:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:30:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:30:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:30:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:30:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:30:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:30:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:30:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:30:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:30:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:30:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:30:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:30:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:30:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:30:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:30:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:30:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:30:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:30:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:30:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:30:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:30:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:30:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:30:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:30:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:30:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:30:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:30:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:30:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:30:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:30:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:30:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:30:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:30:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:30:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:30:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:30:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:30:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:30:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:30:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:30:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:30:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:30:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:30:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:30:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:30:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:30:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:30:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:30:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:30:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:30:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:30:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:30:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:30:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:30:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:30:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:30:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:30:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:30:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:30:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:30:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:30:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:30:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:30:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:30:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:30:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:30:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:30:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:30:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:30:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:30:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:30:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:30:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:30:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:30:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:30:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:30:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:30:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:30:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:30:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:30:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:30:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:30:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:30:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:30:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:30:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:30:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:30:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:30:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:30:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:30:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:30:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:30:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:30:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:30:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:30:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:30:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:30:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:30:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:30:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:30:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:30:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:30:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:30:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:30:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:30:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:30:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:30:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:30:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:30:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:30:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:30:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:30:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:30:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:30:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:30:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:30:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:30:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:30:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:30:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:30:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:30:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:30:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:30:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:30:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:30:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:30:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:30:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:30:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:31:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:31:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:31:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:31:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:31:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:31:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:31:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:31:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:31:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:31:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:31:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:31:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:31:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:31:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:31:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:31:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:31:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:31:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:31:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:31:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:31:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:31:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:31:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:31:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:31:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:31:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:31:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:31:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:31:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:31:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:31:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:31:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:31:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:31:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:31:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:31:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:31:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:31:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:31:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:31:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:31:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:31:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:31:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:31:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:31:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:31:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:31:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:31:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:31:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:31:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:31:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:31:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:31:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:31:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:31:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:31:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:31:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:31:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:31:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:31:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:31:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:31:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:31:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:31:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:31:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:31:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:31:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:31:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:31:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:31:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:31:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:31:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:31:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:31:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:31:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:31:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:31:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:31:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:31:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:31:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:31:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:31:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:31:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:31:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:31:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:31:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:31:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:31:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:31:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:31:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:31:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:31:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:31:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:31:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:31:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:31:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:31:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:31:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:31:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:31:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:31:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:31:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:31:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:31:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:31:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:31:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:31:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:31:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:31:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:31:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:31:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:31:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:31:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:31:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:31:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:31:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:31:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:31:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:31:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:31:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:31:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:31:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:31:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:31:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:31:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:31:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:31:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:31:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:31:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:31:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:31:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:31:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:31:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:31:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:31:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:31:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:31:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:31:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:31:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:31:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:31:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:31:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:31:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:31:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:31:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:31:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:31:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:31:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:31:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:31:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:31:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:31:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:31:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:31:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:31:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:31:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:31:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:31:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:31:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:31:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:31:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:31:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:31:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:31:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:31:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:31:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:32:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:32:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:32:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:32:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:32:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:32:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:32:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:32:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:32:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:32:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:32:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:32:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:32:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:32:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:32:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:32:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:32:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:32:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:32:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:32:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:32:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:32:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:32:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:32:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:32:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:32:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:32:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:32:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:32:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:32:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:32:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:32:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:32:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:32:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:32:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:32:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:32:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:32:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:32:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:32:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:32:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:32:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:32:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:32:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:32:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:32:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:32:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:32:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:32:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:32:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:32:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:32:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:32:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:32:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:32:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:32:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:32:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:32:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:32:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:32:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:32:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:32:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:32:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:32:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:32:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:32:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:32:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:32:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:32:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:32:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:32:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:32:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:32:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:32:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:32:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:32:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:32:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:32:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:32:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:32:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:32:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:32:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:32:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:32:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:32:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:32:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:32:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:32:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:32:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:32:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:32:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:32:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:32:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:32:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:32:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:32:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:32:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:32:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:32:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:32:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:32:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:32:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:32:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:32:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:32:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:32:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:32:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:32:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:32:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:32:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:32:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:32:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:32:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:32:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:32:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:32:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:32:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:32:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:32:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:32:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:32:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:32:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:32:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:32:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:32:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:32:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:32:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:32:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:32:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:32:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:32:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:32:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:32:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:32:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:32:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:32:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:32:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:32:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:32:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:32:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:32:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:32:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:32:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:32:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:32:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:32:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:32:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:32:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:32:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:32:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:32:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:32:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:32:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:32:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:32:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:32:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:32:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:32:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:32:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:32:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:32:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:32:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:32:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:32:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:32:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:32:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:32:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:32:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:32:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:32:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:32:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:32:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:32:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:32:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:32:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:32:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:32:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:32:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:32:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:32:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:32:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:32:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:32:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:32:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:32:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:32:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:32:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:32:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:32:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:32:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:32:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:32:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:32:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:32:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:32:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:32:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:32:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:32:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:32:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:32:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:32:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:32:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:32:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:32:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:32:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:32:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:32:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:32:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:32:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:32:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:32:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:32:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:32:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:32:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:32:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:32:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:33:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:33:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:33:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:33:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:33:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:33:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:33:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:33:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:33:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:33:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:33:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:33:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:33:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:33:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:33:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:33:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:33:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:33:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:33:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:33:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:33:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:33:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:33:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:33:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:33:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:33:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:33:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:33:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:33:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:33:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:33:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:33:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:33:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:33:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:33:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:33:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:33:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:33:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:33:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:33:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:33:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:33:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:33:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:33:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:33:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:33:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:33:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:33:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:33:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:33:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:33:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:33:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:33:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:33:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:33:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:33:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:33:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:33:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:33:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:33:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:33:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:33:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:33:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:33:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:33:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:33:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:33:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:33:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:33:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:33:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:33:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:33:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:33:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:33:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:33:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:33:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:33:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:33:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:33:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:33:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:33:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:33:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:33:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:33:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:33:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:33:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:33:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:33:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:33:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:33:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:33:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:33:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:33:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:33:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:33:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:33:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:33:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:33:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:33:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:33:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:33:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:33:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:33:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:33:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:33:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:33:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:33:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:33:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:33:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:33:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:33:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:33:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:33:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:33:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:33:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:33:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:33:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:33:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:33:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:33:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:33:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:33:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:33:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:33:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:33:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:33:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:33:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:33:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:33:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:33:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:33:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:33:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:33:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:33:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:33:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:33:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:33:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:33:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:33:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:33:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:33:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:33:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:33:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:33:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:33:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:33:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:33:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:33:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:33:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:33:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:33:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:33:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:33:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:33:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:33:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:33:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:33:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:33:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:33:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:33:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:33:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:33:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:33:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:33:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:33:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:33:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:33:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:33:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:33:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:33:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:33:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:33:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:33:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:33:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:33:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:33:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:33:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:33:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:33:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:33:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:33:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:33:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:33:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:33:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:33:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:33:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:33:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:33:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:33:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:33:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:33:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:33:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:33:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:33:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:33:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:33:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:33:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:33:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:33:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:33:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:33:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:33:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:33:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:33:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:33:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:33:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:33:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:33:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:33:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:33:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:33:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:33:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:33:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:33:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:33:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:33:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:34:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:34:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:34:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:34:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:34:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:34:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:34:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:34:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:34:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:34:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:34:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:34:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:34:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:34:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:34:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:34:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:34:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:34:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:34:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:34:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:34:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:34:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:34:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:34:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:34:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:34:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:34:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:34:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:34:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:34:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:34:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:34:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:34:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:34:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:34:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:34:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:34:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:34:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:34:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:34:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:34:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:34:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:34:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:34:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:34:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:34:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:34:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:34:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:34:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:34:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:34:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:34:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:34:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:34:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:34:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:34:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:34:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:34:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:34:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:34:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:34:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:34:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:34:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:34:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:34:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:34:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:34:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:34:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:34:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:34:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:34:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:34:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:34:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:34:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:34:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:34:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:34:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:34:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:34:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:34:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:34:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:34:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:34:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:34:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:34:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:34:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:34:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:34:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:34:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:34:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:34:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:34:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:34:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:34:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:34:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:34:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:34:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:34:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:34:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:34:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:34:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:34:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:34:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:34:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:34:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:34:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:34:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:34:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:34:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:34:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:34:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:34:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:34:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:34:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:34:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:34:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:34:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:34:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:34:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:34:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:34:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:34:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:34:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:34:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:34:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:34:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:34:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:34:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:34:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:34:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:34:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:34:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:34:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:34:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:34:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:34:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:34:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:34:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:34:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:34:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:34:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:34:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:34:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:34:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:34:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:34:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:34:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:34:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:34:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:34:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:34:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:34:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:34:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:34:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:34:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:34:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:34:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:34:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:34:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:34:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:34:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:34:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:34:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:34:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:34:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:34:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:34:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:34:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:34:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:34:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:34:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:34:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:34:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:34:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:34:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:34:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:34:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:34:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:34:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:34:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:34:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:34:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:34:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:34:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:34:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:34:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:34:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:34:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:34:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:34:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:34:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:34:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:34:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:34:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:34:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:34:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:34:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:34:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:34:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:34:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:34:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:34:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:34:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:34:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:34:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:34:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:34:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:34:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:34:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:34:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:34:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:34:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:34:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:34:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:34:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:34:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:35:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:35:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:35:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:35:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:35:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:35:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:35:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:35:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:35:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:35:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:35:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:35:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:35:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:35:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:35:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:35:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:35:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:35:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:35:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:35:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:35:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:35:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:35:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:35:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:35:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:35:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:35:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:35:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:35:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:35:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:35:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:35:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:35:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:35:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:35:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:35:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:35:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:35:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:35:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:35:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:35:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:35:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:35:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:35:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:35:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:35:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:35:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:35:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:35:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:35:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:35:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:35:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:35:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:35:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:35:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:35:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:35:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:35:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:35:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:35:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:35:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:35:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:35:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:35:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:35:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:35:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:35:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:35:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:35:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:35:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:35:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:35:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:35:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:35:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:35:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:35:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:35:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:35:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:35:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:35:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:35:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:35:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:35:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:35:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:35:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:35:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:35:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:35:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:35:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:35:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:35:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:35:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:35:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:35:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:35:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:35:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:35:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:35:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:35:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:35:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:35:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:35:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:35:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:35:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:35:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:35:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:35:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:35:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:35:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:35:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:35:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:35:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:35:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:35:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:35:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:35:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:35:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:35:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:35:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:35:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:35:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:35:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:35:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:35:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:35:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:35:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:35:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:35:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:35:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:35:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:35:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:35:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:35:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:35:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:35:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:35:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:35:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:35:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:35:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:35:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:35:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:35:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:35:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:35:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:35:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:35:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:35:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:35:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:35:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:35:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:35:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:35:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:35:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:35:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:35:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:35:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:35:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:35:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:35:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:35:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:35:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:35:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:35:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:35:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:35:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:35:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:35:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:35:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:35:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:35:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:35:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:35:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:35:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:35:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:35:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:35:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:35:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:35:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:35:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:35:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:35:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:35:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:35:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:35:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:35:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:35:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:35:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:35:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:35:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:35:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:35:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:35:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:35:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:35:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:35:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:35:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:35:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:35:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:35:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:35:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:35:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:35:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:35:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:35:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:35:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:35:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:35:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:35:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:35:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:35:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:35:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:35:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:35:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:35:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:35:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:35:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:36:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:36:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:36:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:36:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:36:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:36:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:36:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:36:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:36:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:36:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:36:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:36:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:36:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:36:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:36:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:36:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:36:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:36:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:36:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:36:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:36:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:36:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:36:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:36:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:36:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:36:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:36:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:36:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:36:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:36:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:36:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:36:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:36:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:36:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:36:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:36:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:36:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:36:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:36:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:36:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:36:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:36:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:36:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:36:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:36:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:36:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:36:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:36:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:36:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:36:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:36:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:36:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:36:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:36:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:36:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:36:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:36:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:36:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:36:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:36:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:36:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:36:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:36:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:36:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:36:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:36:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:36:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:36:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:36:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:36:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:36:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:36:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:36:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:36:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:36:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:36:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:36:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:36:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:36:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:36:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:36:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:36:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:36:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:36:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:36:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:36:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:36:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:36:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:36:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:36:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:36:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:36:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:36:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:36:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:36:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:36:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:36:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:36:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:36:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:36:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:36:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:36:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:36:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:36:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:36:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:36:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:36:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:36:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:36:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:36:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:36:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:36:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:36:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:36:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:36:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:36:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:36:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:36:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:36:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:36:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:36:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:36:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:36:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:36:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:36:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:36:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:36:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:36:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:36:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:36:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:36:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:36:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:36:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:36:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:36:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:36:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:36:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:36:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:36:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:36:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:36:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:36:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:36:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:36:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:36:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:36:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:36:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:36:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:36:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:36:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:36:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:36:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:36:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:36:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:36:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:36:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:36:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:36:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:36:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:36:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:36:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:36:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:36:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:36:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:36:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:36:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:36:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:36:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:36:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:36:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:36:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:36:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:36:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:36:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:36:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:36:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:36:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:36:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:36:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:36:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:36:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:36:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:36:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:36:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:36:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:36:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:36:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:36:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:36:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:36:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:36:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:36:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:36:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:36:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:36:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:36:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:36:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:36:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:36:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:36:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:36:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:36:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:36:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:36:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:36:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:36:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:36:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:36:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:36:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:36:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:36:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:36:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:36:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:36:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:36:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:36:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:37:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:37:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:37:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:37:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:37:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:37:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:37:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:37:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:37:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:37:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:37:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:37:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:37:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:37:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:37:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:37:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:37:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:37:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:37:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:37:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:37:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:37:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:37:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:37:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:37:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:37:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:37:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:37:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:37:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:37:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:37:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:37:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:37:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:37:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:37:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:37:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:37:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:37:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:37:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:37:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:37:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:37:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:37:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:37:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:37:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:37:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:37:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:37:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:37:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:37:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:37:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:37:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:37:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:37:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:37:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:37:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:37:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:37:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:37:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:37:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:37:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:37:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:37:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:37:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:37:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:37:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:37:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:37:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:37:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:37:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:37:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:37:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:37:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:37:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:37:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:37:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:37:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:37:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:37:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:37:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:37:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:37:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:37:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:37:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:37:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:37:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:37:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:37:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:37:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:37:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:37:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:37:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:37:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:37:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:37:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:37:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:37:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:37:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:37:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:37:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:37:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:37:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:37:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:37:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:37:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:37:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:37:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:37:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:39:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:39:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:39:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:39:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:39:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:39:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:39:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:39:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:39:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:39:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:39:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:39:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:39:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:39:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:39:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:39:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:39:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:39:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:39:32 rdma06 rsyslogd[1041]: imjournal: 5082 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 20:39:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:39:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:39:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:39:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:39:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:39:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:39:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:39:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:39:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:39:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:39:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:39:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:39:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:39:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:39:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:39:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:39:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:39:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:39:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:39:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:39:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:39:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:39:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:39:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:39:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:39:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:39:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:39:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:39:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:39:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:39:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:39:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:39:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:39:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:39:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:39:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:39:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:39:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:39:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:39:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:39:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:39:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:39:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:39:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:39:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:39:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:39:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:39:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:39:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:39:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:39:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:39:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:39:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:39:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:39:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:39:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:39:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:39:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:39:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:39:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:39:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:39:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:39:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:39:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:39:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:39:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:39:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:39:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:39:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:39:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:39:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:39:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:39:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:39:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:39:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:39:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:39:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:39:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:39:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:39:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:39:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:39:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:39:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:39:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:39:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:39:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:39:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:39:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:39:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:39:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:40:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:40:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:40:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:40:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:40:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:40:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:40:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:40:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:40:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:40:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:40:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:40:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:40:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:40:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:40:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:40:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:40:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:40:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:40:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:40:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:40:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:40:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:40:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:40:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:40:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:40:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:40:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:40:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:40:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:40:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:40:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:40:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:40:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:40:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:40:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:40:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:40:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:40:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:40:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:40:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:40:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:40:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:40:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:40:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:40:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:40:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:40:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:40:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:40:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:40:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:40:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:40:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:40:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:40:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:40:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:40:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:40:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:40:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:40:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:40:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:40:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:40:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:40:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:40:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:40:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:40:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:40:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:40:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:40:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:40:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:40:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:40:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:40:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:40:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:40:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:40:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:40:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:40:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:40:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:40:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:40:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:40:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:40:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:40:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:40:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:40:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:40:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:40:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:40:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:40:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:40:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:40:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:40:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:40:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:40:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:40:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:40:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:40:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:40:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:40:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:40:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:40:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:40:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:40:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:40:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:40:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:40:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:40:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:40:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:40:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:40:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:40:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:40:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:40:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:40:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:40:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:40:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:40:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:40:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:40:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:40:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:40:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:40:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:40:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:40:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:40:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:40:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:40:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:40:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:40:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:40:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:40:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:40:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:40:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:40:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:40:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:40:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:40:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:40:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:40:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:40:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:40:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:40:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:40:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:40:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:40:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:40:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:40:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:40:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:40:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:40:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:40:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:40:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:40:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:40:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:40:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:40:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:40:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:40:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:40:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:40:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:40:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:40:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:40:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:40:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:40:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:40:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:40:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:40:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:40:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:40:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:40:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:40:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:40:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:40:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:40:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:40:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:40:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:40:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:40:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:40:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:40:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:40:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:40:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:40:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:40:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:40:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:40:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:40:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:40:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:40:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:40:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:40:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:40:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:40:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:40:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:40:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:40:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:40:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:40:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:40:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:40:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:40:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:40:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:40:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:40:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:40:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:40:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:40:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:40:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:40:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:40:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:40:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:40:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:40:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:40:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:41:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:41:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:41:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:41:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:41:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:41:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:41:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:41:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:41:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:41:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:41:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:41:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:41:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:41:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:41:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:41:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:41:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:41:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:41:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:41:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:41:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:41:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:41:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:41:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:41:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:41:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:41:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:41:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:41:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:41:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:41:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:41:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:41:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:41:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:41:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:41:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:41:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:41:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:41:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:41:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:41:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:41:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:41:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:41:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:41:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:41:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:41:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:41:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:41:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:41:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:41:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:41:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:41:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:41:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:41:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:41:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:41:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:41:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:41:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:41:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:41:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:41:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:41:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:41:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:41:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:41:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:41:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:41:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:41:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:41:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:41:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:41:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:41:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:41:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:41:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:41:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:41:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:41:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:41:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:41:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:41:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:41:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:41:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:41:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:41:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:41:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:41:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:41:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:41:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:41:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:41:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:41:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:41:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:41:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:41:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:41:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:41:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:41:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:41:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:41:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:41:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:41:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:41:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:41:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:41:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:41:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:41:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:41:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:41:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:41:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:41:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:41:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:41:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:41:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:41:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:41:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:41:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:41:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:41:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:41:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:41:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:41:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:41:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:41:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:41:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:41:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:41:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:41:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:41:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:41:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:41:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:41:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:41:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:41:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:41:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:41:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:41:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:41:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:41:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:41:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:41:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:41:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:41:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:41:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:41:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:41:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:41:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:41:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:41:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:41:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:41:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:41:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:41:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:41:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:41:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:41:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:41:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:41:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:41:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:41:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:41:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:41:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:41:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:41:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:41:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:41:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:41:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:41:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:41:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:41:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:41:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:41:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:41:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:41:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:41:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:41:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:41:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:41:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:41:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:41:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:41:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:41:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:41:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:41:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:41:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:41:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:41:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:41:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:41:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:41:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:41:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:41:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:41:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:41:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:41:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:41:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:41:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:41:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:41:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:41:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:41:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:41:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:41:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:41:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:41:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:41:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:41:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:41:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:41:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:41:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:41:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:41:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:41:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:41:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:41:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:41:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:42:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:42:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:42:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:42:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:42:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:42:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:42:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:42:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:42:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:42:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:42:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:42:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:42:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:42:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:42:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:42:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:42:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:42:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:42:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:42:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:42:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:42:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:42:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:42:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:42:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:42:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:42:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:42:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:42:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:42:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:42:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:42:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:42:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:42:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:42:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:42:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:42:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:42:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:42:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:42:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:42:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:42:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:42:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:42:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:42:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:42:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:42:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:42:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:42:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:42:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:42:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:42:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:42:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:42:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:42:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:42:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:42:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:42:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:42:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:42:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:42:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:42:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:42:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:42:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:42:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:42:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:42:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:42:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:42:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:42:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:42:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:42:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:42:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:42:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:42:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:42:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:42:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:42:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:42:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:42:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:42:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:42:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:42:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:42:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:42:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:42:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:42:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:42:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:42:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:42:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:42:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:42:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:42:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:42:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:42:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:42:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:42:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:42:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:42:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:42:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:42:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:42:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:42:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:42:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:42:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:42:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:42:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:42:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:42:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:42:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:42:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:42:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:42:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:42:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:42:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:42:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:42:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:42:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:42:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:42:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:42:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:42:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:42:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:42:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:42:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:42:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:42:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:42:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:42:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:42:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:42:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:42:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:42:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:42:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:42:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:42:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:42:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:42:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:42:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:42:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:42:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:42:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:42:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:42:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:42:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:42:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:42:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:42:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:42:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:42:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:42:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:42:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:42:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:42:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:42:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:42:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:42:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:42:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:42:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:42:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:42:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:42:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:42:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:42:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:42:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:42:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:42:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:42:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:42:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:42:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:42:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:42:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:42:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:42:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:42:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:42:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:42:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:42:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:42:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:42:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:42:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:42:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:42:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:42:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:42:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:42:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:42:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:42:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:42:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:42:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:42:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:42:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:42:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:42:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:42:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:42:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:42:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:42:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:42:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:42:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:42:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:42:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:42:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:42:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:42:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:42:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:42:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:42:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:42:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:42:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:42:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:42:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:42:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:42:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:42:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:42:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:43:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:43:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:43:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:43:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:43:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:43:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:43:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:43:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:43:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:43:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:43:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:43:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:43:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:43:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:43:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:43:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:43:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:43:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:43:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:43:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:43:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:43:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:43:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:43:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:43:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:43:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:43:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:43:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:43:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:43:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:43:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:43:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:43:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:43:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:43:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:43:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:43:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:43:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:43:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:43:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:43:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:43:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:43:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:43:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:43:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:43:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:43:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:43:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:43:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:43:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:43:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:43:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:43:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:43:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:43:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:43:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:43:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:43:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:43:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:43:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:43:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:43:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:43:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:43:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:43:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:43:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:43:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:43:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:43:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:43:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:43:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:43:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:43:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:43:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:43:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:43:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:43:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:43:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:43:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:43:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:43:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:43:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:43:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:43:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:43:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:43:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:43:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:43:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:43:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:43:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:43:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:43:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:43:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:43:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:43:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:43:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:43:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:43:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:43:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:43:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:43:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:43:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:43:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:43:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:43:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:43:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:43:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:43:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:43:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:43:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:43:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:43:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:43:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:43:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:43:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:43:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:43:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:43:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:43:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:43:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:43:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:43:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:43:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:43:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:43:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:43:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:43:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:43:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:43:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:43:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:43:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:43:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:43:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:43:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:43:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:43:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:43:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:43:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:43:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:43:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:43:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:43:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:43:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:43:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:43:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:43:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:43:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:43:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:43:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:43:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:43:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:43:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:43:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:43:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:43:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:43:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:43:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:43:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:43:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:43:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:43:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:43:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:43:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:43:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:43:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:43:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:43:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:43:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:43:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:43:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:43:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:43:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:43:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:43:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:43:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:43:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:43:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:43:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:43:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:43:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:43:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:43:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:43:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:43:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:43:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:43:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:43:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:43:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:43:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:43:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:43:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:43:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:43:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:43:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:43:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:43:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:43:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:43:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:43:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:43:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:43:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:43:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:43:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:43:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:43:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:43:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:43:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:43:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:43:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:43:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:43:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:43:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:43:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:43:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:43:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:43:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:44:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:44:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:44:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:44:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:44:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:44:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:44:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:44:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:44:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:44:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:44:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:44:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:44:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:44:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:44:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:44:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:44:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:44:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:44:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:44:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:44:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:44:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:44:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:44:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:44:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:44:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:44:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:44:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:44:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:44:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:44:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:44:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:44:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:44:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:44:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:44:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:44:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:44:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:44:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:44:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:44:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:44:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:44:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:44:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:44:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:44:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:44:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:44:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:44:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:44:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:44:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:44:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:44:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:44:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:44:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:44:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:44:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:44:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:44:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:44:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:44:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:44:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:44:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:44:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:44:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:44:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:44:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:44:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:44:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:44:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:44:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:44:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:44:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:44:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:44:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:44:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:44:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:44:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:44:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:44:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:44:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:44:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:44:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:44:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:44:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:44:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:44:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:44:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:44:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:44:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:44:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:44:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:44:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:44:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:44:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:44:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:44:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:44:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:44:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:44:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:44:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:44:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:44:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:44:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:44:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:44:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:44:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:44:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:44:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:44:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:44:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:44:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:44:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:44:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:44:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:44:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:44:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:44:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:44:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:44:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:44:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:44:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:44:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:44:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:44:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:44:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:44:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:44:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:44:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:44:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:44:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:44:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:44:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:44:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:44:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:44:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:44:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:44:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:44:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:44:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:44:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:44:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:44:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:44:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:44:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:44:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:44:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:44:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:44:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:44:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:44:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:44:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:44:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:44:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:44:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:44:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:44:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:44:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:44:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:44:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:44:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:44:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:44:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:44:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:44:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:44:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:44:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:44:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:44:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:44:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:44:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:44:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:44:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:44:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:44:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:44:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:44:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:44:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:44:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:44:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:44:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:44:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:44:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:44:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:44:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:44:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:44:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:44:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:44:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:44:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:44:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:44:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:44:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:44:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:44:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:44:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:44:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:44:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:44:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:44:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:44:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:44:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:44:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:44:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:44:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:44:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:44:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:44:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:44:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:44:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:44:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:44:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:44:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:44:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:44:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:44:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:45:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:45:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:45:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:45:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:45:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:45:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:45:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:45:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:45:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:45:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:45:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:45:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:45:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:45:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:45:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:45:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:45:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:45:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:45:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:45:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:45:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:45:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:45:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:45:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:45:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:45:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:45:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:45:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:45:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:45:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:45:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:45:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:45:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:45:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:45:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:45:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:45:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:45:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:45:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:45:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:45:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:45:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:45:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:45:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:45:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:45:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:45:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:45:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:45:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:45:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:45:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:45:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:45:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:45:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:45:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:45:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:45:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:45:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:45:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:45:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:45:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:45:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:45:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:45:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:45:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:45:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:45:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:45:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:45:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:45:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:45:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:45:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:45:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:45:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:45:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:45:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:45:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:45:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:45:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:45:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:45:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:45:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:45:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:45:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:45:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:45:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:45:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:45:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:45:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:45:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:45:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:45:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:45:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:45:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:45:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:45:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:45:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:45:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:45:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:45:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:45:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:45:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:45:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:45:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:45:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:45:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:45:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:45:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:45:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:45:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:45:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:45:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:45:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:45:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:45:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:45:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:45:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:45:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:45:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:45:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:45:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:45:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:45:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:45:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:45:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:45:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:45:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:45:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:45:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:45:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:45:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:45:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:45:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:45:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:45:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:45:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:45:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:45:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:45:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:45:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:45:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:45:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:45:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:45:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:45:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:45:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:45:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:45:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:45:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:45:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:45:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:45:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:45:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:45:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:45:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:45:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:45:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:45:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:45:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:45:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:45:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:45:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:45:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:45:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:45:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:45:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:45:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:45:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:45:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:45:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:45:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:45:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:45:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:45:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:45:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:45:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:45:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:45:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:45:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:45:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:45:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:45:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:45:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:45:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:45:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:45:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:45:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:45:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:45:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:45:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:45:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:45:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:45:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:45:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:45:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:45:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:45:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:45:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:45:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:45:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:45:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:45:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:45:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:45:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:45:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:45:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:45:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:45:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:45:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:45:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:45:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:45:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:45:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:45:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:45:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:45:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:46:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:46:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:46:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:46:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:46:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:46:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:46:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:46:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:46:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:46:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:46:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:46:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:46:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:46:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:46:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:46:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:46:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:46:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:46:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:46:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:46:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:46:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:46:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:46:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:46:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:46:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:46:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:46:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:46:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:46:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:46:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:46:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:46:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:46:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:46:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:46:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:46:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:46:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:46:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:46:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:46:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:46:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:46:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:46:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:46:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:46:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:46:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:46:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:46:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:46:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:46:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:46:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:46:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:46:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:46:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:46:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:46:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:46:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:46:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:46:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:46:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:46:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:46:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:46:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:46:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:46:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:46:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:46:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:46:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:46:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:46:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:46:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:46:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:46:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:46:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:46:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:46:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:46:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:46:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:46:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:46:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:46:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:46:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:46:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:46:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:46:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:46:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:46:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:46:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:46:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:46:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:46:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:46:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:46:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:46:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:46:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:46:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:46:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:46:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:46:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:46:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:46:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:46:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:46:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:46:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:46:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:46:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:46:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:46:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:46:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:46:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:46:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:46:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:46:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:46:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:46:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:46:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:46:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:46:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:46:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:46:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:46:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:46:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:46:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:46:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:46:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:46:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:46:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:46:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:46:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:46:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:46:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:46:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:46:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:46:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:46:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:46:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:46:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:46:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:46:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:46:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:46:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:46:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:46:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:46:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:46:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:46:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:46:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:46:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:46:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:46:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:46:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:46:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:46:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:46:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:46:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:46:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:46:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:46:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:46:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:46:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:46:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:46:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:46:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:46:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:46:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:46:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:46:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:46:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:46:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:46:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:46:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:46:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:46:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:46:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:46:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:46:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:46:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:46:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:46:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:46:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:46:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:46:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:46:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:46:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:46:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:46:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:46:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:46:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:46:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:46:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:46:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:46:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:46:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:46:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:46:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:46:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:46:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:46:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:46:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:46:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:46:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:46:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:46:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:46:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:46:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:46:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:46:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:46:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:46:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:46:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:46:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:46:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:46:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:46:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:46:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:47:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:47:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:47:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:47:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:47:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:47:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:47:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:47:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:47:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:47:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:47:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:47:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:47:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:47:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:47:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:47:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:47:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:47:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:47:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:47:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:47:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:47:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:47:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:47:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:47:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:47:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:47:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:47:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:47:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:47:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:47:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:47:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:47:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:47:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:47:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:47:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:47:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:47:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:47:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:47:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:47:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:47:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:47:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:47:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:47:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:47:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:47:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:47:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:47:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:47:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:47:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:47:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:47:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:47:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:47:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:47:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:47:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:47:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:47:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:47:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:47:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:47:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:47:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:47:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:47:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:47:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:47:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:47:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:47:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:47:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:47:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:47:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:47:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:47:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:47:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:47:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:47:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:47:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:47:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:47:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:47:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:47:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:47:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:47:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:47:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:47:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:47:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:47:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:47:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:47:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:47:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:47:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:47:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:47:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:47:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:47:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:47:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:47:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:47:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:47:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:47:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:47:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:47:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:47:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:47:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:47:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:47:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:47:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:49:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:49:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:49:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:49:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:49:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:49:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:49:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:49:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:49:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:49:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:49:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:49:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:49:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:49:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:49:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:49:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:49:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:49:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:49:37 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 20:49:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:49:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:49:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:49:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:49:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:49:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:49:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:49:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:49:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:49:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:49:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:49:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:49:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:49:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:49:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:49:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:49:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:49:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:49:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:49:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:49:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:49:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:49:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:49:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:49:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:49:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:49:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:49:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:49:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:49:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:49:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:49:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:49:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:49:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:49:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:49:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:49:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:49:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:49:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:49:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:49:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:49:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:49:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:49:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:49:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:49:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:49:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:49:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:49:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:49:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:49:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:49:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:49:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:49:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:49:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:49:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:49:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:49:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:49:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:49:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:49:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:49:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:49:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:49:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:49:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:49:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:49:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:49:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:49:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:49:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:49:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:49:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:50:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:50:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:50:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:50:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:50:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:50:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:50:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:50:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:50:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:50:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:50:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:50:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:50:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:50:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:50:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:50:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:50:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:50:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:50:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:50:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:50:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:50:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:50:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:50:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:50:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:50:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:50:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:50:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:50:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:50:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:50:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:50:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:50:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:50:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:50:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:50:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:50:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:50:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:50:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:50:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:50:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:50:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:50:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:50:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:50:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:50:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:50:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:50:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:50:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:50:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:50:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:50:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:50:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:50:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:50:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:50:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:50:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:50:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:50:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:50:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:50:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:50:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:50:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:50:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:50:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:50:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:50:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:50:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:50:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:50:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:50:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:50:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:50:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:50:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:50:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:50:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:50:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:50:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:50:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:50:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:50:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:50:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:50:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:50:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:50:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:50:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:50:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:50:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:50:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:50:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:50:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:50:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:50:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:50:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:50:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:50:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:50:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:50:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:50:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:50:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:50:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:50:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:50:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:50:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:50:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:50:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:50:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:50:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:50:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:50:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:50:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:50:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:50:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:50:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:50:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:50:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:50:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:50:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:50:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:50:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:50:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:50:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:50:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:50:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:50:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:50:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:50:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:50:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:50:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:50:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:50:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:50:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:50:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:50:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:50:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:50:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:50:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:50:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:50:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:50:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:50:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:50:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:50:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:50:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:50:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:50:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:50:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:50:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:50:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:50:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:50:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:50:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:50:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:50:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:50:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:50:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:50:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:50:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:50:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:50:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:50:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:50:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:50:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:50:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:50:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:50:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:50:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:50:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:50:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:50:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:50:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:50:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:50:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:50:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:50:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:50:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:50:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:50:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:50:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:50:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:50:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:50:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:50:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:50:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:50:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:50:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:50:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:50:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:50:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:50:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:50:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:50:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:50:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:50:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:50:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:50:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:50:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:50:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:50:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:50:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:50:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:50:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:50:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:50:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:50:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:50:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:50:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:50:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:50:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:50:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:50:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:50:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:50:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:50:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:50:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:50:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:51:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:51:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:51:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:51:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:51:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:51:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:51:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:51:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:51:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:51:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:51:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:51:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:51:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:51:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:51:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:51:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:51:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:51:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:51:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:51:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:51:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:51:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:51:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:51:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:51:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:51:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:51:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:51:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:51:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:51:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:51:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:51:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:51:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:51:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:51:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:51:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:51:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:51:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:51:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:51:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:51:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:51:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:51:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:51:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:51:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:51:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:51:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:51:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:51:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:51:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:51:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:51:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:51:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:51:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:51:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:51:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:51:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:51:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:51:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:51:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:51:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:51:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:51:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:51:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:51:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:51:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:51:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:51:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:51:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:51:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:51:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:51:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:51:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:51:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:51:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:51:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:51:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:51:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:51:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:51:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:51:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:51:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:51:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:51:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:51:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:51:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:51:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:51:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:51:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:51:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:51:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:51:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:51:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:51:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:51:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:51:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:51:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:51:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:51:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:51:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:51:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:51:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:51:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:51:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:51:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:51:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:51:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:51:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:51:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:51:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:51:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:51:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:51:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:51:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:51:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:51:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:51:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:51:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:51:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:51:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:51:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:51:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:51:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:51:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:51:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:51:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:51:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:51:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:51:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:51:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:51:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:51:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:51:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:51:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:51:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:51:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:51:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:51:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:51:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:51:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:51:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:51:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:51:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:51:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:51:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:51:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:51:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:51:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:51:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:51:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:51:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:51:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:51:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:51:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:51:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:51:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:51:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:51:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:51:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:51:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:51:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:51:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:51:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:51:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:51:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:51:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:51:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:51:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:51:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:51:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:51:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:51:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:51:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:51:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:51:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:51:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:51:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:51:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:51:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:51:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:51:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:51:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:51:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:51:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:51:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:51:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:51:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:51:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:51:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:51:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:51:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:51:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:51:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:51:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:51:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:51:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:51:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:51:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:51:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:51:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:51:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:51:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:51:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:51:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:51:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:51:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:51:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:51:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:51:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:51:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:51:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:51:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:51:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:51:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:51:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:51:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:52:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:52:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:52:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:52:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:52:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:52:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:52:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:52:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:52:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:52:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:52:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:52:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:52:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:52:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:52:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:52:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:52:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:52:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:52:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:52:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:52:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:52:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:52:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:52:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:52:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:52:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:52:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:52:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:52:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:52:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:52:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:52:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:52:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:52:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:52:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:52:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:52:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:52:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:52:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:52:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:52:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:52:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:52:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:52:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:52:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:52:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:52:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:52:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:52:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:52:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:52:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:52:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:52:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:52:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:52:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:52:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:52:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:52:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:52:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:52:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:52:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:52:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:52:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:52:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:52:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:52:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:52:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:52:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:52:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:52:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:52:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:52:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:52:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:52:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:52:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:52:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:52:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:52:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:52:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:52:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:52:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:52:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:52:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:52:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:52:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:52:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:52:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:52:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:52:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:52:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:52:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:52:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:52:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:52:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:52:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:52:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:52:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:52:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:52:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:52:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:52:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:52:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:52:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:52:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:52:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:52:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:52:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:52:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:52:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:52:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:52:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:52:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:52:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:52:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:52:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:52:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:52:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:52:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:52:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:52:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:52:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:52:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:52:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:52:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:52:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:52:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:52:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:52:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:52:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:52:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:52:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:52:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:52:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:52:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:52:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:52:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:52:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:52:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:52:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:52:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:52:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:52:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:52:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:52:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:52:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:52:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:52:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:52:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:52:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:52:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:52:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:52:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:52:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:52:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:52:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:52:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:52:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:52:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:52:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:52:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:52:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:52:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:52:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:52:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:52:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:52:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:52:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:52:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:52:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:52:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:52:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:52:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:52:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:52:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:52:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:52:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:52:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:52:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:52:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:52:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:52:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:52:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:52:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:52:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:52:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:52:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:52:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:52:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:52:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:52:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:52:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:52:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:52:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:52:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:52:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:52:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:52:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:52:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:52:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:52:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:52:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:52:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:52:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:52:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:52:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:52:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:52:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:52:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:52:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:52:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:52:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:52:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:52:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:52:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:52:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:52:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:53:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:53:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:53:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:53:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:53:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:53:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:53:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:53:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:53:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:53:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:53:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:53:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:53:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:53:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:53:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:53:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:53:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:53:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:53:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:53:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:53:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:53:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:53:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:53:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:53:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:53:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:53:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:53:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:53:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:53:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:53:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:53:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:53:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:53:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:53:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:53:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:53:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:53:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:53:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:53:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:53:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:53:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:53:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:53:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:53:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:53:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:53:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:53:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:53:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:53:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:53:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:53:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:53:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:53:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:53:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:53:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:53:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:53:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:53:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:53:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:53:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:53:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:53:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:53:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:53:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:53:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:53:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:53:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:53:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:53:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:53:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:53:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:53:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:53:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:53:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:53:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:53:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:53:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:53:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:53:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:53:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:53:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:53:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:53:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:53:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:53:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:53:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:53:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:53:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:53:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:53:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:53:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:53:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:53:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:53:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:53:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:53:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:53:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:53:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:53:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:53:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:53:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:53:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:53:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:53:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:53:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:53:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:53:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:53:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:53:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:53:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:53:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:53:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:53:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:53:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:53:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:53:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:53:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:53:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:53:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:53:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:53:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:53:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:53:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:53:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:53:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:53:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:53:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:53:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:53:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:53:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:53:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:53:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:53:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:53:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:53:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:53:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:53:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:53:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:53:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:53:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:53:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:53:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:53:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:53:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:53:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:53:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:53:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:53:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:53:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:53:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:53:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:53:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:53:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:53:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:53:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:53:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:53:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:53:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:53:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:53:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:53:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:53:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:53:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:53:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:53:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:53:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:53:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:53:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:53:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:53:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:53:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:53:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:53:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:53:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:53:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:53:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:53:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:53:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:53:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:53:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:53:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:53:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:53:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:53:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:53:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:53:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:53:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:53:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:53:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:53:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:53:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:53:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:53:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:53:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:53:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:53:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:53:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:53:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:53:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:53:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:53:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:53:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:53:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:53:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:53:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:53:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:53:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:53:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:53:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:53:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:53:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:53:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:53:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:53:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:53:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:54:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:54:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:54:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:54:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:54:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:54:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:54:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:54:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:54:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:54:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:54:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:54:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:54:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:54:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:54:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:54:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:54:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:54:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:54:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:54:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:54:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:54:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:54:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:54:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:54:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:54:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:54:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:54:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:54:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:54:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:54:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:54:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:54:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:54:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:54:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:54:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:54:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:54:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:54:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:54:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:54:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:54:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:54:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:54:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:54:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:54:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:54:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:54:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:54:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:54:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:54:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:54:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:54:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:54:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:54:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:54:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:54:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:54:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:54:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:54:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:54:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:54:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:54:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:54:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:54:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:54:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:54:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:54:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:54:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:54:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:54:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:54:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:54:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:54:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:54:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:54:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:54:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:54:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:54:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:54:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:54:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:54:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:54:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:54:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:54:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:54:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:54:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:54:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:54:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:54:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:54:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:54:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:54:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:54:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:54:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:54:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:54:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:54:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:54:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:54:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:54:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:54:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:54:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:54:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:54:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:54:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:54:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:54:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:54:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:54:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:54:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:54:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:54:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:54:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:54:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:54:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:54:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:54:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:54:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:54:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:54:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:54:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:54:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:54:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:54:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:54:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:54:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:54:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:54:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:54:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:54:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:54:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:54:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:54:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:54:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:54:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:54:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:54:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:54:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:54:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:54:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:54:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:54:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:54:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:54:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:54:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:54:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:54:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:54:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:54:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:54:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:54:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:54:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:54:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:54:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:54:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:54:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:54:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:54:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:54:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:54:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:54:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:54:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:54:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:54:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:54:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:54:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:54:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:54:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:54:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:54:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:54:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:54:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:54:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:54:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:54:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:54:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:54:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:54:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:54:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:54:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:54:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:54:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:54:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:54:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:54:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:54:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:54:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:54:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:54:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:54:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:54:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:54:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:54:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:54:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:54:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:54:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:54:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:54:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:54:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:54:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:54:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:54:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:54:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:54:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:54:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:54:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:54:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:54:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:54:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:54:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:54:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:54:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:54:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:54:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:54:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:55:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:55:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:55:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:55:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:55:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:55:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:55:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:55:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:55:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:55:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:55:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:55:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:55:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:55:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:55:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:55:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:55:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:55:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:55:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:55:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:55:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:55:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:55:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:55:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:55:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:55:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:55:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:55:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:55:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:55:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:55:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:55:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:55:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:55:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:55:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:55:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:55:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:55:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:55:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:55:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:55:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:55:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:55:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:55:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:55:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:55:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:55:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:55:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:55:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:55:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:55:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:55:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:55:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:55:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:55:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:55:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:55:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:55:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:55:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:55:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:55:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:55:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:55:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:55:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:55:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:55:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:55:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:55:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:55:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:55:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:55:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:55:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:55:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:55:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:55:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:55:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:55:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:55:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:55:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:55:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:55:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:55:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:55:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:55:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:55:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:55:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:55:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:55:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:55:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:55:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:55:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:55:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:55:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:55:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:55:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:55:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:55:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:55:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:55:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:55:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:55:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:55:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:55:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:55:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:55:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:55:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:55:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:55:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:55:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:55:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:55:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:55:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:55:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:55:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:55:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:55:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:55:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:55:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:55:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:55:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:55:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:55:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:55:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:55:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:55:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:55:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:55:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:55:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:55:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:55:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:55:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:55:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:55:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:55:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:55:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:55:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:55:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:55:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:55:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:55:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:55:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:55:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:55:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:55:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:55:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:55:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:55:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:55:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:55:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:55:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:55:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:55:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:55:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:55:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:55:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:55:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:55:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:55:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:55:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:55:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:55:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:55:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:55:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:55:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:55:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:55:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:55:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:55:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:55:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:55:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:55:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:55:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:55:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:55:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:55:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:55:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:55:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:55:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:55:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:55:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:55:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:55:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:55:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:55:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:55:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:55:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:55:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:55:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:55:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:55:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:55:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:55:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:55:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:55:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:55:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:55:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:55:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:55:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:55:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:55:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:55:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:55:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:55:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:55:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:55:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:55:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:55:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:55:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:55:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:55:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:55:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:55:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:55:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:55:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:55:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:55:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:56:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:56:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:56:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:56:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:56:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:56:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:56:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:56:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:56:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:56:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:56:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:56:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:56:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:56:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:56:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:56:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:56:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:56:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:56:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:56:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:56:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:56:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:56:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:56:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:56:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:56:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:56:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:56:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:56:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:56:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:56:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:56:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:56:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:56:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:56:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:56:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:56:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:56:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:56:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:56:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:56:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:56:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:56:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:56:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:56:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:56:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:56:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:56:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:56:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:56:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:56:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:56:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:56:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:56:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:56:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:56:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:56:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:56:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:56:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:56:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:56:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:56:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:56:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:56:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:56:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:56:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:56:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:56:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:56:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:56:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:56:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:56:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:56:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:56:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:56:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:56:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:56:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:56:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:56:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:56:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:56:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:56:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:56:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:56:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:56:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:56:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:56:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:56:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:56:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:56:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:56:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:56:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:56:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:56:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:56:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:56:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:56:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:56:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:56:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:56:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:56:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:56:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:56:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:56:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:56:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:56:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:56:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:56:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:56:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:56:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:56:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:56:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:56:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:56:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:56:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:56:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:56:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:56:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:56:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:56:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:56:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:56:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:56:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:56:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:56:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:56:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:56:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:56:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:56:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:56:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:56:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:56:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:56:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:56:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:56:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:56:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:56:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:56:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:56:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:56:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:56:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:56:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:56:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:56:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:56:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:56:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:56:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:56:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:56:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:56:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:56:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:56:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:56:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:56:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:56:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:56:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:56:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:56:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:56:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:56:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:56:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:56:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:56:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:56:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:56:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:56:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:56:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:56:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:56:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:56:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:56:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:56:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:56:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:56:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:56:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:56:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:56:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:56:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:56:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:56:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:56:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:56:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:56:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:56:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:56:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:56:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:56:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:56:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:56:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:56:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:56:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:56:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:56:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:56:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:56:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:56:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:56:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:56:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:56:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:56:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:56:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:56:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:56:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:56:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:56:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:56:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:56:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:56:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:56:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:56:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:56:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:56:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:56:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:56:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:56:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:56:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:57:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:57:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:57:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:57:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:57:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:57:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:57:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:57:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:57:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:57:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:57:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:57:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:57:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:57:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:57:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:57:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:57:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:57:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:57:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:57:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:57:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:57:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:57:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:57:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:57:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:57:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:57:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:57:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:57:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:57:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:57:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:57:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:57:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:57:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:57:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:57:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:57:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:57:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:57:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:57:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:57:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:57:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:57:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:57:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:57:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:57:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:57:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:57:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:57:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:57:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:57:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:57:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:57:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:57:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:57:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:57:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:57:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:57:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:57:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:57:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:57:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:57:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:57:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:57:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:57:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:57:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:57:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:57:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:57:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:57:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:57:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:57:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:57:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:57:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:57:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:57:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:57:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:57:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:57:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:57:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:57:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:57:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:57:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:57:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:57:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:57:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:57:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:57:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:57:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:57:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:57:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:57:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:57:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:57:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:57:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:57:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:57:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:57:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:57:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:57:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:57:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:57:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:57:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:57:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:57:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:57:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:57:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:57:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:57:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:57:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:57:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:57:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:57:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:57:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:57:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:57:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:57:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:57:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:57:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:57:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:57:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:57:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:57:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:57:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:57:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:57:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:59:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:59:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:59:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:59:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:59:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:59:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:59:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:59:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:59:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:59:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:59:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:59:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:59:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:59:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:59:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:59:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:59:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:59:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:59:42 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 20:59:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:59:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:59:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:59:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:59:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:59:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:59:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:59:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:59:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:59:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:59:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:59:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:59:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:59:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:59:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:59:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:59:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:59:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:59:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:59:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:59:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:59:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:59:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:59:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:59:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:59:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:59:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:59:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:59:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:59:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:59:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:59:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:59:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:59:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:59:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:59:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 20:59:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 20:59:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:59:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:59:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:59:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 20:59:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 20:59:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 20:59:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 20:59:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 20:59:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 20:59:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 20:59:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 20:59:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 20:59:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 20:59:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 20:59:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 20:59:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 20:59:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:00:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:00:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:00:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:00:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:00:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:00:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:00:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:00:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:00:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:00:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:00:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:00:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:00:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:00:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:00:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:00:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:00:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:00:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:00:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:00:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:00:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:00:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:00:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:00:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:00:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:00:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:00:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:00:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:00:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:00:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:00:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:00:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:00:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:00:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:00:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:00:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:00:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:00:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:00:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:00:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:00:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:00:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:00:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:00:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:00:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:00:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:00:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:00:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:00:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:00:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:00:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:00:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:00:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:00:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:00:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:00:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:00:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:00:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:00:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:00:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:00:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:00:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:00:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:00:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:00:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:00:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:00:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:00:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:00:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:00:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:00:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:00:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:00:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:00:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:00:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:00:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:00:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:00:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:00:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:00:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:00:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:00:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:00:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:00:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:00:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:00:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:00:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:00:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:00:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:00:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:00:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:00:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:00:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:00:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:00:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:00:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:00:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:00:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:00:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:00:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:00:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:00:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:00:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:00:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:00:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:00:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:00:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:00:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:00:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:00:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:00:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:00:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:00:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:00:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:00:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:00:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:00:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:00:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:00:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:00:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:00:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:00:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:00:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:00:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:00:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:00:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:00:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:00:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:00:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:00:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:00:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:00:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:00:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:00:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:00:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:00:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:00:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:00:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:00:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:00:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:00:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:00:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:00:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:00:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:00:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:00:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:00:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:00:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:00:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:00:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:00:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:00:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:00:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:00:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:00:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:00:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:00:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:00:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:00:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:00:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:00:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:00:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:00:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:00:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:00:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:00:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:00:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:00:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:00:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:00:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:00:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:00:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:00:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:00:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:00:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:00:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:00:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:00:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:00:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:00:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:00:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:00:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:00:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:00:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:00:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:00:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:00:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:00:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:00:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:00:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:00:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:00:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:00:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:00:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:00:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:00:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:00:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:00:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:00:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:00:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:00:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:00:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:00:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:00:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:00:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:00:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:00:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:00:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:00:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:00:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:00:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:00:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:00:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:00:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:00:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:00:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:01:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:01:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:01:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:01:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:01:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:01:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:01:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:01:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:01:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:01:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:01:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:01:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:01:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:01:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:01:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:01:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:01:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:01:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:01:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:01:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:01:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:01:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:01:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:01:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:01:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:01:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:01:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:01:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:01:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:01:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:01:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:01:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:01:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:01:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:01:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:01:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:01:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:01:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:01:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:01:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:01:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:01:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:01:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:01:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:01:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:01:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:01:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:01:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:01:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:01:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:01:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:01:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:01:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:01:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:01:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:01:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:01:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:01:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:01:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:01:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:01:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:01:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:01:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:01:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:01:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:01:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:01:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:01:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:01:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:01:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:01:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:01:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:01:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:01:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:01:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:01:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:01:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:01:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:01:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:01:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:01:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:01:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:01:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:01:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:01:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:01:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:01:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:01:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:01:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:01:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:01:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:01:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:01:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:01:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:01:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:01:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:01:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:01:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:01:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:01:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:01:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:01:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:01:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:01:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:01:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:01:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:01:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:01:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:01:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:01:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:01:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:01:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:01:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:01:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:01:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:01:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:01:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:01:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:01:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:01:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:01:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:01:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:01:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:01:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:01:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:01:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:01:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:01:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:01:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:01:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:01:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:01:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:01:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:01:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:01:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:01:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:01:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:01:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:01:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:01:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:01:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:01:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:01:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:01:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:01:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:01:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:01:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:01:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:01:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:01:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:01:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:01:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:01:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:01:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:01:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:01:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:01:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:01:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:01:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:01:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:01:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:01:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:01:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:01:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:01:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:01:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:01:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:01:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:01:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:01:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:01:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:01:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:01:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:01:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:01:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:01:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:01:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:01:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:01:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:01:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:01:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:01:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:01:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:01:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:01:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:01:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:01:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:01:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:01:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:01:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:01:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:01:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:01:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:01:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:01:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:01:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:01:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:01:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:01:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:01:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:01:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:01:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:01:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:01:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:01:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:01:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:01:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:01:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:01:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:01:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:01:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:01:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:01:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:01:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:01:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:01:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:02:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:02:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:02:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:02:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:02:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:02:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:02:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:02:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:02:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:02:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:02:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:02:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:02:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:02:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:02:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:02:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:02:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:02:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:02:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:02:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:02:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:02:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:02:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:02:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:02:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:02:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:02:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:02:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:02:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:02:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:02:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:02:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:02:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:02:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:02:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:02:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:02:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:02:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:02:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:02:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:02:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:02:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:02:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:02:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:02:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:02:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:02:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:02:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:02:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:02:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:02:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:02:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:02:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:02:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:02:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:02:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:02:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:02:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:02:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:02:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:02:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:02:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:02:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:02:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:02:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:02:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:02:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:02:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:02:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:02:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:02:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:02:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:02:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:02:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:02:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:02:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:02:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:02:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:02:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:02:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:02:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:02:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:02:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:02:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:02:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:02:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:02:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:02:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:02:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:02:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:02:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:02:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:02:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:02:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:02:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:02:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:02:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:02:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:02:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:02:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:02:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:02:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:02:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:02:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:02:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:02:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:02:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:02:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:02:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:02:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:02:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:02:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:02:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:02:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:02:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:02:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:02:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:02:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:02:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:02:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:02:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:02:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:02:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:02:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:02:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:02:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:02:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:02:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:02:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:02:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:02:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:02:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:02:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:02:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:02:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:02:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:02:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:02:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:02:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:02:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:02:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:02:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:02:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:02:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:02:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:02:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:02:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:02:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:02:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:02:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:02:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:02:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:02:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:02:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:02:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:02:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:02:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:02:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:02:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:02:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:02:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:02:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:02:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:02:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:02:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:02:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:02:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:02:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:02:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:02:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:02:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:02:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:02:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:02:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:02:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:02:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:02:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:02:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:02:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:02:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:02:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:02:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:02:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:02:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:02:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:02:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:02:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:02:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:02:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:02:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:02:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:02:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:02:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:02:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:02:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:02:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:02:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:02:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:02:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:02:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:02:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:02:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:02:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:02:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:02:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:02:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:02:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:02:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:02:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:02:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:02:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:02:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:02:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:02:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:02:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:02:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:03:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:03:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:03:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:03:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:03:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:03:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:03:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:03:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:03:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:03:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:03:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:03:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:03:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:03:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:03:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:03:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:03:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:03:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:03:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:03:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:03:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:03:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:03:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:03:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:03:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:03:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:03:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:03:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:03:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:03:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:03:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:03:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:03:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:03:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:03:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:03:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:03:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:03:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:03:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:03:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:03:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:03:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:03:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:03:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:03:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:03:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:03:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:03:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:03:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:03:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:03:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:03:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:03:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:03:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:03:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:03:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:03:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:03:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:03:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:03:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:03:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:03:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:03:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:03:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:03:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:03:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:03:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:03:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:03:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:03:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:03:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:03:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:03:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:03:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:03:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:03:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:03:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:03:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:03:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:03:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:03:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:03:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:03:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:03:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:03:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:03:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:03:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:03:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:03:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:03:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:03:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:03:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:03:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:03:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:03:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:03:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:03:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:03:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:03:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:03:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:03:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:03:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:03:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:03:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:03:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:03:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:03:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:03:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:03:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:03:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:03:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:03:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:03:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:03:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:03:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:03:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:03:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:03:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:03:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:03:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:03:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:03:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:03:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:03:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:03:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:03:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:03:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:03:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:03:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:03:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:03:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:03:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:03:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:03:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:03:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:03:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:03:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:03:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:03:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:03:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:03:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:03:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:03:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:03:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:03:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:03:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:03:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:03:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:03:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:03:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:03:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:03:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:03:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:03:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:03:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:03:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:03:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:03:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:03:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:03:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:03:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:03:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:03:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:03:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:03:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:03:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:03:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:03:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:03:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:03:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:03:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:03:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:03:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:03:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:03:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:03:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:03:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:03:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:03:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:03:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:03:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:03:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:03:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:03:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:03:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:03:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:03:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:03:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:03:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:03:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:03:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:03:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:03:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:03:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:03:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:03:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:03:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:03:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:03:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:03:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:03:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:03:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:03:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:03:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:03:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:03:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:03:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:03:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:03:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:03:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:03:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:03:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:03:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:03:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:03:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:03:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:04:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:04:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:04:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:04:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:04:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:04:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:04:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:04:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:04:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:04:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:04:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:04:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:04:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:04:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:04:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:04:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:04:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:04:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:04:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:04:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:04:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:04:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:04:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:04:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:04:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:04:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:04:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:04:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:04:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:04:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:04:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:04:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:04:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:04:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:04:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:04:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:04:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:04:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:04:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:04:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:04:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:04:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:04:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:04:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:04:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:04:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:04:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:04:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:04:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:04:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:04:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:04:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:04:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:04:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:04:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:04:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:04:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:04:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:04:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:04:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:04:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:04:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:04:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:04:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:04:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:04:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:04:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:04:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:04:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:04:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:04:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:04:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:04:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:04:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:04:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:04:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:04:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:04:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:04:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:04:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:04:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:04:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:04:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:04:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:04:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:04:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:04:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:04:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:04:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:04:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:04:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:04:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:04:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:04:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:04:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:04:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:04:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:04:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:04:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:04:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:04:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:04:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:04:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:04:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:04:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:04:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:04:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:04:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:04:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:04:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:04:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:04:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:04:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:04:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:04:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:04:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:04:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:04:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:04:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:04:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:04:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:04:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:04:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:04:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:04:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:04:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:04:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:04:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:04:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:04:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:04:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:04:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:04:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:04:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:04:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:04:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:04:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:04:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:04:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:04:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:04:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:04:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:04:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:04:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:04:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:04:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:04:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:04:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:04:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:04:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:04:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:04:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:04:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:04:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:04:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:04:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:04:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:04:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:04:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:04:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:04:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:04:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:04:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:04:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:04:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:04:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:04:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:04:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:04:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:04:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:04:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:04:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:04:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:04:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:04:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:04:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:04:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:04:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:04:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:04:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:04:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:04:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:04:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:04:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:04:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:04:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:04:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:04:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:04:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:04:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:04:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:04:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:04:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:04:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:04:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:04:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:04:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:04:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:04:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:04:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:04:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:04:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:04:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:04:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:04:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:04:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:04:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:04:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:04:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:04:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:04:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:04:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:04:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:04:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:04:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:04:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:05:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:05:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:05:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:05:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:05:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:05:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:05:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:05:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:05:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:05:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:05:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:05:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:05:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:05:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:05:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:05:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:05:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:05:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:05:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:05:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:05:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:05:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:05:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:05:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:05:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:05:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:05:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:05:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:05:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:05:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:05:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:05:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:05:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:05:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:05:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:05:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:05:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:05:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:05:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:05:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:05:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:05:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:05:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:05:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:05:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:05:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:05:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:05:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:05:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:05:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:05:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:05:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:05:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:05:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:05:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:05:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:05:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:05:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:05:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:05:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:05:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:05:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:05:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:05:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:05:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:05:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:05:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:05:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:05:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:05:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:05:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:05:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:05:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:05:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:05:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:05:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:05:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:05:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:05:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:05:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:05:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:05:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:05:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:05:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:05:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:05:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:05:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:05:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:05:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:05:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:05:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:05:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:05:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:05:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:05:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:05:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:05:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:05:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:05:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:05:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:05:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:05:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:05:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:05:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:05:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:05:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:05:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:05:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:05:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:05:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:05:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:05:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:05:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:05:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:05:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:05:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:05:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:05:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:05:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:05:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:05:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:05:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:05:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:05:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:05:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:05:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:05:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:05:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:05:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:05:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:05:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:05:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:05:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:05:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:05:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:05:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:05:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:05:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:05:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:05:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:05:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:05:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:05:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:05:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:05:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:05:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:05:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:05:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:05:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:05:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:05:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:05:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:05:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:05:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:05:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:05:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:05:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:05:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:05:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:05:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:05:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:05:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:05:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:05:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:05:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:05:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:05:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:05:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:05:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:05:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:05:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:05:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:05:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:05:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:05:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:05:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:05:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:05:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:05:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:05:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:05:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:05:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:05:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:05:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:05:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:05:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:05:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:05:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:05:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:05:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:05:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:05:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:05:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:05:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:05:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:05:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:05:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:05:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:05:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:05:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:05:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:05:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:05:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:05:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:05:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:05:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:05:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:05:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:05:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:05:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:05:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:05:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:05:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:05:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:05:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:05:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:06:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:06:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:06:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:06:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:06:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:06:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:06:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:06:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:06:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:06:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:06:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:06:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:06:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:06:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:06:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:06:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:06:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:06:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:06:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:06:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:06:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:06:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:06:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:06:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:06:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:06:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:06:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:06:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:06:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:06:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:06:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:06:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:06:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:06:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:06:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:06:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:06:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:06:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:06:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:06:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:06:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:06:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:06:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:06:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:06:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:06:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:06:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:06:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:06:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:06:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:06:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:06:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:06:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:06:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:06:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:06:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:06:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:06:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:06:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:06:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:06:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:06:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:06:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:06:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:06:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:06:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:06:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:06:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:06:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:06:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:06:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:06:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:06:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:06:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:06:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:06:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:06:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:06:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:06:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:06:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:06:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:06:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:06:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:06:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:06:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:06:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:06:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:06:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:06:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:06:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:06:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:06:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:06:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:06:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:06:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:06:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:06:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:06:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:06:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:06:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:06:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:06:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:06:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:06:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:06:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:06:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:06:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:06:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:06:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:06:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:06:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:06:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:06:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:06:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:06:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:06:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:06:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:06:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:06:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:06:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:06:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:06:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:06:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:06:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:06:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:06:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:06:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:06:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:06:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:06:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:06:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:06:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:06:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:06:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:06:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:06:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:06:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:06:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:06:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:06:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:06:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:06:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:06:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:06:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:06:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:06:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:06:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:06:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:06:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:06:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:06:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:06:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:06:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:06:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:06:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:06:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:06:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:06:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:06:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:06:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:06:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:06:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:06:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:06:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:06:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:06:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:06:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:06:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:06:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:06:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:06:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:06:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:06:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:06:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:06:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:06:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:06:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:06:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:06:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:06:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:06:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:06:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:06:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:06:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:06:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:06:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:06:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:06:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:06:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:06:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:06:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:06:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:06:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:06:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:06:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:06:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:06:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:06:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:06:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:06:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:06:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:06:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:06:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:06:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:06:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:06:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:06:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:06:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:06:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:06:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:06:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:06:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:06:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:06:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:06:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:06:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:07:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:07:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:07:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:07:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:07:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:07:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:07:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:07:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:07:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:07:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:07:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:07:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:07:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:07:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:07:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:07:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:07:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:07:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:07:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:07:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:07:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:07:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:07:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:07:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:07:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:07:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:07:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:07:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:07:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:07:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:07:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:07:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:07:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:07:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:07:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:07:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:07:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:07:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:07:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:07:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:07:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:07:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:07:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:07:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:07:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:07:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:07:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:07:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:07:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:07:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:07:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:07:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:07:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:07:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:07:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:07:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:07:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:07:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:07:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:07:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:07:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:07:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:07:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:07:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:07:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:07:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:07:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:07:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:07:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:07:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:07:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:07:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:07:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:07:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:07:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:07:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:07:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:07:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:07:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:07:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:07:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:07:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:07:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:07:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:07:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:07:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:07:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:07:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:07:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:07:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:07:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:07:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:07:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:07:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:07:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:07:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:07:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:07:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:07:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:07:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:07:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:07:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:07:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:07:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:07:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:07:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:07:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:07:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:07:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:07:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:07:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:07:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:07:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:07:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:07:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:07:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:07:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:07:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:07:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:07:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:07:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:07:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:07:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:07:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:07:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:07:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:07:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:07:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:07:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:07:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:07:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:07:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:07:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:07:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:07:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:07:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:07:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:07:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:07:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:07:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:07:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:07:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:07:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:07:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:09:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:09:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:09:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:09:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:09:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:09:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:09:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:09:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:09:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:09:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:09:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:09:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:09:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:09:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:09:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:09:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:09:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:09:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:09:43 rdma06 rsyslogd[1041]: imjournal: 5086 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 21:09:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:09:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:09:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:09:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:09:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:09:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:09:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:09:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:09:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:09:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:09:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:09:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:09:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:09:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:09:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:09:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:09:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:09:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:09:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:09:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:09:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:09:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:09:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:09:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:09:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:09:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:09:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:09:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:09:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:09:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:09:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:09:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:09:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:09:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:09:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:09:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:09:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:09:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:09:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:09:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:09:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:09:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:09:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:09:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:09:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:09:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:09:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:09:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:09:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:09:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:09:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:09:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:09:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:09:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:10:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:10:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:10:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:10:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:10:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:10:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:10:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:10:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:10:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:10:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:10:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:10:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:10:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:10:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:10:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:10:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:10:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:10:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:10:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:10:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:10:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:10:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:10:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:10:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:10:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:10:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:10:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:10:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:10:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:10:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:10:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:10:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:10:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:10:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:10:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:10:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:10:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:10:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:10:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:10:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:10:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:10:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:10:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:10:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:10:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:10:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:10:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:10:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:10:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:10:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:10:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:10:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:10:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:10:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:10:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:10:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:10:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:10:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:10:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:10:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:10:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:10:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:10:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:10:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:10:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:10:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:10:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:10:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:10:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:10:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:10:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:10:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:10:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:10:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:10:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:10:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:10:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:10:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:10:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:10:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:10:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:10:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:10:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:10:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:10:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:10:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:10:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:10:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:10:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:10:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:10:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:10:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:10:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:10:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:10:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:10:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:10:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:10:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:10:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:10:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:10:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:10:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:10:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:10:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:10:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:10:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:10:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:10:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:10:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:10:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:10:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:10:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:10:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:10:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:10:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:10:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:10:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:10:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:10:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:10:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:10:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:10:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:10:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:10:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:10:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:10:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:10:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:10:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:10:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:10:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:10:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:10:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:10:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:10:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:10:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:10:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:10:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:10:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:10:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:10:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:10:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:10:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:10:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:10:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:10:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:10:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:10:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:10:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:10:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:10:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:10:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:10:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:10:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:10:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:10:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:10:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:10:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:10:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:10:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:10:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:10:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:10:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:10:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:10:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:10:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:10:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:10:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:10:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:10:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:10:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:10:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:10:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:10:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:10:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:10:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:10:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:10:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:10:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:10:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:10:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:10:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:10:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:10:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:10:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:10:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:10:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:10:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:10:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:10:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:10:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:10:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:10:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:10:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:10:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:10:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:10:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:10:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:10:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:10:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:10:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:10:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:10:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:10:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:10:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:10:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:10:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:10:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:10:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:10:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:10:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:10:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:10:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:10:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:10:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:10:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:10:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:11:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:11:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:11:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:11:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:11:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:11:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:11:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:11:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:11:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:11:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:11:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:11:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:11:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:11:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:11:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:11:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:11:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:11:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:11:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:11:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:11:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:11:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:11:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:11:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:11:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:11:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:11:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:11:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:11:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:11:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:11:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:11:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:11:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:11:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:11:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:11:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:11:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:11:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:11:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:11:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:11:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:11:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:11:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:11:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:11:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:11:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:11:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:11:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:11:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:11:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:11:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:11:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:11:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:11:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:11:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:11:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:11:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:11:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:11:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:11:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:11:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:11:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:11:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:11:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:11:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:11:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:11:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:11:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:11:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:11:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:11:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:11:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:11:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:11:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:11:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:11:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:11:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:11:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:11:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:11:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:11:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:11:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:11:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:11:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:11:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:11:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:11:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:11:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:11:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:11:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:11:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:11:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:11:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:11:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:11:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:11:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:11:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:11:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:11:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:11:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:11:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:11:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:11:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:11:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:11:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:11:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:11:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:11:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:11:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:11:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:11:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:11:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:11:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:11:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:11:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:11:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:11:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:11:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:11:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:11:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:11:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:11:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:11:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:11:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:11:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:11:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:11:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:11:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:11:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:11:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:11:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:11:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:11:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:11:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:11:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:11:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:11:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:11:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:11:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:11:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:11:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:11:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:11:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:11:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:11:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:11:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:11:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:11:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:11:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:11:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:11:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:11:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:11:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:11:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:11:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:11:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:11:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:11:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:11:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:11:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:11:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:11:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:11:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:11:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:11:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:11:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:11:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:11:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:11:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:11:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:11:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:11:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:11:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:11:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:11:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:11:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:11:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:11:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:11:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:11:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:11:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:11:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:11:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:11:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:11:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:11:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:11:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:11:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:11:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:11:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:11:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:11:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:11:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:11:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:11:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:11:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:11:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:11:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:11:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:11:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:11:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:11:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:11:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:11:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:11:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:11:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:11:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:11:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:11:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:11:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:11:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:11:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:11:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:11:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:11:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:11:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:12:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:12:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:12:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:12:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:12:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:12:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:12:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:12:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:12:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:12:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:12:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:12:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:12:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:12:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:12:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:12:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:12:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:12:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:12:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:12:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:12:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:12:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:12:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:12:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:12:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:12:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:12:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:12:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:12:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:12:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:12:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:12:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:12:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:12:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:12:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:12:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:12:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:12:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:12:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:12:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:12:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:12:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:12:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:12:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:12:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:12:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:12:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:12:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:12:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:12:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:12:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:12:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:12:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:12:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:12:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:12:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:12:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:12:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:12:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:12:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:12:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:12:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:12:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:12:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:12:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:12:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:12:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:12:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:12:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:12:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:12:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:12:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:12:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:12:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:12:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:12:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:12:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:12:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:12:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:12:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:12:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:12:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:12:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:12:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:12:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:12:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:12:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:12:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:12:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:12:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:12:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:12:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:12:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:12:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:12:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:12:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:12:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:12:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:12:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:12:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:12:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:12:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:12:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:12:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:12:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:12:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:12:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:12:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:12:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:12:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:12:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:12:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:12:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:12:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:12:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:12:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:12:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:12:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:12:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:12:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:12:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:12:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:12:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:12:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:12:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:12:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:12:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:12:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:12:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:12:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:12:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:12:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:12:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:12:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:12:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:12:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:12:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:12:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:12:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:12:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:12:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:12:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:12:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:12:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:12:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:12:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:12:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:12:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:12:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:12:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:12:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:12:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:12:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:12:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:12:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:12:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:12:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:12:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:12:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:12:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:12:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:12:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:12:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:12:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:12:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:12:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:12:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:12:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:12:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:12:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:12:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:12:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:12:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:12:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:12:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:12:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:12:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:12:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:12:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:12:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:12:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:12:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:12:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:12:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:12:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:12:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:12:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:12:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:12:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:12:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:12:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:12:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:12:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:12:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:12:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:12:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:12:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:12:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:12:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:12:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:12:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:12:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:12:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:12:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:12:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:12:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:12:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:12:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:12:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:12:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:12:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:12:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:12:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:12:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:12:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:12:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:13:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:13:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:13:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:13:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:13:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:13:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:13:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:13:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:13:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:13:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:13:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:13:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:13:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:13:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:13:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:13:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:13:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:13:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:13:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:13:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:13:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:13:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:13:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:13:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:13:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:13:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:13:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:13:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:13:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:13:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:13:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:13:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:13:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:13:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:13:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:13:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:13:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:13:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:13:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:13:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:13:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:13:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:13:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:13:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:13:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:13:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:13:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:13:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:13:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:13:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:13:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:13:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:13:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:13:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:13:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:13:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:13:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:13:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:13:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:13:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:13:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:13:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:13:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:13:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:13:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:13:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:13:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:13:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:13:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:13:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:13:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:13:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:13:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:13:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:13:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:13:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:13:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:13:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:13:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:13:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:13:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:13:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:13:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:13:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:13:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:13:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:13:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:13:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:13:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:13:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:13:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:13:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:13:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:13:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:13:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:13:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:13:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:13:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:13:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:13:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:13:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:13:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:13:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:13:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:13:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:13:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:13:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:13:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:13:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:13:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:13:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:13:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:13:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:13:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:13:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:13:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:13:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:13:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:13:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:13:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:13:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:13:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:13:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:13:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:13:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:13:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:13:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:13:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:13:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:13:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:13:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:13:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:13:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:13:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:13:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:13:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:13:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:13:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:13:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:13:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:13:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:13:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:13:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:13:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:13:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:13:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:13:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:13:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:13:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:13:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:13:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:13:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:13:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:13:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:13:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:13:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:13:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:13:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:13:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:13:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:13:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:13:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:13:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:13:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:13:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:13:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:13:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:13:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:13:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:13:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:13:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:13:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:13:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:13:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:13:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:13:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:13:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:13:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:13:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:13:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:13:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:13:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:13:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:13:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:13:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:13:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:13:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:13:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:13:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:13:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:13:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:13:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:13:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:13:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:13:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:13:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:13:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:13:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:13:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:13:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:13:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:13:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:13:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:13:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:13:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:13:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:13:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:13:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:13:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:13:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:13:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:13:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:13:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:13:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:13:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:13:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:14:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:14:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:14:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:14:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:14:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:14:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:14:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:14:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:14:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:14:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:14:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:14:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:14:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:14:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:14:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:14:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:14:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:14:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:14:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:14:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:14:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:14:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:14:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:14:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:14:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:14:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:14:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:14:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:14:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:14:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:14:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:14:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:14:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:14:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:14:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:14:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:14:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:14:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:14:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:14:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:14:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:14:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:14:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:14:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:14:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:14:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:14:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:14:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:14:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:14:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:14:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:14:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:14:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:14:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:14:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:14:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:14:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:14:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:14:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:14:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:14:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:14:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:14:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:14:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:14:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:14:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:14:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:14:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:14:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:14:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:14:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:14:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:14:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:14:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:14:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:14:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:14:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:14:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:14:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:14:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:14:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:14:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:14:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:14:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:14:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:14:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:14:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:14:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:14:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:14:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:14:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:14:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:14:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:14:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:14:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:14:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:14:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:14:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:14:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:14:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:14:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:14:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:14:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:14:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:14:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:14:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:14:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:14:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:14:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:14:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:14:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:14:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:14:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:14:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:14:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:14:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:14:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:14:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:14:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:14:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:14:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:14:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:14:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:14:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:14:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:14:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:14:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:14:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:14:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:14:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:14:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:14:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:14:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:14:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:14:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:14:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:14:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:14:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:14:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:14:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:14:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:14:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:14:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:14:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:14:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:14:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:14:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:14:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:14:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:14:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:14:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:14:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:14:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:14:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:14:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:14:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:14:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:14:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:14:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:14:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:14:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:14:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:14:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:14:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:14:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:14:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:14:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:14:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:14:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:14:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:14:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:14:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:14:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:14:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:14:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:14:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:14:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:14:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:14:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:14:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:14:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:14:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:14:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:14:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:14:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:14:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:14:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:14:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:14:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:14:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:14:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:14:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:14:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:14:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:14:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:14:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:14:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:14:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:14:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:14:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:14:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:14:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:14:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:14:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:14:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:14:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:14:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:14:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:14:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:14:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:14:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:14:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:14:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:14:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:14:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:14:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:15:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:15:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:15:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:15:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:15:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:15:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:15:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:15:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:15:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:15:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:15:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:15:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:15:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:15:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:15:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:15:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:15:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:15:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:15:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:15:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:15:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:15:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:15:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:15:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:15:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:15:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:15:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:15:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:15:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:15:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:15:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:15:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:15:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:15:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:15:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:15:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:15:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:15:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:15:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:15:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:15:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:15:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:15:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:15:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:15:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:15:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:15:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:15:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:15:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:15:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:15:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:15:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:15:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:15:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:15:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:15:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:15:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:15:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:15:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:15:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:15:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:15:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:15:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:15:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:15:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:15:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:15:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:15:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:15:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:15:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:15:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:15:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:15:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:15:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:15:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:15:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:15:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:15:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:15:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:15:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:15:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:15:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:15:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:15:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:15:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:15:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:15:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:15:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:15:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:15:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:15:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:15:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:15:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:15:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:15:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:15:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:15:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:15:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:15:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:15:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:15:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:15:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:15:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:15:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:15:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:15:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:15:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:15:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:15:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:15:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:15:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:15:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:15:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:15:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:15:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:15:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:15:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:15:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:15:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:15:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:15:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:15:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:15:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:15:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:15:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:15:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:15:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:15:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:15:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:15:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:15:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:15:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:15:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:15:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:15:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:15:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:15:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:15:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:15:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:15:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:15:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:15:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:15:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:15:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:15:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:15:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:15:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:15:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:15:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:15:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:15:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:15:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:15:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:15:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:15:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:15:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:15:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:15:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:15:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:15:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:15:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:15:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:15:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:15:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:15:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:15:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:15:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:15:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:15:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:15:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:15:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:15:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:15:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:15:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:15:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:15:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:15:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:15:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:15:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:15:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:15:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:15:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:15:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:15:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:15:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:15:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:15:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:15:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:15:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:15:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:15:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:15:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:15:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:15:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:15:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:15:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:15:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:15:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:15:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:15:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:15:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:15:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:15:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:15:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:15:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:15:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:15:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:15:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:15:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:15:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:15:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:15:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:15:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:15:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:15:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:15:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:16:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:16:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:16:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:16:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:16:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:16:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:16:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:16:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:16:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:16:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:16:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:16:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:16:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:16:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:16:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:16:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:16:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:16:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:16:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:16:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:16:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:16:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:16:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:16:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:16:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:16:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:16:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:16:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:16:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:16:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:16:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:16:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:16:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:16:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:16:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:16:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:16:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:16:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:16:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:16:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:16:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:16:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:16:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:16:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:16:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:16:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:16:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:16:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:16:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:16:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:16:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:16:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:16:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:16:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:16:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:16:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:16:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:16:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:16:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:16:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:16:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:16:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:16:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:16:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:16:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:16:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:16:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:16:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:16:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:16:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:16:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:16:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:16:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:16:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:16:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:16:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:16:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:16:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:16:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:16:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:16:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:16:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:16:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:16:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:16:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:16:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:16:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:16:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:16:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:16:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:16:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:16:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:16:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:16:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:16:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:16:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:16:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:16:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:16:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:16:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:16:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:16:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:16:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:16:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:16:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:16:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:16:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:16:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:16:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:16:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:16:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:16:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:16:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:16:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:16:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:16:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:16:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:16:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:16:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:16:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:16:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:16:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:16:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:16:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:16:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:16:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:16:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:16:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:16:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:16:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:16:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:16:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:16:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:16:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:16:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:16:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:16:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:16:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:16:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:16:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:16:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:16:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:16:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:16:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:16:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:16:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:16:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:16:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:16:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:16:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:16:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:16:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:16:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:16:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:16:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:16:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:16:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:16:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:16:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:16:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:16:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:16:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:16:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:16:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:16:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:16:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:16:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:16:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:16:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:16:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:16:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:16:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:16:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:16:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:16:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:16:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:16:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:16:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:16:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:16:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:16:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:16:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:16:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:16:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:16:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:16:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:16:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:16:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:16:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:16:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:16:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:16:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:16:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:16:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:16:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:16:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:16:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:16:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:16:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:16:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:16:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:16:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:16:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:16:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:16:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:16:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:16:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:16:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:16:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:16:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:16:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:16:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:16:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:16:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:16:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:16:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:17:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:17:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:17:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:17:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:17:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:17:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:17:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:17:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:17:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:17:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:17:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:17:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:17:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:17:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:17:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:17:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:17:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:17:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:17:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:17:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:17:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:17:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:17:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:17:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:17:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:17:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:17:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:17:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:17:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:17:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:17:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:17:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:17:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:17:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:17:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:17:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:17:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:17:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:17:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:17:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:17:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:17:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:17:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:17:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:17:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:17:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:17:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:17:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:17:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:17:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:17:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:17:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:17:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:17:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:17:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:17:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:17:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:17:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:17:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:17:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:17:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:17:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:17:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:17:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:17:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:17:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:17:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:17:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:17:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:17:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:17:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:17:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:17:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:17:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:17:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:17:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:17:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:17:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:17:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:17:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:17:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:17:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:17:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:17:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:17:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:17:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:17:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:17:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:17:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:17:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:17:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:17:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:17:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:17:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:17:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:17:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:17:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:17:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:17:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:17:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:17:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:17:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:17:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:17:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:17:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:17:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:17:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:17:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:17:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:17:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:17:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:17:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:17:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:17:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:17:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:17:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:17:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:17:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:17:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:17:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:17:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:17:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:17:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:17:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:17:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:17:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:17:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:17:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:17:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:17:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:17:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:17:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:17:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:17:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:17:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:17:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:17:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:17:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:17:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:17:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:17:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:17:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:17:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:17:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:19:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:19:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:19:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:19:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:19:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:19:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:19:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:19:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:19:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:19:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:19:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:19:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:19:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:19:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:19:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:19:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:19:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:19:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:19:48 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 21:19:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:19:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:19:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:19:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:19:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:19:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:19:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:19:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:19:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:19:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:19:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:19:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:19:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:19:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:19:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:19:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:19:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:19:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:19:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:19:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:19:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:19:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:19:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:19:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:19:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:19:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:19:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:19:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:19:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:19:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:19:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:19:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:19:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:19:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:19:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:19:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:20:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:20:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:20:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:20:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:20:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:20:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:20:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:20:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:20:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:20:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:20:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:20:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:20:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:20:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:20:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:20:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:20:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:20:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:20:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:20:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:20:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:20:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:20:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:20:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:20:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:20:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:20:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:20:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:20:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:20:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:20:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:20:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:20:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:20:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:20:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:20:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:20:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:20:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:20:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:20:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:20:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:20:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:20:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:20:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:20:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:20:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:20:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:20:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:20:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:20:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:20:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:20:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:20:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:20:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:20:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:20:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:20:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:20:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:20:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:20:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:20:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:20:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:20:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:20:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:20:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:20:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:20:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:20:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:20:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:20:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:20:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:20:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:20:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:20:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:20:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:20:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:20:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:20:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:20:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:20:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:20:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:20:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:20:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:20:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:20:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:20:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:20:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:20:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:20:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:20:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:20:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:20:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:20:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:20:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:20:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:20:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:20:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:20:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:20:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:20:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:20:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:20:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:20:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:20:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:20:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:20:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:20:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:20:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:20:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:20:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:20:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:20:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:20:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:20:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:20:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:20:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:20:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:20:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:20:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:20:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:20:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:20:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:20:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:20:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:20:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:20:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:20:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:20:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:20:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:20:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:20:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:20:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:20:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:20:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:20:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:20:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:20:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:20:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:20:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:20:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:20:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:20:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:20:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:20:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:20:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:20:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:20:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:20:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:20:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:20:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:20:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:20:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:20:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:20:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:20:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:20:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:20:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:20:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:20:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:20:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:20:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:20:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:20:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:20:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:20:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:20:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:20:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:20:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:20:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:20:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:20:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:20:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:20:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:20:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:20:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:20:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:20:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:20:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:20:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:20:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:20:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:20:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:20:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:20:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:20:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:20:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:20:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:20:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:20:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:20:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:20:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:20:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:20:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:20:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:20:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:20:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:20:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:20:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:20:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:20:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:20:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:20:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:20:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:20:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:20:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:20:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:20:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:20:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:20:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:20:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:20:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:20:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:20:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:20:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:20:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:20:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:21:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:21:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:21:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:21:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:21:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:21:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:21:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:21:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:21:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:21:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:21:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:21:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:21:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:21:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:21:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:21:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:21:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:21:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:21:03 rdma06 systemd-journald[790]: Data hash table of /run/log/journal/95379d2f723847b68d9e976dee8967a9/system.journal has a fill level at 75.0 (163750 of 218332 items, 75497472 file size, 461 bytes per hash table item), suggesting rotation.
Apr 17 21:21:03 rdma06 systemd-journald[790]: /run/log/journal/95379d2f723847b68d9e976dee8967a9/system.journal: Journal header limits reached or header out-of-date, rotating.
Apr 17 21:21:03 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 17 21:21:03 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 17 21:21:03 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 17 21:21:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:21:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:21:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:21:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:21:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:21:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:21:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:21:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:21:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:21:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:21:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:21:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:21:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:21:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:21:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:21:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:21:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:21:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:21:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:21:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:21:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:21:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:21:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:21:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:21:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:21:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:21:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:21:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:21:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:21:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:21:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:21:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:21:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:21:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:21:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:21:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:21:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:21:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:21:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:21:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:21:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:21:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:21:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:21:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:21:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:21:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:21:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:21:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:21:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:21:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:21:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:21:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:21:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:21:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:21:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:21:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:21:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:21:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:21:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:21:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:21:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:21:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:21:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:21:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:21:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:21:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:21:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:21:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:21:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:21:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:21:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:21:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:21:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:21:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:21:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:21:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:21:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:21:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:21:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:21:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:21:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:21:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:21:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:21:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:21:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:21:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:21:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:21:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:21:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:21:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:21:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:21:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:21:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:21:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:21:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:21:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:21:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:21:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:21:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:21:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:21:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:21:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:21:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:21:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:21:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:21:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:21:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:21:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:21:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:21:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:21:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:21:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:21:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:21:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:21:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:21:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:21:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:21:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:21:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:21:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:21:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:21:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:21:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:21:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:21:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:21:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:21:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:21:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:21:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:21:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:21:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:21:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:21:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:21:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:21:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:21:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:21:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:21:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:21:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:21:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:21:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:21:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:21:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:21:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:21:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:21:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:21:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:21:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:21:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:21:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:21:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:21:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:21:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:21:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:21:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:21:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:21:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:21:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:21:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:21:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:21:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:21:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:21:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:21:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:21:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:21:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:21:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:21:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:21:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:21:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:21:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:21:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:21:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:21:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:21:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:21:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:21:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:21:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:21:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:21:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:21:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:21:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:21:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:21:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:21:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:21:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:21:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:21:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:21:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:21:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:21:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:21:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:21:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:21:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:21:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:21:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:21:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:21:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:22:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:22:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:22:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:22:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:22:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:22:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:22:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:22:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:22:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:22:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:22:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:22:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:22:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:22:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:22:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:22:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:22:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:22:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:22:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:22:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:22:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:22:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:22:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:22:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:22:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:22:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:22:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:22:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:22:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:22:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:22:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:22:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:22:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:22:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:22:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:22:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:22:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:22:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:22:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:22:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:22:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:22:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:22:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:22:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:22:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:22:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:22:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:22:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:22:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:22:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:22:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:22:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:22:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:22:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:22:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:22:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:22:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:22:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:22:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:22:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:22:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:22:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:22:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:22:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:22:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:22:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:22:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:22:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:22:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:22:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:22:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:22:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:22:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:22:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:22:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:22:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:22:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:22:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:22:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:22:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:22:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:22:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:22:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:22:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:22:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:22:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:22:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:22:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:22:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:22:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:22:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:22:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:22:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:22:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:22:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:22:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:22:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:22:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:22:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:22:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:22:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:22:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:22:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:22:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:22:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:22:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:22:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:22:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:22:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:22:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:22:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:22:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:22:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:22:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:22:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:22:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:22:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:22:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:22:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:22:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:22:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:22:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:22:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:22:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:22:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:22:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:22:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:22:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:22:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:22:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:22:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:22:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:22:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:22:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:22:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:22:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:22:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:22:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:22:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:22:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:22:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:22:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:22:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:22:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:22:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:22:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:22:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:22:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:22:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:22:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:22:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:22:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:22:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:22:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:22:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:22:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:22:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:22:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:22:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:22:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:22:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:22:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:22:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:22:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:22:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:22:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:22:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:22:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:22:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:22:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:22:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:22:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:22:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:22:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:22:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:22:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:22:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:22:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:22:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:22:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:22:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:22:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:22:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:22:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:22:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:22:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:22:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:22:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:22:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:22:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:22:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:22:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:22:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:22:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:22:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:22:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:22:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:22:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:22:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:22:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:22:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:22:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:22:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:22:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:22:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:22:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:22:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:22:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:22:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:22:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:22:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:22:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:22:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:22:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:22:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:22:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:23:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:23:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:23:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:23:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:23:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:23:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:23:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:23:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:23:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:23:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:23:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:23:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:23:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:23:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:23:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:23:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:23:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:23:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:23:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:23:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:23:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:23:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:23:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:23:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:23:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:23:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:23:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:23:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:23:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:23:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:23:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:23:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:23:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:23:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:23:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:23:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:23:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:23:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:23:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:23:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:23:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:23:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:23:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:23:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:23:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:23:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:23:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:23:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:23:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:23:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:23:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:23:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:23:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:23:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:23:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:23:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:23:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:23:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:23:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:23:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:23:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:23:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:23:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:23:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:23:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:23:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:23:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:23:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:23:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:23:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:23:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:23:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:23:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:23:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:23:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:23:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:23:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:23:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:23:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:23:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:23:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:23:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:23:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:23:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:23:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:23:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:23:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:23:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:23:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:23:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:23:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:23:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:23:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:23:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:23:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:23:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:23:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:23:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:23:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:23:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:23:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:23:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:23:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:23:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:23:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:23:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:23:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:23:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:23:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:23:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:23:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:23:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:23:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:23:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:23:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:23:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:23:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:23:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:23:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:23:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:23:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:23:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:23:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:23:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:23:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:23:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:23:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:23:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:23:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:23:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:23:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:23:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:23:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:23:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:23:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:23:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:23:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:23:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:23:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:23:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:23:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:23:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:23:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:23:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:23:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:23:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:23:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:23:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:23:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:23:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:23:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:23:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:23:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:23:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:23:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:23:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:23:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:23:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:23:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:23:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:23:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:23:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:23:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:23:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:23:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:23:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:24:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:24:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:24:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:24:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:24:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:24:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:24:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:24:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:24:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:24:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:24:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:24:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:24:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:24:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:24:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:24:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:24:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:24:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:24:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:24:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:24:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:24:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:24:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:24:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:24:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:24:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:24:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:24:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:24:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:24:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:24:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:24:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:24:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:24:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:24:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:24:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:24:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:24:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:24:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:24:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:24:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:24:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:24:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:24:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:24:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:24:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:24:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:24:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:24:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:24:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:24:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:24:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:24:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:24:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:24:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:24:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:24:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:24:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:24:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:24:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:24:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:24:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:24:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:24:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:24:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:24:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:24:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:24:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:24:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:24:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:24:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:24:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:24:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:24:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:24:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:24:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:24:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:24:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:24:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:24:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:24:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:24:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:24:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:24:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:24:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:24:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:24:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:24:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:24:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:24:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:24:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:24:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:24:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:24:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:24:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:24:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:24:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:24:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:24:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:24:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:24:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:24:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:24:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:24:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:24:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:24:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:24:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:24:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:24:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:24:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:24:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:24:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:24:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:24:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:24:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:24:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:24:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:24:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:24:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:24:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:24:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:24:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:24:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:24:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:24:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:24:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:24:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:24:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:24:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:24:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:24:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:24:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:24:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:24:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:24:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:24:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:24:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:24:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:24:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:24:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:24:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:24:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:24:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:24:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:24:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:24:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:24:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:24:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:24:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:24:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:24:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:24:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:24:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:24:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:24:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:24:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:24:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:24:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:24:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:24:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:24:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:24:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:24:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:24:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:24:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:24:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:24:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:24:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:24:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:24:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:24:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:24:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:24:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:24:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:24:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:24:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:24:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:24:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:24:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:24:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:24:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:24:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:24:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:24:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:24:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:24:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:24:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:24:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:24:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:24:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:24:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:24:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:24:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:24:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:24:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:24:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:25:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:25:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:25:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:25:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:25:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:25:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:25:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:25:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:25:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:25:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:25:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:25:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:25:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:25:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:25:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:25:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:25:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:25:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:25:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:25:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:25:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:25:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:25:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:25:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:25:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:25:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:25:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:25:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:25:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:25:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:25:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:25:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:25:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:25:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:25:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:25:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:25:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:25:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:25:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:25:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:25:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:25:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:25:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:25:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:25:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:25:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:25:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:25:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:25:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:25:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:25:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:25:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:25:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:25:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:25:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:25:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:25:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:25:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:25:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:25:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:25:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:25:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:25:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:25:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:25:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:25:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:25:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:25:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:25:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:25:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:25:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:25:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:25:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:25:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:25:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:25:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:25:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:25:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:25:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:25:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:25:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:25:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:25:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:25:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:25:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:25:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:25:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:25:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:25:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:25:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:25:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:25:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:25:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:25:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:25:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:25:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:25:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:25:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:25:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:25:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:25:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:25:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:25:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:25:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:25:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:25:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:25:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:25:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:25:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:25:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:25:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:25:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:25:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:25:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:25:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:25:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:25:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:25:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:25:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:25:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:25:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:25:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:25:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:25:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:25:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:25:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:25:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:25:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:25:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:25:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:25:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:25:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:25:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:25:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:25:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:25:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:25:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:25:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:25:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:25:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:25:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:25:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:25:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:25:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:25:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:25:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:25:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:25:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:25:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:25:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:25:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:25:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:25:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:25:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:25:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:25:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:25:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:25:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:25:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:25:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:25:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:25:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:25:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:25:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:25:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:25:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:26:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:26:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:26:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:26:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:26:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:26:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:26:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:26:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:26:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:26:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:26:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:26:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:26:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:26:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:26:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:26:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:26:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:26:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:26:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:26:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:26:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:26:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:26:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:26:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:26:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:26:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:26:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:26:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:26:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:26:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:26:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:26:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:26:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:26:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:26:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:26:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:26:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:26:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:26:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:26:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:26:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:26:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:26:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:26:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:26:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:26:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:26:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:26:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:26:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:26:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:26:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:26:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:26:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:26:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:26:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:26:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:26:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:26:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:26:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:26:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:26:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:26:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:26:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:26:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:26:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:26:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:26:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:26:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:26:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:26:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:26:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:26:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:26:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:26:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:26:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:26:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:26:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:26:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:26:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:26:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:26:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:26:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:26:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:26:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:26:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:26:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:26:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:26:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:26:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:26:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:26:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:26:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:26:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:26:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:26:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:26:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:26:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:26:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:26:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:26:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:26:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:26:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:26:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:26:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:26:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:26:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:26:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:26:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:26:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:26:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:26:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:26:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:26:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:26:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:26:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:26:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:26:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:26:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:26:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:26:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:26:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:26:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:26:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:26:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:26:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:26:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:26:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:26:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:26:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:26:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:26:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:26:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:26:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:26:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:26:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:26:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:26:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:26:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:26:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:26:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:26:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:26:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:26:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:26:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:26:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:26:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:26:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:26:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:26:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:26:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:26:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:26:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:26:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:26:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:26:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:26:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:26:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:26:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:26:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:26:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:26:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:26:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:26:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:26:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:26:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:26:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:26:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:26:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:26:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:26:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:26:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:26:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:26:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:26:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:26:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:26:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:26:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:26:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:26:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:26:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:26:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:26:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:26:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:26:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:26:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:26:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:26:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:26:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:26:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:26:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:26:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:26:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:26:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:26:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:26:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:26:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:26:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:26:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:26:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:26:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:26:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:26:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:26:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:26:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:26:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:26:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:26:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:26:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:26:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:26:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:26:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:26:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:26:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:26:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:26:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:26:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:27:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:27:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:27:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:27:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:27:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:27:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:27:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:27:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:27:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:27:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:27:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:27:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:27:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:27:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:27:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:27:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:27:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:27:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:27:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:27:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:27:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:27:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:27:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:27:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:27:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:27:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:27:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:27:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:27:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:27:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:27:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:27:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:27:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:27:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:27:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:27:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:27:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:27:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:27:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:27:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:27:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:27:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:27:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:27:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:27:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:27:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:27:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:27:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:27:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:27:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:27:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:27:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:27:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:27:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:27:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:27:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:27:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:27:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:27:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:27:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:27:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:27:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:27:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:27:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:27:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:27:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:27:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:27:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:27:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:27:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:27:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:27:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:27:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:27:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:27:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:27:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:27:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:27:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:27:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:27:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:27:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:27:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:27:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:27:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:27:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:27:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:27:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:27:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:27:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:27:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:27:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:27:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:27:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:27:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:27:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:27:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:27:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:27:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:27:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:27:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:27:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:27:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:27:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:27:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:27:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:27:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:27:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:27:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:27:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:27:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:27:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:27:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:27:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:27:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:27:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:27:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:27:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:27:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:27:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:27:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:27:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:27:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:27:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:27:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:27:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:27:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:27:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:27:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:27:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:27:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:27:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:27:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:27:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:27:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:27:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:27:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:27:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:27:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:27:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:27:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:27:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:27:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:27:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:27:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:27:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:27:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:27:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:27:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:27:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:27:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:27:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:27:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:27:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:27:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:27:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:27:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:27:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:27:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:27:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:27:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:27:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:27:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:29:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:29:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:29:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:29:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:29:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:29:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:29:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:29:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:29:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:29:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:29:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:29:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:29:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:29:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:29:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:29:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:29:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:29:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:29:53 rdma06 rsyslogd[1041]: imjournal: 5296 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 21:29:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:29:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:29:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:29:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:29:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:29:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:29:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:29:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:29:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:29:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:29:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:29:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:29:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:29:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:29:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:29:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:29:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:29:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:30:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:30:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:30:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:30:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:30:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:30:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:30:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:30:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:30:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:30:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:30:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:30:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:30:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:30:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:30:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:30:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:30:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:30:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:30:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:30:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:30:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:30:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:30:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:30:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:30:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:30:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:30:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:30:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:30:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:30:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:30:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:30:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:30:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:30:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:30:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:30:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:30:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:30:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:30:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:30:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:30:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:30:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:30:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:30:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:30:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:30:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:30:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:30:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:30:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:30:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:30:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:30:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:30:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:30:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:30:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:30:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:30:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:30:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:30:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:30:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:30:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:30:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:30:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:30:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:30:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:30:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:30:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:30:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:30:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:30:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:30:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:30:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:30:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:30:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:30:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:30:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:30:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:30:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:30:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:30:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:30:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:30:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:30:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:30:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:30:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:30:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:30:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:30:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:30:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:30:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:30:48 rdma06 systemd[1]: Starting dnf makecache...
Apr 17 21:30:48 rdma06 dnf[1168503]: Updating Subscription Management repositories.
Apr 17 21:30:48 rdma06 dnf[1168503]: Unable to read consumer identity
Apr 17 21:30:48 rdma06 dnf[1168503]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 17 21:30:48 rdma06 dnf[1168503]: Metadata cache refreshed recently.
Apr 17 21:30:49 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 17 21:30:49 rdma06 systemd[1]: Finished dnf makecache.
Apr 17 21:30:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:30:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:30:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:30:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:30:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:30:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:30:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:30:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:30:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:30:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:30:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:30:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:30:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:30:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:30:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:30:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:30:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:30:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:30:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:30:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:30:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:30:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:30:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:30:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:30:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:30:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:31:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:31:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:31:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:31:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:31:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:31:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:31:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:31:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:31:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:31:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:31:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:31:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:31:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:31:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:31:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:31:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:31:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:31:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:31:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:31:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:31:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:31:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:31:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:31:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:31:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:31:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:31:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:31:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:31:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:31:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:31:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:31:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:31:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:31:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:31:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:31:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:31:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:31:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:31:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:31:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:31:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:31:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:31:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:31:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:31:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:31:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:31:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:31:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:31:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:31:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:31:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:31:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:31:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:31:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:31:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:31:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:31:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:31:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:31:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:31:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:31:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:31:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:31:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:31:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:31:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:31:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:31:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:31:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:31:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:31:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:31:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:31:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:31:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:31:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:31:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:31:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:31:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:31:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:31:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:31:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:31:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:31:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:31:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:31:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:31:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:31:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:31:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:31:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:31:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:31:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:31:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:31:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:31:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:31:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:31:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:31:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:31:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:31:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:31:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:31:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:31:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:31:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:31:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:31:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:31:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:31:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:31:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:31:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:31:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:31:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:31:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:31:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:31:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:31:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:31:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:31:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:31:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:31:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:31:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:31:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:31:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:31:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:31:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:31:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:31:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:31:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:31:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:31:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:31:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:31:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:31:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:31:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:31:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:31:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:31:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:31:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:31:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:31:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:31:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:31:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:31:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:31:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:31:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:31:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:31:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:31:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:31:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:31:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:31:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:31:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:31:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:31:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:31:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:31:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:31:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:31:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:31:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:31:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:31:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:31:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:31:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:31:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:31:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:31:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:31:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:31:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:31:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:31:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:31:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:31:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:31:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:31:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:31:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:31:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:31:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:31:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:31:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:31:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:31:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:31:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:31:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:31:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:31:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:31:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:31:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:31:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:31:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:31:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:31:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:31:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:31:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:31:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:31:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:31:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:31:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:31:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:31:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:31:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:31:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:31:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:31:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:31:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:31:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:31:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:31:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:31:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:31:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:31:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:31:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:31:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:31:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:31:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:31:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:31:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:31:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:31:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:32:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:32:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:32:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:32:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:32:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:32:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:32:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:32:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:32:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:32:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:32:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:32:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:32:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:32:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:32:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:32:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:32:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:32:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:32:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:32:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:32:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:32:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:32:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:32:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:32:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:32:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:32:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:32:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:32:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:32:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:32:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:32:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:32:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:32:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:32:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:32:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:32:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:32:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:32:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:32:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:32:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:32:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:32:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:32:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:32:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:32:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:32:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:32:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:32:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:32:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:32:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:32:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:32:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:32:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:32:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:32:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:32:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:32:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:32:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:32:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:32:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:32:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:32:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:32:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:32:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:32:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:32:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:32:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:32:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:32:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:32:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:32:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:32:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:32:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:32:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:32:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:32:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:32:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:32:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:32:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:32:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:32:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:32:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:32:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:32:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:32:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:32:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:32:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:32:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:32:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:32:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:32:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:32:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:32:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:32:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:32:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:32:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:32:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:32:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:32:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:32:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:32:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:32:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:32:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:32:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:32:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:32:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:32:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:32:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:32:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:32:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:32:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:32:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:32:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:32:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:32:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:32:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:32:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:32:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:32:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:32:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:32:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:32:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:32:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:32:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:32:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:32:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:32:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:32:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:32:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:32:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:32:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:32:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:32:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:32:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:32:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:32:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:32:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:32:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:32:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:32:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:32:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:32:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:32:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:32:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:32:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:32:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:32:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:32:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:32:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:32:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:32:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:32:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:32:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:32:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:32:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:32:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:32:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:32:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:32:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:32:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:32:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:32:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:32:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:32:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:32:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:32:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:32:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:32:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:32:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:32:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:32:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:32:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:32:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:32:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:32:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:32:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:32:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:32:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:32:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:32:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:32:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:32:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:32:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:32:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:32:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:32:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:32:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:32:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:32:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:32:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:32:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:32:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:32:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:32:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:32:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:32:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:32:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:32:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:32:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:32:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:32:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:32:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:32:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:32:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:32:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:32:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:32:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:32:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:32:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:32:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:32:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:32:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:32:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:32:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:32:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:33:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:33:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:33:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:33:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:33:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:33:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:33:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:33:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:33:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:33:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:33:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:33:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:33:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:33:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:33:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:33:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:33:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:33:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:33:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:33:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:33:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:33:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:33:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:33:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:33:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:33:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:33:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:33:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:33:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:33:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:33:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:33:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:33:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:33:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:33:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:33:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:33:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:33:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:33:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:33:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:33:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:33:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:33:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:33:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:33:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:33:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:33:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:33:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:33:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:33:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:33:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:33:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:33:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:33:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:33:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:33:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:33:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:33:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:33:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:33:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:33:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:33:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:33:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:33:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:33:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:33:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:33:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:33:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:33:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:33:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:33:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:33:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:33:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:33:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:33:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:33:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:33:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:33:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:33:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:33:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:33:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:33:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:33:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:33:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:33:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:33:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:33:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:33:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:33:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:33:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:33:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:33:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:33:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:33:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:33:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:33:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:33:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:33:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:33:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:33:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:33:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:33:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:33:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:33:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:33:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:33:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:33:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:33:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:33:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:33:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:33:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:33:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:33:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:33:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:33:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:33:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:33:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:33:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:33:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:33:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:33:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:33:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:33:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:33:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:33:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:33:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:33:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:33:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:33:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:33:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:33:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:33:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:33:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:33:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:33:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:33:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:33:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:33:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:33:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:33:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:33:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:33:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:33:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:33:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:33:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:33:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:33:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:33:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:33:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:33:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:33:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:33:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:33:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:33:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:33:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:33:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:33:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:33:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:33:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:33:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:33:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:33:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:33:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:33:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:33:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:33:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:33:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:33:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:33:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:33:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:33:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:33:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:33:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:33:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:33:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:33:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:33:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:33:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:33:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:33:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:33:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:33:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:33:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:33:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:33:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:33:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:33:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:33:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:33:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:33:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:33:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:33:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:33:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:33:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:33:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:33:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:33:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:33:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:33:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:33:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:33:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:33:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:33:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:33:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:33:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:33:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:33:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:33:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:33:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:33:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:33:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:33:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:33:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:33:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:33:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:33:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:34:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:34:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:34:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:34:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:34:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:34:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:34:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:34:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:34:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:34:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:34:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:34:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:34:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:34:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:34:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:34:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:34:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:34:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:34:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:34:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:34:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:34:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:34:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:34:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:34:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:34:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:34:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:34:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:34:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:34:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:34:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:34:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:34:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:34:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:34:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:34:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:34:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:34:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:34:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:34:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:34:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:34:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:34:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:34:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:34:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:34:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:34:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:34:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:34:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:34:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:34:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:34:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:34:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:34:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:34:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:34:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:34:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:34:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:34:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:34:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:34:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:34:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:34:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:34:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:34:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:34:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:34:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:34:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:34:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:34:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:34:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:34:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:34:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:34:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:34:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:34:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:34:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:34:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:34:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:34:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:34:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:34:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:34:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:34:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:34:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:34:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:34:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:34:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:34:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:34:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:34:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:34:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:34:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:34:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:34:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:34:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:34:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:34:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:34:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:34:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:34:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:34:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:34:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:34:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:34:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:34:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:34:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:34:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:34:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:34:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:34:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:34:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:34:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:34:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:34:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:34:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:34:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:34:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:34:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:34:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:34:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:34:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:34:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:34:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:34:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:34:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:34:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:34:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:34:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:34:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:34:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:34:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:34:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:34:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:34:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:34:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:34:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:34:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:34:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:34:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:34:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:34:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:34:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:34:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:34:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:34:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:34:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:34:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:34:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:34:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:34:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:34:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:34:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:34:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:34:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:34:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:34:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:34:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:34:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:34:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:34:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:34:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:34:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:34:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:34:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:34:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:34:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:34:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:34:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:34:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:34:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:34:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:34:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:34:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:34:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:34:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:34:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:34:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:34:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:34:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:34:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:34:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:34:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:34:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:34:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:34:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:34:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:34:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:34:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:34:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:34:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:34:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:34:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:34:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:34:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:34:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:34:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:34:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:34:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:34:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:34:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:34:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:34:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:34:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:34:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:34:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:34:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:34:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:34:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:34:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:34:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:34:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:34:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:34:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:34:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:34:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:35:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:35:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:35:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:35:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:35:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:35:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:35:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:35:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:35:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:35:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:35:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:35:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:35:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:35:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:35:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:35:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:35:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:35:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:35:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:35:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:35:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:35:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:35:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:35:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:35:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:35:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:35:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:35:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:35:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:35:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:35:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:35:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:35:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:35:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:35:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:35:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:35:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:35:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:35:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:35:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:35:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:35:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:35:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:35:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:35:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:35:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:35:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:35:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:35:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:35:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:35:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:35:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:35:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:35:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:35:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:35:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:35:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:35:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:35:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:35:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:35:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:35:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:35:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:35:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:35:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:35:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:35:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:35:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:35:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:35:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:35:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:35:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:35:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:35:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:35:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:35:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:35:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:35:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:35:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:35:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:35:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:35:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:35:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:35:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:35:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:35:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:35:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:35:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:35:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:35:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:35:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:35:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:35:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:35:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:35:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:35:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:35:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:35:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:35:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:35:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:35:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:35:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:35:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:35:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:35:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:35:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:35:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:35:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:35:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:35:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:35:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:35:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:35:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:35:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:35:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:35:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:35:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:35:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:35:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:35:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:35:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:35:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:35:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:35:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:35:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:35:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:35:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:35:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:35:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:35:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:35:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:35:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:35:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:35:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:35:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:35:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:35:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:35:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:35:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:35:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:35:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:35:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:35:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:35:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:35:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:35:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:35:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:35:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:35:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:35:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:35:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:35:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:35:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:35:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:35:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:35:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:35:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:35:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:35:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:35:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:35:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:35:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:35:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:35:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:35:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:35:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:35:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:35:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:35:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:35:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:35:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:35:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:35:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:35:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:35:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:35:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:35:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:35:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:35:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:35:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:35:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:35:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:35:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:35:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:35:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:35:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:35:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:35:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:35:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:35:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:35:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:35:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:35:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:35:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:35:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:35:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:35:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:35:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:35:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:35:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:35:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:35:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:35:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:35:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:35:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:35:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:35:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:35:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:35:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:35:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:35:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:35:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:35:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:35:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:35:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:35:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:36:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:36:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:36:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:36:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:36:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:36:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:36:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:36:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:36:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:36:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:36:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:36:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:36:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:36:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:36:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:36:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:36:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:36:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:36:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:36:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:36:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:36:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:36:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:36:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:36:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:36:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:36:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:36:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:36:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:36:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:36:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:36:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:36:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:36:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:36:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:36:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:36:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:36:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:36:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:36:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:36:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:36:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:36:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:36:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:36:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:36:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:36:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:36:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:36:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:36:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:36:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:36:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:36:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:36:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:36:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:36:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:36:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:36:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:36:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:36:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:36:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:36:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:36:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:36:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:36:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:36:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:36:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:36:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:36:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:36:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:36:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:36:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:36:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:36:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:36:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:36:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:36:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:36:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:36:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:36:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:36:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:36:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:36:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:36:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:36:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:36:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:36:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:36:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:36:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:36:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:36:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:36:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:36:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:36:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:36:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:36:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:36:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:36:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:36:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:36:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:36:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:36:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:36:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:36:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:36:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:36:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:36:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:36:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:36:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:36:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:36:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:36:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:36:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:36:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:36:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:36:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:36:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:36:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:36:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:36:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:36:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:36:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:36:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:36:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:36:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:36:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:36:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:36:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:36:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:36:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:36:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:36:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:36:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:36:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:36:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:36:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:36:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:36:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:36:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:36:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:36:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:36:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:36:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:36:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:36:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:36:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:36:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:36:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:36:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:36:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:36:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:36:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:36:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:36:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:36:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:36:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:36:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:36:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:36:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:36:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:36:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:36:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:36:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:36:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:36:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:36:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:36:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:36:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:36:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:36:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:36:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:36:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:36:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:36:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:36:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:36:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:36:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:36:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:36:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:36:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:36:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:36:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:36:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:36:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:36:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:36:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:36:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:36:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:36:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:36:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:36:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:36:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:36:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:36:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:36:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:36:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:36:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:36:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:36:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:36:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:36:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:36:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:36:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:36:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:36:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:36:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:36:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:36:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:36:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:36:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:36:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:36:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:36:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:36:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:36:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:36:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:37:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:37:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:37:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:37:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:37:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:37:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:37:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:37:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:37:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:37:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:37:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:37:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:37:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:37:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:37:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:37:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:37:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:37:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:37:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:37:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:37:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:37:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:37:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:37:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:37:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:37:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:37:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:37:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:37:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:37:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:37:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:37:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:37:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:37:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:37:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:37:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:37:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:37:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:37:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:37:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:37:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:37:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:37:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:37:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:37:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:37:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:37:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:37:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:37:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:37:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:37:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:37:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:37:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:37:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:37:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:37:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:37:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:37:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:37:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:37:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:37:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:37:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:37:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:37:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:37:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:37:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:37:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:37:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:37:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:37:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:37:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:37:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:37:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:37:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:37:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:37:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:37:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:37:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:37:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:37:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:37:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:37:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:37:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:37:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:37:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:37:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:37:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:37:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:37:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:37:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:37:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:37:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:37:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:37:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:37:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:37:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:37:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:37:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:37:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:37:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:37:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:37:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:37:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:37:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:37:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:37:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:37:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:37:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:37:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:37:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:37:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:37:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:37:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:37:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:37:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:37:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:37:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:37:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:37:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:37:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:37:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:37:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:37:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:37:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:37:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:37:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:37:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:37:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:37:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:37:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:37:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:37:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:37:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:37:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:37:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:37:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:37:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:37:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:37:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:37:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:37:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:37:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:37:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:37:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:37:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:37:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:37:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:37:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:37:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:37:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:37:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:37:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:37:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:37:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:37:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:37:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:37:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:37:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:37:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:37:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:37:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:37:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:37:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:37:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:37:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:37:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:37:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:37:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:37:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:37:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:37:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:37:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:37:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:37:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:37:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:37:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:37:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:37:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:37:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:37:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:39:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:39:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:39:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:39:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:39:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:39:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:39:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:39:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:39:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:39:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:39:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:39:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:39:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:39:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:39:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:39:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:39:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:39:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:39:54 rdma06 rsyslogd[1041]: imjournal: 5089 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 21:39:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:39:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:39:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:39:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:39:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:39:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:39:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:39:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:39:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:39:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:39:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:39:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:39:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:39:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:39:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:39:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:39:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:39:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:40:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:40:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:40:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:40:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:40:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:40:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:40:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:40:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:40:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:40:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:40:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:40:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:40:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:40:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:40:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:40:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:40:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:40:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:40:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:40:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:40:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:40:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:40:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:40:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:40:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:40:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:40:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:40:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:40:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:40:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:40:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:40:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:40:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:40:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:40:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:40:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:40:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:40:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:40:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:40:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:40:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:40:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:40:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:40:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:40:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:40:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:40:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:40:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:40:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:40:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:40:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:40:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:40:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:40:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:40:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:40:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:40:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:40:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:40:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:40:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:40:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:40:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:40:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:40:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:40:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:40:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:40:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:40:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:40:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:40:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:40:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:40:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:40:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:40:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:40:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:40:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:40:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:40:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:40:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:40:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:40:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:40:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:40:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:40:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:40:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:40:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:40:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:40:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:40:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:40:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:40:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:40:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:40:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:40:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:40:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:40:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:40:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:40:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:40:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:40:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:40:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:40:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:40:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:40:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:40:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:40:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:40:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:40:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:40:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:40:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:40:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:40:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:40:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:40:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:40:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:40:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:40:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:40:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:40:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:40:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:40:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:40:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:40:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:40:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:40:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:40:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:40:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:40:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:40:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:40:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:40:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:40:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:40:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:40:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:40:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:40:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:40:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:40:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:40:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:40:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:40:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:40:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:40:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:40:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:40:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:40:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:40:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:40:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:40:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:40:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:40:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:40:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:40:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:40:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:40:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:40:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:40:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:40:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:40:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:40:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:40:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:40:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:40:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:40:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:40:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:40:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:40:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:40:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:40:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:40:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:40:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:40:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:40:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:40:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:40:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:40:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:40:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:40:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:40:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:40:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:40:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:40:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:40:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:40:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:40:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:40:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:40:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:40:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:40:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:40:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:40:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:40:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:40:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:40:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:40:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:40:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:40:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:40:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:40:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:40:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:40:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:40:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:40:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:40:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:40:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:40:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:40:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:40:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:40:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:40:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:40:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:40:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:40:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:40:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:40:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:40:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:41:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:41:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:41:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:41:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:41:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:41:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:41:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:41:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:41:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:41:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:41:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:41:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:41:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:41:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:41:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:41:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:41:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:41:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:41:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:41:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:41:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:41:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:41:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:41:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:41:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:41:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:41:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:41:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:41:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:41:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:41:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:41:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:41:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:41:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:41:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:41:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:41:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:41:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:41:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:41:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:41:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:41:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:41:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:41:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:41:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:41:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:41:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:41:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:41:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:41:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:41:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:41:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:41:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:41:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:41:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:41:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:41:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:41:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:41:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:41:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:41:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:41:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:41:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:41:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:41:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:41:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:41:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:41:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:41:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:41:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:41:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:41:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:41:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:41:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:41:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:41:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:41:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:41:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:41:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:41:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:41:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:41:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:41:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:41:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:41:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:41:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:41:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:41:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:41:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:41:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:41:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:41:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:41:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:41:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:41:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:41:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:41:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:41:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:41:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:41:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:41:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:41:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:41:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:41:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:41:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:41:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:41:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:41:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:41:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:41:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:41:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:41:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:41:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:41:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:41:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:41:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:41:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:41:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:41:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:41:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:41:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:41:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:41:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:41:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:41:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:41:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:41:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:41:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:41:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:41:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:41:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:41:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:41:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:41:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:41:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:41:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:41:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:41:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:41:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:41:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:41:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:41:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:41:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:41:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:41:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:41:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:41:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:41:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:41:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:41:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:41:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:41:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:41:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:41:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:41:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:41:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:41:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:41:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:41:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:41:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:41:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:41:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:41:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:41:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:41:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:41:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:41:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:41:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:41:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:41:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:41:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:41:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:41:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:41:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:41:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:41:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:41:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:41:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:41:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:41:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:41:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:41:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:41:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:41:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:41:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:41:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:41:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:41:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:41:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:41:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:41:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:41:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:41:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:41:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:41:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:41:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:41:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:41:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:41:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:41:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:41:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:41:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:41:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:41:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:41:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:41:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:41:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:41:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:41:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:41:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:41:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:41:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:41:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:41:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:41:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:41:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:42:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:42:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:42:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:42:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:42:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:42:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:42:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:42:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:42:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:42:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:42:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:42:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:42:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:42:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:42:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:42:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:42:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:42:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:42:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:42:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:42:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:42:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:42:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:42:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:42:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:42:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:42:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:42:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:42:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:42:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:42:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:42:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:42:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:42:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:42:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:42:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:42:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:42:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:42:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:42:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:42:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:42:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:42:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:42:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:42:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:42:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:42:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:42:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:42:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:42:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:42:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:42:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:42:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:42:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:42:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:42:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:42:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:42:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:42:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:42:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:42:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:42:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:42:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:42:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:42:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:42:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:42:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:42:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:42:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:42:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:42:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:42:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:42:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:42:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:42:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:42:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:42:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:42:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:42:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:42:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:42:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:42:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:42:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:42:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:42:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:42:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:42:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:42:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:42:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:42:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:42:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:42:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:42:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:42:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:42:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:42:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:42:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:42:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:42:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:42:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:42:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:42:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:42:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:42:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:42:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:42:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:42:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:42:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:42:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:42:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:42:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:42:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:42:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:42:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:42:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:42:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:42:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:42:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:42:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:42:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:42:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:42:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:42:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:42:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:42:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:42:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:42:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:42:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:42:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:42:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:42:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:42:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:42:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:42:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:42:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:42:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:42:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:42:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:42:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:42:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:42:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:42:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:42:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:42:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:42:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:42:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:42:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:42:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:42:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:42:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:42:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:42:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:42:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:42:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:42:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:42:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:42:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:42:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:42:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:42:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:42:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:42:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:42:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:42:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:42:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:42:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:42:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:42:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:42:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:42:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:42:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:42:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:42:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:42:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:42:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:42:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:42:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:42:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:42:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:42:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:42:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:42:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:42:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:42:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:42:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:42:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:42:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:42:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:42:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:42:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:42:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:42:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:42:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:42:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:42:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:42:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:42:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:42:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:42:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:42:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:42:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:42:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:42:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:42:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:42:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:42:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:42:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:42:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:42:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:42:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:42:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:42:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:42:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:42:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:42:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:42:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:43:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:43:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:43:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:43:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:43:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:43:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:43:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:43:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:43:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:43:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:43:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:43:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:43:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:43:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:43:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:43:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:43:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:43:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:43:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:43:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:43:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:43:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:43:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:43:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:43:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:43:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:43:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:43:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:43:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:43:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:43:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:43:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:43:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:43:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:43:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:43:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:43:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:43:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:43:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:43:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:43:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:43:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:43:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:43:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:43:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:43:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:43:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:43:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:43:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:43:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:43:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:43:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:43:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:43:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:43:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:43:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:43:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:43:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:43:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:43:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:43:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:43:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:43:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:43:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:43:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:43:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:43:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:43:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:43:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:43:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:43:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:43:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:43:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:43:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:43:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:43:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:43:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:43:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:43:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:43:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:43:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:43:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:43:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:43:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:43:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:43:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:43:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:43:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:43:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:43:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:43:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:43:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:43:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:43:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:43:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:43:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:43:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:43:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:43:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:43:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:43:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:43:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:43:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:43:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:43:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:43:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:43:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:43:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:43:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:43:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:43:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:43:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:43:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:43:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:43:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:43:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:43:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:43:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:43:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:43:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:43:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:43:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:43:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:43:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:43:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:43:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:43:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:43:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:43:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:43:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:43:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:43:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:43:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:43:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:43:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:43:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:43:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:43:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:43:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:43:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:43:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:43:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:43:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:43:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:43:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:43:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:43:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:43:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:43:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:43:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:43:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:43:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:43:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:43:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:43:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:43:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:43:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:43:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:43:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:43:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:43:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:43:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:43:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:43:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:43:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:43:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:43:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:43:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:43:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:43:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:43:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:43:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:43:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:43:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:43:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:43:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:43:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:43:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:43:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:43:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:43:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:43:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:43:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:43:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:43:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:43:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:43:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:43:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:43:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:43:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:43:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:43:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:43:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:43:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:43:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:43:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:43:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:43:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:43:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:43:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:43:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:43:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:43:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:43:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:43:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:43:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:43:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:43:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:43:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:43:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:43:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:43:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:43:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:43:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:43:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:43:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:44:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:44:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:44:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:44:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:44:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:44:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:44:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:44:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:44:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:44:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:44:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:44:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:44:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:44:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:44:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:44:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:44:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:44:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:44:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:44:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:44:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:44:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:44:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:44:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:44:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:44:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:44:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:44:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:44:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:44:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:44:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:44:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:44:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:44:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:44:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:44:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:44:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:44:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:44:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:44:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:44:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:44:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:44:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:44:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:44:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:44:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:44:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:44:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:44:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:44:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:44:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:44:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:44:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:44:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:44:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:44:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:44:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:44:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:44:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:44:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:44:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:44:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:44:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:44:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:44:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:44:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:44:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:44:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:44:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:44:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:44:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:44:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:44:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:44:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:44:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:44:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:44:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:44:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:44:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:44:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:44:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:44:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:44:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:44:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:44:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:44:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:44:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:44:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:44:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:44:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:44:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:44:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:44:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:44:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:44:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:44:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:44:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:44:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:44:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:44:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:44:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:44:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:44:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:44:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:44:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:44:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:44:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:44:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:44:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:44:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:44:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:44:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:44:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:44:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:44:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:44:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:44:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:44:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:44:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:44:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:44:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:44:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:44:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:44:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:44:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:44:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:44:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:44:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:44:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:44:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:44:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:44:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:44:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:44:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:44:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:44:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:44:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:44:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:44:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:44:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:44:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:44:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:44:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:44:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:44:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:44:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:44:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:44:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:44:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:44:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:44:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:44:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:44:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:44:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:44:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:44:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:44:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:44:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:44:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:44:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:44:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:44:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:44:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:44:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:44:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:44:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:44:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:44:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:44:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:44:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:44:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:44:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:44:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:44:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:44:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:44:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:44:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:44:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:44:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:44:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:44:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:44:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:44:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:44:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:44:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:44:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:44:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:44:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:44:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:44:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:44:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:44:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:44:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:44:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:44:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:44:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:44:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:44:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:44:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:44:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:44:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:44:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:44:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:44:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:44:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:44:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:44:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:44:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:44:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:44:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:44:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:44:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:44:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:44:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:44:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:44:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:45:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:45:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:45:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:45:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:45:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:45:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:45:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:45:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:45:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:45:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:45:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:45:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:45:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:45:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:45:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:45:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:45:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:45:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:45:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:45:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:45:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:45:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:45:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:45:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:45:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:45:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:45:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:45:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:45:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:45:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:45:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:45:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:45:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:45:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:45:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:45:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:45:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:45:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:45:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:45:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:45:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:45:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:45:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:45:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:45:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:45:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:45:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:45:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:45:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:45:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:45:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:45:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:45:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:45:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:45:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:45:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:45:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:45:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:45:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:45:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:45:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:45:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:45:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:45:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:45:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:45:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:45:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:45:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:45:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:45:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:45:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:45:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:45:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:45:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:45:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:45:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:45:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:45:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:45:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:45:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:45:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:45:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:45:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:45:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:45:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:45:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:45:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:45:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:45:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:45:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:45:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:45:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:45:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:45:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:45:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:45:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:45:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:45:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:45:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:45:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:45:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:45:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:45:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:45:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:45:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:45:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:45:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:45:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:45:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:45:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:45:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:45:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:45:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:45:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:45:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:45:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:45:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:45:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:45:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:45:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:45:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:45:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:45:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:45:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:45:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:45:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:45:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:45:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:45:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:45:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:45:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:45:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:45:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:45:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:45:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:45:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:45:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:45:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:45:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:45:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:45:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:45:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:45:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:45:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:45:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:45:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:45:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:45:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:45:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:45:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:45:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:45:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:45:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:45:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:45:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:45:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:45:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:45:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:45:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:45:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:45:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:45:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:45:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:45:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:45:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:45:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:45:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:45:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:45:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:45:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:45:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:45:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:45:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:45:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:45:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:45:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:45:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:45:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:45:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:45:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:45:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:45:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:45:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:45:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:45:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:45:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:45:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:45:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:45:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:45:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:45:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:45:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:45:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:45:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:45:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:45:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:45:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:45:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:45:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:45:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:45:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:45:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:45:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:45:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:45:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:45:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:45:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:45:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:45:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:45:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:45:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:45:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:45:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:45:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:45:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:45:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:46:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:46:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:46:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:46:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:46:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:46:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:46:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:46:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:46:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:46:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:46:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:46:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:46:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:46:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:46:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:46:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:46:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:46:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:46:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:46:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:46:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:46:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:46:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:46:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:46:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:46:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:46:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:46:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:46:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:46:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:46:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:46:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:46:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:46:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:46:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:46:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:46:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:46:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:46:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:46:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:46:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:46:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:46:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:46:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:46:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:46:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:46:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:46:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:46:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:46:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:46:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:46:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:46:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:46:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:46:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:46:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:46:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:46:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:46:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:46:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:46:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:46:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:46:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:46:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:46:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:46:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:46:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:46:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:46:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:46:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:46:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:46:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:46:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:46:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:46:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:46:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:46:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:46:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:46:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:46:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:46:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:46:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:46:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:46:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:46:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:46:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:46:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:46:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:46:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:46:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:46:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:46:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:46:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:46:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:46:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:46:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:46:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:46:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:46:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:46:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:46:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:46:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:46:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:46:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:46:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:46:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:46:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:46:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:46:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:46:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:46:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:46:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:46:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:46:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:46:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:46:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:46:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:46:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:46:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:46:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:46:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:46:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:46:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:46:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:46:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:46:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:46:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:46:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:46:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:46:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:46:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:46:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:46:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:46:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:46:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:46:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:46:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:46:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:46:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:46:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:46:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:46:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:46:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:46:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:46:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:46:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:46:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:46:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:46:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:46:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:46:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:46:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:46:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:46:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:46:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:46:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:46:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:46:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:46:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:46:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:46:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:46:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:46:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:46:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:46:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:46:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:46:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:46:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:46:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:46:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:46:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:46:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:46:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:46:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:46:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:46:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:46:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:46:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:46:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:46:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:46:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:46:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:46:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:46:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:46:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:46:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:46:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:46:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:46:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:46:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:46:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:46:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:46:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:46:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:46:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:46:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:46:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:46:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:46:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:46:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:46:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:46:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:46:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:46:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:46:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:46:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:46:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:46:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:46:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:46:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:46:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:46:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:46:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:46:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:46:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:46:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:47:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:47:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:47:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:47:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:47:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:47:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:47:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:47:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:47:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:47:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:47:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:47:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:47:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:47:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:47:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:47:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:47:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:47:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:47:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:47:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:47:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:47:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:47:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:47:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:47:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:47:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:47:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:47:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:47:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:47:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:47:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:47:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:47:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:47:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:47:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:47:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:47:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:47:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:47:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:47:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:47:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:47:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:47:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:47:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:47:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:47:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:47:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:47:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:47:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:47:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:47:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:47:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:47:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:47:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:47:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:47:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:47:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:47:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:47:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:47:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:47:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:47:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:47:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:47:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:47:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:47:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:47:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:47:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:47:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:47:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:47:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:47:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:47:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:47:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:47:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:47:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:47:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:47:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:47:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:47:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:47:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:47:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:47:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:47:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:47:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:47:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:47:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:47:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:47:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:47:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:47:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:47:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:47:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:47:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:47:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:47:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:47:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:47:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:47:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:47:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:47:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:47:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:47:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:47:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:47:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:47:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:47:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:47:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:47:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:47:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:47:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:47:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:47:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:47:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:47:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:47:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:47:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:47:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:47:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:47:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:47:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:47:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:47:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:47:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:47:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:47:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:47:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:47:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:47:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:47:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:47:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:47:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:47:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:47:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:47:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:47:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:47:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:47:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:47:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:47:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:49:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:49:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:49:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:49:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:49:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:49:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:49:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:49:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:49:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:49:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:49:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:49:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:49:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:49:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:49:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:49:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:49:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:49:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:49:59 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 21:50:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:50:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:50:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:50:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:50:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:50:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:50:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:50:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:50:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:50:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:50:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:50:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:50:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:50:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:50:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:50:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:50:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:50:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:50:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:50:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:50:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:50:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:50:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:50:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:50:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:50:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:50:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:50:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:50:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:50:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:50:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:50:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:50:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:50:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:50:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:50:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:50:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:50:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:50:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:50:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:50:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:50:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:50:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:50:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:50:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:50:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:50:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:50:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:50:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:50:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:50:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:50:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:50:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:50:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:50:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:50:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:50:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:50:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:50:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:50:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:50:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:50:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:50:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:50:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:50:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:50:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:50:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:50:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:50:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:50:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:50:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:50:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:50:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:50:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:50:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:50:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:50:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:50:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:50:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:50:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:50:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:50:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:50:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:50:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:50:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:50:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:50:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:50:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:50:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:50:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:50:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:50:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:50:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:50:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:50:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:50:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:50:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:50:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:50:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:50:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:50:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:50:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:50:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:50:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:50:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:50:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:50:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:50:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:50:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:50:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:50:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:50:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:50:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:50:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:50:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:50:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:50:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:50:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:50:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:50:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:50:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:50:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:50:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:50:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:50:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:50:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:50:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:50:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:50:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:50:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:50:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:50:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:50:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:50:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:50:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:50:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:50:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:50:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:50:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:50:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:50:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:50:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:50:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:50:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:50:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:50:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:50:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:50:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:50:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:50:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:50:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:50:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:50:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:50:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:50:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:50:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:50:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:50:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:50:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:50:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:50:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:50:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:50:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:50:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:50:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:50:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:50:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:50:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:50:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:50:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:50:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:50:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:50:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:50:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:50:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:50:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:50:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:50:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:50:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:50:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:50:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:50:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:50:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:50:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:50:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:50:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:50:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:50:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:50:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:50:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:50:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:50:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:50:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:50:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:50:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:50:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:50:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:50:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:50:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:50:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:50:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:50:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:50:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:50:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:50:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:50:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:50:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:50:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:50:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:50:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:50:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:50:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:50:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:50:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:50:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:50:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:51:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:51:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:51:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:51:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:51:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:51:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:51:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:51:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:51:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:51:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:51:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:51:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:51:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:51:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:51:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:51:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:51:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:51:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:51:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:51:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:51:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:51:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:51:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:51:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:51:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:51:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:51:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:51:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:51:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:51:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:51:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:51:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:51:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:51:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:51:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:51:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:51:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:51:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:51:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:51:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:51:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:51:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:51:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:51:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:51:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:51:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:51:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:51:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:51:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:51:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:51:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:51:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:51:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:51:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:51:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:51:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:51:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:51:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:51:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:51:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:51:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:51:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:51:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:51:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:51:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:51:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:51:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:51:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:51:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:51:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:51:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:51:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:51:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:51:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:51:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:51:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:51:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:51:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:51:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:51:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:51:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:51:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:51:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:51:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:51:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:51:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:51:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:51:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:51:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:51:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:51:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:51:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:51:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:51:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:51:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:51:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:51:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:51:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:51:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:51:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:51:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:51:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:51:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:51:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:51:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:51:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:51:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:51:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:51:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:51:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:51:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:51:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:51:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:51:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:51:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:51:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:51:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:51:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:51:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:51:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:51:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:51:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:51:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:51:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:51:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:51:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:51:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:51:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:51:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:51:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:51:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:51:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:51:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:51:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:51:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:51:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:51:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:51:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:51:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:51:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:51:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:51:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:51:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:51:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:51:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:51:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:51:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:51:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:51:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:51:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:51:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:51:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:51:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:51:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:51:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:51:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:51:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:51:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:51:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:51:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:51:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:51:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:51:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:51:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:51:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:51:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:51:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:51:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:51:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:51:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:51:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:51:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:51:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:51:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:51:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:51:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:51:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:51:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:51:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:51:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:51:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:51:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:51:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:51:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:51:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:51:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:51:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:51:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:51:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:51:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:51:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:51:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:51:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:51:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:51:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:51:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:51:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:51:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:51:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:51:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:51:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:51:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:51:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:51:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:51:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:51:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:51:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:51:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:51:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:51:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:51:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:51:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:51:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:51:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:51:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:51:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:52:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:52:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:52:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:52:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:52:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:52:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:52:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:52:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:52:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:52:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:52:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:52:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:52:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:52:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:52:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:52:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:52:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:52:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:52:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:52:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:52:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:52:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:52:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:52:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:52:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:52:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:52:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:52:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:52:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:52:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:52:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:52:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:52:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:52:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:52:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:52:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:52:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:52:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:52:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:52:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:52:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:52:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:52:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:52:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:52:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:52:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:52:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:52:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:52:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:52:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:52:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:52:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:52:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:52:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:52:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:52:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:52:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:52:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:52:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:52:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:52:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:52:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:52:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:52:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:52:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:52:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:52:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:52:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:52:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:52:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:52:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:52:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:52:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:52:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:52:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:52:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:52:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:52:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:52:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:52:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:52:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:52:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:52:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:52:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:52:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:52:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:52:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:52:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:52:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:52:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:52:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:52:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:52:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:52:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:52:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:52:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:52:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:52:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:52:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:52:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:52:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:52:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:52:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:52:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:52:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:52:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:52:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:52:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:52:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:52:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:52:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:52:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:52:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:52:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:52:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:52:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:52:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:52:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:52:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:52:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:52:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:52:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:52:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:52:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:52:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:52:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:52:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:52:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:52:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:52:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:52:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:52:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:52:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:52:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:52:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:52:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:52:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:52:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:52:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:52:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:52:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:52:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:52:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:52:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:52:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:52:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:52:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:52:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:52:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:52:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:52:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:52:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:52:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:52:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:52:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:52:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:52:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:52:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:52:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:52:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:52:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:52:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:52:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:52:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:52:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:52:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:52:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:52:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:52:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:52:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:52:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:52:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:52:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:52:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:52:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:52:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:52:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:52:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:52:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:52:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:52:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:52:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:52:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:52:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:52:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:52:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:52:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:52:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:52:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:52:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:52:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:52:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:52:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:52:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:52:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:52:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:52:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:52:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:52:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:52:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:52:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:52:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:52:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:52:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:52:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:52:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:52:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:52:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:52:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:52:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:52:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:52:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:52:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:52:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:52:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:52:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:53:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:53:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:53:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:53:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:53:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:53:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:53:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:53:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:53:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:53:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:53:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:53:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:53:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:53:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:53:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:53:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:53:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:53:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:53:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:53:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:53:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:53:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:53:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:53:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:53:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:53:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:53:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:53:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:53:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:53:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:53:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:53:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:53:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:53:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:53:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:53:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:53:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:53:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:53:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:53:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:53:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:53:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:53:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:53:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:53:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:53:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:53:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:53:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:53:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:53:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:53:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:53:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:53:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:53:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:53:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:53:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:53:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:53:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:53:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:53:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:53:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:53:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:53:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:53:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:53:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:53:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:53:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:53:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:53:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:53:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:53:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:53:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:53:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:53:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:53:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:53:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:53:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:53:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:53:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:53:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:53:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:53:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:53:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:53:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:53:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:53:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:53:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:53:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:53:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:53:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:53:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:53:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:53:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:53:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:53:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:53:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:53:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:53:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:53:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:53:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:53:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:53:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:53:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:53:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:53:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:53:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:53:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:53:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:53:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:53:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:53:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:53:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:53:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:53:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:53:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:53:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:53:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:53:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:53:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:53:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:53:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:53:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:53:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:53:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:53:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:53:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:53:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:53:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:53:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:53:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:53:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:53:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:53:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:53:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:53:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:53:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:54:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:54:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:54:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:54:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:54:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:54:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:54:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:54:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:54:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:54:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:54:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:54:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:54:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:54:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:54:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:54:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:54:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:54:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:54:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:54:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:54:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:54:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:54:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:54:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:54:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:54:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:54:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:54:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:54:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:54:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:54:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:54:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:54:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:54:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:54:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:54:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:54:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:54:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:54:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:54:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:54:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:54:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:54:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:54:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:54:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:54:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:54:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:54:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:54:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:54:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:54:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:54:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:54:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:54:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:54:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:54:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:54:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:54:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:54:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:54:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:54:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:54:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:54:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:54:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:54:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:54:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:54:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:54:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:54:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:54:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:54:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:54:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:54:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:54:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:54:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:54:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:54:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:54:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:54:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:54:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:54:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:54:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:54:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:54:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:54:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:54:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:54:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:54:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:54:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:54:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:54:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:54:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:54:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:54:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:54:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:54:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:54:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:54:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:54:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:54:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:54:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:54:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:54:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:54:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:54:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:54:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:54:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:54:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:54:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:54:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:54:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:54:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:54:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:54:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:54:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:54:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:54:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:54:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:54:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:54:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:54:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:54:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:54:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:54:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:54:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:54:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:54:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:54:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:54:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:54:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:54:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:54:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:54:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:54:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:54:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:54:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:54:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:54:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:54:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:54:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:54:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:54:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:54:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:54:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:54:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:54:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:54:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:54:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:54:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:54:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:54:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:54:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:54:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:54:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:54:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:54:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:54:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:54:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:54:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:54:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:54:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:54:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:54:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:54:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:54:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:54:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:54:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:54:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:54:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:54:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:54:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:54:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:54:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:54:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:54:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:54:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:55:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:55:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:55:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:55:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:55:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:55:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:55:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:55:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:55:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:55:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:55:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:55:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:55:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:55:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:55:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:55:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:55:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:55:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:55:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:55:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:55:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:55:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:55:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:55:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:55:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:55:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:55:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:55:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:55:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:55:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:55:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:55:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:55:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:55:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:55:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:55:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:55:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:55:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:55:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:55:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:55:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:55:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:55:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:55:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:55:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:55:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:55:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:55:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:55:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:55:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:55:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:55:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:55:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:55:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:55:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:55:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:55:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:55:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:55:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:55:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:55:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:55:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:55:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:55:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:55:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:55:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:55:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:55:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:55:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:55:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:55:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:55:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:55:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:55:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:55:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:55:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:55:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:55:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:55:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:55:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:55:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:55:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:55:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:55:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:55:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:55:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:55:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:55:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:55:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:55:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:55:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:55:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:55:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:55:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:55:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:55:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:55:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:55:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:55:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:55:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:55:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:55:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:55:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:55:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:55:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:55:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:55:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:55:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:55:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:55:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:55:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:55:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:55:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:55:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:55:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:55:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:55:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:55:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:55:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:55:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:55:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:55:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:55:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:55:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:55:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:55:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:55:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:55:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:55:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:55:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:55:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:55:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:55:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:55:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:55:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:55:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:55:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:55:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:55:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:55:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:55:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:55:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:55:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:55:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:55:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:55:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:55:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:55:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:55:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:55:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:55:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:55:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:55:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:55:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:55:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:55:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:55:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:55:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:55:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:55:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:55:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:55:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:55:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:55:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:55:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:55:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:55:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:55:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:55:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:55:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:55:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:55:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:55:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:55:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:55:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:55:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:55:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:55:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:55:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:55:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:55:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:55:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:55:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:55:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:55:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:55:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:55:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:55:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:55:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:55:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:55:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:55:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:55:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:55:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:55:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:55:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:55:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:55:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:55:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:55:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:55:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:55:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:55:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:55:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:55:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:55:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:55:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:55:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:55:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:55:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:55:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:55:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:55:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:55:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:55:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:55:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:56:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:56:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:56:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:56:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:56:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:56:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:56:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:56:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:56:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:56:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:56:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:56:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:56:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:56:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:56:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:56:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:56:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:56:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:56:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:56:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:56:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:56:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:56:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:56:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:56:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:56:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:56:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:56:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:56:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:56:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:56:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:56:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:56:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:56:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:56:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:56:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:56:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:56:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:56:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:56:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:56:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:56:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:56:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:56:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:56:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:56:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:56:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:56:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:56:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:56:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:56:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:56:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:56:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:56:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:56:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:56:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:56:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:56:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:56:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:56:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:56:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:56:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:56:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:56:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:56:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:56:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:56:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:56:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:56:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:56:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:56:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:56:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:56:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:56:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:56:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:56:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:56:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:56:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:56:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:56:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:56:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:56:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:56:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:56:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:56:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:56:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:56:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:56:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:56:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:56:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:56:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:56:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:56:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:56:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:56:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:56:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:56:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:56:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:56:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:56:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:56:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:56:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:56:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:56:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:56:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:56:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:56:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:56:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:56:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:56:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:56:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:56:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:56:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:56:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:56:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:56:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:56:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:56:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:56:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:56:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:56:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:56:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:56:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:56:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:56:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:56:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:56:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:56:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:56:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:56:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:56:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:56:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:56:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:56:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:56:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:56:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:56:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:56:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:56:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:56:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:56:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:56:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:56:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:56:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:56:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:56:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:56:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:56:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:56:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:56:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:56:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:56:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:56:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:56:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:56:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:56:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:56:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:56:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:56:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:56:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:56:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:56:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:56:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:56:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:56:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:56:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:56:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:56:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:56:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:56:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:56:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:56:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:56:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:56:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:56:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:56:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:56:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:56:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:56:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:56:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:56:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:56:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:56:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:56:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:56:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:56:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:56:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:56:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:56:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:56:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:56:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:56:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:56:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:56:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:56:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:56:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:56:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:56:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:56:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:56:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:56:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:56:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:56:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:56:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:56:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:56:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:56:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:56:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:56:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:56:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:56:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:56:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:56:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:56:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:56:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:56:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:57:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:57:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:57:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:57:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:57:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:57:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:57:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:57:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:57:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:57:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:57:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:57:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:57:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:57:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:57:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:57:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:57:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:57:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:57:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:57:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:57:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:57:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:57:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:57:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:57:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:57:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:57:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:57:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:57:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:57:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:57:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:57:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:57:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:57:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:57:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:57:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:57:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:57:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:57:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:57:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:57:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:57:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:57:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:57:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:57:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:57:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:57:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:57:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:57:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:57:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:57:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:57:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:57:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:57:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:57:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:57:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:57:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:57:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:57:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:57:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:57:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:57:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:57:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:57:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:57:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:57:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:57:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:57:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:57:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:57:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:57:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:57:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:57:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:57:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:57:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:57:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:57:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:57:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:57:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:57:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:57:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:57:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:57:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:57:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:57:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:57:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:57:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:57:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:57:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:57:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:57:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:57:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:57:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:57:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:57:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:57:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:57:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:57:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:57:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:57:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:57:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:57:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:57:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:57:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:57:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:57:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:57:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:57:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:57:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:57:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:57:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:57:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:57:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:57:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:57:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:57:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:57:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:57:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:57:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:57:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:57:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:57:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:57:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:57:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:57:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:57:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:57:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:57:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:57:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:57:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:57:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:57:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:57:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:57:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:57:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:57:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:57:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:57:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:57:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:57:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:57:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:57:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:57:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:57:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:57:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:57:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:57:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:57:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:57:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:57:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:57:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:57:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:57:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:57:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:57:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:57:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:57:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:57:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:57:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:57:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:57:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:57:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:57:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:57:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:57:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:57:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:57:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:57:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:57:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:57:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:57:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:57:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:57:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:57:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:57:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:57:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:57:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:57:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:57:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:57:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 21:57:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 21:57:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:57:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:57:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:57:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 21:57:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 21:57:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 21:57:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 21:57:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 21:57:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 21:57:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 21:57:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 21:57:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 21:57:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 21:57:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 21:57:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 21:57:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 21:57:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:00:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:00:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:00:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:00:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:00:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:00:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:00:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:00:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:00:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:00:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:00:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:00:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:00:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:00:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:00:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:00:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:00:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:00:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:00:04 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 22:00:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:00:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:00:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:00:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:00:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:00:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:00:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:00:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:00:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:00:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:00:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:00:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:00:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:00:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:00:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:00:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:00:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:00:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:00:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:00:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:00:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:00:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:00:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:00:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:00:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:00:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:00:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:00:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:00:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:00:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:00:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:00:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:00:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:00:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:00:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:00:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:00:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:00:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:00:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:00:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:00:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:00:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:00:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:00:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:00:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:00:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:00:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:00:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:00:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:00:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:00:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:00:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:00:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:00:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:00:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:00:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:00:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:00:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:00:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:00:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:00:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:00:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:00:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:00:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:00:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:00:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:00:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:00:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:00:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:00:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:00:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:00:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:00:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:00:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:00:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:00:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:00:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:00:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:00:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:00:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:00:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:00:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:00:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:00:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:00:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:00:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:00:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:00:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:00:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:00:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:00:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:00:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:00:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:00:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:00:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:00:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:00:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:00:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:00:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:00:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:00:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:00:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:00:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:00:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:00:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:00:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:00:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:00:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:00:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:00:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:00:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:00:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:00:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:00:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:00:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:00:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:00:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:00:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:00:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:00:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:00:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:00:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:00:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:00:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:00:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:00:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:00:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:00:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:00:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:00:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:00:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:00:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:00:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:00:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:00:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:00:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:00:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:00:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:00:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:00:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:00:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:00:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:00:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:00:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:00:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:00:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:00:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:00:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:00:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:00:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:00:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:00:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:00:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:00:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:00:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:00:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:00:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:00:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:00:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:00:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:00:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:00:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:00:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:00:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:00:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:00:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:00:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:00:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:01:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:01:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:01:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:01:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:01:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:01:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:01:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:01:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:01:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:01:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:01:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:01:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:01:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:01:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:01:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:01:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:01:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:01:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:01:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:01:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:01:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:01:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:01:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:01:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:01:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:01:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:01:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:01:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:01:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:01:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:01:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:01:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:01:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:01:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:01:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:01:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:01:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:01:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:01:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:01:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:01:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:01:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:01:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:01:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:01:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:01:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:01:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:01:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:01:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:01:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:01:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:01:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:01:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:01:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:01:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:01:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:01:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:01:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:01:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:01:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:01:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:01:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:01:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:01:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:01:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:01:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:01:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:01:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:01:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:01:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:01:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:01:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:01:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:01:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:01:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:01:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:01:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:01:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:01:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:01:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:01:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:01:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:01:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:01:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:01:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:01:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:01:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:01:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:01:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:01:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:01:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:01:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:01:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:01:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:01:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:01:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:01:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:01:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:01:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:01:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:01:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:01:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:01:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:01:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:01:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:01:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:01:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:01:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:01:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:01:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:01:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:01:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:01:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:01:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:01:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:01:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:01:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:01:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:01:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:01:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:01:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:01:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:01:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:01:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:01:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:01:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:01:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:01:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:01:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:01:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:01:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:01:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:01:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:01:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:01:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:01:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:01:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:01:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:01:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:01:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:01:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:01:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:01:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:01:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:01:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:01:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:01:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:01:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:01:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:01:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:01:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:01:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:01:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:01:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:01:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:01:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:01:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:01:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:01:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:01:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:01:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:01:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:01:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:01:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:01:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:01:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:01:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:01:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:01:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:01:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:01:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:01:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:01:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:01:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:01:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:01:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:01:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:01:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:01:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:01:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:01:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:01:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:01:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:01:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:01:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:01:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:01:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:01:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:01:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:01:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:01:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:01:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:01:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:01:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:01:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:01:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:01:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:01:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:01:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:01:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:01:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:01:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:01:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:01:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:01:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:01:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:01:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:01:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:01:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:01:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:01:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:01:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:01:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:01:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:01:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:01:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:02:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:02:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:02:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:02:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:02:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:02:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:02:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:02:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:02:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:02:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:02:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:02:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:02:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:02:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:02:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:02:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:02:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:02:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:02:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:02:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:02:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:02:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:02:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:02:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:02:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:02:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:02:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:02:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:02:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:02:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:02:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:02:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:02:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:02:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:02:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:02:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:02:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:02:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:02:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:02:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:02:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:02:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:02:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:02:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:02:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:02:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:02:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:02:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:02:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:02:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:02:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:02:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:02:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:02:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:02:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:02:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:02:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:02:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:02:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:02:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:02:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:02:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:02:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:02:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:02:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:02:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:02:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:02:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:02:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:02:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:02:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:02:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:02:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:02:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:02:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:02:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:02:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:02:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:02:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:02:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:02:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:02:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:02:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:02:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:02:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:02:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:02:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:02:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:02:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:02:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:02:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:02:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:02:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:02:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:02:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:02:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:02:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:02:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:02:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:02:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:02:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:02:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:02:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:02:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:02:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:02:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:02:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:02:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:02:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:02:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:02:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:02:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:02:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:02:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:02:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:02:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:02:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:02:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:02:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:02:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:02:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:02:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:02:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:02:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:02:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:02:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:02:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:02:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:02:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:02:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:02:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:02:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:02:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:02:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:02:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:02:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:02:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:02:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:02:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:02:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:02:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:02:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:02:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:02:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:02:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:02:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:02:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:02:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:02:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:02:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:02:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:02:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:02:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:02:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:02:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:02:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:02:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:02:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:02:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:02:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:02:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:02:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:02:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:02:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:02:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:02:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:02:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:02:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:02:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:02:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:02:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:02:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:02:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:02:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:02:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:02:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:02:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:02:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:02:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:02:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:02:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:02:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:02:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:02:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:02:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:02:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:02:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:02:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:02:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:02:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:02:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:02:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:02:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:02:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:02:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:02:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:02:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:02:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:02:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:02:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:02:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:02:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:02:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:02:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:02:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:02:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:02:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:02:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:02:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:02:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:02:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:02:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:02:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:02:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:02:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:02:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:03:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:03:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:03:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:03:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:03:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:03:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:03:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:03:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:03:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:03:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:03:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:03:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:03:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:03:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:03:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:03:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:03:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:03:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:03:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:03:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:03:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:03:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:03:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:03:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:03:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:03:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:03:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:03:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:03:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:03:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:03:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:03:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:03:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:03:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:03:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:03:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:03:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:03:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:03:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:03:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:03:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:03:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:03:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:03:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:03:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:03:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:03:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:03:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:03:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:03:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:03:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:03:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:03:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:03:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:03:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:03:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:03:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:03:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:03:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:03:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:03:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:03:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:03:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:03:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:03:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:03:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:03:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:03:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:03:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:03:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:03:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:03:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:03:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:03:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:03:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:03:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:03:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:03:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:03:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:03:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:03:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:03:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:03:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:03:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:03:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:03:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:03:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:03:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:03:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:03:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:03:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:03:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:03:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:03:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:03:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:03:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:03:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:03:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:03:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:03:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:03:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:03:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:03:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:03:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:03:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:03:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:03:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:03:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:03:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:03:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:03:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:03:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:03:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:03:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:03:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:03:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:03:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:03:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:03:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:03:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:03:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:03:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:03:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:03:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:03:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:03:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:03:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:03:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:03:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:03:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:03:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:03:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:03:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:03:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:03:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:03:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:03:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:03:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:03:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:03:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:03:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:03:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:03:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:03:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:03:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:03:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:03:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:03:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:03:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:03:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:03:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:03:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:03:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:03:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:03:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:03:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:03:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:03:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:03:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:03:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:03:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:03:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:03:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:03:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:03:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:03:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:03:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:03:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:03:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:03:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:03:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:03:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:03:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:03:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:03:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:03:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:03:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:03:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:03:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:03:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:03:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:03:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:03:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:03:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:03:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:03:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:03:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:03:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:03:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:03:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:03:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:03:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:03:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:03:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:03:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:03:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:03:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:03:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:03:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:03:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:03:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:03:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:03:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:03:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:03:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:03:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:03:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:03:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:03:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:03:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:03:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:03:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:03:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:03:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:03:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:03:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:04:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:04:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:04:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:04:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:04:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:04:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:04:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:04:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:04:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:04:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:04:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:04:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:04:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:04:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:04:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:04:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:04:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:04:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:04:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:04:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:04:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:04:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:04:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:04:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:04:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:04:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:04:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:04:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:04:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:04:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:04:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:04:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:04:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:04:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:04:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:04:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:04:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:04:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:04:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:04:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:04:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:04:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:04:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:04:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:04:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:04:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:04:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:04:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:04:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:04:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:04:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:04:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:04:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:04:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:04:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:04:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:04:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:04:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:04:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:04:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:04:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:04:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:04:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:04:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:04:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:04:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:04:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:04:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:04:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:04:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:04:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:04:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:04:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:04:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:04:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:04:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:04:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:04:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:04:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:04:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:04:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:04:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:04:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:04:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:04:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:04:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:04:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:04:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:04:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:04:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:04:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:04:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:04:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:04:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:04:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:04:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:04:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:04:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:04:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:04:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:04:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:04:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:04:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:04:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:04:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:04:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:04:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:04:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:04:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:04:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:04:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:04:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:04:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:04:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:04:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:04:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:04:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:04:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:04:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:04:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:04:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:04:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:04:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:04:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:04:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:04:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:04:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:04:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:04:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:04:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:04:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:04:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:04:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:04:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:04:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:04:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:04:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:04:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:04:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:04:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:04:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:04:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:04:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:04:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:04:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:04:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:04:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:04:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:04:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:04:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:04:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:04:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:04:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:04:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:04:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:04:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:04:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:04:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:04:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:04:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:04:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:04:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:04:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:04:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:04:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:04:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:04:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:04:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:04:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:04:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:04:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:04:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:04:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:04:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:04:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:04:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:04:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:04:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:04:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:04:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:04:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:04:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:04:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:04:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:04:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:04:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:04:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:04:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:04:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:04:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:04:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:04:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:04:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:04:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:04:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:04:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:04:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:04:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:04:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:04:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:04:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:04:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:04:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:04:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:04:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:04:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:04:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:04:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:04:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:04:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:04:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:04:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:04:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:04:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:04:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:04:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:05:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:05:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:05:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:05:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:05:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:05:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:05:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:05:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:05:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:05:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:05:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:05:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:05:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:05:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:05:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:05:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:05:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:05:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:05:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:05:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:05:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:05:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:05:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:05:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:05:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:05:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:05:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:05:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:05:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:05:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:05:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:05:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:05:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:05:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:05:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:05:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:05:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:05:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:05:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:05:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:05:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:05:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:05:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:05:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:05:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:05:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:05:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:05:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:05:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:05:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:05:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:05:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:05:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:05:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:05:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:05:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:05:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:05:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:05:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:05:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:05:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:05:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:05:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:05:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:05:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:05:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:05:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:05:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:05:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:05:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:05:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:05:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:05:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:05:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:05:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:05:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:05:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:05:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:05:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:05:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:05:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:05:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:05:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:05:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:05:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:05:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:05:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:05:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:05:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:05:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:05:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:05:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:05:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:05:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:05:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:05:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:05:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:05:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:05:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:05:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:05:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:05:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:05:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:05:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:05:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:05:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:05:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:05:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:05:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:05:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:05:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:05:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:05:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:05:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:05:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:05:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:05:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:05:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:05:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:05:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:05:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:05:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:05:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:05:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:05:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:05:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:05:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:05:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:05:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:05:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:05:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:05:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:05:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:05:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:05:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:05:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:05:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:05:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:05:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:05:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:05:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:05:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:05:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:05:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:05:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:05:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:05:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:05:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:05:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:05:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:05:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:05:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:05:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:05:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:05:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:05:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:05:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:05:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:05:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:05:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:05:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:05:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:05:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:05:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:05:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:05:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:05:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:05:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:05:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:05:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:05:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:05:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:05:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:05:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:05:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:05:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:05:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:05:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:05:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:05:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:05:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:05:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:05:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:05:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:05:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:05:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:05:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:05:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:05:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:05:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:05:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:05:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:05:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:05:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:05:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:05:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:05:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:05:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:05:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:05:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:05:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:05:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:05:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:05:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:05:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:05:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:05:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:05:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:05:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:05:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:05:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:05:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:05:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:05:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:05:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:05:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:06:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:06:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:06:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:06:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:06:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:06:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:06:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:06:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:06:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:06:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:06:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:06:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:06:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:06:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:06:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:06:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:06:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:06:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:06:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:06:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:06:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:06:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:06:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:06:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:06:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:06:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:06:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:06:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:06:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:06:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:06:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:06:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:06:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:06:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:06:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:06:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:06:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:06:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:06:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:06:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:06:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:06:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:06:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:06:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:06:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:06:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:06:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:06:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:06:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:06:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:06:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:06:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:06:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:06:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:06:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:06:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:06:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:06:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:06:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:06:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:06:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:06:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:06:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:06:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:06:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:06:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:06:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:06:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:06:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:06:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:06:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:06:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:06:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:06:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:06:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:06:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:06:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:06:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:06:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:06:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:06:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:06:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:06:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:06:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:06:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:06:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:06:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:06:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:06:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:06:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:06:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:06:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:06:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:06:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:06:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:06:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:06:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:06:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:06:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:06:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:06:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:06:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:06:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:06:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:06:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:06:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:06:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:06:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:06:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:06:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:06:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:06:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:06:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:06:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:06:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:06:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:06:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:06:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:06:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:06:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:06:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:06:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:06:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:06:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:06:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:06:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:06:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:06:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:06:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:06:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:06:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:06:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:06:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:06:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:06:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:06:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:06:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:06:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:06:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:06:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:06:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:06:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:06:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:06:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:06:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:06:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:06:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:06:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:06:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:06:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:06:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:06:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:06:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:06:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:06:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:06:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:06:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:06:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:06:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:06:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:06:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:06:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:06:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:06:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:06:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:06:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:06:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:06:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:06:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:06:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:06:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:06:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:06:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:06:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:06:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:06:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:06:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:06:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:06:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:06:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:06:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:06:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:06:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:06:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:06:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:06:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:06:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:06:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:06:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:06:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:06:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:06:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:06:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:06:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:06:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:06:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:06:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:06:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:06:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:06:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:06:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:06:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:06:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:06:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:06:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:06:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:06:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:06:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:06:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:06:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:06:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:06:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:06:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:06:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:06:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:06:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:07:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:07:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:07:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:07:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:07:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:07:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:07:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:07:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:07:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:07:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:07:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:07:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:07:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:07:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:07:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:07:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:07:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:07:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:07:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:07:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:07:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:07:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:07:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:07:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:07:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:07:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:07:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:07:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:07:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:07:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:07:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:07:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:07:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:07:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:07:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:07:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:07:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:07:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:07:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:07:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:07:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:07:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:07:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:07:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:07:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:07:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:07:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:07:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:07:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:07:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:07:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:07:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:07:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:07:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:07:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:07:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:07:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:07:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:07:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:07:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:07:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:07:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:07:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:07:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:07:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:07:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:07:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:07:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:07:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:07:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:07:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:07:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:07:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:07:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:07:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:07:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:07:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:07:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:07:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:07:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:07:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:07:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:07:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:07:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:07:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:07:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:07:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:07:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:07:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:07:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:07:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:07:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:07:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:07:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:07:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:07:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:07:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:07:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:07:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:07:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:07:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:07:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:07:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:07:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:07:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:07:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:07:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:07:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:07:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:07:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:07:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:07:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:07:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:07:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:07:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:07:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:07:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:07:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:07:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:07:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:07:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:07:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:07:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:07:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:07:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:07:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:07:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:07:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:07:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:07:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:07:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:07:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:07:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:07:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:07:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:07:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:07:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:07:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:07:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:07:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:07:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:07:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:07:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:07:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:07:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:07:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:07:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:07:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:07:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:07:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:07:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:07:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:07:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:07:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:07:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:07:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:07:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:07:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:07:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:07:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:07:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:07:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:07:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:07:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:07:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:07:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:07:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:07:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:07:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:07:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:07:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:07:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:07:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:07:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:07:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:07:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:07:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:07:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:07:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:07:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:07:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:07:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:07:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:07:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:07:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:07:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:07:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:07:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:07:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:07:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:07:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:07:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:07:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:07:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:07:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:07:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:07:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:07:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:07:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:07:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:07:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:07:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:07:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:07:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:07:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:07:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:07:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:07:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:07:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:07:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:07:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:07:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:07:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:07:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:07:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:07:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:10:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:10:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:10:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:10:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:10:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:10:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:10:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:10:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:10:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:10:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:10:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:10:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:10:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:10:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:10:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:10:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:10:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:10:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:10:05 rdma06 rsyslogd[1041]: imjournal: 5086 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 22:10:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:10:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:10:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:10:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:10:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:10:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:10:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:10:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:10:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:10:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:10:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:10:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:10:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:10:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:10:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:10:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:10:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:10:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:10:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:10:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:10:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:10:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:10:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:10:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:10:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:10:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:10:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:10:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:10:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:10:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:10:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:10:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:10:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:10:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:10:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:10:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:10:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:10:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:10:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:10:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:10:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:10:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:10:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:10:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:10:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:10:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:10:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:10:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:10:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:10:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:10:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:10:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:10:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:10:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:10:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:10:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:10:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:10:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:10:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:10:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:10:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:10:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:10:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:10:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:10:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:10:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:10:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:10:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:10:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:10:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:10:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:10:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:10:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:10:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:10:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:10:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:10:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:10:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:10:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:10:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:10:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:10:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:10:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:10:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:10:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:10:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:10:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:10:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:10:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:10:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:10:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:10:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:10:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:10:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:10:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:10:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:10:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:10:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:10:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:10:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:10:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:10:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:10:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:10:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:10:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:10:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:10:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:10:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:10:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:10:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:10:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:10:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:10:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:10:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:10:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:10:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:10:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:10:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:10:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:10:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:10:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:10:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:10:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:10:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:10:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:10:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:10:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:10:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:10:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:10:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:10:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:10:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:10:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:10:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:10:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:10:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:10:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:10:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:10:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:10:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:10:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:10:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:10:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:10:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:10:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:10:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:10:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:10:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:10:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:10:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:10:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:10:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:10:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:10:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:10:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:10:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:10:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:10:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:10:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:10:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:10:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:10:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:10:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:10:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:10:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:10:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:10:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:10:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:10:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:10:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:10:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:10:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:10:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:10:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:10:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:10:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:10:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:10:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:10:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:10:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:10:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:10:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:10:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:10:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:10:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:10:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:10:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:10:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:10:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:10:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:10:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:10:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:10:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:10:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:10:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:10:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:10:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:10:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:11:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:11:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:11:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:11:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:11:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:11:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:11:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:11:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:11:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:11:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:11:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:11:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:11:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:11:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:11:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:11:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:11:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:11:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:11:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:11:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:11:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:11:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:11:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:11:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:11:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:11:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:11:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:11:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:11:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:11:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:11:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:11:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:11:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:11:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:11:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:11:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:11:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:11:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:11:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:11:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:11:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:11:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:11:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:11:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:11:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:11:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:11:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:11:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:11:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:11:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:11:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:11:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:11:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:11:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:11:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:11:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:11:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:11:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:11:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:11:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:11:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:11:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:11:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:11:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:11:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:11:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:11:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:11:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:11:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:11:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:11:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:11:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:11:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:11:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:11:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:11:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:11:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:11:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:11:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:11:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:11:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:11:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:11:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:11:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:11:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:11:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:11:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:11:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:11:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:11:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:11:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:11:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:11:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:11:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:11:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:11:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:11:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:11:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:11:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:11:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:11:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:11:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:11:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:11:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:11:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:11:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:11:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:11:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:11:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:11:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:11:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:11:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:11:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:11:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:11:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:11:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:11:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:11:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:11:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:11:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:11:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:11:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:11:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:11:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:11:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:11:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:11:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:11:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:11:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:11:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:11:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:11:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:11:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:11:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:11:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:11:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:11:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:11:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:11:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:11:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:11:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:11:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:11:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:11:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:11:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:11:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:11:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:11:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:11:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:11:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:11:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:11:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:11:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:11:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:11:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:11:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:11:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:11:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:11:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:11:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:11:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:11:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:11:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:11:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:11:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:11:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:11:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:11:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:11:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:11:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:11:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:11:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:11:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:11:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:11:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:11:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:11:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:11:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:11:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:11:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:11:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:11:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:11:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:11:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:11:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:11:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:11:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:11:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:11:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:11:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:11:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:11:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:11:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:11:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:11:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:11:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:11:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:11:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:11:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:11:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:11:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:11:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:11:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:11:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:11:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:11:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:11:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:11:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:11:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:11:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:11:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:11:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:11:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:11:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:11:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:11:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:12:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:12:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:12:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:12:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:12:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:12:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:12:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:12:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:12:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:12:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:12:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:12:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:12:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:12:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:12:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:12:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:12:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:12:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:12:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:12:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:12:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:12:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:12:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:12:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:12:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:12:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:12:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:12:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:12:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:12:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:12:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:12:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:12:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:12:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:12:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:12:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:12:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:12:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:12:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:12:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:12:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:12:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:12:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:12:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:12:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:12:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:12:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:12:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:12:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:12:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:12:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:12:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:12:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:12:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:12:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:12:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:12:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:12:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:12:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:12:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:12:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:12:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:12:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:12:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:12:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:12:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:12:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:12:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:12:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:12:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:12:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:12:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:12:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:12:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:12:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:12:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:12:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:12:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:12:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:12:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:12:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:12:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:12:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:12:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:12:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:12:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:12:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:12:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:12:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:12:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:12:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:12:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:12:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:12:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:12:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:12:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:12:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:12:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:12:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:12:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:12:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:12:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:12:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:12:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:12:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:12:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:12:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:12:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:12:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:12:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:12:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:12:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:12:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:12:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:12:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:12:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:12:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:12:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:12:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:12:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:12:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:12:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:12:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:12:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:12:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:12:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:12:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:12:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:12:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:12:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:12:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:12:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:12:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:12:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:12:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:12:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:12:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:12:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:12:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:12:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:12:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:12:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:12:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:12:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:12:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:12:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:12:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:12:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:12:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:12:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:12:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:12:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:12:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:12:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:12:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:12:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:12:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:12:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:12:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:12:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:12:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:12:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:12:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:12:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:12:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:12:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:12:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:12:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:12:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:12:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:12:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:12:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:12:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:12:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:12:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:12:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:12:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:12:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:12:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:12:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:12:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:12:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:12:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:12:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:12:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:12:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:12:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:12:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:12:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:12:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:12:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:12:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:12:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:12:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:12:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:12:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:12:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:12:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:12:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:12:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:12:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:12:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:12:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:12:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:12:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:12:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:12:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:12:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:12:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:12:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:12:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:12:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:12:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:12:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:12:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:12:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:13:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:13:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:13:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:13:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:13:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:13:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:13:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:13:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:13:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:13:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:13:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:13:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:13:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:13:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:13:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:13:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:13:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:13:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:13:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:13:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:13:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:13:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:13:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:13:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:13:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:13:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:13:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:13:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:13:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:13:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:13:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:13:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:13:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:13:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:13:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:13:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:13:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:13:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:13:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:13:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:13:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:13:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:13:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:13:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:13:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:13:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:13:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:13:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:13:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:13:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:13:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:13:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:13:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:13:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:13:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:13:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:13:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:13:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:13:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:13:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:13:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:13:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:13:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:13:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:13:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:13:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:13:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:13:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:13:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:13:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:13:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:13:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:13:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:13:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:13:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:13:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:13:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:13:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:13:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:13:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:13:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:13:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:13:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:13:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:13:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:13:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:13:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:13:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:13:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:13:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:13:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:13:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:13:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:13:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:13:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:13:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:13:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:13:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:13:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:13:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:13:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:13:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:13:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:13:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:13:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:13:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:13:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:13:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:13:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:13:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:13:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:13:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:13:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:13:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:13:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:13:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:13:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:13:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:13:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:13:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:13:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:13:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:13:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:13:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:13:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:13:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:13:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:13:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:13:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:13:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:13:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:13:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:13:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:13:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:13:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:13:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:13:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:13:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:13:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:13:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:13:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:13:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:13:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:13:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:13:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:13:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:13:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:13:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:13:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:13:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:13:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:13:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:13:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:13:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:13:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:13:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:13:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:13:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:13:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:13:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:13:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:13:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:13:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:13:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:13:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:13:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:13:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:13:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:13:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:13:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:13:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:13:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:13:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:13:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:13:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:13:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:13:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:13:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:13:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:13:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:13:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:13:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:13:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:13:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:13:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:13:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:13:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:13:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:13:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:13:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:13:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:13:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:13:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:13:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:13:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:13:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:13:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:13:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:13:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:13:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:13:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:13:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:13:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:13:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:13:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:13:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:13:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:13:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:13:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:13:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:13:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:13:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:13:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:13:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:13:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:13:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:14:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:14:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:14:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:14:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:14:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:14:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:14:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:14:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:14:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:14:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:14:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:14:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:14:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:14:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:14:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:14:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:14:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:14:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:14:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:14:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:14:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:14:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:14:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:14:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:14:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:14:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:14:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:14:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:14:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:14:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:14:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:14:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:14:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:14:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:14:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:14:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:14:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:14:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:14:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:14:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:14:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:14:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:14:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:14:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:14:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:14:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:14:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:14:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:14:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:14:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:14:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:14:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:14:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:14:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:14:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:14:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:14:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:14:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:14:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:14:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:14:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:14:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:14:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:14:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:14:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:14:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:14:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:14:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:14:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:14:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:14:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:14:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:14:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:14:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:14:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:14:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:14:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:14:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:14:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:14:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:14:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:14:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:14:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:14:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:14:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:14:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:14:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:14:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:14:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:14:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:14:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:14:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:14:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:14:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:14:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:14:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:14:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:14:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:14:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:14:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:14:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:14:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:14:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:14:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:14:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:14:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:14:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:14:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:14:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:14:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:14:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:14:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:14:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:14:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:14:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:14:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:14:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:14:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:14:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:14:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:14:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:14:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:14:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:14:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:14:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:14:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:14:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:14:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:14:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:14:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:14:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:14:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:14:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:14:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:14:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:14:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:14:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:14:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:14:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:14:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:14:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:14:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:14:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:14:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:14:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:14:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:14:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:14:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:14:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:14:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:14:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:14:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:14:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:14:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:14:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:14:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:14:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:14:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:14:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:14:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:14:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:14:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:14:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:14:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:14:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:14:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:14:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:14:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:14:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:14:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:14:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:14:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:14:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:14:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:14:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:14:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:14:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:14:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:14:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:14:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:14:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:14:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:14:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:14:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:14:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:14:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:14:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:14:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:14:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:14:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:14:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:14:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:14:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:14:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:14:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:14:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:14:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:14:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:14:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:14:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:14:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:14:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:14:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:14:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:14:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:14:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:14:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:14:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:14:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:14:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:14:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:14:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:14:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:14:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:14:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:14:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:15:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:15:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:15:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:15:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:15:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:15:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:15:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:15:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:15:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:15:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:15:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:15:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:15:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:15:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:15:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:15:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:15:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:15:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:15:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:15:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:15:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:15:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:15:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:15:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:15:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:15:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:15:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:15:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:15:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:15:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:15:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:15:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:15:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:15:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:15:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:15:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:15:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:15:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:15:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:15:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:15:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:15:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:15:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:15:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:15:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:15:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:15:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:15:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:15:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:15:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:15:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:15:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:15:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:15:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:15:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:15:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:15:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:15:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:15:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:15:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:15:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:15:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:15:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:15:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:15:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:15:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:15:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:15:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:15:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:15:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:15:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:15:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:15:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:15:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:15:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:15:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:15:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:15:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:15:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:15:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:15:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:15:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:15:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:15:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:15:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:15:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:15:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:15:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:15:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:15:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:15:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:15:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:15:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:15:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:15:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:15:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:15:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:15:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:15:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:15:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:15:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:15:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:15:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:15:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:15:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:15:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:15:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:15:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:15:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:15:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:15:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:15:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:15:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:15:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:15:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:15:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:15:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:15:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:15:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:15:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:15:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:15:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:15:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:15:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:15:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:15:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:15:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:15:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:15:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:15:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:15:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:15:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:15:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:15:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:15:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:15:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:15:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:15:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:15:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:15:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:15:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:15:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:15:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:15:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:15:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:15:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:15:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:15:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:15:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:15:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:15:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:15:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:15:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:15:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:15:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:15:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:15:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:15:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:15:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:15:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:15:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:15:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:15:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:15:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:15:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:15:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:15:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:15:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:15:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:15:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:15:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:15:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:15:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:15:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:15:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:15:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:15:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:15:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:15:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:15:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:15:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:15:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:15:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:15:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:15:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:15:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:15:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:15:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:15:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:15:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:15:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:15:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:15:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:15:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:15:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:15:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:15:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:15:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:15:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:15:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:15:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:15:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:15:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:15:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:15:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:15:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:15:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:15:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:15:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:15:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:15:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:15:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:15:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:15:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:15:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:15:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:16:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:16:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:16:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:16:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:16:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:16:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:16:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:16:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:16:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:16:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:16:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:16:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:16:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:16:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:16:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:16:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:16:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:16:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:16:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:16:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:16:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:16:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:16:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:16:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:16:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:16:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:16:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:16:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:16:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:16:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:16:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:16:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:16:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:16:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:16:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:16:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:16:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:16:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:16:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:16:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:16:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:16:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:16:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:16:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:16:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:16:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:16:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:16:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:16:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:16:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:16:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:16:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:16:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:16:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:16:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:16:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:16:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:16:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:16:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:16:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:16:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:16:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:16:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:16:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:16:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:16:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:16:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:16:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:16:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:16:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:16:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:16:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:16:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:16:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:16:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:16:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:16:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:16:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:16:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:16:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:16:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:16:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:16:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:16:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:16:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:16:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:16:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:16:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:16:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:16:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:16:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:16:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:16:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:16:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:16:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:16:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:16:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:16:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:16:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:16:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:16:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:16:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:16:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:16:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:16:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:16:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:16:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:16:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:16:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:16:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:16:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:16:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:16:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:16:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:16:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:16:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:16:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:16:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:16:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:16:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:16:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:16:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:16:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:16:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:16:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:16:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:16:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:16:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:16:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:16:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:16:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:16:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:16:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:16:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:16:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:16:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:16:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:16:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:16:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:16:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:16:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:16:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:16:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:16:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:16:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:16:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:16:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:16:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:16:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:16:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:16:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:16:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:16:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:16:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:16:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:16:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:16:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:16:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:16:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:16:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:16:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:16:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:16:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:16:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:16:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:16:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:16:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:16:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:16:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:16:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:16:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:16:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:16:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:16:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:16:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:16:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:16:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:16:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:16:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:16:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:16:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:16:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:16:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:16:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:16:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:16:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:16:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:16:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:16:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:16:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:16:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:16:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:16:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:16:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:16:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:16:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:16:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:16:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:17:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:17:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:17:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:17:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:17:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:17:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:17:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:17:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:17:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:17:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:17:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:17:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:17:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:17:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:17:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:17:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:17:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:17:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:17:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:17:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:17:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:17:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:17:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:17:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:17:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:17:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:17:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:17:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:17:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:17:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:17:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:17:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:17:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:17:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:17:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:17:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:17:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:17:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:17:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:17:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:17:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:17:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:17:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:17:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:17:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:17:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:17:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:17:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:17:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:17:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:17:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:17:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:17:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:17:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:17:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:17:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:17:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:17:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:17:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:17:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:17:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:17:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:17:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:17:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:17:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:17:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:17:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:17:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:17:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:17:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:17:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:17:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:17:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:17:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:17:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:17:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:17:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:17:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:17:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:17:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:17:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:17:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:17:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:17:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:17:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:17:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:17:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:17:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:17:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:17:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:17:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:17:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:17:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:17:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:17:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:17:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:17:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:17:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:17:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:17:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:17:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:17:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:17:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:17:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:17:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:17:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:17:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:17:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:17:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:17:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:17:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:17:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:17:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:17:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:17:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:17:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:17:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:17:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:17:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:17:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:17:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:17:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:17:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:17:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:17:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:17:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:17:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:17:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:17:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:17:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:17:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:17:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:17:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:17:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:17:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:17:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:17:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:17:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:17:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:17:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:17:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:17:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:17:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:17:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:17:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:17:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:17:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:17:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:17:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:17:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:17:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:17:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:17:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:17:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:17:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:17:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:17:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:17:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:17:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:17:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:17:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:17:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:17:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:17:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:17:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:17:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:17:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:17:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:17:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:17:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:17:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:17:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:17:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:17:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:17:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:17:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:17:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:17:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:17:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:17:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:17:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:17:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:17:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:17:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:17:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:17:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:17:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:17:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:17:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:17:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:17:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:17:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:17:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:17:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:17:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:17:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:17:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:17:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:17:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:17:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:17:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:17:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:17:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:17:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:17:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:17:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:17:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:17:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:17:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:17:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:17:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:17:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:17:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:17:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:17:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:17:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:18:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:18:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:18:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:18:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:18:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:18:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:18:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:18:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:18:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:18:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:18:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:18:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:18:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:18:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:18:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:18:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:18:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:18:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:20:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:20:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:20:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:20:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:20:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:20:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:20:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:20:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:20:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:20:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:20:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:20:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:20:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:20:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:20:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:20:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:20:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:20:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:20:10 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 22:20:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:20:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:20:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:20:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:20:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:20:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:20:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:20:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:20:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:20:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:20:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:20:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:20:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:20:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:20:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:20:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:20:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:20:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:20:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:20:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:20:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:20:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:20:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:20:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:20:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:20:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:20:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:20:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:20:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:20:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:20:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:20:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:20:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:20:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:20:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:20:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:20:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:20:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:20:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:20:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:20:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:20:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:20:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:20:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:20:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:20:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:20:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:20:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:20:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:20:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:20:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:20:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:20:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:20:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:20:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:20:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:20:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:20:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:20:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:20:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:20:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:20:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:20:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:20:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:20:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:20:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:20:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:20:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:20:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:20:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:20:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:20:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:20:32 rdma06 NetworkManager[1025]: <info>  [1681784432.9480] dhcp4 (eno1): state changed new lease, address=10.73.131.45
Apr 17 22:20:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:20:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:20:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:20:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:20:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:20:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:20:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:20:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:20:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:20:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:20:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:20:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:20:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:20:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:20:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:20:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:20:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:20:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:20:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:20:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:20:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:20:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:20:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:20:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:20:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:20:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:20:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:20:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:20:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:20:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:20:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:20:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:20:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:20:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:20:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:20:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:20:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:20:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:20:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:20:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:20:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:20:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:20:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:20:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:20:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:20:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:20:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:20:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:20:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:20:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:20:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:20:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:20:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:20:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:20:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:20:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:20:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:20:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:20:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:20:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:20:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:20:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:20:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:20:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:20:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:20:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:20:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:20:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:20:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:20:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:20:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:20:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:20:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:20:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:20:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:20:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:20:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:20:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:20:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:20:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:20:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:20:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:20:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:20:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:20:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:20:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:20:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:20:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:20:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:20:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:21:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:21:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:21:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:21:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:21:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:21:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:21:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:21:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:21:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:21:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:21:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:21:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:21:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:21:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:21:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:21:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:21:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:21:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:21:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:21:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:21:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:21:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:21:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:21:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:21:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:21:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:21:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:21:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:21:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:21:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:21:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:21:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:21:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:21:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:21:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:21:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:21:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:21:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:21:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:21:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:21:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:21:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:21:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:21:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:21:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:21:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:21:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:21:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:21:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:21:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:21:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:21:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:21:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:21:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:21:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:21:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:21:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:21:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:21:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:21:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:21:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:21:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:21:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:21:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:21:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:21:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:21:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:21:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:21:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:21:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:21:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:21:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:21:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:21:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:21:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:21:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:21:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:21:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:21:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:21:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:21:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:21:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:21:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:21:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:21:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:21:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:21:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:21:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:21:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:21:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:21:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:21:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:21:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:21:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:21:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:21:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:21:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:21:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:21:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:21:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:21:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:21:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:21:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:21:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:21:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:21:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:21:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:21:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:21:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:21:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:21:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:21:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:21:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:21:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:21:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:21:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:21:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:21:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:21:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:21:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:21:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:21:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:21:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:21:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:21:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:21:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:21:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:21:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:21:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:21:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:21:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:21:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:21:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:21:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:21:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:21:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:21:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:21:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:21:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:21:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:21:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:21:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:21:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:21:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:21:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:21:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:21:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:21:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:21:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:21:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:21:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:21:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:21:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:21:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:21:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:21:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:21:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:21:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:21:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:21:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:21:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:21:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:21:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:21:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:21:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:21:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:21:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:21:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:21:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:21:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:21:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:21:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:21:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:21:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:21:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:21:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:21:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:21:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:21:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:21:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:21:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:21:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:21:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:21:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:21:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:21:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:21:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:21:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:21:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:21:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:21:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:21:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:21:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:21:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:21:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:21:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:21:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:21:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:21:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:21:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:21:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:21:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:21:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:21:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:21:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:21:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:21:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:21:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:21:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:21:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:21:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:21:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:21:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:21:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:21:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:21:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:22:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:22:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:22:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:22:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:22:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:22:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:22:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:22:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:22:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:22:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:22:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:22:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:22:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:22:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:22:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:22:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:22:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:22:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:22:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:22:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:22:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:22:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:22:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:22:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:22:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:22:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:22:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:22:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:22:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:22:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:22:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:22:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:22:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:22:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:22:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:22:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:22:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:22:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:22:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:22:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:22:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:22:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:22:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:22:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:22:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:22:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:22:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:22:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:22:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:22:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:22:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:22:10 rdma06 systemd-journald[790]: Data hash table of /run/log/journal/95379d2f723847b68d9e976dee8967a9/system.journal has a fill level at 75.0 (163750 of 218332 items, 75497472 file size, 461 bytes per hash table item), suggesting rotation.
Apr 17 22:22:10 rdma06 systemd-journald[790]: /run/log/journal/95379d2f723847b68d9e976dee8967a9/system.journal: Journal header limits reached or header out-of-date, rotating.
Apr 17 22:22:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:22:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:22:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:22:10 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 17 22:22:10 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 17 22:22:10 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 17 22:22:10 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 17 22:22:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:22:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:22:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:22:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:22:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:22:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:22:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:22:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:22:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:22:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:22:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:22:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:22:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:22:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:22:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:22:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:22:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:22:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:22:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:22:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:22:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:22:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:22:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:22:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:22:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:22:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:22:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:22:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:22:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:22:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:22:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:22:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:22:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:22:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:22:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:22:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:22:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:22:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:22:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:22:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:22:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:22:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:22:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:22:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:22:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:22:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:22:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:22:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:22:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:22:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:22:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:22:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:22:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:22:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:22:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:22:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:22:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:22:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:22:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:22:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:22:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:22:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:22:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:22:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:22:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:22:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:22:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:22:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:22:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:22:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:22:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:22:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:22:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:22:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:22:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:22:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:22:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:22:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:22:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:22:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:22:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:22:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:22:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:22:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:22:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:22:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:22:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:22:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:22:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:22:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:22:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:22:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:22:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:22:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:22:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:22:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:22:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:22:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:22:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:22:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:22:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:22:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:22:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:22:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:22:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:22:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:22:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:22:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:22:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:22:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:22:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:22:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:22:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:22:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:22:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:22:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:22:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:22:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:22:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:22:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:22:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:22:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:22:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:22:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:22:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:22:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:22:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:22:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:22:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:22:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:22:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:22:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:22:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:22:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:22:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:22:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:22:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:22:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:22:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:22:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:22:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:22:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:22:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:22:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:22:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:22:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:22:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:22:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:22:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:22:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:22:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:22:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:22:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:22:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:22:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:22:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:22:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:22:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:22:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:22:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:22:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:22:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:23:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:23:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:23:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:23:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:23:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:23:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:23:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:23:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:23:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:23:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:23:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:23:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:23:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:23:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:23:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:23:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:23:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:23:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:23:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:23:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:23:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:23:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:23:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:23:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:23:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:23:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:23:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:23:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:23:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:23:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:23:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:23:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:23:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:23:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:23:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:23:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:23:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:23:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:23:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:23:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:23:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:23:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:23:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:23:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:23:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:23:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:23:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:23:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:23:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:23:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:23:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:23:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:23:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:23:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:23:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:23:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:23:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:23:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:23:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:23:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:23:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:23:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:23:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:23:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:23:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:23:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:23:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:23:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:23:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:23:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:23:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:23:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:23:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:23:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:23:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:23:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:23:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:23:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:23:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:23:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:23:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:23:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:23:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:23:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:23:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:23:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:23:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:23:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:23:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:23:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:23:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:23:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:23:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:23:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:23:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:23:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:23:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:23:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:23:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:23:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:23:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:23:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:23:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:23:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:23:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:23:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:23:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:23:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:23:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:23:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:23:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:23:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:23:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:23:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:23:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:23:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:23:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:23:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:23:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:23:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:23:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:23:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:23:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:23:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:23:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:23:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:23:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:23:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:23:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:23:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:23:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:23:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:23:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:23:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:23:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:23:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:23:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:23:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:23:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:23:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:23:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:23:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:23:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:23:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:23:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:23:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:23:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:23:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:23:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:23:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:23:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:23:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:23:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:23:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:23:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:23:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:23:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:23:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:23:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:23:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:23:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:23:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:23:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:23:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:23:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:23:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:23:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:23:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:23:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:23:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:23:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:23:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:23:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:23:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:23:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:23:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:23:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:23:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:23:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:23:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:23:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:23:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:23:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:23:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:23:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:23:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:23:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:23:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:23:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:23:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:23:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:23:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:23:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:23:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:23:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:23:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:23:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:23:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:23:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:23:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:23:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:23:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:23:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:23:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:23:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:23:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:23:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:23:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:23:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:23:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:23:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:23:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:23:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:23:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:23:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:23:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:24:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:24:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:24:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:24:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:24:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:24:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:24:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:24:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:24:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:24:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:24:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:24:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:24:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:24:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:24:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:24:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:24:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:24:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:24:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:24:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:24:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:24:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:24:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:24:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:24:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:24:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:24:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:24:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:24:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:24:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:24:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:24:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:24:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:24:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:24:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:24:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:24:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:24:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:24:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:24:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:24:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:24:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:24:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:24:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:24:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:24:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:24:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:24:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:24:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:24:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:24:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:24:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:24:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:24:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:24:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:24:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:24:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:24:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:24:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:24:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:24:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:24:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:24:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:24:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:24:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:24:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:24:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:24:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:24:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:24:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:24:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:24:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:24:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:24:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:24:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:24:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:24:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:24:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:24:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:24:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:24:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:24:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:24:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:24:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:24:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:24:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:24:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:24:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:24:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:24:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:24:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:24:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:24:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:24:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:24:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:24:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:24:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:24:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:24:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:24:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:24:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:24:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:24:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:24:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:24:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:24:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:24:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:24:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:24:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:24:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:24:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:24:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:24:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:24:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:24:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:24:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:24:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:24:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:24:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:24:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:24:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:24:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:24:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:24:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:24:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:24:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:24:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:24:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:24:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:24:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:24:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:24:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:24:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:24:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:24:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:24:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:24:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:24:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:24:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:24:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:24:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:24:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:24:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:24:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:24:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:24:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:24:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:24:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:24:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:24:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:24:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:24:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:24:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:24:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:24:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:24:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:24:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:24:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:24:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:24:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:24:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:24:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:24:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:24:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:24:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:24:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:24:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:24:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:24:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:24:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:24:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:24:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:24:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:24:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:24:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:24:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:24:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:24:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:24:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:24:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:24:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:24:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:24:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:24:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:24:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:24:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:24:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:24:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:24:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:24:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:24:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:24:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:24:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:24:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:24:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:24:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:24:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:24:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:24:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:24:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:24:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:24:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:24:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:24:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:24:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:24:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:24:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:24:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:24:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:24:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:24:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:24:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:24:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:24:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:24:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:24:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:25:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:25:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:25:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:25:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:25:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:25:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:25:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:25:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:25:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:25:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:25:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:25:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:25:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:25:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:25:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:25:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:25:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:25:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:25:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:25:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:25:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:25:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:25:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:25:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:25:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:25:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:25:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:25:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:25:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:25:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:25:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:25:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:25:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:25:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:25:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:25:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:25:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:25:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:25:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:25:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:25:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:25:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:25:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:25:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:25:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:25:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:25:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:25:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:25:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:25:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:25:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:25:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:25:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:25:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:25:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:25:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:25:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:25:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:25:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:25:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:25:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:25:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:25:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:25:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:25:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:25:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:25:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:25:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:25:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:25:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:25:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:25:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:25:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:25:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:25:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:25:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:25:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:25:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:25:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:25:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:25:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:25:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:25:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:25:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:25:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:25:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:25:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:25:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:25:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:25:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:25:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:25:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:25:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:25:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:25:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:25:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:25:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:25:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:25:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:25:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:25:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:25:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:25:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:25:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:25:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:25:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:25:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:25:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:25:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:25:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:25:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:25:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:25:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:25:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:25:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:25:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:25:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:25:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:25:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:25:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:25:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:25:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:25:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:25:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:25:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:25:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:25:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:25:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:25:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:25:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:25:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:25:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:25:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:25:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:25:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:25:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:25:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:25:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:25:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:25:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:25:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:25:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:25:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:25:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:25:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:25:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:25:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:25:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:25:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:25:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:25:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:25:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:25:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:25:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:25:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:25:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:25:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:25:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:25:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:25:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:25:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:25:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:25:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:25:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:25:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:25:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:25:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:25:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:25:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:25:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:25:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:25:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:25:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:25:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:25:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:25:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:25:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:25:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:25:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:25:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:25:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:25:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:25:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:25:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:25:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:25:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:25:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:25:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:25:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:25:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:25:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:25:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:25:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:25:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:25:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:25:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:25:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:25:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:25:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:25:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:25:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:25:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:25:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:25:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:25:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:25:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:25:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:25:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:25:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:25:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:25:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:25:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:25:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:25:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:25:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:25:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:26:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:26:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:26:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:26:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:26:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:26:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:26:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:26:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:26:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:26:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:26:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:26:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:26:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:26:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:26:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:26:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:26:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:26:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:26:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:26:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:26:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:26:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:26:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:26:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:26:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:26:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:26:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:26:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:26:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:26:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:26:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:26:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:26:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:26:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:26:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:26:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:26:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:26:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:26:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:26:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:26:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:26:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:26:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:26:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:26:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:26:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:26:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:26:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:26:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:26:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:26:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:26:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:26:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:26:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:26:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:26:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:26:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:26:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:26:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:26:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:26:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:26:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:26:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:26:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:26:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:26:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:26:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:26:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:26:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:26:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:26:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:26:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:26:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:26:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:26:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:26:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:26:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:26:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:26:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:26:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:26:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:26:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:26:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:26:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:26:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:26:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:26:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:26:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:26:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:26:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:26:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:26:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:26:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:26:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:26:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:26:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:26:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:26:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:26:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:26:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:26:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:26:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:26:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:26:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:26:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:26:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:26:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:26:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:26:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:26:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:26:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:26:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:26:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:26:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:26:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:26:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:26:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:26:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:26:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:26:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:26:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:26:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:26:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:26:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:26:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:26:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:26:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:26:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:26:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:26:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:26:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:26:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:26:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:26:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:26:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:26:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:26:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:26:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:26:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:26:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:26:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:26:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:26:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:26:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:26:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:26:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:26:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:26:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:26:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:26:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:26:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:26:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:26:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:26:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:26:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:26:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:26:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:26:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:26:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:26:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:26:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:26:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:26:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:26:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:26:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:26:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:26:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:26:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:26:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:26:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:26:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:26:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:26:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:26:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:26:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:26:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:26:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:26:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:26:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:26:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:26:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:26:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:26:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:26:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:26:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:26:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:26:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:26:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:26:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:26:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:26:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:26:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:26:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:26:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:26:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:26:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:26:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:26:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:26:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:26:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:26:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:26:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:26:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:26:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:26:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:26:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:26:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:26:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:26:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:26:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:26:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:26:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:26:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:26:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:26:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:26:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:27:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:27:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:27:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:27:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:27:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:27:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:27:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:27:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:27:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:27:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:27:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:27:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:27:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:27:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:27:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:27:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:27:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:27:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:27:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:27:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:27:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:27:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:27:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:27:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:27:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:27:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:27:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:27:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:27:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:27:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:27:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:27:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:27:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:27:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:27:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:27:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:27:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:27:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:27:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:27:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:27:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:27:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:27:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:27:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:27:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:27:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:27:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:27:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:27:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:27:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:27:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:27:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:27:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:27:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:27:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:27:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:27:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:27:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:27:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:27:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:27:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:27:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:27:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:27:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:27:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:27:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:27:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:27:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:27:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:27:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:27:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:27:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:27:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:27:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:27:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:27:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:27:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:27:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:27:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:27:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:27:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:27:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:27:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:27:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:27:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:27:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:27:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:27:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:27:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:27:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:27:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:27:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:27:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:27:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:27:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:27:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:27:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:27:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:27:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:27:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:27:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:27:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:27:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:27:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:27:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:27:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:27:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:27:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:27:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:27:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:27:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:27:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:27:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:27:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:27:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:27:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:27:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:27:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:27:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:27:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:27:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:27:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:27:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:27:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:27:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:27:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:27:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:27:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:27:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:27:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:27:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:27:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:27:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:27:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:27:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:27:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:27:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:27:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:27:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:27:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:27:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:27:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:27:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:27:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:27:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:27:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:27:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:27:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:27:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:27:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:27:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:27:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:27:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:27:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:27:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:27:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:27:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:27:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:27:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:27:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:27:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:27:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:27:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:27:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:27:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:27:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:27:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:27:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:27:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:27:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:27:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:27:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:27:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:27:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:27:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:27:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:27:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:27:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:27:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:27:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:27:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:27:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:27:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:27:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:27:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:27:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:27:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:27:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:27:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:27:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:27:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:27:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:27:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:27:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:27:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:27:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:27:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:27:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:27:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:27:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:27:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:27:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:27:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:27:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:27:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:27:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:27:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:27:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:27:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:27:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:27:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:27:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:27:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:27:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:27:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:27:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:28:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:28:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:28:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:28:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:28:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:28:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:28:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:28:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:28:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:28:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:28:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:28:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:28:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:28:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:28:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:28:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:28:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:28:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:28:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:28:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:28:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:28:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:28:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:28:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:28:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:28:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:28:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:28:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:28:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:28:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:28:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:28:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:28:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:28:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:28:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:28:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:30:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:30:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:30:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:30:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:30:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:30:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:30:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:30:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:30:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:30:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:30:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:30:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:30:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:30:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:30:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:30:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:30:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:30:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:30:15 rdma06 rsyslogd[1041]: imjournal: 5298 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 22:30:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:30:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:30:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:30:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:30:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:30:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:30:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:30:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:30:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:30:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:30:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:30:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:30:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:30:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:30:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:30:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:30:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:30:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:30:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:30:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:30:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:30:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:30:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:30:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:30:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:30:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:30:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:30:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:30:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:30:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:30:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:30:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:30:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:30:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:30:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:30:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:30:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:30:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:30:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:30:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:30:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:30:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:30:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:30:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:30:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:30:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:30:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:30:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:30:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:30:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:30:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:30:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:30:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:30:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:30:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:30:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:30:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:30:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:30:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:30:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:30:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:30:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:30:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:30:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:30:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:30:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:30:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:30:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:30:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:30:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:30:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:30:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:30:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:30:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:30:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:30:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:30:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:30:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:30:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:30:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:30:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:30:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:30:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:30:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:30:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:30:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:30:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:30:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:30:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:30:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:30:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:30:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:30:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:30:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:30:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:30:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:30:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:30:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:30:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:30:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:30:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:30:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:30:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:30:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:30:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:30:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:30:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:30:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:30:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:30:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:30:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:30:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:30:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:30:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:30:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:30:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:30:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:30:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:30:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:30:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:30:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:30:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:30:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:30:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:30:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:30:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:30:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:30:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:30:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:30:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:30:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:30:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:30:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:30:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:30:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:30:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:30:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:30:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:30:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:30:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:30:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:30:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:30:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:30:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:31:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:31:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:31:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:31:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:31:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:31:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:31:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:31:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:31:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:31:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:31:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:31:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:31:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:31:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:31:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:31:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:31:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:31:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:31:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:31:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:31:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:31:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:31:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:31:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:31:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:31:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:31:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:31:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:31:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:31:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:31:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:31:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:31:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:31:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:31:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:31:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:31:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:31:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:31:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:31:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:31:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:31:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:31:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:31:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:31:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:31:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:31:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:31:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:31:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:31:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:31:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:31:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:31:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:31:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:31:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:31:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:31:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:31:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:31:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:31:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:31:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:31:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:31:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:31:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:31:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:31:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:31:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:31:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:31:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:31:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:31:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:31:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:31:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:31:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:31:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:31:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:31:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:31:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:31:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:31:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:31:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:31:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:31:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:31:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:31:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:31:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:31:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:31:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:31:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:31:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:31:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:31:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:31:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:31:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:31:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:31:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:31:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:31:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:31:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:31:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:31:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:31:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:31:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:31:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:31:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:31:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:31:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:31:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:31:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:31:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:31:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:31:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:31:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:31:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:31:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:31:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:31:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:31:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:31:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:31:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:31:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:31:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:31:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:31:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:31:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:31:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:31:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:31:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:31:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:31:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:31:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:31:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:31:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:31:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:31:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:31:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:31:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:31:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:31:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:31:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:31:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:31:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:31:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:31:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:31:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:31:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:31:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:31:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:31:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:31:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:31:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:31:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:31:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:31:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:31:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:31:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:31:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:31:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:31:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:31:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:31:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:31:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:31:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:31:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:31:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:31:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:31:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:31:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:31:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:31:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:31:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:31:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:31:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:31:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:31:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:31:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:31:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:31:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:31:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:31:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:31:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:31:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:31:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:31:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:31:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:31:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:31:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:31:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:31:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:31:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:31:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:31:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:31:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:31:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:31:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:31:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:31:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:31:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:31:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:31:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:31:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:31:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:31:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:31:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:31:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:31:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:31:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:31:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:31:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:31:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:31:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:31:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:31:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:31:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:31:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:31:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:32:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:32:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:32:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:32:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:32:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:32:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:32:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:32:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:32:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:32:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:32:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:32:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:32:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:32:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:32:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:32:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:32:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:32:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:32:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:32:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:32:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:32:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:32:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:32:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:32:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:32:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:32:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:32:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:32:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:32:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:32:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:32:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:32:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:32:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:32:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:32:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:32:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:32:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:32:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:32:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:32:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:32:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:32:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:32:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:32:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:32:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:32:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:32:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:32:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:32:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:32:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:32:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:32:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:32:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:32:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:32:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:32:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:32:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:32:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:32:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:32:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:32:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:32:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:32:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:32:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:32:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:32:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:32:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:32:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:32:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:32:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:32:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:32:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:32:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:32:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:32:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:32:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:32:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:32:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:32:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:32:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:32:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:32:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:32:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:32:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:32:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:32:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:32:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:32:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:32:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:32:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:32:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:32:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:32:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:32:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:32:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:32:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:32:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:32:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:32:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:32:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:32:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:32:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:32:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:32:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:32:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:32:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:32:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:32:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:32:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:32:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:32:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:32:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:32:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:32:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:32:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:32:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:32:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:32:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:32:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:32:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:32:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:32:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:32:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:32:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:32:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:32:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:32:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:32:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:32:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:32:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:32:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:32:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:32:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:32:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:32:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:32:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:32:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:32:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:32:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:32:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:32:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:32:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:32:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:32:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:32:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:32:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:32:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:32:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:32:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:32:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:32:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:32:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:32:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:32:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:32:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:32:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:32:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:32:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:32:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:32:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:32:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:32:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:32:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:32:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:32:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:32:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:32:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:32:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:32:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:32:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:32:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:32:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:32:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:32:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:32:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:32:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:32:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:32:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:32:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:32:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:32:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:32:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:32:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:32:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:32:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:32:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:32:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:32:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:32:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:32:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:32:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:32:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:32:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:32:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:32:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:32:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:32:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:32:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:32:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:32:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:32:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:32:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:32:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:32:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:32:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:32:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:32:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:32:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:32:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:32:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:32:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:32:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:32:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:32:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:32:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:33:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:33:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:33:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:33:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:33:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:33:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:33:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:33:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:33:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:33:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:33:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:33:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:33:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:33:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:33:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:33:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:33:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:33:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:33:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:33:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:33:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:33:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:33:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:33:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:33:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:33:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:33:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:33:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:33:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:33:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:33:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:33:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:33:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:33:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:33:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:33:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:33:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:33:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:33:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:33:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:33:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:33:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:33:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:33:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:33:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:33:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:33:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:33:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:33:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:33:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:33:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:33:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:33:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:33:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:33:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:33:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:33:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:33:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:33:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:33:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:33:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:33:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:33:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:33:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:33:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:33:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:33:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:33:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:33:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:33:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:33:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:33:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:33:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:33:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:33:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:33:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:33:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:33:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:33:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:33:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:33:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:33:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:33:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:33:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:33:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:33:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:33:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:33:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:33:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:33:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:33:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:33:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:33:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:33:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:33:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:33:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:33:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:33:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:33:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:33:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:33:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:33:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:33:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:33:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:33:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:33:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:33:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:33:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:33:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:33:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:33:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:33:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:33:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:33:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:33:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:33:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:33:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:33:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:33:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:33:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:33:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:33:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:33:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:33:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:33:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:33:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:33:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:33:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:33:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:33:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:33:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:33:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:33:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:33:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:33:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:33:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:33:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:33:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:33:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:33:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:33:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:33:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:33:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:33:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:33:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:33:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:33:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:33:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:33:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:33:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:33:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:33:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:33:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:33:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:33:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:33:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:33:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:33:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:33:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:33:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:33:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:33:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:33:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:33:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:33:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:33:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:33:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:33:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:33:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:33:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:33:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:33:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:33:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:33:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:33:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:33:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:33:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:33:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:33:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:33:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:33:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:33:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:33:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:33:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:33:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:33:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:33:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:33:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:33:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:33:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:33:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:33:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:33:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:33:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:33:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:33:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:33:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:33:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:33:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:33:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:33:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:33:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:33:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:33:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:33:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:33:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:33:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:33:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:33:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:33:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:33:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:33:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:33:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:33:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:33:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:33:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:34:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:34:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:34:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:34:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:34:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:34:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:34:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:34:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:34:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:34:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:34:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:34:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:34:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:34:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:34:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:34:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:34:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:34:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:34:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:34:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:34:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:34:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:34:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:34:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:34:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:34:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:34:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:34:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:34:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:34:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:34:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:34:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:34:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:34:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:34:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:34:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:34:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:34:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:34:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:34:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:34:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:34:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:34:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:34:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:34:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:34:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:34:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:34:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:34:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:34:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:34:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:34:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:34:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:34:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:34:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:34:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:34:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:34:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:34:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:34:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:34:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:34:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:34:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:34:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:34:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:34:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:34:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:34:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:34:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:34:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:34:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:34:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:34:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:34:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:34:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:34:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:34:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:34:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:34:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:34:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:34:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:34:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:34:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:34:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:34:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:34:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:34:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:34:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:34:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:34:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:34:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:34:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:34:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:34:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:34:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:34:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:34:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:34:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:34:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:34:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:34:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:34:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:34:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:34:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:34:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:34:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:34:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:34:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:34:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:34:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:34:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:34:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:34:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:34:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:34:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:34:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:34:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:34:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:34:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:34:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:34:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:34:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:34:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:34:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:34:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:34:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:34:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:34:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:34:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:34:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:34:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:34:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:34:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:34:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:34:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:34:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:34:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:34:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:34:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:34:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:34:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:34:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:34:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:34:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:34:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:34:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:34:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:34:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:34:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:34:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:34:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:34:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:34:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:34:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:34:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:34:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:34:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:34:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:34:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:34:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:34:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:34:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:34:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:34:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:34:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:34:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:34:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:34:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:34:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:34:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:34:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:34:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:34:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:34:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:34:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:34:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:34:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:34:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:34:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:34:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:34:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:34:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:34:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:34:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:34:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:34:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:34:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:34:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:34:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:34:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:34:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:34:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:34:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:34:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:34:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:34:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:34:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:34:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:34:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:34:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:34:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:34:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:34:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:34:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:34:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:34:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:34:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:34:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:34:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:34:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:34:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:34:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:34:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:34:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:34:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:34:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:35:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:35:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:35:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:35:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:35:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:35:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:35:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:35:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:35:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:35:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:35:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:35:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:35:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:35:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:35:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:35:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:35:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:35:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:35:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:35:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:35:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:35:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:35:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:35:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:35:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:35:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:35:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:35:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:35:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:35:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:35:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:35:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:35:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:35:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:35:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:35:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:35:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:35:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:35:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:35:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:35:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:35:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:35:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:35:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:35:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:35:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:35:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:35:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:35:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:35:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:35:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:35:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:35:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:35:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:35:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:35:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:35:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:35:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:35:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:35:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:35:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:35:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:35:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:35:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:35:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:35:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:35:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:35:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:35:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:35:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:35:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:35:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:35:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:35:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:35:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:35:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:35:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:35:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:35:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:35:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:35:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:35:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:35:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:35:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:35:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:35:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:35:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:35:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:35:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:35:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:35:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:35:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:35:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:35:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:35:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:35:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:35:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:35:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:35:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:35:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:35:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:35:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:35:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:35:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:35:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:35:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:35:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:35:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:35:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:35:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:35:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:35:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:35:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:35:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:35:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:35:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:35:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:35:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:35:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:35:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:35:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:35:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:35:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:35:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:35:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:35:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:35:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:35:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:35:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:35:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:35:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:35:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:35:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:35:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:35:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:35:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:35:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:35:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:35:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:35:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:35:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:35:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:35:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:35:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:35:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:35:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:35:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:35:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:35:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:35:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:35:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:35:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:35:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:35:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:35:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:35:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:35:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:35:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:35:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:35:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:35:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:35:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:35:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:35:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:35:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:35:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:35:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:35:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:35:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:35:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:35:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:35:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:35:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:35:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:35:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:35:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:35:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:35:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:35:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:35:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:35:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:35:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:35:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:35:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:35:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:35:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:35:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:35:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:35:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:35:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:35:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:35:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:35:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:35:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:35:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:35:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:35:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:35:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:35:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:35:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:35:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:35:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:35:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:35:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:35:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:35:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:35:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:35:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:35:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:35:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:35:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:35:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:35:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:35:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:35:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:35:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:36:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:36:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:36:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:36:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:36:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:36:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:36:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:36:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:36:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:36:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:36:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:36:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:36:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:36:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:36:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:36:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:36:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:36:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:36:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:36:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:36:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:36:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:36:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:36:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:36:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:36:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:36:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:36:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:36:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:36:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:36:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:36:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:36:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:36:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:36:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:36:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:36:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:36:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:36:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:36:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:36:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:36:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:36:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:36:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:36:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:36:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:36:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:36:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:36:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:36:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:36:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:36:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:36:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:36:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:36:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:36:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:36:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:36:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:36:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:36:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:36:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:36:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:36:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:36:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:36:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:36:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:36:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:36:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:36:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:36:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:36:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:36:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:36:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:36:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:36:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:36:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:36:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:36:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:36:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:36:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:36:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:36:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:36:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:36:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:36:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:36:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:36:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:36:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:36:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:36:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:36:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:36:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:36:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:36:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:36:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:36:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:36:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:36:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:36:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:36:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:36:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:36:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:36:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:36:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:36:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:36:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:36:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:36:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:36:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:36:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:36:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:36:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:36:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:36:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:36:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:36:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:36:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:36:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:36:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:36:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:36:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:36:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:36:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:36:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:36:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:36:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:36:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:36:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:36:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:36:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:36:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:36:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:36:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:36:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:36:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:36:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:36:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:36:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:36:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:36:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:36:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:36:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:36:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:36:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:36:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:36:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:36:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:36:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:36:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:36:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:36:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:36:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:36:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:36:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:36:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:36:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:36:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:36:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:36:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:36:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:36:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:36:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:36:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:36:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:36:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:36:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:36:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:36:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:36:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:36:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:36:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:36:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:36:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:36:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:36:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:36:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:36:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:36:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:36:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:36:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:36:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:36:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:36:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:36:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:36:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:36:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:36:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:36:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:36:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:36:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:36:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:36:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:36:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:36:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:36:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:36:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:36:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:36:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:36:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:36:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:36:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:36:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:36:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:36:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:36:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:36:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:36:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:36:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:36:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:36:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:36:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:36:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:36:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:36:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:36:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:36:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:37:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:37:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:37:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:37:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:37:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:37:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:37:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:37:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:37:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:37:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:37:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:37:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:37:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:37:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:37:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:37:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:37:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:37:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:37:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:37:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:37:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:37:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:37:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:37:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:37:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:37:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:37:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:37:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:37:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:37:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:37:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:37:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:37:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:37:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:37:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:37:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:37:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:37:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:37:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:37:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:37:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:37:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:37:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:37:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:37:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:37:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:37:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:37:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:37:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:37:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:37:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:37:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:37:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:37:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:37:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:37:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:37:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:37:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:37:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:37:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:37:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:37:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:37:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:37:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:37:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:37:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:37:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:37:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:37:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:37:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:37:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:37:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:37:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:37:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:37:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:37:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:37:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:37:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:37:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:37:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:37:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:37:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:37:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:37:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:37:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:37:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:37:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:37:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:37:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:37:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:37:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:37:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:37:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:37:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:37:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:37:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:37:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:37:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:37:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:37:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:37:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:37:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:37:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:37:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:37:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:37:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:37:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:37:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:37:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:37:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:37:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:37:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:37:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:37:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:37:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:37:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:37:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:37:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:37:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:37:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:37:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:37:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:37:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:37:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:37:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:37:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:37:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:37:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:37:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:37:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:37:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:37:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:37:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:37:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:37:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:37:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:37:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:37:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:37:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:37:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:37:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:37:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:37:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:37:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:37:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:37:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:37:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:37:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:37:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:37:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:37:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:37:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:37:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:37:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:37:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:37:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:37:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:37:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:37:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:37:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:37:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:37:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:37:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:37:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:37:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:37:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:37:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:37:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:37:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:37:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:37:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:37:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:37:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:37:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:37:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:37:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:37:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:37:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:37:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:37:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:37:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:37:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:37:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:37:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:37:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:37:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:37:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:37:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:37:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:37:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:37:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:37:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:37:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:37:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:37:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:37:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:37:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:37:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:37:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:37:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:37:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:37:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:37:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:37:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:37:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:37:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:37:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:37:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:37:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:37:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:37:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:37:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:37:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:37:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:37:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:37:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:38:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:38:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:38:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:38:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:38:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:38:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:38:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:38:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:38:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:38:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:38:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:38:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:38:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:38:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:38:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:38:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:38:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:38:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:38:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:38:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:38:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:38:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:38:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:38:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:38:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:38:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:38:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:38:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:38:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:38:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:38:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:38:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:38:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:38:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:38:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:38:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:38:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:38:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:38:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:38:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:38:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:38:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:38:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:38:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:38:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:38:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:38:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:38:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:38:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:38:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:38:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:38:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:38:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:38:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:40:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:40:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:40:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:40:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:40:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:40:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:40:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:40:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:40:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:40:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:40:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:40:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:40:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:40:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:40:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:40:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:40:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:40:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:40:21 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 22:40:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:40:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:40:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:40:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:40:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:40:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:40:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:40:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:40:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:40:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:40:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:40:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:40:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:40:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:40:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:40:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:40:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:40:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:40:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:40:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:40:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:40:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:40:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:40:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:40:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:40:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:40:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:40:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:40:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:40:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:40:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:40:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:40:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:40:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:40:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:40:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:40:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:40:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:40:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:40:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:40:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:40:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:40:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:40:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:40:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:40:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:40:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:40:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:40:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:40:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:40:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:40:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:40:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:40:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:40:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:40:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:40:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:40:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:40:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:40:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:40:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:40:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:40:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:40:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:40:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:40:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:40:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:40:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:40:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:40:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:40:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:40:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:40:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:40:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:40:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:40:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:40:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:40:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:40:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:40:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:40:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:40:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:40:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:40:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:40:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:40:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:40:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:40:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:40:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:40:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:40:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:40:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:40:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:40:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:40:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:40:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:40:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:40:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:40:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:40:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:40:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:40:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:40:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:40:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:40:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:40:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:40:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:40:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:40:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:40:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:40:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:40:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:40:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:40:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:40:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:40:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:40:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:40:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:40:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:40:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:40:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:40:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:40:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:40:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:40:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:40:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:41:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:41:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:41:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:41:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:41:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:41:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:41:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:41:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:41:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:41:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:41:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:41:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:41:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:41:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:41:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:41:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:41:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:41:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:41:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:41:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:41:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:41:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:41:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:41:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:41:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:41:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:41:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:41:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:41:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:41:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:41:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:41:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:41:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:41:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:41:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:41:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:41:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:41:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:41:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:41:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:41:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:41:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:41:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:41:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:41:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:41:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:41:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:41:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:41:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:41:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:41:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:41:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:41:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:41:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:41:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:41:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:41:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:41:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:41:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:41:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:41:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:41:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:41:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:41:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:41:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:41:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:41:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:41:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:41:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:41:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:41:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:41:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:41:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:41:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:41:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:41:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:41:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:41:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:41:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:41:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:41:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:41:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:41:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:41:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:41:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:41:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:41:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:41:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:41:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:41:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:41:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:41:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:41:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:41:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:41:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:41:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:41:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:41:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:41:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:41:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:41:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:41:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:41:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:41:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:41:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:41:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:41:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:41:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:41:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:41:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:41:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:41:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:41:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:41:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:41:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:41:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:41:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:41:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:41:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:41:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:41:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:41:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:41:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:41:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:41:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:41:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:41:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:41:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:41:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:41:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:41:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:41:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:41:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:41:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:41:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:41:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:41:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:41:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:41:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:41:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:41:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:41:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:41:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:41:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:41:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:41:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:41:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:41:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:41:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:41:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:41:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:41:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:41:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:41:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:41:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:41:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:41:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:41:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:41:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:41:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:41:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:41:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:41:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:41:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:41:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:41:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:41:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:41:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:41:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:41:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:41:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:41:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:41:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:41:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:41:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:41:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:41:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:41:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:41:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:41:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:41:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:41:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:41:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:41:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:41:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:41:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:41:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:41:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:41:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:41:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:41:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:41:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:41:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:41:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:41:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:41:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:41:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:41:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:41:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:41:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:41:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:41:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:41:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:41:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:41:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:41:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:41:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:41:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:41:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:41:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:41:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:41:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:41:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:41:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:41:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:41:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:42:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:42:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:42:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:42:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:42:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:42:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:42:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:42:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:42:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:42:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:42:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:42:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:42:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:42:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:42:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:42:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:42:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:42:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:42:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:42:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:42:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:42:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:42:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:42:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:42:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:42:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:42:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:42:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:42:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:42:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:42:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:42:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:42:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:42:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:42:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:42:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:42:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:42:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:42:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:42:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:42:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:42:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:42:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:42:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:42:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:42:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:42:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:42:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:42:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:42:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:42:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:42:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:42:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:42:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:42:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:42:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:42:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:42:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:42:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:42:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:42:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:42:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:42:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:42:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:42:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:42:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:42:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:42:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:42:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:42:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:42:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:42:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:42:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:42:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:42:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:42:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:42:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:42:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:42:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:42:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:42:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:42:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:42:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:42:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:42:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:42:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:42:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:42:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:42:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:42:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:42:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:42:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:42:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:42:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:42:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:42:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:42:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:42:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:42:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:42:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:42:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:42:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:42:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:42:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:42:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:42:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:42:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:42:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:42:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:42:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:42:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:42:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:42:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:42:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:42:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:42:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:42:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:42:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:42:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:42:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:42:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:42:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:42:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:42:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:42:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:42:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:42:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:42:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:42:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:42:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:42:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:42:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:42:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:42:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:42:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:42:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:42:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:42:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:42:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:42:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:42:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:42:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:42:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:42:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:42:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:42:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:42:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:42:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:42:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:42:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:42:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:42:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:42:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:42:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:42:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:42:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:42:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:42:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:42:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:42:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:42:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:42:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:42:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:42:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:42:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:42:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:42:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:42:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:42:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:42:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:42:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:42:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:42:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:42:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:42:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:42:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:42:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:42:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:42:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:42:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:42:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:42:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:42:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:42:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:42:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:42:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:42:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:42:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:42:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:42:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:42:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:42:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:42:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:42:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:42:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:42:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:42:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:42:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:42:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:42:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:42:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:42:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:42:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:42:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:42:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:42:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:42:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:42:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:42:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:42:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:42:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:42:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:42:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:42:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:42:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:42:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:43:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:43:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:43:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:43:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:43:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:43:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:43:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:43:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:43:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:43:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:43:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:43:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:43:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:43:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:43:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:43:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:43:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:43:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:43:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:43:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:43:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:43:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:43:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:43:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:43:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:43:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:43:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:43:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:43:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:43:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:43:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:43:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:43:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:43:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:43:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:43:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:43:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:43:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:43:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:43:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:43:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:43:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:43:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:43:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:43:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:43:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:43:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:43:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:43:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:43:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:43:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:43:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:43:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:43:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:43:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:43:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:43:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:43:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:43:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:43:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:43:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:43:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:43:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:43:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:43:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:43:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:43:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:43:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:43:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:43:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:43:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:43:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:43:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:43:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:43:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:43:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:43:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:43:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:43:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:43:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:43:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:43:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:43:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:43:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:43:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:43:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:43:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:43:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:43:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:43:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:43:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:43:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:43:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:43:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:43:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:43:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:43:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:43:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:43:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:43:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:43:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:43:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:43:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:43:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:43:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:43:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:43:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:43:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:43:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:43:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:43:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:43:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:43:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:43:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:43:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:43:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:43:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:43:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:43:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:43:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:43:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:43:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:43:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:43:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:43:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:43:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:43:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:43:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:43:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:43:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:43:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:43:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:43:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:43:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:43:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:43:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:43:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:43:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:43:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:43:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:43:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:43:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:43:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:43:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:43:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:43:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:43:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:43:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:43:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:43:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:43:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:43:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:43:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:43:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:43:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:43:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:43:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:43:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:43:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:43:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:43:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:43:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:43:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:43:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:43:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:43:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:43:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:43:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:43:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:43:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:43:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:43:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:43:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:43:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:43:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:43:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:43:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:43:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:43:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:43:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:43:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:43:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:43:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:43:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:43:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:43:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:43:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:43:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:43:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:43:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:43:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:43:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:43:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:43:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:43:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:43:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:43:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:43:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:43:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:43:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:43:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:43:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:43:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:43:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:43:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:43:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:43:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:43:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:43:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:43:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:43:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:43:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:43:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:43:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:43:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:43:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:44:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:44:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:44:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:44:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:44:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:44:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:44:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:44:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:44:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:44:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:44:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:44:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:44:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:44:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:44:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:44:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:44:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:44:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:44:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:44:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:44:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:44:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:44:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:44:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:44:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:44:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:44:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:44:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:44:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:44:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:44:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:44:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:44:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:44:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:44:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:44:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:44:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:44:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:44:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:44:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:44:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:44:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:44:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:44:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:44:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:44:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:44:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:44:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:44:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:44:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:44:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:44:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:44:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:44:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:44:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:44:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:44:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:44:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:44:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:44:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:44:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:44:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:44:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:44:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:44:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:44:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:44:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:44:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:44:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:44:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:44:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:44:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:44:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:44:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:44:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:44:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:44:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:44:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:44:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:44:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:44:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:44:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:44:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:44:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:44:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:44:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:44:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:44:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:44:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:44:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:44:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:44:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:44:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:44:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:44:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:44:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:44:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:44:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:44:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:44:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:44:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:44:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:44:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:44:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:44:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:44:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:44:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:44:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:44:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:44:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:44:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:44:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:44:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:44:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:44:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:44:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:44:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:44:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:44:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:44:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:44:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:44:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:44:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:44:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:44:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:44:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:44:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:44:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:44:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:44:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:44:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:44:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:44:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:44:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:44:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:44:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:44:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:44:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:44:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:44:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:44:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:44:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:44:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:44:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:44:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:44:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:44:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:44:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:44:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:44:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:44:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:44:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:44:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:44:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:44:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:44:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:44:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:44:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:44:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:44:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:44:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:44:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:44:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:44:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:44:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:44:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:44:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:44:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:44:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:44:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:44:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:44:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:44:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:44:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:44:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:44:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:44:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:44:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:44:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:44:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:44:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:44:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:44:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:44:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:44:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:44:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:44:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:44:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:44:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:44:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:44:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:44:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:44:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:44:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:44:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:44:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:44:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:44:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:44:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:44:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:44:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:44:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:44:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:44:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:44:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:44:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:44:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:44:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:44:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:44:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:44:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:44:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:44:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:44:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:44:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:44:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:45:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:45:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:45:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:45:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:45:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:45:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:45:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:45:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:45:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:45:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:45:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:45:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:45:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:45:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:45:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:45:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:45:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:45:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:45:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:45:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:45:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:45:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:45:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:45:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:45:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:45:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:45:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:45:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:45:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:45:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:45:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:45:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:45:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:45:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:45:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:45:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:45:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:45:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:45:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:45:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:45:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:45:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:45:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:45:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:45:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:45:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:45:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:45:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:45:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:45:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:45:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:45:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:45:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:45:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:45:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:45:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:45:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:45:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:45:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:45:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:45:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:45:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:45:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:45:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:45:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:45:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:45:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:45:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:45:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:45:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:45:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:45:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:45:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:45:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:45:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:45:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:45:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:45:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:45:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:45:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:45:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:45:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:45:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:45:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:45:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:45:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:45:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:45:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:45:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:45:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:45:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:45:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:45:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:45:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:45:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:45:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:45:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:45:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:45:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:45:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:45:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:45:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:45:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:45:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:45:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:45:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:45:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:45:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:45:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:45:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:45:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:45:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:45:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:45:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:45:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:45:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:45:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:45:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:45:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:45:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:45:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:45:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:45:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:45:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:45:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:45:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:45:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:45:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:45:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:45:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:45:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:45:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:45:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:45:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:45:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:45:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:45:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:45:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:45:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:45:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:45:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:45:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:45:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:45:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:45:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:45:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:45:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:45:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:45:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:45:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:45:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:45:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:45:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:45:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:45:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:45:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:45:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:45:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:45:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:45:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:45:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:45:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:45:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:45:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:45:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:45:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:45:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:45:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:45:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:45:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:45:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:45:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:45:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:45:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:45:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:45:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:45:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:45:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:45:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:45:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:45:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:45:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:45:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:45:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:45:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:45:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:45:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:45:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:45:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:45:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:45:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:45:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:45:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:45:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:45:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:45:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:45:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:45:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:45:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:45:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:45:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:45:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:45:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:45:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:45:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:45:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:45:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:45:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:45:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:45:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:45:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:45:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:45:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:45:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:45:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:45:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:46:00 rdma06 systemd[5685]: Starting Cleanup of User's Temporary Files and Directories...
Apr 17 22:46:00 rdma06 systemd[1]: Starting Cleanup of Temporary Directories...
Apr 17 22:46:00 rdma06 systemd[5685]: Finished Cleanup of User's Temporary Files and Directories.
Apr 17 22:46:00 rdma06 systemd[1]: systemd-tmpfiles-clean.service: Deactivated successfully.
Apr 17 22:46:00 rdma06 systemd[1]: Finished Cleanup of Temporary Directories.
Apr 17 22:46:00 rdma06 systemd[1]: run-credentials-systemd\x2dtmpfiles\x2dclean.service.mount: Deactivated successfully.
Apr 17 22:46:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:46:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:46:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:46:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:46:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:46:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:46:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:46:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:46:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:46:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:46:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:46:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:46:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:46:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:46:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:46:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:46:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:46:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:46:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:46:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:46:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:46:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:46:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:46:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:46:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:46:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:46:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:46:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:46:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:46:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:46:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:46:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:46:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:46:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:46:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:46:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:46:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:46:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:46:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:46:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:46:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:46:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:46:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:46:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:46:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:46:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:46:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:46:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:46:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:46:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:46:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:46:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:46:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:46:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:46:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:46:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:46:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:46:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:46:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:46:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:46:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:46:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:46:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:46:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:46:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:46:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:46:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:46:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:46:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:46:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:46:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:46:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:46:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:46:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:46:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:46:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:46:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:46:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:46:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:46:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:46:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:46:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:46:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:46:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:46:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:46:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:46:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:46:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:46:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:46:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:46:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:46:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:46:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:46:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:46:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:46:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:46:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:46:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:46:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:46:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:46:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:46:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:46:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:46:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:46:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:46:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:46:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:46:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:46:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:46:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:46:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:46:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:46:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:46:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:46:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:46:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:46:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:46:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:46:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:46:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:46:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:46:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:46:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:46:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:46:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:46:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:46:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:46:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:46:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:46:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:46:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:46:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:46:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:46:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:46:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:46:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:46:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:46:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:46:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:46:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:46:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:46:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:46:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:46:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:46:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:46:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:46:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:46:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:46:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:46:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:46:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:46:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:46:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:46:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:46:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:46:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:46:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:46:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:46:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:46:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:46:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:46:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:46:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:46:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:46:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:46:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:46:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:46:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:46:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:46:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:46:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:46:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:46:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:46:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:46:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:46:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:46:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:46:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:46:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:46:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:46:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:46:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:46:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:46:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:46:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:46:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:46:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:46:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:46:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:46:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:46:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:46:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:46:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:46:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:46:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:46:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:46:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:46:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:46:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:46:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:46:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:46:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:46:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:46:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:46:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:46:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:46:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:46:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:46:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:46:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:46:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:46:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:46:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:46:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:46:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:46:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:47:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:47:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:47:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:47:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:47:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:47:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:47:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:47:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:47:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:47:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:47:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:47:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:47:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:47:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:47:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:47:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:47:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:47:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:47:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:47:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:47:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:47:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:47:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:47:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:47:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:47:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:47:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:47:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:47:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:47:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:47:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:47:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:47:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:47:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:47:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:47:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:47:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:47:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:47:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:47:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:47:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:47:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:47:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:47:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:47:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:47:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:47:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:47:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:47:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:47:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:47:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:47:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:47:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:47:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:47:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:47:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:47:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:47:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:47:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:47:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:47:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:47:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:47:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:47:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:47:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:47:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:47:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:47:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:47:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:47:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:47:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:47:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:47:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:47:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:47:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:47:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:47:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:47:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:47:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:47:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:47:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:47:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:47:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:47:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:47:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:47:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:47:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:47:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:47:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:47:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:47:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:47:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:47:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:47:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:47:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:47:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:47:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:47:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:47:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:47:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:47:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:47:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:47:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:47:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:47:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:47:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:47:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:47:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:47:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:47:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:47:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:47:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:47:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:47:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:47:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:47:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:47:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:47:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:47:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:47:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:47:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:47:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:47:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:47:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:47:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:47:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:47:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:47:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:47:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:47:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:47:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:47:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:47:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:47:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:47:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:47:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:47:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:47:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:47:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:47:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:47:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:47:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:47:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:47:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:47:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:47:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:47:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:47:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:47:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:47:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:47:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:47:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:47:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:47:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:47:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:47:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:47:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:47:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:47:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:47:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:47:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:47:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:47:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:47:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:47:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:47:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:47:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:47:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:47:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:47:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:47:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:47:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:47:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:47:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:47:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:47:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:47:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:47:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:47:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:47:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:47:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:47:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:47:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:47:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:47:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:47:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:47:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:47:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:47:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:47:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:47:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:47:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:47:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:47:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:47:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:47:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:47:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:47:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:47:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:47:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:47:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:47:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:47:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:47:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:47:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:47:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:47:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:47:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:47:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:47:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:47:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:47:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:47:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:47:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:47:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:47:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:48:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:48:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:48:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:48:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:48:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:48:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:48:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:48:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:48:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:48:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:48:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:48:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:48:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:48:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:48:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:48:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:48:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:48:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:48:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:48:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:48:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:48:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:48:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:48:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:48:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:48:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:48:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:48:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:48:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:48:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:48:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:48:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:48:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:48:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:48:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:48:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:48:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:48:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:48:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:48:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:48:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:48:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:48:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:48:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:48:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:48:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:48:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:48:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:48:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:48:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:48:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:48:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:48:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:48:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:48:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:48:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:48:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:48:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:48:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:48:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:48:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:48:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:48:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:48:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:48:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:48:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:48:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:48:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:48:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:48:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:48:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:48:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:50:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:50:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:50:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:50:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:50:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:50:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:50:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:50:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:50:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:50:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:50:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:50:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:50:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:50:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:50:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:50:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:50:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:50:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:50:26 rdma06 rsyslogd[1041]: imjournal: 5304 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 22:50:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:50:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:50:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:50:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:50:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:50:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:50:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:50:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:50:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:50:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:50:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:50:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:50:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:50:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:50:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:50:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:50:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:50:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:50:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:50:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:50:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:50:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:50:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:50:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:50:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:50:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:50:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:50:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:50:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:50:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:50:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:50:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:50:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:50:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:50:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:50:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:50:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:50:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:50:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:50:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:50:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:50:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:50:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:50:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:50:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:50:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:50:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:50:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:50:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:50:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:50:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:50:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:50:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:50:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:50:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:50:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:50:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:50:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:50:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:50:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:50:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:50:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:50:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:50:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:50:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:50:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:50:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:50:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:50:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:50:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:50:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:50:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:50:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:50:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:50:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:50:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:50:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:50:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:50:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:50:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:50:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:50:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:50:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:50:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:50:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:50:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:50:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:50:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:50:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:50:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:50:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:50:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:50:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:50:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:50:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:50:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:50:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:50:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:50:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:50:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:50:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:50:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:50:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:50:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:50:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:50:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:50:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:50:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:51:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:51:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:51:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:51:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:51:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:51:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:51:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:51:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:51:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:51:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:51:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:51:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:51:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:51:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:51:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:51:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:51:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:51:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:51:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:51:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:51:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:51:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:51:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:51:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:51:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:51:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:51:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:51:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:51:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:51:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:51:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:51:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:51:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:51:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:51:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:51:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:51:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:51:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:51:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:51:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:51:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:51:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:51:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:51:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:51:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:51:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:51:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:51:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:51:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:51:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:51:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:51:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:51:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:51:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:51:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:51:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:51:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:51:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:51:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:51:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:51:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:51:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:51:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:51:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:51:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:51:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:51:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:51:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:51:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:51:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:51:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:51:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:51:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:51:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:51:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:51:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:51:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:51:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:51:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:51:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:51:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:51:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:51:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:51:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:51:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:51:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:51:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:51:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:51:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:51:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:51:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:51:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:51:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:51:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:51:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:51:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:51:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:51:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:51:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:51:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:51:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:51:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:51:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:51:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:51:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:51:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:51:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:51:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:51:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:51:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:51:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:51:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:51:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:51:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:51:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:51:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:51:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:51:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:51:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:51:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:51:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:51:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:51:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:51:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:51:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:51:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:51:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:51:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:51:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:51:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:51:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:51:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:51:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:51:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:51:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:51:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:51:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:51:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:51:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:51:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:51:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:51:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:51:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:51:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:51:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:51:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:51:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:51:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:51:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:51:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:51:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:51:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:51:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:51:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:51:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:51:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:51:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:51:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:51:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:51:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:51:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:51:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:51:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:51:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:51:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:51:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:51:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:51:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:51:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:51:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:51:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:51:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:51:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:51:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:51:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:51:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:51:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:51:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:51:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:51:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:51:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:51:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:51:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:51:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:51:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:51:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:51:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:51:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:51:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:51:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:51:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:51:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:51:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:51:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:51:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:51:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:51:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:51:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:51:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:51:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:51:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:51:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:51:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:51:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:51:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:51:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:51:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:51:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:51:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:51:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:51:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:51:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:51:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:51:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:51:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:51:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:52:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:52:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:52:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:52:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:52:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:52:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:52:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:52:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:52:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:52:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:52:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:52:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:52:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:52:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:52:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:52:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:52:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:52:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:52:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:52:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:52:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:52:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:52:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:52:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:52:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:52:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:52:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:52:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:52:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:52:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:52:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:52:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:52:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:52:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:52:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:52:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:52:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:52:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:52:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:52:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:52:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:52:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:52:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:52:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:52:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:52:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:52:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:52:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:52:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:52:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:52:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:52:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:52:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:52:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:52:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:52:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:52:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:52:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:52:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:52:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:52:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:52:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:52:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:52:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:52:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:52:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:52:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:52:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:52:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:52:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:52:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:52:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:52:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:52:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:52:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:52:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:52:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:52:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:52:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:52:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:52:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:52:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:52:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:52:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:52:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:52:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:52:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:52:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:52:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:52:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:52:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:52:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:52:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:52:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:52:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:52:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:52:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:52:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:52:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:52:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:52:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:52:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:52:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:52:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:52:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:52:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:52:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:52:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:52:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:52:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:52:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:52:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:52:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:52:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:52:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:52:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:52:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:52:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:52:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:52:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:52:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:52:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:52:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:52:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:52:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:52:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:52:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:52:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:52:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:52:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:52:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:52:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:52:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:52:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:52:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:52:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:52:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:52:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:52:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:52:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:52:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:52:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:52:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:52:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:52:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:52:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:52:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:52:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:52:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:52:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:52:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:52:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:52:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:52:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:52:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:52:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:52:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:52:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:52:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:52:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:52:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:52:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:52:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:52:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:52:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:52:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:52:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:52:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:52:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:52:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:52:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:52:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:52:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:52:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:52:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:52:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:52:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:52:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:52:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:52:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:52:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:52:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:52:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:52:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:52:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:52:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:52:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:52:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:52:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:52:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:52:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:52:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:52:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:52:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:52:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:52:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:52:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:52:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:52:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:52:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:52:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:52:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:52:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:52:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:52:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:52:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:52:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:52:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:52:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:52:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:52:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:52:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:52:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:52:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:52:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:52:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:53:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:53:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:53:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:53:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:53:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:53:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:53:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:53:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:53:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:53:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:53:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:53:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:53:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:53:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:53:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:53:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:53:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:53:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:53:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:53:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:53:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:53:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:53:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:53:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:53:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:53:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:53:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:53:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:53:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:53:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:53:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:53:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:53:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:53:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:53:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:53:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:53:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:53:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:53:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:53:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:53:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:53:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:53:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:53:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:53:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:53:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:53:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:53:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:53:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:53:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:53:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:53:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:53:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:53:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:53:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:53:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:53:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:53:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:53:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:53:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:53:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:53:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:53:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:53:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:53:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:53:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:53:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:53:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:53:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:53:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:53:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:53:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:53:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:53:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:53:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:53:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:53:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:53:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:53:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:53:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:53:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:53:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:53:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:53:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:53:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:53:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:53:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:53:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:53:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:53:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:53:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:53:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:53:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:53:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:53:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:53:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:53:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:53:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:53:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:53:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:53:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:53:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:53:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:53:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:53:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:53:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:53:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:53:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:53:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:53:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:53:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:53:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:53:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:53:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:53:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:53:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:53:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:53:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:53:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:53:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:53:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:53:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:53:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:53:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:53:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:53:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:53:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:53:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:53:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:53:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:53:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:53:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:53:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:53:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:53:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:53:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:53:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:53:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:53:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:53:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:53:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:53:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:53:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:53:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:53:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:53:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:53:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:53:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:53:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:53:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:53:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:53:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:53:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:53:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:53:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:53:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:53:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:53:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:53:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:53:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:53:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:53:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:53:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:53:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:53:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:53:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:53:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:53:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:53:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:53:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:53:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:53:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:53:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:53:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:53:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:53:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:53:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:53:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:53:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:53:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:53:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:53:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:53:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:53:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:53:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:53:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:53:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:53:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:53:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:53:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:53:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:53:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:53:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:53:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:53:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:53:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:53:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:53:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:53:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:53:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:53:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:53:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:53:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:53:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:53:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:53:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:53:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:53:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:53:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:53:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:53:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:53:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:53:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:53:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:53:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:53:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:54:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:54:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:54:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:54:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:54:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:54:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:54:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:54:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:54:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:54:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:54:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:54:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:54:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:54:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:54:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:54:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:54:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:54:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:54:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:54:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:54:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:54:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:54:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:54:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:54:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:54:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:54:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:54:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:54:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:54:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:54:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:54:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:54:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:54:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:54:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:54:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:54:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:54:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:54:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:54:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:54:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:54:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:54:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:54:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:54:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:54:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:54:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:54:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:54:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:54:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:54:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:54:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:54:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:54:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:54:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:54:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:54:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:54:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:54:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:54:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:54:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:54:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:54:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:54:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:54:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:54:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:54:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:54:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:54:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:54:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:54:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:54:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:54:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:54:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:54:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:54:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:54:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:54:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:54:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:54:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:54:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:54:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:54:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:54:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:54:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:54:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:54:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:54:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:54:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:54:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:54:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:54:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:54:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:54:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:54:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:54:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:54:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:54:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:54:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:54:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:54:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:54:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:54:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:54:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:54:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:54:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:54:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:54:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:54:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:54:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:54:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:54:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:54:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:54:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:54:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:54:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:54:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:54:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:54:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:54:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:54:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:54:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:54:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:54:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:54:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:54:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:54:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:54:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:54:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:54:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:54:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:54:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:54:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:54:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:54:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:54:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:54:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:54:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:54:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:54:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:54:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:54:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:54:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:54:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:54:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:54:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:54:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:54:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:54:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:54:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:54:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:54:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:54:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:54:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:54:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:54:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:54:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:54:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:54:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:54:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:54:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:54:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:54:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:54:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:54:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:54:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:54:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:54:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:54:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:54:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:54:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:54:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:54:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:54:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:54:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:54:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:54:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:54:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:54:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:54:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:54:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:54:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:54:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:54:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:54:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:54:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:54:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:54:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:54:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:54:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:54:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:54:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:54:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:54:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:54:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:54:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:54:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:54:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:54:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:54:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:54:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:54:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:54:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:54:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:54:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:54:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:54:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:54:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:54:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:54:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:54:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:54:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:54:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:54:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:54:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:54:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:55:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:55:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:55:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:55:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:55:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:55:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:55:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:55:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:55:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:55:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:55:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:55:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:55:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:55:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:55:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:55:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:55:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:55:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:55:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:55:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:55:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:55:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:55:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:55:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:55:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:55:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:55:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:55:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:55:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:55:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:55:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:55:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:55:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:55:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:55:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:55:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:55:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:55:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:55:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:55:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:55:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:55:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:55:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:55:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:55:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:55:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:55:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:55:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:55:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:55:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:55:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:55:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:55:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:55:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:55:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:55:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:55:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:55:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:55:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:55:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:55:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:55:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:55:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:55:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:55:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:55:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:55:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:55:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:55:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:55:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:55:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:55:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:55:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:55:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:55:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:55:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:55:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:55:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:55:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:55:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:55:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:55:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:55:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:55:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:55:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:55:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:55:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:55:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:55:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:55:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:55:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:55:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:55:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:55:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:55:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:55:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:55:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:55:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:55:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:55:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:55:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:55:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:55:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:55:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:55:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:55:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:55:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:55:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:55:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:55:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:55:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:55:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:55:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:55:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:55:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:55:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:55:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:55:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:55:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:55:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:55:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:55:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:55:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:55:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:55:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:55:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:55:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:55:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:55:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:55:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:55:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:55:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:55:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:55:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:55:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:55:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:55:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:55:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:55:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:55:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:55:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:55:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:55:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:55:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:55:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:55:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:55:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:55:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:55:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:55:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:55:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:55:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:55:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:55:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:55:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:55:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:55:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:55:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:55:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:55:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:55:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:55:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:55:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:55:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:55:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:55:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:55:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:55:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:55:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:55:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:55:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:55:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:55:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:55:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:55:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:55:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:55:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:55:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:55:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:55:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:55:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:55:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:55:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:55:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:55:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:55:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:55:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:55:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:55:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:55:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:55:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:55:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:55:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:55:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:55:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:55:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:55:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:55:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:55:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:55:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:55:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:55:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:55:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:55:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:55:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:55:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:55:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:55:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:55:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:55:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:55:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:55:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:55:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:55:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:55:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:55:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:56:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:56:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:56:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:56:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:56:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:56:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:56:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:56:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:56:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:56:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:56:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:56:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:56:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:56:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:56:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:56:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:56:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:56:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:56:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:56:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:56:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:56:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:56:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:56:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:56:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:56:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:56:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:56:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:56:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:56:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:56:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:56:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:56:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:56:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:56:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:56:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:56:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:56:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:56:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:56:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:56:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:56:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:56:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:56:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:56:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:56:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:56:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:56:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:56:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:56:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:56:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:56:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:56:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:56:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:56:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:56:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:56:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:56:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:56:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:56:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:56:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:56:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:56:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:56:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:56:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:56:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:56:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:56:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:56:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:56:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:56:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:56:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:56:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:56:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:56:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:56:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:56:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:56:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:56:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:56:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:56:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:56:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:56:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:56:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:56:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:56:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:56:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:56:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:56:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:56:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:56:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:56:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:56:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:56:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:56:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:56:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:56:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:56:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:56:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:56:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:56:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:56:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:56:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:56:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:56:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:56:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:56:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:56:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:56:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:56:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:56:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:56:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:56:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:56:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:56:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:56:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:56:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:56:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:56:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:56:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:56:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:56:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:56:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:56:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:56:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:56:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:56:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:56:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:56:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:56:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:56:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:56:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:56:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:56:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:56:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:56:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:56:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:56:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:56:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:56:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:56:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:56:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:56:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:56:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:56:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:56:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:56:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:56:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:56:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:56:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:56:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:56:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:56:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:56:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:56:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:56:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:56:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:56:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:56:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:56:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:56:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:56:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:56:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:56:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:56:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:56:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:56:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:56:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:56:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:56:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:56:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:56:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:56:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:56:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:56:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:56:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:56:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:56:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:56:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:56:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:56:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:56:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:56:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:56:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:56:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:56:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:56:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:56:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:56:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:56:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:56:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:56:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:56:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:56:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:56:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:56:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:56:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:56:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:56:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:56:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:56:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:56:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:56:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:56:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:56:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:56:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:56:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:56:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:56:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:56:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:56:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:56:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:56:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:56:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:56:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:56:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:57:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:57:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:57:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:57:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:57:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:57:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:57:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:57:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:57:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:57:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:57:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:57:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:57:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:57:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:57:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:57:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:57:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:57:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:57:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:57:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:57:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:57:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:57:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:57:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:57:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:57:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:57:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:57:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:57:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:57:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:57:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:57:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:57:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:57:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:57:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:57:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:57:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:57:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:57:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:57:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:57:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:57:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:57:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:57:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:57:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:57:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:57:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:57:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:57:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:57:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:57:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:57:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:57:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:57:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:57:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:57:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:57:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:57:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:57:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:57:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:57:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:57:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:57:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:57:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:57:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:57:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:57:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:57:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:57:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:57:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:57:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:57:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:57:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:57:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:57:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:57:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:57:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:57:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:57:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:57:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:57:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:57:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:57:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:57:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:57:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:57:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:57:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:57:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:57:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:57:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:57:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:57:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:57:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:57:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:57:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:57:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:57:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:57:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:57:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:57:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:57:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:57:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:57:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:57:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:57:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:57:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:57:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:57:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:57:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:57:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:57:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:57:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:57:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:57:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:57:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:57:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:57:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:57:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:57:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:57:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:57:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:57:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:57:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:57:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:57:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:57:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:57:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:57:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:57:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:57:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:57:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:57:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:57:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:57:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:57:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:57:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:57:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:57:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:57:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:57:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:57:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:57:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:57:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:57:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:57:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:57:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:57:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:57:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:57:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:57:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:57:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:57:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:57:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:57:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:57:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:57:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:57:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:57:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:57:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:57:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:57:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:57:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:57:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:57:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:57:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:57:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:57:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:57:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:57:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:57:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:57:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:57:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:57:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:57:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:57:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:57:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:57:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:57:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:57:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:57:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:57:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:57:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:57:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:57:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:57:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:57:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:57:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:57:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:57:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:57:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:57:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:57:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:57:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:57:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:57:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:57:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:57:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:57:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:57:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:57:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:57:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:57:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:57:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:57:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:57:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:57:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:57:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:57:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:57:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:57:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:57:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:57:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:57:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:57:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:57:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:57:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:58:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:58:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:58:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:58:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:58:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:58:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:58:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:58:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:58:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:58:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:58:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:58:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:58:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:58:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:58:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:58:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:58:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:58:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:58:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:58:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:58:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:58:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:58:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:58:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:58:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:58:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:58:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:58:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:58:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:58:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:58:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:58:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:58:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:58:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:58:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:58:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:58:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:58:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:58:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:58:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:58:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:58:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:58:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:58:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:58:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:58:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:58:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:58:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:58:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:58:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:58:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:58:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:58:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:58:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:58:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:58:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:58:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:58:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:58:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:58:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:58:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:58:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:58:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:58:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:58:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:58:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:58:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:58:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:58:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:58:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:58:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:58:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 22:58:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 22:58:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:58:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:58:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:58:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 22:58:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 22:58:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 22:58:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 22:58:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 22:58:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 22:58:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 22:58:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 22:58:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 22:58:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 22:58:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 22:58:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 22:58:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 22:58:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:00:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:00:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:00:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:00:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:00:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:00:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:00:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:00:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:00:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:00:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:00:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:00:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:00:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:00:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:00:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:00:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:00:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:00:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:00:31 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 23:00:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:00:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:00:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:00:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:00:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:00:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:00:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:00:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:00:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:00:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:00:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:00:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:00:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:00:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:00:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:00:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:00:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:00:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:00:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:00:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:00:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:00:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:00:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:00:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:00:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:00:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:00:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:00:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:00:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:00:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:00:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:00:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:00:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:00:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:00:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:00:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:00:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:00:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:00:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:00:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:00:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:00:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:00:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:00:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:00:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:00:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:00:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:00:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:00:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:00:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:00:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:00:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:00:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:00:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:00:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:00:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:00:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:00:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:00:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:00:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:00:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:00:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:00:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:00:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:00:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:00:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:00:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:00:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:00:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:00:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:00:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:00:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:00:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:00:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:00:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:00:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:00:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:00:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:00:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:00:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:00:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:00:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:00:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:00:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:00:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:00:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:00:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:00:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:00:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:00:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:01:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:01:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:01:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:01:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:01:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:01:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:01:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:01:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:01:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:01:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:01:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:01:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:01:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:01:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:01:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:01:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:01:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:01:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:01:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:01:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:01:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:01:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:01:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:01:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:01:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:01:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:01:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:01:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:01:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:01:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:01:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:01:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:01:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:01:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:01:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:01:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:01:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:01:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:01:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:01:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:01:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:01:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:01:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:01:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:01:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:01:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:01:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:01:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:01:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:01:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:01:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:01:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:01:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:01:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:01:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:01:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:01:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:01:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:01:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:01:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:01:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:01:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:01:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:01:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:01:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:01:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:01:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:01:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:01:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:01:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:01:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:01:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:01:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:01:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:01:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:01:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:01:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:01:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:01:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:01:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:01:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:01:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:01:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:01:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:01:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:01:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:01:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:01:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:01:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:01:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:01:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:01:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:01:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:01:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:01:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:01:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:01:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:01:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:01:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:01:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:01:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:01:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:01:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:01:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:01:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:01:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:01:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:01:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:01:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:01:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:01:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:01:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:01:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:01:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:01:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:01:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:01:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:01:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:01:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:01:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:01:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:01:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:01:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:01:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:01:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:01:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:01:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:01:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:01:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:01:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:01:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:01:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:01:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:01:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:01:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:01:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:01:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:01:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:01:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:01:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:01:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:01:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:01:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:01:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:01:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:01:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:01:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:01:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:01:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:01:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:01:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:01:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:01:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:01:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:01:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:01:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:01:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:01:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:01:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:01:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:01:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:01:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:01:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:01:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:01:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:01:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:01:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:01:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:01:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:01:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:01:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:01:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:01:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:01:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:01:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:01:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:01:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:01:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:01:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:01:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:01:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:01:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:01:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:01:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:01:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:01:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:01:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:01:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:01:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:01:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:01:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:01:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:01:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:01:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:01:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:01:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:01:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:01:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:01:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:01:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:01:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:01:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:01:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:01:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:01:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:01:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:01:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:01:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:01:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:01:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:01:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:01:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:01:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:01:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:01:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:01:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:02:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:02:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:02:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:02:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:02:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:02:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:02:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:02:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:02:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:02:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:02:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:02:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:02:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:02:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:02:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:02:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:02:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:02:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:02:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:02:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:02:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:02:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:02:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:02:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:02:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:02:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:02:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:02:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:02:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:02:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:02:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:02:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:02:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:02:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:02:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:02:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:02:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:02:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:02:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:02:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:02:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:02:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:02:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:02:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:02:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:02:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:02:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:02:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:02:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:02:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:02:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:02:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:02:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:02:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:02:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:02:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:02:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:02:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:02:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:02:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:02:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:02:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:02:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:02:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:02:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:02:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:02:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:02:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:02:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:02:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:02:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:02:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:02:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:02:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:02:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:02:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:02:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:02:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:02:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:02:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:02:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:02:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:02:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:02:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:02:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:02:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:02:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:02:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:02:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:02:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:02:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:02:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:02:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:02:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:02:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:02:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:02:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:02:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:02:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:02:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:02:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:02:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:02:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:02:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:02:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:02:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:02:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:02:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:02:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:02:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:02:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:02:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:02:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:02:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:02:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:02:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:02:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:02:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:02:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:02:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:02:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:02:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:02:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:02:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:02:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:02:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:02:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:02:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:02:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:02:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:02:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:02:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:02:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:02:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:02:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:02:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:02:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:02:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:02:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:02:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:02:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:02:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:02:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:02:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:02:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:02:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:02:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:02:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:02:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:02:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:02:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:02:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:02:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:02:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:02:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:02:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:02:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:02:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:02:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:02:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:02:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:02:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:02:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:02:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:02:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:02:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:02:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:02:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:02:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:02:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:02:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:02:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:02:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:02:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:02:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:02:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:02:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:02:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:02:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:02:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:02:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:02:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:02:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:02:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:02:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:02:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:02:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:02:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:02:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:02:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:02:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:02:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:02:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:02:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:02:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:02:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:02:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:02:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:02:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:02:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:02:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:02:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:02:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:02:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:02:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:02:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:02:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:02:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:02:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:02:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:02:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:02:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:02:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:02:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:02:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:02:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:03:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:03:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:03:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:03:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:03:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:03:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:03:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:03:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:03:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:03:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:03:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:03:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:03:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:03:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:03:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:03:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:03:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:03:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:03:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:03:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:03:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:03:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:03:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:03:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:03:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:03:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:03:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:03:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:03:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:03:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:03:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:03:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:03:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:03:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:03:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:03:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:03:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:03:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:03:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:03:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:03:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:03:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:03:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:03:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:03:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:03:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:03:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:03:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:03:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:03:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:03:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:03:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:03:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:03:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:03:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:03:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:03:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:03:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:03:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:03:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:03:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:03:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:03:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:03:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:03:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:03:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:03:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:03:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:03:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:03:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:03:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:03:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:03:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:03:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:03:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:03:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:03:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:03:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:03:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:03:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:03:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:03:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:03:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:03:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:03:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:03:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:03:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:03:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:03:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:03:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:03:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:03:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:03:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:03:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:03:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:03:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:03:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:03:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:03:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:03:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:03:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:03:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:03:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:03:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:03:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:03:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:03:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:03:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:03:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:03:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:03:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:03:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:03:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:03:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:03:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:03:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:03:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:03:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:03:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:03:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:03:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:03:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:03:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:03:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:03:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:03:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:03:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:03:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:03:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:03:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:03:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:03:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:03:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:03:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:03:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:03:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:03:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:03:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:03:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:03:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:03:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:03:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:03:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:03:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:03:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:03:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:03:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:03:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:03:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:03:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:03:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:03:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:03:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:03:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:03:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:03:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:03:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:03:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:03:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:03:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:03:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:03:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:03:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:03:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:03:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:03:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:03:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:03:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:03:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:03:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:03:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:03:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:03:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:03:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:03:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:03:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:03:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:03:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:03:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:03:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:03:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:03:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:03:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:03:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:03:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:03:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:03:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:03:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:03:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:03:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:03:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:03:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:03:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:03:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:03:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:03:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:03:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:03:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:03:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:03:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:03:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:03:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:03:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:03:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:03:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:03:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:03:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:03:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:03:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:03:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:03:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:03:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:03:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:03:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:03:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:03:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:04:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:04:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:04:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:04:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:04:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:04:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:04:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:04:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:04:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:04:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:04:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:04:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:04:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:04:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:04:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:04:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:04:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:04:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:04:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:04:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:04:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:04:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:04:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:04:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:04:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:04:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:04:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:04:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:04:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:04:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:04:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:04:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:04:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:04:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:04:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:04:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:04:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:04:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:04:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:04:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:04:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:04:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:04:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:04:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:04:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:04:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:04:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:04:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:04:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:04:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:04:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:04:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:04:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:04:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:04:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:04:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:04:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:04:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:04:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:04:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:04:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:04:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:04:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:04:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:04:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:04:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:04:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:04:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:04:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:04:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:04:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:04:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:04:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:04:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:04:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:04:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:04:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:04:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:04:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:04:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:04:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:04:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:04:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:04:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:04:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:04:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:04:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:04:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:04:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:04:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:04:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:04:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:04:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:04:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:04:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:04:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:04:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:04:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:04:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:04:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:04:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:04:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:04:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:04:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:04:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:04:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:04:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:04:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:04:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:04:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:04:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:04:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:04:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:04:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:04:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:04:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:04:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:04:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:04:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:04:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:04:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:04:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:04:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:04:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:04:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:04:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:04:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:04:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:04:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:04:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:04:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:04:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:04:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:04:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:04:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:04:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:04:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:04:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:04:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:04:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:04:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:04:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:04:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:04:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:04:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:04:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:04:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:04:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:04:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:04:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:04:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:04:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:04:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:04:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:04:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:04:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:04:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:04:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:04:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:04:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:04:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:04:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:04:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:04:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:04:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:04:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:04:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:04:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:04:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:04:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:04:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:04:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:04:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:04:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:04:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:04:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:04:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:04:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:04:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:04:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:04:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:04:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:04:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:04:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:04:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:04:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:04:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:04:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:04:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:04:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:04:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:04:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:04:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:04:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:04:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:04:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:04:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:04:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:04:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:04:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:04:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:04:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:04:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:04:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:04:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:04:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:04:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:04:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:04:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:04:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:04:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:04:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:04:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:04:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:04:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:04:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:05:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:05:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:05:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:05:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:05:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:05:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:05:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:05:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:05:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:05:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:05:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:05:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:05:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:05:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:05:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:05:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:05:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:05:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:05:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:05:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:05:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:05:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:05:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:05:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:05:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:05:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:05:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:05:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:05:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:05:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:05:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:05:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:05:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:05:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:05:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:05:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:05:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:05:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:05:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:05:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:05:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:05:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:05:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:05:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:05:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:05:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:05:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:05:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:05:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:05:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:05:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:05:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:05:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:05:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:05:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:05:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:05:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:05:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:05:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:05:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:05:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:05:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:05:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:05:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:05:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:05:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:05:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:05:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:05:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:05:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:05:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:05:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:05:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:05:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:05:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:05:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:05:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:05:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:05:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:05:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:05:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:05:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:05:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:05:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:05:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:05:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:05:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:05:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:05:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:05:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:05:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:05:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:05:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:05:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:05:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:05:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:05:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:05:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:05:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:05:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:05:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:05:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:05:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:05:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:05:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:05:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:05:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:05:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:05:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:05:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:05:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:05:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:05:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:05:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:05:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:05:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:05:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:05:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:05:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:05:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:05:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:05:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:05:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:05:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:05:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:05:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:05:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:05:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:05:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:05:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:05:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:05:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:05:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:05:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:05:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:05:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:05:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:05:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:05:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:05:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:05:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:05:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:05:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:05:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:05:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:05:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:05:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:05:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:05:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:05:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:05:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:05:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:05:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:05:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:05:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:05:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:05:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:05:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:05:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:05:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:05:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:05:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:05:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:05:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:05:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:05:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:05:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:05:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:05:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:05:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:05:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:05:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:05:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:05:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:05:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:05:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:05:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:05:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:05:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:05:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:05:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:05:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:05:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:05:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:05:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:05:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:05:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:05:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:05:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:05:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:05:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:05:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:05:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:05:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:05:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:05:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:05:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:05:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:05:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:05:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:05:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:05:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:05:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:05:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:05:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:05:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:05:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:05:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:05:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:05:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:05:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:05:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:05:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:05:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:05:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:05:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:06:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:06:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:06:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:06:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:06:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:06:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:06:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:06:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:06:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:06:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:06:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:06:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:06:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:06:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:06:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:06:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:06:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:06:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:06:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:06:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:06:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:06:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:06:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:06:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:06:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:06:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:06:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:06:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:06:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:06:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:06:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:06:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:06:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:06:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:06:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:06:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:06:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:06:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:06:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:06:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:06:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:06:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:06:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:06:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:06:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:06:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:06:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:06:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:06:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:06:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:06:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:06:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:06:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:06:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:06:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:06:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:06:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:06:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:06:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:06:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:06:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:06:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:06:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:06:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:06:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:06:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:06:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:06:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:06:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:06:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:06:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:06:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:06:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:06:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:06:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:06:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:06:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:06:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:06:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:06:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:06:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:06:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:06:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:06:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:06:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:06:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:06:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:06:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:06:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:06:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:06:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:06:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:06:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:06:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:06:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:06:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:06:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:06:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:06:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:06:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:06:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:06:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:06:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:06:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:06:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:06:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:06:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:06:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:06:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:06:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:06:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:06:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:06:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:06:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:06:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:06:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:06:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:06:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:06:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:06:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:06:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:06:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:06:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:06:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:06:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:06:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:06:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:06:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:06:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:06:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:06:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:06:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:06:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:06:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:06:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:06:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:06:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:06:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:06:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:06:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:06:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:06:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:06:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:06:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:06:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:06:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:06:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:06:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:06:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:06:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:06:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:06:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:06:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:06:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:06:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:06:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:06:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:06:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:06:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:06:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:06:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:06:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:06:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:06:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:06:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:06:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:06:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:06:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:06:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:06:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:06:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:06:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:06:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:06:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:06:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:06:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:06:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:06:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:06:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:06:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:06:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:06:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:06:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:06:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:06:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:06:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:06:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:06:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:06:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:06:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:06:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:06:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:06:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:06:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:06:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:06:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:06:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:06:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:06:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:06:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:06:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:06:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:06:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:06:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:06:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:06:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:06:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:06:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:06:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:06:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:06:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:06:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:06:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:06:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:06:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:06:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:07:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:07:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:07:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:07:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:07:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:07:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:07:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:07:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:07:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:07:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:07:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:07:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:07:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:07:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:07:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:07:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:07:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:07:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:07:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:07:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:07:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:07:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:07:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:07:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:07:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:07:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:07:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:07:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:07:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:07:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:07:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:07:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:07:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:07:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:07:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:07:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:07:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:07:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:07:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:07:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:07:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:07:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:07:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:07:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:07:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:07:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:07:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:07:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:07:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:07:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:07:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:07:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:07:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:07:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:07:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:07:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:07:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:07:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:07:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:07:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:07:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:07:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:07:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:07:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:07:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:07:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:07:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:07:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:07:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:07:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:07:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:07:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:07:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:07:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:07:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:07:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:07:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:07:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:07:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:07:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:07:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:07:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:07:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:07:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:07:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:07:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:07:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:07:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:07:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:07:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:07:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:07:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:07:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:07:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:07:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:07:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:07:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:07:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:07:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:07:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:07:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:07:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:07:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:07:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:07:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:07:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:07:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:07:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:07:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:07:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:07:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:07:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:07:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:07:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:07:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:07:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:07:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:07:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:07:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:07:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:07:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:07:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:07:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:07:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:07:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:07:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:07:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:07:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:07:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:07:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:07:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:07:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:07:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:07:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:07:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:07:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:07:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:07:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:07:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:07:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:07:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:07:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:07:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:07:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:07:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:07:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:07:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:07:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:07:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:07:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:07:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:07:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:07:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:07:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:07:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:07:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:07:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:07:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:07:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:07:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:07:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:07:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:07:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:07:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:07:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:07:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:07:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:07:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:07:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:07:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:07:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:07:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:07:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:07:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:07:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:07:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:07:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:07:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:07:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:07:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:07:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:07:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:07:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:07:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:07:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:07:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:07:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:07:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:07:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:07:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:07:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:07:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:07:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:07:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:07:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:07:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:07:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:07:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:07:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:07:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:07:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:07:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:07:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:07:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:07:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:07:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:07:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:07:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:07:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:07:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:07:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:07:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:07:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:07:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:07:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:07:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:08:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:08:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:08:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:08:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:08:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:08:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:08:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:08:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:08:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:08:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:08:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:08:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:08:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:08:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:08:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:08:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:08:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:08:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:08:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:08:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:08:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:08:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:08:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:08:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:08:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:08:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:08:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:08:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:08:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:08:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:08:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:08:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:08:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:08:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:08:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:08:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:08:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:08:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:08:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:08:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:08:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:08:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:08:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:08:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:08:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:08:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:08:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:08:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:08:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:08:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:08:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:08:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:08:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:08:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:08:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:08:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:08:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:08:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:08:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:08:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:08:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:08:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:08:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:08:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:08:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:08:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:08:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:08:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:08:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:08:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:08:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:08:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:08:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:08:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:08:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:08:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:08:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:08:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:08:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:08:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:08:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:08:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:08:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:08:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:08:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:08:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:08:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:08:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:08:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:08:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:08:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:08:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:08:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:08:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:08:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:08:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:08:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:08:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:08:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:08:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:08:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:08:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:08:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:08:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:08:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:08:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:08:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:08:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:10:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:10:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:10:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:10:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:10:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:10:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:10:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:10:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:10:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:10:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:10:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:10:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:10:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:10:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:10:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:10:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:10:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:10:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:10:32 rdma06 rsyslogd[1041]: imjournal: 5086 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 23:10:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:10:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:10:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:10:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:10:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:10:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:10:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:10:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:10:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:10:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:10:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:10:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:10:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:10:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:10:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:10:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:10:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:10:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:10:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:10:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:10:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:10:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:10:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:10:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:10:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:10:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:10:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:10:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:10:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:10:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:10:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:10:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:10:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:10:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:10:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:10:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:10:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:10:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:10:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:10:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:10:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:10:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:10:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:10:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:10:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:10:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:10:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:10:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:10:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:10:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:10:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:10:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:10:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:10:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:10:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:10:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:10:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:10:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:10:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:10:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:10:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:10:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:10:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:10:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:10:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:10:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:10:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:10:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:10:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:10:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:10:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:10:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:10:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:10:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:10:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:10:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:10:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:10:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:10:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:10:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:10:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:10:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:10:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:10:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:10:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:10:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:10:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:10:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:10:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:10:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:11:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:11:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:11:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:11:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:11:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:11:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:11:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:11:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:11:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:11:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:11:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:11:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:11:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:11:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:11:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:11:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:11:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:11:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:11:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:11:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:11:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:11:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:11:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:11:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:11:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:11:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:11:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:11:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:11:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:11:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:11:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:11:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:11:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:11:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:11:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:11:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:11:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:11:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:11:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:11:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:11:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:11:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:11:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:11:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:11:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:11:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:11:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:11:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:11:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:11:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:11:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:11:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:11:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:11:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:11:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:11:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:11:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:11:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:11:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:11:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:11:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:11:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:11:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:11:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:11:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:11:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:11:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:11:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:11:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:11:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:11:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:11:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:11:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:11:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:11:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:11:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:11:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:11:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:11:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:11:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:11:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:11:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:11:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:11:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:11:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:11:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:11:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:11:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:11:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:11:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:11:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:11:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:11:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:11:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:11:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:11:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:11:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:11:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:11:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:11:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:11:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:11:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:11:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:11:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:11:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:11:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:11:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:11:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:11:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:11:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:11:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:11:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:11:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:11:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:11:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:11:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:11:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:11:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:11:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:11:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:11:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:11:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:11:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:11:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:11:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:11:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:11:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:11:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:11:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:11:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:11:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:11:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:11:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:11:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:11:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:11:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:11:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:11:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:11:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:11:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:11:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:11:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:11:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:11:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:11:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:11:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:11:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:11:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:11:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:11:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:11:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:11:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:11:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:11:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:11:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:11:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:11:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:11:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:11:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:11:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:11:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:11:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:11:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:11:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:11:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:11:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:11:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:11:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:11:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:11:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:11:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:11:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:11:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:11:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:11:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:11:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:11:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:11:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:11:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:11:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:11:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:11:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:11:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:11:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:11:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:11:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:11:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:11:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:11:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:11:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:11:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:11:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:11:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:11:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:11:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:11:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:11:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:11:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:11:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:11:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:11:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:11:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:11:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:11:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:11:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:11:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:11:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:11:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:11:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:11:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:11:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:11:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:11:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:11:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:11:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:11:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:12:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:12:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:12:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:12:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:12:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:12:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:12:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:12:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:12:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:12:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:12:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:12:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:12:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:12:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:12:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:12:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:12:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:12:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:12:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:12:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:12:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:12:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:12:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:12:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:12:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:12:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:12:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:12:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:12:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:12:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:12:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:12:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:12:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:12:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:12:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:12:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:12:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:12:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:12:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:12:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:12:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:12:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:12:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:12:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:12:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:12:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:12:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:12:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:12:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:12:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:12:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:12:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:12:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:12:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:12:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:12:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:12:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:12:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:12:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:12:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:12:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:12:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:12:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:12:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:12:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:12:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:12:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:12:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:12:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:12:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:12:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:12:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:12:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:12:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:12:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:12:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:12:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:12:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:12:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:12:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:12:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:12:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:12:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:12:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:12:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:12:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:12:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:12:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:12:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:12:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:12:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:12:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:12:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:12:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:12:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:12:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:12:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:12:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:12:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:12:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:12:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:12:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:12:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:12:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:12:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:12:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:12:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:12:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:12:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:12:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:12:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:12:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:12:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:12:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:12:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:12:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:12:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:12:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:12:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:12:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:12:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:12:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:12:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:12:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:12:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:12:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:12:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:12:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:12:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:12:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:12:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:12:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:12:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:12:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:12:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:12:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:12:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:12:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:12:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:12:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:12:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:12:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:12:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:12:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:12:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:12:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:12:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:12:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:12:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:12:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:12:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:12:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:12:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:12:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:12:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:12:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:12:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:12:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:12:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:12:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:12:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:12:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:12:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:12:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:12:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:12:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:12:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:12:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:12:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:12:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:12:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:12:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:12:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:12:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:12:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:12:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:12:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:12:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:12:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:12:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:12:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:12:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:12:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:12:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:12:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:12:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:12:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:12:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:12:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:12:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:12:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:12:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:12:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:12:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:12:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:12:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:12:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:12:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:12:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:12:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:12:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:12:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:12:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:12:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:12:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:12:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:12:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:12:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:12:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:12:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:12:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:12:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:12:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:12:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:12:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:12:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:13:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:13:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:13:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:13:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:13:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:13:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:13:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:13:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:13:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:13:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:13:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:13:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:13:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:13:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:13:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:13:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:13:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:13:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:13:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:13:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:13:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:13:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:13:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:13:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:13:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:13:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:13:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:13:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:13:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:13:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:13:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:13:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:13:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:13:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:13:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:13:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:13:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:13:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:13:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:13:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:13:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:13:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:13:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:13:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:13:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:13:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:13:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:13:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:13:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:13:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:13:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:13:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:13:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:13:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:13:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:13:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:13:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:13:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:13:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:13:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:13:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:13:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:13:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:13:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:13:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:13:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:13:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:13:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:13:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:13:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:13:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:13:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:13:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:13:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:13:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:13:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:13:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:13:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:13:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:13:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:13:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:13:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:13:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:13:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:13:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:13:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:13:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:13:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:13:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:13:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:13:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:13:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:13:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:13:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:13:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:13:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:13:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:13:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:13:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:13:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:13:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:13:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:13:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:13:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:13:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:13:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:13:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:13:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:13:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:13:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:13:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:13:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:13:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:13:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:13:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:13:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:13:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:13:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:13:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:13:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:13:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:13:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:13:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:13:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:13:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:13:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:13:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:13:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:13:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:13:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:13:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:13:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:13:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:13:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:13:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:13:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:13:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:13:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:13:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:13:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:13:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:13:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:13:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:13:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:13:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:13:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:13:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:13:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:13:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:13:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:13:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:13:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:13:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:13:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:13:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:13:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:13:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:13:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:13:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:13:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:13:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:13:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:13:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:13:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:13:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:13:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:13:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:13:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:13:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:13:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:13:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:13:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:13:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:13:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:13:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:13:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:13:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:13:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:13:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:13:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:13:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:13:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:13:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:13:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:13:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:13:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:13:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:13:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:13:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:13:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:13:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:13:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:13:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:13:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:13:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:13:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:13:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:13:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:13:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:13:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:13:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:13:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:13:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:13:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:13:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:13:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:13:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:13:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:13:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:13:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:13:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:13:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:13:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:13:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:13:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:13:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:14:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:14:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:14:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:14:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:14:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:14:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:14:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:14:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:14:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:14:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:14:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:14:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:14:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:14:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:14:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:14:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:14:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:14:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:14:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:14:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:14:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:14:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:14:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:14:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:14:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:14:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:14:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:14:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:14:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:14:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:14:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:14:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:14:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:14:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:14:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:14:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:14:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:14:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:14:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:14:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:14:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:14:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:14:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:14:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:14:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:14:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:14:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:14:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:14:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:14:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:14:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:14:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:14:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:14:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:14:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:14:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:14:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:14:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:14:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:14:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:14:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:14:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:14:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:14:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:14:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:14:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:14:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:14:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:14:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:14:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:14:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:14:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:14:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:14:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:14:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:14:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:14:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:14:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:14:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:14:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:14:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:14:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:14:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:14:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:14:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:14:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:14:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:14:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:14:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:14:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:14:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:14:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:14:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:14:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:14:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:14:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:14:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:14:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:14:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:14:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:14:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:14:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:14:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:14:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:14:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:14:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:14:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:14:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:14:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:14:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:14:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:14:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:14:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:14:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:14:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:14:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:14:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:14:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:14:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:14:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:14:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:14:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:14:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:14:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:14:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:14:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:14:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:14:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:14:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:14:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:14:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:14:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:14:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:14:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:14:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:14:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:14:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:14:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:14:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:14:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:14:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:14:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:14:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:14:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:14:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:14:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:14:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:14:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:14:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:14:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:14:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:14:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:14:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:14:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:14:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:14:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:14:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:14:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:14:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:14:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:14:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:14:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:14:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:14:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:14:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:14:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:14:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:14:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:14:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:14:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:14:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:14:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:14:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:14:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:14:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:14:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:14:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:14:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:14:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:14:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:14:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:14:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:14:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:14:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:14:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:14:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:14:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:14:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:14:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:14:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:14:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:14:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:14:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:14:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:14:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:14:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:14:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:14:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:14:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:14:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:14:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:14:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:14:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:14:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:14:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:14:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:14:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:14:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:14:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:14:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:14:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:14:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:14:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:14:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:14:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:14:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:15:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:15:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:15:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:15:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:15:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:15:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:15:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:15:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:15:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:15:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:15:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:15:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:15:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:15:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:15:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:15:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:15:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:15:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:15:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:15:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:15:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:15:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:15:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:15:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:15:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:15:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:15:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:15:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:15:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:15:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:15:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:15:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:15:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:15:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:15:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:15:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:15:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:15:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:15:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:15:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:15:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:15:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:15:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:15:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:15:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:15:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:15:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:15:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:15:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:15:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:15:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:15:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:15:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:15:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:15:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:15:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:15:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:15:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:15:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:15:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:15:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:15:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:15:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:15:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:15:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:15:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:15:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:15:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:15:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:15:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:15:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:15:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:15:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:15:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:15:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:15:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:15:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:15:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:15:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:15:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:15:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:15:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:15:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:15:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:15:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:15:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:15:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:15:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:15:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:15:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:15:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:15:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:15:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:15:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:15:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:15:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:15:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:15:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:15:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:15:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:15:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:15:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:15:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:15:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:15:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:15:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:15:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:15:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:15:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:15:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:15:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:15:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:15:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:15:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:15:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:15:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:15:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:15:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:15:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:15:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:15:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:15:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:15:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:15:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:15:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:15:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:15:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:15:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:15:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:15:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:15:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:15:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:15:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:15:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:15:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:15:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:15:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:15:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:15:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:15:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:15:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:15:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:15:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:15:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:15:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:15:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:15:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:15:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:15:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:15:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:15:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:15:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:15:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:15:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:15:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:15:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:15:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:15:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:15:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:15:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:15:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:15:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:15:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:15:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:15:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:15:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:15:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:15:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:15:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:15:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:15:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:15:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:15:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:15:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:15:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:15:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:15:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:15:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:15:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:15:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:15:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:15:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:15:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:15:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:15:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:15:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:15:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:15:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:15:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:15:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:15:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:15:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:15:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:15:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:15:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:15:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:15:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:15:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:15:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:15:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:15:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:15:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:15:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:15:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:15:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:15:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:15:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:15:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:15:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:15:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:15:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:15:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:15:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:15:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:15:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:15:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:16:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:16:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:16:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:16:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:16:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:16:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:16:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:16:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:16:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:16:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:16:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:16:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:16:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:16:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:16:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:16:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:16:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:16:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:16:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:16:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:16:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:16:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:16:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:16:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:16:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:16:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:16:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:16:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:16:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:16:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:16:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:16:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:16:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:16:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:16:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:16:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:16:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:16:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:16:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:16:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:16:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:16:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:16:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:16:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:16:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:16:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:16:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:16:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:16:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:16:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:16:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:16:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:16:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:16:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:16:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:16:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:16:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:16:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:16:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:16:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:16:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:16:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:16:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:16:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:16:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:16:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:16:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:16:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:16:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:16:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:16:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:16:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:16:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:16:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:16:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:16:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:16:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:16:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:16:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:16:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:16:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:16:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:16:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:16:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:16:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:16:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:16:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:16:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:16:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:16:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:16:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:16:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:16:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:16:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:16:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:16:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:16:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:16:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:16:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:16:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:16:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:16:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:16:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:16:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:16:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:16:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:16:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:16:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:16:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:16:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:16:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:16:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:16:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:16:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:16:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:16:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:16:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:16:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:16:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:16:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:16:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:16:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:16:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:16:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:16:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:16:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:16:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:16:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:16:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:16:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:16:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:16:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:16:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:16:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:16:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:16:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:16:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:16:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:16:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:16:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:16:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:16:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:16:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:16:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:16:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:16:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:16:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:16:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:16:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:16:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:16:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:16:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:16:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:16:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:16:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:16:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:16:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:16:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:16:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:16:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:16:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:16:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:16:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:16:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:16:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:16:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:16:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:16:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:16:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:16:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:16:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:16:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:16:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:16:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:16:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:16:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:16:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:16:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:16:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:16:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:16:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:16:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:16:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:16:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:16:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:16:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:16:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:16:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:16:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:16:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:16:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:16:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:16:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:16:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:16:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:16:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:16:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:16:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:16:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:16:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:16:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:16:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:16:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:16:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:16:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:16:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:16:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:16:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:16:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:16:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:16:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:16:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:16:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:16:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:16:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:16:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:17:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:17:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:17:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:17:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:17:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:17:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:17:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:17:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:17:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:17:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:17:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:17:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:17:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:17:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:17:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:17:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:17:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:17:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:17:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:17:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:17:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:17:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:17:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:17:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:17:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:17:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:17:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:17:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:17:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:17:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:17:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:17:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:17:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:17:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:17:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:17:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:17:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:17:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:17:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:17:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:17:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:17:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:17:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:17:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:17:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:17:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:17:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:17:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:17:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:17:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:17:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:17:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:17:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:17:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:17:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:17:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:17:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:17:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:17:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:17:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:17:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:17:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:17:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:17:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:17:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:17:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:17:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:17:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:17:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:17:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:17:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:17:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:17:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:17:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:17:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:17:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:17:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:17:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:17:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:17:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:17:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:17:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:17:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:17:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:17:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:17:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:17:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:17:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:17:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:17:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:17:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:17:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:17:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:17:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:17:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:17:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:17:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:17:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:17:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:17:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:17:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:17:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:17:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:17:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:17:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:17:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:17:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:17:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:17:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:17:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:17:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:17:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:17:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:17:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:17:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:17:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:17:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:17:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:17:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:17:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:17:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:17:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:17:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:17:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:17:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:17:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:17:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:17:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:17:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:17:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:17:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:17:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:17:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:17:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:17:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:17:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:17:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:17:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:17:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:17:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:17:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:17:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:17:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:17:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:17:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:17:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:17:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:17:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:17:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:17:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:17:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:17:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:17:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:17:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:17:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:17:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:17:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:17:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:17:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:17:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:17:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:17:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:17:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:17:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:17:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:17:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:17:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:17:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:17:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:17:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:17:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:17:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:17:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:17:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:17:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:17:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:17:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:17:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:17:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:17:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:17:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:17:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:17:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:17:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:17:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:17:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:17:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:17:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:17:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:17:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:17:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:17:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:17:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:17:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:17:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:17:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:17:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:17:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:17:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:17:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:17:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:17:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:17:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:17:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:17:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:17:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:17:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:17:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:17:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:17:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:17:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:17:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:17:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:17:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:17:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:17:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:18:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:18:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:18:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:18:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:18:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:18:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:18:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:18:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:18:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:18:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:18:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:18:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:18:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:18:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:18:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:18:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:18:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:18:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:18:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:18:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:18:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:18:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:18:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:18:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:18:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:18:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:18:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:18:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:18:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:18:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:18:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:18:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:18:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:18:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:18:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:18:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:18:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:18:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:18:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:18:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:18:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:18:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:18:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:18:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:18:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:18:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:18:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:18:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:18:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:18:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:18:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:18:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:18:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:18:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:18:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:18:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:18:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:18:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:18:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:18:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:18:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:18:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:18:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:18:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:18:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:18:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:18:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:18:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:18:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:18:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:18:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:18:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:18:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:18:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:18:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:18:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:18:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:18:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:18:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:18:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:18:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:18:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:18:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:18:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:18:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:18:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:18:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:18:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:18:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:18:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:18:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:18:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:18:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:18:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:18:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:18:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:18:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:18:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:18:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:18:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:18:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:18:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:18:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:18:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:18:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:18:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:18:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:18:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:20:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:20:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:20:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:20:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:20:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:20:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:20:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:20:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:20:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:20:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:20:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:20:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:20:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:20:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:20:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:20:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:20:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:20:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:20:37 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 23:20:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:20:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:20:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:20:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:20:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:20:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:20:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:20:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:20:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:20:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:20:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:20:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:20:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:20:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:20:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:20:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:20:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:20:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:20:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:20:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:20:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:20:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:20:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:20:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:20:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:20:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:20:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:20:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:20:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:20:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:20:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:20:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:20:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:20:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:20:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:20:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:20:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:20:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:20:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:20:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:20:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:20:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:20:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:20:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:20:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:20:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:20:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:20:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:20:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:20:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:20:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:20:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:20:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:20:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:20:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:20:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:20:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:20:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:20:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:20:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:20:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:20:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:20:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:20:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:20:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:20:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:20:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:20:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:20:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:20:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:20:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:20:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:21:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:21:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:21:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:21:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:21:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:21:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:21:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:21:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:21:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:21:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:21:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:21:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:21:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:21:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:21:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:21:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:21:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:21:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:21:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:21:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:21:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:21:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:21:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:21:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:21:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:21:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:21:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:21:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:21:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:21:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:21:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:21:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:21:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:21:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:21:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:21:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:21:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:21:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:21:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:21:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:21:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:21:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:21:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:21:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:21:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:21:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:21:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:21:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:21:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:21:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:21:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:21:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:21:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:21:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:21:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:21:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:21:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:21:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:21:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:21:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:21:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:21:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:21:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:21:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:21:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:21:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:21:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:21:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:21:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:21:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:21:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:21:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:21:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:21:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:21:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:21:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:21:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:21:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:21:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:21:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:21:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:21:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:21:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:21:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:21:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:21:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:21:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:21:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:21:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:21:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:21:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:21:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:21:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:21:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:21:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:21:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:21:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:21:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:21:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:21:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:21:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:21:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:21:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:21:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:21:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:21:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:21:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:21:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:21:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:21:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:21:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:21:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:21:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:21:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:21:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:21:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:21:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:21:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:21:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:21:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:21:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:21:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:21:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:21:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:21:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:21:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:21:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:21:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:21:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:21:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:21:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:21:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:21:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:21:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:21:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:21:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:21:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:21:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:21:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:21:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:21:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:21:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:21:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:21:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:21:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:21:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:21:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:21:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:21:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:21:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:21:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:21:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:21:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:21:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:21:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:21:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:21:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:21:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:21:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:21:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:21:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:21:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:21:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:21:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:21:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:21:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:21:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:21:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:21:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:21:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:21:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:21:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:21:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:21:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:21:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:21:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:21:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:21:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:21:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:21:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:21:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:21:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:21:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:21:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:21:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:21:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:21:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:21:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:21:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:21:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:21:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:21:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:21:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:21:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:21:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:21:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:21:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:21:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:21:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:21:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:21:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:21:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:21:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:21:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:21:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:21:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:22:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:22:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:22:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:22:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:22:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:22:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:22:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:22:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:22:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:22:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:22:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:22:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:22:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:22:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:22:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:22:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:22:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:22:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:22:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:22:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:22:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:22:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:22:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:22:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:22:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:22:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:22:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:22:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:22:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:22:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:22:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:22:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:22:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:22:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:22:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:22:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:22:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:22:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:22:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:22:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:22:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:22:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:22:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:22:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:22:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:22:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:22:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:22:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:22:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:22:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:22:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:22:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:22:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:22:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:22:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:22:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:22:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:22:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:22:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:22:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:22:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:22:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:22:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:22:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:22:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:22:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:22:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:22:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:22:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:22:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:22:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:22:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:22:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:22:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:22:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:22:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:22:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:22:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:22:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:22:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:22:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:22:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:22:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:22:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:22:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:22:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:22:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:22:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:22:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:22:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:22:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:22:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:22:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:22:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:22:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:22:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:22:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:22:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:22:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:22:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:22:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:22:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:22:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:22:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:22:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:22:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:22:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:22:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:22:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:22:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:22:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:22:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:22:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:22:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:22:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:22:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:22:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:22:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:22:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:22:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:22:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:22:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:22:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:22:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:22:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:22:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:22:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:22:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:22:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:22:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:22:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:22:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:22:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:22:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:22:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:22:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:22:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:22:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:22:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:22:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:22:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:22:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:22:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:22:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:22:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:22:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:22:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:22:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:22:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:22:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:22:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:22:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:22:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:22:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:22:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:22:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:22:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:22:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:22:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:22:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:22:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:22:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:22:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:22:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:22:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:22:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:22:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:22:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:22:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:22:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:22:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:22:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:22:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:22:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:22:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:22:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:22:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:22:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:22:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:22:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:22:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:22:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:22:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:22:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:22:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:22:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:22:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:22:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:22:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:22:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:22:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:22:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:22:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:22:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:22:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:22:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:23:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:23:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:23:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:23:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:23:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:23:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:23:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:23:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:23:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:23:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:23:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:23:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:23:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:23:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:23:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:23:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:23:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:23:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:23:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:23:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:23:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:23:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:23:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:23:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:23:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:23:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:23:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:23:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:23:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:23:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:23:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:23:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:23:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:23:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:23:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:23:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:23:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:23:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:23:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:23:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:23:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:23:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:23:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:23:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:23:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:23:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:23:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:23:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:23:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:23:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:23:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:23:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:23:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:23:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:23:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:23:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:23:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:23:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:23:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:23:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:23:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:23:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:23:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:23:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:23:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:23:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:23:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:23:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:23:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:23:17 rdma06 systemd-journald[790]: Data hash table of /run/log/journal/95379d2f723847b68d9e976dee8967a9/system.journal has a fill level at 75.0 (163750 of 218332 items, 75497472 file size, 461 bytes per hash table item), suggesting rotation.
Apr 17 23:23:17 rdma06 systemd-journald[790]: /run/log/journal/95379d2f723847b68d9e976dee8967a9/system.journal: Journal header limits reached or header out-of-date, rotating.
Apr 17 23:23:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:23:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:23:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:23:17 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 17 23:23:17 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 17 23:23:17 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 17 23:23:17 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 17 23:23:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:23:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:23:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:23:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:23:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:23:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:23:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:23:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:23:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:23:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:23:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:23:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:23:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:23:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:23:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:23:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:23:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:23:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:23:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:23:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:23:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:23:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:23:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:23:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:23:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:23:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:23:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:23:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:23:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:23:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:23:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:23:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:23:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:23:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:23:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:23:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:23:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:23:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:23:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:23:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:23:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:23:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:23:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:23:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:23:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:23:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:23:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:23:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:23:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:23:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:23:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:23:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:23:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:23:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:23:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:23:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:23:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:23:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:23:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:23:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:23:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:23:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:23:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:23:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:23:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:23:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:23:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:23:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:23:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:23:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:23:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:23:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:23:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:23:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:23:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:23:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:23:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:23:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:23:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:23:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:23:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:23:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:23:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:23:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:23:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:23:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:23:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:23:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:23:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:23:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:23:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:23:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:23:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:23:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:23:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:23:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:23:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:23:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:23:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:23:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:23:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:23:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:23:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:23:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:23:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:23:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:23:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:23:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:23:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:23:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:23:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:23:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:23:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:23:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:23:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:23:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:23:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:23:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:23:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:23:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:23:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:23:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:23:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:23:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:23:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:23:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:23:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:23:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:23:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:23:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:23:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:23:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:23:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:23:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:23:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:23:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:23:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:23:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:23:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:23:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:23:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:23:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:23:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:23:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:24:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:24:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:24:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:24:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:24:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:24:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:24:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:24:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:24:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:24:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:24:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:24:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:24:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:24:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:24:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:24:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:24:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:24:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:24:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:24:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:24:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:24:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:24:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:24:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:24:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:24:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:24:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:24:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:24:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:24:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:24:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:24:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:24:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:24:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:24:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:24:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:24:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:24:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:24:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:24:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:24:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:24:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:24:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:24:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:24:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:24:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:24:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:24:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:24:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:24:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:24:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:24:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:24:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:24:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:24:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:24:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:24:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:24:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:24:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:24:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:24:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:24:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:24:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:24:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:24:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:24:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:24:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:24:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:24:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:24:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:24:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:24:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:24:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:24:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:24:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:24:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:24:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:24:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:24:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:24:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:24:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:24:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:24:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:24:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:24:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:24:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:24:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:24:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:24:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:24:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:24:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:24:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:24:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:24:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:24:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:24:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:24:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:24:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:24:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:24:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:24:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:24:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:24:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:24:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:24:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:24:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:24:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:24:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:24:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:24:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:24:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:24:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:24:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:24:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:24:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:24:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:24:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:24:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:24:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:24:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:24:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:24:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:24:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:24:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:24:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:24:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:24:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:24:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:24:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:24:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:24:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:24:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:24:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:24:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:24:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:24:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:24:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:24:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:24:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:24:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:24:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:24:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:24:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:24:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:24:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:24:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:24:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:24:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:24:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:24:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:24:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:24:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:24:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:24:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:24:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:24:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:24:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:24:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:24:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:24:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:24:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:24:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:24:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:24:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:24:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:24:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:24:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:24:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:24:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:24:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:24:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:24:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:24:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:24:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:24:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:24:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:24:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:24:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:24:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:24:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:24:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:24:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:24:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:24:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:24:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:24:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:24:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:24:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:24:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:24:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:24:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:24:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:24:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:24:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:24:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:24:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:24:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:24:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:24:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:24:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:24:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:24:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:24:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:24:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:24:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:24:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:24:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:24:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:24:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:24:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:24:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:24:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:24:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:24:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:24:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:24:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:25:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:25:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:25:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:25:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:25:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:25:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:25:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:25:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:25:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:25:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:25:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:25:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:25:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:25:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:25:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:25:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:25:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:25:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:25:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:25:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:25:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:25:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:25:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:25:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:25:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:25:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:25:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:25:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:25:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:25:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:25:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:25:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:25:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:25:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:25:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:25:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:25:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:25:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:25:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:25:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:25:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:25:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:25:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:25:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:25:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:25:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:25:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:25:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:25:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:25:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:25:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:25:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:25:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:25:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:25:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:25:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:25:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:25:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:25:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:25:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:25:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:25:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:25:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:25:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:25:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:25:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:25:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:25:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:25:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:25:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:25:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:25:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:25:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:25:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:25:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:25:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:25:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:25:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:25:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:25:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:25:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:25:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:25:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:25:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:25:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:25:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:25:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:25:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:25:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:25:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:25:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:25:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:25:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:25:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:25:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:25:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:25:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:25:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:25:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:25:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:25:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:25:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:25:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:25:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:25:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:25:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:25:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:25:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:25:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:25:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:25:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:25:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:25:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:25:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:25:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:25:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:25:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:25:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:25:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:25:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:25:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:25:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:25:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:25:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:25:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:25:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:25:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:25:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:25:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:25:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:25:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:25:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:25:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:25:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:25:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:25:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:25:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:25:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:25:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:25:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:25:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:25:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:25:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:25:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:25:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:25:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:25:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:25:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:25:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:25:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:25:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:25:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:25:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:25:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:25:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:25:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:25:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:25:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:25:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:25:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:25:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:25:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:25:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:25:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:25:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:25:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:25:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:25:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:25:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:25:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:25:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:25:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:25:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:25:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:25:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:25:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:25:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:25:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:25:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:25:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:25:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:25:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:25:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:25:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:25:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:25:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:25:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:25:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:25:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:25:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:25:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:25:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:25:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:25:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:25:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:25:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:25:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:25:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:25:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:25:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:25:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:25:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:25:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:25:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:25:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:25:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:25:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:25:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:25:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:25:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:25:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:25:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:25:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:25:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:25:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:25:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:26:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:26:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:26:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:26:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:26:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:26:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:26:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:26:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:26:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:26:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:26:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:26:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:26:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:26:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:26:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:26:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:26:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:26:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:26:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:26:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:26:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:26:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:26:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:26:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:26:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:26:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:26:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:26:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:26:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:26:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:26:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:26:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:26:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:26:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:26:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:26:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:26:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:26:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:26:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:26:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:26:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:26:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:26:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:26:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:26:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:26:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:26:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:26:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:26:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:26:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:26:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:26:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:26:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:26:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:26:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:26:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:26:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:26:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:26:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:26:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:26:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:26:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:26:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:26:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:26:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:26:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:26:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:26:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:26:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:26:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:26:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:26:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:26:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:26:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:26:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:26:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:26:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:26:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:26:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:26:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:26:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:26:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:26:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:26:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:26:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:26:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:26:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:26:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:26:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:26:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:26:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:26:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:26:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:26:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:26:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:26:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:26:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:26:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:26:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:26:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:26:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:26:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:26:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:26:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:26:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:26:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:26:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:26:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:26:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:26:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:26:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:26:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:26:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:26:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:26:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:26:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:26:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:26:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:26:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:26:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:26:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:26:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:26:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:26:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:26:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:26:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:26:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:26:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:26:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:26:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:26:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:26:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:26:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:26:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:26:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:26:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:26:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:26:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:26:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:26:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:26:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:26:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:26:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:26:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:26:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:26:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:26:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:26:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:26:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:26:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:26:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:26:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:26:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:26:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:26:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:26:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:26:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:26:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:26:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:26:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:26:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:26:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:26:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:26:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:26:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:26:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:26:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:26:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:26:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:26:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:26:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:26:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:26:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:26:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:26:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:26:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:26:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:26:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:26:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:26:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:26:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:26:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:26:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:26:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:26:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:26:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:26:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:26:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:26:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:26:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:26:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:26:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:26:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:26:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:26:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:26:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:26:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:26:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:26:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:26:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:26:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:26:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:26:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:26:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:26:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:26:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:26:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:26:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:26:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:26:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:26:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:26:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:26:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:26:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:26:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:26:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:27:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:27:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:27:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:27:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:27:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:27:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:27:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:27:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:27:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:27:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:27:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:27:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:27:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:27:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:27:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:27:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:27:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:27:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:27:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:27:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:27:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:27:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:27:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:27:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:27:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:27:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:27:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:27:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:27:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:27:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:27:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:27:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:27:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:27:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:27:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:27:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:27:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:27:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:27:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:27:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:27:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:27:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:27:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:27:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:27:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:27:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:27:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:27:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:27:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:27:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:27:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:27:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:27:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:27:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:27:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:27:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:27:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:27:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:27:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:27:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:27:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:27:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:27:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:27:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:27:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:27:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:27:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:27:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:27:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:27:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:27:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:27:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:27:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:27:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:27:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:27:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:27:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:27:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:27:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:27:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:27:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:27:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:27:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:27:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:27:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:27:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:27:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:27:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:27:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:27:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:27:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:27:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:27:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:27:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:27:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:27:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:27:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:27:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:27:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:27:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:27:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:27:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:27:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:27:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:27:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:27:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:27:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:27:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:27:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:27:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:27:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:27:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:27:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:27:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:27:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:27:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:27:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:27:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:27:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:27:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:27:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:27:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:27:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:27:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:27:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:27:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:27:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:27:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:27:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:27:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:27:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:27:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:27:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:27:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:27:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:27:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:27:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:27:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:27:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:27:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:27:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:27:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:27:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:27:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:27:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:27:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:27:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:27:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:27:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:27:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:27:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:27:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:27:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:27:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:27:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:27:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:27:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:27:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:27:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:27:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:27:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:27:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:27:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:27:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:27:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:27:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:27:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:27:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:27:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:27:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:27:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:27:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:27:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:27:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:27:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:27:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:27:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:27:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:27:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:27:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:27:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:27:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:27:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:27:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:27:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:27:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:27:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:27:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:27:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:27:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:27:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:27:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:27:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:27:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:27:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:27:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:27:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:27:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:27:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:27:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:27:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:27:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:27:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:27:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:27:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:27:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:27:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:27:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:27:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:27:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:27:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:27:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:27:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:27:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:27:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:27:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:28:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:28:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:28:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:28:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:28:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:28:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:28:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:28:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:28:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:28:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:28:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:28:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:28:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:28:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:28:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:28:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:28:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:28:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:28:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:28:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:28:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:28:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:28:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:28:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:28:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:28:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:28:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:28:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:28:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:28:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:28:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:28:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:28:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:28:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:28:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:28:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:28:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:28:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:28:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:28:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:28:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:28:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:28:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:28:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:28:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:28:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:28:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:28:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:28:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:28:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:28:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:28:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:28:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:28:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:28:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:28:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:28:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:28:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:28:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:28:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:28:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:28:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:28:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:28:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:28:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:28:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:28:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:28:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:28:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:28:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:28:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:28:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:28:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:28:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:28:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:28:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:28:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:28:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:28:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:28:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:28:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:28:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:28:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:28:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:28:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:28:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:28:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:28:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:28:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:28:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:28:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:28:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:28:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:28:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:28:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:28:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:28:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:28:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:28:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:28:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:28:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:28:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:28:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:28:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:28:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:28:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:28:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:28:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:28:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:28:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:28:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:28:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:28:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:28:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:28:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:28:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:28:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:28:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:28:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:28:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:28:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:28:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:28:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:28:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:28:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:28:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:30:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:30:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:30:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:30:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:30:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:30:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:30:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:30:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:30:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:30:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:30:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:30:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:30:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:30:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:30:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:30:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:30:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:30:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:30:43 rdma06 rsyslogd[1041]: imjournal: 5297 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 23:30:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:30:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:30:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:30:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:30:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:30:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:30:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:30:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:30:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:30:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:30:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:30:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:30:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:30:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:30:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:30:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:30:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:30:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:30:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:30:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:30:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:30:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:30:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:30:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:30:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:30:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:30:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:30:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:30:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:30:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:30:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:30:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:30:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:30:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:30:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:30:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:30:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:30:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:30:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:30:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:30:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:30:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:30:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:30:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:30:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:30:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:30:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:30:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:30:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:30:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:30:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:30:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:30:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:30:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:31:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:31:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:31:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:31:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:31:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:31:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:31:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:31:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:31:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:31:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:31:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:31:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:31:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:31:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:31:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:31:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:31:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:31:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:31:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:31:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:31:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:31:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:31:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:31:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:31:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:31:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:31:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:31:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:31:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:31:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:31:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:31:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:31:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:31:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:31:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:31:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:31:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:31:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:31:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:31:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:31:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:31:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:31:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:31:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:31:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:31:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:31:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:31:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:31:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:31:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:31:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:31:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:31:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:31:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:31:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:31:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:31:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:31:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:31:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:31:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:31:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:31:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:31:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:31:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:31:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:31:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:31:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:31:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:31:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:31:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:31:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:31:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:31:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:31:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:31:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:31:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:31:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:31:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:31:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:31:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:31:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:31:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:31:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:31:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:31:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:31:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:31:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:31:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:31:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:31:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:31:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:31:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:31:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:31:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:31:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:31:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:31:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:31:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:31:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:31:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:31:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:31:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:31:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:31:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:31:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:31:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:31:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:31:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:31:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:31:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:31:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:31:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:31:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:31:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:31:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:31:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:31:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:31:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:31:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:31:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:31:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:31:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:31:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:31:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:31:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:31:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:31:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:31:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:31:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:31:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:31:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:31:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:31:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:31:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:31:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:31:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:31:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:31:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:31:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:31:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:31:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:31:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:31:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:31:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:31:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:31:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:31:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:31:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:31:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:31:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:31:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:31:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:31:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:31:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:31:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:31:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:31:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:31:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:31:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:31:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:31:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:31:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:31:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:31:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:31:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:31:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:31:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:31:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:31:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:31:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:31:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:31:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:31:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:31:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:31:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:31:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:31:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:31:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:31:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:31:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:31:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:31:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:31:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:31:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:31:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:31:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:31:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:31:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:31:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:31:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:31:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:31:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:31:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:31:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:31:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:31:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:31:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:31:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:31:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:31:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:31:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:31:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:31:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:31:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:31:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:31:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:31:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:31:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:31:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:31:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:31:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:31:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:31:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:31:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:31:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:31:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:32:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:32:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:32:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:32:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:32:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:32:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:32:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:32:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:32:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:32:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:32:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:32:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:32:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:32:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:32:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:32:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:32:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:32:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:32:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:32:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:32:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:32:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:32:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:32:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:32:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:32:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:32:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:32:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:32:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:32:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:32:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:32:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:32:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:32:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:32:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:32:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:32:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:32:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:32:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:32:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:32:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:32:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:32:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:32:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:32:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:32:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:32:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:32:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:32:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:32:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:32:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:32:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:32:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:32:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:32:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:32:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:32:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:32:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:32:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:32:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:32:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:32:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:32:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:32:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:32:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:32:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:32:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:32:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:32:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:32:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:32:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:32:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:32:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:32:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:32:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:32:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:32:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:32:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:32:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:32:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:32:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:32:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:32:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:32:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:32:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:32:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:32:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:32:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:32:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:32:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:32:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:32:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:32:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:32:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:32:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:32:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:32:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:32:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:32:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:32:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:32:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:32:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:32:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:32:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:32:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:32:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:32:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:32:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:32:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:32:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:32:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:32:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:32:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:32:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:32:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:32:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:32:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:32:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:32:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:32:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:32:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:32:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:32:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:32:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:32:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:32:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:32:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:32:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:32:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:32:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:32:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:32:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:32:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:32:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:32:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:32:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:32:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:32:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:32:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:32:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:32:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:32:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:32:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:32:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:32:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:32:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:32:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:32:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:32:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:32:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:32:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:32:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:32:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:32:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:32:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:32:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:32:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:32:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:32:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:32:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:32:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:32:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:32:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:32:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:32:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:32:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:32:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:32:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:32:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:32:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:32:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:32:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:32:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:32:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:32:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:32:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:32:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:32:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:32:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:32:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:32:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:32:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:32:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:32:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:32:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:32:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:32:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:32:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:32:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:32:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:32:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:32:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:32:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:32:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:32:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:32:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:32:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:32:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:32:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:32:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:32:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:32:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:32:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:32:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:32:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:32:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:32:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:32:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:32:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:32:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:32:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:32:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:32:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:32:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:32:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:32:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:33:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:33:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:33:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:33:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:33:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:33:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:33:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:33:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:33:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:33:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:33:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:33:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:33:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:33:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:33:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:33:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:33:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:33:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:33:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:33:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:33:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:33:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:33:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:33:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:33:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:33:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:33:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:33:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:33:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:33:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:33:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:33:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:33:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:33:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:33:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:33:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:33:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:33:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:33:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:33:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:33:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:33:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:33:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:33:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:33:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:33:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:33:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:33:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:33:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:33:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:33:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:33:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:33:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:33:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:33:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:33:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:33:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:33:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:33:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:33:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:33:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:33:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:33:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:33:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:33:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:33:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:33:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:33:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:33:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:33:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:33:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:33:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:33:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:33:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:33:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:33:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:33:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:33:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:33:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:33:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:33:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:33:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:33:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:33:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:33:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:33:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:33:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:33:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:33:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:33:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:33:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:33:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:33:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:33:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:33:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:33:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:33:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:33:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:33:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:33:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:33:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:33:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:33:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:33:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:33:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:33:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:33:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:33:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:33:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:33:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:33:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:33:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:33:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:33:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:33:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:33:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:33:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:33:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:33:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:33:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:33:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:33:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:33:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:33:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:33:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:33:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:33:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:33:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:33:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:33:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:33:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:33:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:33:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:33:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:33:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:33:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:33:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:33:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:33:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:33:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:33:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:33:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:33:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:33:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:33:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:33:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:33:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:33:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:33:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:33:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:33:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:33:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:33:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:33:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:33:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:33:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:33:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:33:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:33:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:33:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:33:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:33:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:33:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:33:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:33:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:33:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:33:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:33:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:33:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:33:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:33:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:33:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:33:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:33:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:33:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:33:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:33:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:33:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:33:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:33:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:33:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:33:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:33:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:33:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:33:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:33:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:33:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:33:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:33:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:33:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:33:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:33:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:33:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:33:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:33:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:33:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:33:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:33:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:33:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:33:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:33:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:33:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:33:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:33:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:33:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:33:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:33:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:33:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:33:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:33:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:33:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:33:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:33:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:33:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:33:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:33:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:34:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:34:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:34:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:34:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:34:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:34:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:34:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:34:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:34:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:34:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:34:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:34:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:34:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:34:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:34:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:34:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:34:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:34:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:34:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:34:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:34:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:34:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:34:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:34:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:34:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:34:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:34:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:34:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:34:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:34:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:34:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:34:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:34:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:34:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:34:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:34:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:34:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:34:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:34:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:34:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:34:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:34:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:34:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:34:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:34:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:34:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:34:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:34:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:34:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:34:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:34:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:34:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:34:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:34:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:34:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:34:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:34:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:34:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:34:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:34:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:34:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:34:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:34:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:34:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:34:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:34:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:34:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:34:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:34:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:34:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:34:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:34:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:34:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:34:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:34:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:34:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:34:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:34:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:34:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:34:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:34:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:34:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:34:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:34:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:34:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:34:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:34:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:34:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:34:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:34:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:34:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:34:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:34:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:34:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:34:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:34:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:34:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:34:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:34:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:34:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:34:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:34:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:34:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:34:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:34:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:34:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:34:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:34:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:34:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:34:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:34:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:34:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:34:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:34:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:34:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:34:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:34:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:34:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:34:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:34:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:34:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:34:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:34:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:34:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:34:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:34:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:34:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:34:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:34:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:34:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:34:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:34:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:34:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:34:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:34:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:34:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:34:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:34:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:34:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:34:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:34:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:34:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:34:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:34:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:34:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:34:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:34:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:34:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:34:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:34:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:34:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:34:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:34:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:34:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:34:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:34:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:34:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:34:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:34:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:34:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:34:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:34:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:34:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:34:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:34:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:34:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:34:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:34:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:34:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:34:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:34:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:34:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:34:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:34:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:34:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:34:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:34:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:34:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:34:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:34:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:34:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:34:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:34:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:34:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:34:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:34:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:34:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:34:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:34:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:34:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:34:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:34:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:34:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:34:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:34:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:34:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:34:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:34:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:34:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:34:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:34:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:34:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:34:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:34:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:34:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:34:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:34:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:34:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:34:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:34:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:34:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:34:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:34:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:34:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:34:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:34:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:35:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:35:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:35:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:35:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:35:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:35:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:35:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:35:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:35:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:35:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:35:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:35:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:35:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:35:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:35:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:35:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:35:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:35:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:35:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:35:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:35:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:35:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:35:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:35:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:35:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:35:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:35:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:35:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:35:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:35:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:35:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:35:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:35:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:35:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:35:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:35:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:35:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:35:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:35:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:35:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:35:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:35:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:35:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:35:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:35:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:35:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:35:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:35:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:35:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:35:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:35:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:35:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:35:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:35:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:35:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:35:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:35:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:35:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:35:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:35:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:35:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:35:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:35:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:35:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:35:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:35:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:35:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:35:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:35:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:35:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:35:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:35:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:35:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:35:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:35:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:35:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:35:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:35:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:35:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:35:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:35:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:35:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:35:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:35:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:35:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:35:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:35:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:35:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:35:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:35:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:35:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:35:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:35:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:35:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:35:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:35:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:35:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:35:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:35:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:35:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:35:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:35:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:35:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:35:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:35:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:35:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:35:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:35:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:35:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:35:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:35:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:35:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:35:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:35:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:35:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:35:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:35:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:35:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:35:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:35:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:35:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:35:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:35:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:35:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:35:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:35:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:35:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:35:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:35:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:35:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:35:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:35:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:35:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:35:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:35:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:35:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:35:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:35:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:35:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:35:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:35:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:35:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:35:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:35:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:35:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:35:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:35:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:35:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:35:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:35:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:35:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:35:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:35:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:35:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:35:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:35:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:35:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:35:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:35:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:35:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:35:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:35:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:35:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:35:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:35:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:35:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:35:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:35:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:35:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:35:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:35:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:35:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:35:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:35:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:35:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:35:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:35:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:35:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:35:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:35:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:35:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:35:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:35:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:35:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:35:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:35:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:35:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:35:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:35:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:35:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:35:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:35:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:35:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:35:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:35:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:35:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:35:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:35:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:35:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:35:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:35:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:35:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:35:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:35:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:35:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:35:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:35:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:35:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:35:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:35:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:35:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:35:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:35:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:35:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:35:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:35:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:36:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:36:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:36:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:36:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:36:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:36:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:36:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:36:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:36:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:36:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:36:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:36:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:36:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:36:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:36:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:36:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:36:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:36:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:36:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:36:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:36:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:36:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:36:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:36:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:36:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:36:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:36:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:36:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:36:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:36:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:36:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:36:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:36:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:36:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:36:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:36:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:36:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:36:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:36:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:36:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:36:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:36:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:36:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:36:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:36:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:36:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:36:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:36:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:36:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:36:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:36:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:36:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:36:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:36:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:36:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:36:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:36:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:36:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:36:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:36:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:36:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:36:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:36:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:36:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:36:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:36:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:36:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:36:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:36:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:36:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:36:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:36:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:36:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:36:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:36:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:36:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:36:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:36:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:36:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:36:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:36:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:36:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:36:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:36:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:36:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:36:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:36:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:36:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:36:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:36:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:36:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:36:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:36:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:36:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:36:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:36:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:36:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:36:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:36:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:36:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:36:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:36:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:36:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:36:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:36:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:36:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:36:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:36:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:36:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:36:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:36:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:36:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:36:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:36:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:36:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:36:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:36:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:36:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:36:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:36:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:36:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:36:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:36:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:36:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:36:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:36:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:36:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:36:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:36:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:36:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:36:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:36:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:36:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:36:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:36:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:36:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:36:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:36:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:36:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:36:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:36:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:36:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:36:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:36:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:36:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:36:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:36:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:36:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:36:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:36:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:36:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:36:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:36:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:36:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:36:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:36:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:36:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:36:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:36:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:36:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:36:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:36:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:36:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:36:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:36:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:36:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:36:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:36:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:36:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:36:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:36:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:36:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:36:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:36:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:36:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:36:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:36:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:36:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:36:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:36:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:36:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:36:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:36:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:36:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:36:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:36:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:36:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:36:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:36:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:36:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:36:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:36:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:36:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:36:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:36:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:36:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:36:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:36:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:36:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:36:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:36:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:36:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:36:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:36:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:36:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:36:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:36:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:36:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:36:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:36:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:36:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:36:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:36:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:36:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:36:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:36:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:37:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:37:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:37:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:37:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:37:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:37:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:37:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:37:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:37:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:37:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:37:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:37:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:37:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:37:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:37:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:37:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:37:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:37:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:37:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:37:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:37:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:37:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:37:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:37:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:37:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:37:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:37:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:37:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:37:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:37:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:37:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:37:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:37:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:37:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:37:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:37:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:37:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:37:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:37:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:37:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:37:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:37:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:37:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:37:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:37:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:37:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:37:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:37:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:37:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:37:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:37:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:37:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:37:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:37:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:37:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:37:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:37:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:37:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:37:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:37:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:37:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:37:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:37:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:37:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:37:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:37:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:37:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:37:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:37:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:37:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:37:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:37:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:37:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:37:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:37:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:37:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:37:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:37:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:37:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:37:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:37:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:37:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:37:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:37:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:37:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:37:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:37:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:37:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:37:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:37:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:37:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:37:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:37:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:37:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:37:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:37:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:37:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:37:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:37:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:37:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:37:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:37:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:37:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:37:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:37:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:37:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:37:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:37:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:37:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:37:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:37:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:37:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:37:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:37:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:37:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:37:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:37:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:37:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:37:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:37:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:37:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:37:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:37:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:37:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:37:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:37:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:37:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:37:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:37:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:37:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:37:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:37:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:37:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:37:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:37:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:37:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:37:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:37:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:37:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:37:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:37:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:37:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:37:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:37:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:37:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:37:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:37:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:37:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:37:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:37:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:37:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:37:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:37:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:37:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:37:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:37:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:37:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:37:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:37:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:37:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:37:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:37:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:37:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:37:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:37:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:37:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:37:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:37:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:37:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:37:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:37:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:37:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:37:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:37:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:37:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:37:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:37:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:37:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:37:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:37:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:37:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:37:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:37:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:37:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:37:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:37:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:37:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:37:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:37:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:37:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:37:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:37:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:37:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:37:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:37:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:37:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:37:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:37:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:37:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:37:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:37:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:37:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:37:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:37:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:37:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:37:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:37:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:37:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:37:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:37:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:37:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:37:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:37:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:37:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:37:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:37:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:38:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:38:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:38:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:38:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:38:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:38:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:38:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:38:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:38:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:38:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:38:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:38:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:38:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:38:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:38:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:38:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:38:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:38:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:38:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:38:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:38:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:38:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:38:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:38:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:38:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:38:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:38:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:38:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:38:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:38:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:38:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:38:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:38:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:38:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:38:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:38:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:38:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:38:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:38:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:38:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:38:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:38:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:38:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:38:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:38:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:38:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:38:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:38:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:38:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:38:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:38:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:38:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:38:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:38:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:38:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:38:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:38:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:38:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:38:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:38:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:38:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:38:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:38:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:38:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:38:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:38:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:38:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:38:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:38:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:38:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:38:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:38:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:38:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:38:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:38:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:38:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:38:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:38:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:38:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:38:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:38:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:38:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:38:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:38:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:38:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:38:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:38:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:38:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:38:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:38:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:38:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:38:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:38:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:38:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:38:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:38:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:38:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:38:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:38:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:38:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:38:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:38:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:38:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:38:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:38:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:38:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:38:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:38:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:38:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:38:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:38:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:38:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:38:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:38:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:38:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:38:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:38:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:38:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:38:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:38:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:38:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:38:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:38:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:38:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:38:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:38:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:38:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:38:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:38:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:38:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:38:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:38:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:38:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:38:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:38:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:38:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:38:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:38:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:38:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:38:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:38:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:38:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:38:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:38:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:40:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:40:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:40:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:40:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:40:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:40:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:40:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:40:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:40:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:40:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:40:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:40:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:40:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:40:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:40:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:40:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:40:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:40:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:40:48 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 23:40:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:40:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:40:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:40:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:40:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:40:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:40:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:40:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:40:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:40:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:40:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:40:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:40:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:40:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:40:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:40:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:40:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:40:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:40:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:40:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:40:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:40:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:40:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:40:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:40:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:40:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:40:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:40:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:40:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:40:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:40:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:40:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:40:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:40:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:40:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:40:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:41:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:41:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:41:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:41:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:41:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:41:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:41:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:41:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:41:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:41:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:41:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:41:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:41:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:41:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:41:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:41:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:41:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:41:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:41:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:41:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:41:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:41:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:41:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:41:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:41:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:41:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:41:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:41:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:41:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:41:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:41:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:41:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:41:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:41:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:41:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:41:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:41:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:41:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:41:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:41:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:41:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:41:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:41:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:41:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:41:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:41:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:41:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:41:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:41:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:41:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:41:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:41:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:41:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:41:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:41:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:41:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:41:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:41:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:41:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:41:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:41:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:41:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:41:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:41:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:41:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:41:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:41:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:41:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:41:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:41:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:41:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:41:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:41:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:41:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:41:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:41:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:41:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:41:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:41:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:41:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:41:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:41:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:41:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:41:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:41:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:41:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:41:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:41:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:41:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:41:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:41:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:41:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:41:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:41:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:41:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:41:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:41:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:41:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:41:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:41:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:41:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:41:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:41:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:41:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:41:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:41:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:41:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:41:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:41:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:41:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:41:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:41:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:41:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:41:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:41:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:41:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:41:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:41:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:41:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:41:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:41:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:41:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:41:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:41:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:41:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:41:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:41:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:41:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:41:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:41:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:41:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:41:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:41:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:41:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:41:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:41:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:41:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:41:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:41:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:41:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:41:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:41:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:41:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:41:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:41:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:41:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:41:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:41:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:41:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:41:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:41:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:41:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:41:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:41:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:41:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:41:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:41:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:41:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:41:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:41:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:41:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:41:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:41:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:41:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:41:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:41:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:41:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:41:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:41:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:41:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:41:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:41:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:41:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:41:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:41:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:41:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:41:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:41:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:41:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:41:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:41:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:41:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:41:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:41:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:41:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:41:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:41:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:41:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:41:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:41:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:41:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:41:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:41:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:41:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:41:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:41:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:41:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:41:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:41:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:41:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:41:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:41:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:41:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:41:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:41:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:41:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:41:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:41:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:41:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:41:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:41:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:41:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:41:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:41:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:41:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:41:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:42:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:42:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:42:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:42:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:42:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:42:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:42:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:42:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:42:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:42:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:42:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:42:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:42:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:42:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:42:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:42:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:42:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:42:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:42:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:42:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:42:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:42:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:42:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:42:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:42:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:42:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:42:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:42:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:42:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:42:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:42:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:42:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:42:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:42:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:42:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:42:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:42:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:42:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:42:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:42:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:42:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:42:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:42:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:42:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:42:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:42:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:42:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:42:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:42:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:42:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:42:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:42:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:42:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:42:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:42:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:42:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:42:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:42:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:42:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:42:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:42:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:42:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:42:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:42:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:42:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:42:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:42:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:42:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:42:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:42:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:42:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:42:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:42:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:42:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:42:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:42:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:42:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:42:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:42:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:42:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:42:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:42:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:42:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:42:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:42:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:42:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:42:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:42:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:42:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:42:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:42:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:42:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:42:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:42:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:42:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:42:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:42:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:42:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:42:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:42:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:42:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:42:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:42:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:42:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:42:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:42:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:42:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:42:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:42:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:42:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:42:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:42:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:42:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:42:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:42:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:42:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:42:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:42:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:42:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:42:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:42:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:42:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:42:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:42:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:42:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:42:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:42:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:42:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:42:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:42:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:42:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:42:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:42:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:42:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:42:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:42:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:42:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:42:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:42:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:42:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:42:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:42:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:42:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:42:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:42:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:42:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:42:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:42:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:42:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:42:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:42:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:42:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:42:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:42:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:42:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:42:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:42:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:42:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:42:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:42:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:42:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:42:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:42:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:42:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:42:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:42:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:42:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:42:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:42:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:42:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:42:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:42:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:42:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:42:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:42:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:42:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:42:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:42:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:42:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:42:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:42:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:42:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:42:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:42:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:42:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:42:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:42:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:42:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:42:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:42:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:42:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:42:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:42:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:42:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:42:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:42:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:42:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:42:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:42:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:42:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:42:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:42:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:42:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:42:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:42:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:42:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:42:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:42:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:42:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:42:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:42:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:42:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:42:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:42:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:42:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:42:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:43:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:43:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:43:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:43:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:43:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:43:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:43:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:43:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:43:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:43:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:43:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:43:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:43:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:43:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:43:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:43:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:43:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:43:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:43:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:43:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:43:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:43:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:43:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:43:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:43:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:43:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:43:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:43:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:43:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:43:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:43:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:43:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:43:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:43:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:43:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:43:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:43:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:43:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:43:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:43:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:43:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:43:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:43:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:43:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:43:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:43:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:43:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:43:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:43:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:43:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:43:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:43:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:43:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:43:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:43:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:43:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:43:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:43:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:43:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:43:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:43:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:43:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:43:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:43:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:43:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:43:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:43:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:43:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:43:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:43:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:43:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:43:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:43:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:43:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:43:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:43:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:43:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:43:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:43:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:43:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:43:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:43:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:43:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:43:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:43:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:43:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:43:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:43:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:43:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:43:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:43:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:43:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:43:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:43:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:43:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:43:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:43:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:43:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:43:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:43:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:43:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:43:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:43:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:43:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:43:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:43:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:43:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:43:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:43:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:43:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:43:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:43:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:43:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:43:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:43:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:43:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:43:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:43:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:43:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:43:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:43:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:43:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:43:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:43:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:43:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:43:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:43:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:43:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:43:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:43:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:43:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:43:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:43:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:43:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:43:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:43:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:43:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:43:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:43:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:43:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:43:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:43:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:43:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:43:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:43:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:43:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:43:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:43:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:43:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:43:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:43:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:43:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:43:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:43:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:43:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:43:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:43:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:43:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:43:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:43:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:43:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:43:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:43:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:43:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:43:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:43:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:43:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:43:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:43:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:43:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:43:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:43:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:43:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:43:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:43:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:43:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:43:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:43:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:43:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:43:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:43:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:43:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:43:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:43:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:43:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:43:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:43:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:43:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:43:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:43:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:43:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:43:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:43:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:43:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:43:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:43:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:43:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:43:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:43:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:43:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:43:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:43:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:43:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:43:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:43:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:43:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:43:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:43:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:43:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:43:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:43:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:43:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:43:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:43:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:43:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:43:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:44:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:44:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:44:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:44:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:44:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:44:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:44:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:44:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:44:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:44:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:44:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:44:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:44:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:44:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:44:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:44:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:44:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:44:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:44:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:44:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:44:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:44:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:44:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:44:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:44:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:44:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:44:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:44:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:44:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:44:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:44:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:44:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:44:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:44:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:44:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:44:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:44:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:44:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:44:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:44:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:44:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:44:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:44:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:44:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:44:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:44:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:44:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:44:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:44:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:44:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:44:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:44:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:44:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:44:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:44:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:44:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:44:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:44:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:44:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:44:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:44:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:44:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:44:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:44:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:44:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:44:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:44:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:44:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:44:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:44:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:44:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:44:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:44:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:44:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:44:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:44:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:44:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:44:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:44:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:44:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:44:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:44:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:44:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:44:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:44:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:44:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:44:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:44:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:44:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:44:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:44:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:44:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:44:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:44:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:44:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:44:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:44:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:44:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:44:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:44:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:44:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:44:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:44:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:44:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:44:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:44:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:44:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:44:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:44:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:44:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:44:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:44:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:44:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:44:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:44:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:44:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:44:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:44:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:44:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:44:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:44:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:44:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:44:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:44:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:44:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:44:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:44:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:44:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:44:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:44:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:44:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:44:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:44:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:44:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:44:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:44:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:44:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:44:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:44:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:44:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:44:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:44:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:44:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:44:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:44:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:44:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:44:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:44:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:44:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:44:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:44:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:44:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:44:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:44:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:44:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:44:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:44:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:44:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:44:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:44:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:44:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:44:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:44:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:44:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:44:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:44:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:44:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:44:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:44:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:44:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:44:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:44:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:44:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:44:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:44:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:44:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:44:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:44:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:44:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:44:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:44:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:44:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:44:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:44:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:44:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:44:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:44:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:44:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:44:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:44:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:44:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:44:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:44:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:44:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:44:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:44:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:44:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:44:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:44:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:44:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:44:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:44:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:44:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:44:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:44:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:44:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:44:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:44:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:44:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:44:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:44:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:44:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:44:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:44:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:44:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:44:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:45:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:45:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:45:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:45:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:45:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:45:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:45:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:45:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:45:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:45:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:45:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:45:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:45:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:45:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:45:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:45:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:45:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:45:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:45:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:45:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:45:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:45:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:45:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:45:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:45:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:45:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:45:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:45:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:45:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:45:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:45:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:45:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:45:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:45:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:45:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:45:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:45:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:45:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:45:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:45:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:45:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:45:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:45:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:45:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:45:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:45:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:45:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:45:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:45:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:45:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:45:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:45:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:45:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:45:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:45:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:45:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:45:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:45:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:45:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:45:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:45:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:45:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:45:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:45:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:45:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:45:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:45:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:45:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:45:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:45:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:45:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:45:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:45:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:45:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:45:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:45:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:45:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:45:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:45:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:45:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:45:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:45:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:45:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:45:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:45:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:45:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:45:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:45:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:45:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:45:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:45:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:45:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:45:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:45:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:45:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:45:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:45:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:45:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:45:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:45:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:45:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:45:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:45:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:45:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:45:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:45:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:45:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:45:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:45:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:45:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:45:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:45:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:45:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:45:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:45:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:45:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:45:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:45:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:45:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:45:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:45:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:45:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:45:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:45:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:45:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:45:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:45:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:45:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:45:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:45:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:45:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:45:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:45:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:45:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:45:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:45:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:45:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:45:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:45:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:45:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:45:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:45:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:45:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:45:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:45:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:45:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:45:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:45:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:45:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:45:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:45:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:45:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:45:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:45:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:45:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:45:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:45:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:45:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:45:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:45:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:45:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:45:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:45:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:45:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:45:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:45:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:45:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:45:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:45:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:45:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:45:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:45:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:45:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:45:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:45:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:45:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:45:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:45:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:45:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:45:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:45:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:45:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:45:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:45:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:45:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:45:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:45:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:45:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:45:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:45:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:45:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:45:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:45:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:45:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:45:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:45:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:45:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:45:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:45:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:45:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:45:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:45:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:45:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:45:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:45:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:45:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:45:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:45:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:45:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:45:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:45:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:45:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:45:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:45:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:45:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:45:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:46:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:46:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:46:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:46:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:46:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:46:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:46:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:46:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:46:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:46:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:46:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:46:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:46:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:46:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:46:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:46:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:46:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:46:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:46:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:46:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:46:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:46:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:46:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:46:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:46:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:46:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:46:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:46:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:46:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:46:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:46:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:46:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:46:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:46:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:46:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:46:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:46:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:46:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:46:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:46:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:46:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:46:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:46:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:46:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:46:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:46:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:46:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:46:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:46:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:46:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:46:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:46:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:46:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:46:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:46:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:46:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:46:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:46:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:46:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:46:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:46:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:46:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:46:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:46:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:46:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:46:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:46:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:46:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:46:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:46:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:46:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:46:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:46:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:46:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:46:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:46:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:46:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:46:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:46:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:46:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:46:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:46:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:46:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:46:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:46:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:46:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:46:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:46:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:46:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:46:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:46:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:46:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:46:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:46:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:46:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:46:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:46:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:46:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:46:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:46:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:46:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:46:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:46:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:46:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:46:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:46:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:46:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:46:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:46:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:46:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:46:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:46:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:46:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:46:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:46:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:46:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:46:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:46:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:46:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:46:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:46:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:46:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:46:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:46:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:46:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:46:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:46:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:46:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:46:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:46:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:46:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:46:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:46:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:46:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:46:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:46:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:46:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:46:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:46:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:46:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:46:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:46:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:46:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:46:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:46:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:46:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:46:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:46:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:46:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:46:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:46:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:46:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:46:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:46:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:46:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:46:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:46:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:46:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:46:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:46:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:46:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:46:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:46:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:46:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:46:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:46:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:46:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:46:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:46:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:46:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:46:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:46:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:46:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:46:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:46:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:46:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:46:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:46:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:46:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:46:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:46:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:46:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:46:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:46:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:46:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:46:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:46:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:46:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:46:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:46:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:46:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:46:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:46:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:46:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:46:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:46:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:46:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:46:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:46:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:46:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:46:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:46:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:46:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:46:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:46:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:46:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:46:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:46:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:46:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:46:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:46:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:46:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:46:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:46:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:46:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:46:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:47:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:47:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:47:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:47:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:47:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:47:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:47:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:47:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:47:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:47:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:47:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:47:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:47:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:47:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:47:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:47:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:47:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:47:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:47:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:47:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:47:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:47:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:47:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:47:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:47:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:47:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:47:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:47:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:47:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:47:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:47:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:47:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:47:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:47:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:47:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:47:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:47:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:47:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:47:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:47:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:47:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:47:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:47:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:47:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:47:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:47:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:47:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:47:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:47:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:47:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:47:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:47:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:47:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:47:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:47:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:47:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:47:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:47:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:47:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:47:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:47:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:47:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:47:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:47:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:47:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:47:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:47:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:47:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:47:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:47:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:47:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:47:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:47:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:47:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:47:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:47:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:47:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:47:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:47:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:47:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:47:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:47:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:47:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:47:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:47:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:47:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:47:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:47:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:47:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:47:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:47:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:47:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:47:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:47:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:47:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:47:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:47:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:47:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:47:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:47:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:47:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:47:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:47:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:47:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:47:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:47:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:47:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:47:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:47:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:47:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:47:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:47:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:47:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:47:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:47:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:47:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:47:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:47:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:47:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:47:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:47:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:47:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:47:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:47:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:47:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:47:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:47:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:47:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:47:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:47:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:47:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:47:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:47:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:47:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:47:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:47:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:47:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:47:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:47:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:47:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:47:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:47:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:47:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:47:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:47:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:47:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:47:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:47:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:47:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:47:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:47:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:47:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:47:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:47:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:47:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:47:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:47:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:47:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:47:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:47:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:47:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:47:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:47:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:47:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:47:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:47:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:47:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:47:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:47:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:47:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:47:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:47:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:47:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:47:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:47:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:47:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:47:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:47:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:47:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:47:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:47:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:47:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:47:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:47:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:47:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:47:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:47:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:47:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:47:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:47:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:47:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:47:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:47:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:47:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:47:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:47:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:47:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:47:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:47:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:47:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:47:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:47:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:47:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:47:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:47:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:47:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:47:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:47:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:47:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:47:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:47:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:47:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:47:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:47:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:47:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:47:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:48:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:48:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:48:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:48:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:48:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:48:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:48:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:48:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:48:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:48:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:48:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:48:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:48:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:48:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:48:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:48:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:48:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:48:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:48:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:48:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:48:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:48:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:48:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:48:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:48:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:48:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:48:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:48:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:48:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:48:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:48:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:48:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:48:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:48:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:48:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:48:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:48:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:48:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:48:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:48:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:48:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:48:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:48:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:48:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:48:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:48:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:48:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:48:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:48:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:48:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:48:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:48:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:48:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:48:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:48:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:48:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:48:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:48:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:48:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:48:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:48:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:48:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:48:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:48:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:48:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:48:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:48:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:48:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:48:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:48:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:48:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:48:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:48:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:48:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:48:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:48:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:48:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:48:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:48:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:48:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:48:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:48:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:48:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:48:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:48:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:48:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:48:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:48:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:48:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:48:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:48:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:48:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:48:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:48:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:48:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:48:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:48:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:48:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:48:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:48:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:48:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:48:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:48:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:48:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:48:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:48:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:48:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:48:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:48:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:48:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:48:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:48:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:48:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:48:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:48:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:48:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:48:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:48:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:48:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:48:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:48:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:48:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:48:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:48:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:48:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:48:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:48:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:48:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:48:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:48:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:48:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:48:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:48:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:48:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:48:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:48:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:48:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:48:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:48:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:48:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:48:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:48:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:48:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:48:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:48:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:48:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:48:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:48:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:48:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:48:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:48:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:48:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:48:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:48:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:48:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:48:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:48:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:48:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:48:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:48:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:48:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:48:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:50:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:50:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:50:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:50:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:50:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:50:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:50:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:50:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:50:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:50:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:50:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:50:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:50:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:50:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:50:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:50:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:50:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:50:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:50:53 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 17 23:50:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:50:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:50:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:50:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:50:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:50:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:50:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:50:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:50:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:50:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:50:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:50:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:50:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:50:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:50:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:50:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:50:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:50:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:51:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:51:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:51:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:51:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:51:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:51:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:51:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:51:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:51:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:51:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:51:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:51:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:51:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:51:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:51:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:51:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:51:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:51:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:51:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:51:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:51:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:51:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:51:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:51:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:51:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:51:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:51:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:51:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:51:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:51:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:51:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:51:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:51:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:51:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:51:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:51:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:51:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:51:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:51:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:51:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:51:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:51:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:51:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:51:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:51:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:51:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:51:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:51:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:51:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:51:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:51:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:51:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:51:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:51:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:51:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:51:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:51:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:51:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:51:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:51:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:51:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:51:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:51:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:51:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:51:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:51:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:51:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:51:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:51:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:51:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:51:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:51:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:51:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:51:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:51:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:51:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:51:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:51:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:51:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:51:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:51:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:51:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:51:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:51:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:51:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:51:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:51:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:51:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:51:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:51:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:51:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:51:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:51:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:51:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:51:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:51:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:51:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:51:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:51:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:51:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:51:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:51:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:51:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:51:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:51:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:51:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:51:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:51:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:51:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:51:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:51:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:51:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:51:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:51:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:51:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:51:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:51:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:51:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:51:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:51:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:51:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:51:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:51:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:51:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:51:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:51:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:51:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:51:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:51:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:51:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:51:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:51:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:51:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:51:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:51:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:51:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:51:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:51:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:51:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:51:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:51:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:51:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:51:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:51:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:51:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:51:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:52:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:52:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:52:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:52:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:52:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:52:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:52:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:52:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:52:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:52:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:52:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:52:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:52:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:52:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:52:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:52:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:52:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:52:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:52:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:52:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:52:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:52:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:52:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:52:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:52:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:52:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:52:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:52:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:52:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:52:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:52:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:52:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:52:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:52:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:52:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:52:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:52:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:52:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:52:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:52:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:52:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:52:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:52:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:52:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:52:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:52:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:52:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:52:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:52:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:52:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:52:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:52:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:52:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:52:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:52:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:52:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:52:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:52:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:52:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:52:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:52:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:52:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:52:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:52:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:52:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:52:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:52:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:52:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:52:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:52:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:52:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:52:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:52:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:52:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:52:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:52:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:52:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:52:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:52:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:52:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:52:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:52:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:52:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:52:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:52:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:52:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:52:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:52:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:52:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:52:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:52:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:52:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:52:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:52:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:52:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:52:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:52:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:52:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:52:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:52:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:52:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:52:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:52:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:52:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:52:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:52:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:52:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:52:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:52:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:52:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:52:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:52:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:52:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:52:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:52:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:52:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:52:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:52:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:52:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:52:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:52:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:52:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:52:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:52:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:52:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:52:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:52:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:52:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:52:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:52:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:52:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:52:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:52:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:52:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:52:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:52:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:52:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:52:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:52:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:52:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:52:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:52:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:52:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:52:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:52:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:52:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:52:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:52:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:52:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:52:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:52:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:52:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:52:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:52:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:52:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:52:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:52:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:52:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:52:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:52:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:52:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:52:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:52:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:52:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:52:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:52:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:52:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:52:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:52:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:52:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:52:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:52:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:52:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:52:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:52:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:52:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:53:00 rdma06 systemd[1]: Starting dnf makecache...
Apr 17 23:53:01 rdma06 dnf[1171978]: Updating Subscription Management repositories.
Apr 17 23:53:01 rdma06 dnf[1171978]: Unable to read consumer identity
Apr 17 23:53:01 rdma06 dnf[1171978]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 17 23:53:03 rdma06 dnf[1171978]: Copr repo for fmf owned by psss                 1.9 kB/s | 3.3 kB     00:01
Apr 17 23:53:03 rdma06 dnf[1171978]: beaker-AppStream                                1.8 kB/s | 168  B     00:00
Apr 17 23:53:03 rdma06 dnf[1171978]: Errors during downloading metadata for repository 'beaker-AppStream':
Apr 17 23:53:03 rdma06 dnf[1171978]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 23:53:03 rdma06 dnf[1171978]: Error: Failed to download metadata for repo 'beaker-AppStream': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 23:53:03 rdma06 dnf[1171978]: beaker-AppStream-debuginfo                      5.0 kB/s | 168  B     00:00
Apr 17 23:53:03 rdma06 dnf[1171978]: Errors during downloading metadata for repository 'beaker-AppStream-debuginfo':
Apr 17 23:53:03 rdma06 dnf[1171978]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 23:53:03 rdma06 dnf[1171978]: Error: Failed to download metadata for repo 'beaker-AppStream-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 23:53:03 rdma06 dnf[1171978]: beaker-BaseOS                                   4.9 kB/s | 168  B     00:00
Apr 17 23:53:03 rdma06 dnf[1171978]: Errors during downloading metadata for repository 'beaker-BaseOS':
Apr 17 23:53:03 rdma06 dnf[1171978]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 23:53:03 rdma06 dnf[1171978]: Error: Failed to download metadata for repo 'beaker-BaseOS': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 23:53:03 rdma06 dnf[1171978]: beaker-BaseOS-debuginfo                          56 kB/s | 168  B     00:00
Apr 17 23:53:03 rdma06 dnf[1171978]: Errors during downloading metadata for repository 'beaker-BaseOS-debuginfo':
Apr 17 23:53:03 rdma06 dnf[1171978]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 23:53:03 rdma06 dnf[1171978]: Error: Failed to download metadata for repo 'beaker-BaseOS-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 23:53:03 rdma06 dnf[1171978]: beaker-CRB                                       18 kB/s | 168  B     00:00
Apr 17 23:53:03 rdma06 dnf[1171978]: Errors during downloading metadata for repository 'beaker-CRB':
Apr 17 23:53:03 rdma06 dnf[1171978]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 23:53:03 rdma06 dnf[1171978]: Error: Failed to download metadata for repo 'beaker-CRB': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 23:53:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:53:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:53:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:53:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:53:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:53:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:53:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:53:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:53:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:53:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:53:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:53:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:53:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:53:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:53:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:53:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:53:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:53:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:53:03 rdma06 dnf[1171978]: beaker-CRB-debuginfo                            498  B/s | 168  B     00:00
Apr 17 23:53:03 rdma06 dnf[1171978]: Errors during downloading metadata for repository 'beaker-CRB-debuginfo':
Apr 17 23:53:03 rdma06 dnf[1171978]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 23:53:03 rdma06 dnf[1171978]: Error: Failed to download metadata for repo 'beaker-CRB-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 23:53:03 rdma06 dnf[1171978]: beaker-HighAvailability                          19 kB/s | 168  B     00:00
Apr 17 23:53:03 rdma06 dnf[1171978]: Errors during downloading metadata for repository 'beaker-HighAvailability':
Apr 17 23:53:03 rdma06 dnf[1171978]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 23:53:03 rdma06 dnf[1171978]: Error: Failed to download metadata for repo 'beaker-HighAvailability': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 23:53:03 rdma06 dnf[1171978]: beaker-HighAvailability-debuginfo               7.5 kB/s | 168  B     00:00
Apr 17 23:53:03 rdma06 dnf[1171978]: Errors during downloading metadata for repository 'beaker-HighAvailability-debuginfo':
Apr 17 23:53:03 rdma06 dnf[1171978]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 23:53:03 rdma06 dnf[1171978]: Error: Failed to download metadata for repo 'beaker-HighAvailability-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 23:53:03 rdma06 dnf[1171978]: beaker-NFV                                      3.4 kB/s | 168  B     00:00
Apr 17 23:53:03 rdma06 dnf[1171978]: Errors during downloading metadata for repository 'beaker-NFV':
Apr 17 23:53:03 rdma06 dnf[1171978]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 23:53:03 rdma06 dnf[1171978]: Error: Failed to download metadata for repo 'beaker-NFV': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 23:53:03 rdma06 dnf[1171978]: beaker-NFV-debuginfo                             55 kB/s | 168  B     00:00
Apr 17 23:53:03 rdma06 dnf[1171978]: Errors during downloading metadata for repository 'beaker-NFV-debuginfo':
Apr 17 23:53:03 rdma06 dnf[1171978]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 23:53:03 rdma06 dnf[1171978]: Error: Failed to download metadata for repo 'beaker-NFV-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 23:53:03 rdma06 dnf[1171978]: beaker-RT                                       4.4 kB/s | 168  B     00:00
Apr 17 23:53:03 rdma06 dnf[1171978]: Errors during downloading metadata for repository 'beaker-RT':
Apr 17 23:53:03 rdma06 dnf[1171978]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 23:53:03 rdma06 dnf[1171978]: Error: Failed to download metadata for repo 'beaker-RT': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 23:53:04 rdma06 dnf[1171978]: beaker-RT-debuginfo                             737  B/s | 168  B     00:00
Apr 17 23:53:04 rdma06 dnf[1171978]: Errors during downloading metadata for repository 'beaker-RT-debuginfo':
Apr 17 23:53:04 rdma06 dnf[1171978]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 23:53:04 rdma06 dnf[1171978]: Error: Failed to download metadata for repo 'beaker-RT-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 23:53:04 rdma06 dnf[1171978]: beaker-ResilientStorage                          26 kB/s | 168  B     00:00
Apr 17 23:53:04 rdma06 dnf[1171978]: Errors during downloading metadata for repository 'beaker-ResilientStorage':
Apr 17 23:53:04 rdma06 dnf[1171978]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 23:53:04 rdma06 dnf[1171978]: Error: Failed to download metadata for repo 'beaker-ResilientStorage': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 23:53:04 rdma06 dnf[1171978]: beaker-ResilientStorage-debuginfo                61 kB/s | 168  B     00:00
Apr 17 23:53:04 rdma06 dnf[1171978]: Errors during downloading metadata for repository 'beaker-ResilientStorage-debuginfo':
Apr 17 23:53:04 rdma06 dnf[1171978]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 23:53:04 rdma06 dnf[1171978]: Error: Failed to download metadata for repo 'beaker-ResilientStorage-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 23:53:04 rdma06 dnf[1171978]: beaker-SAP                                      3.5 kB/s | 168  B     00:00
Apr 17 23:53:04 rdma06 dnf[1171978]: Errors during downloading metadata for repository 'beaker-SAP':
Apr 17 23:53:04 rdma06 dnf[1171978]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 23:53:04 rdma06 dnf[1171978]: Error: Failed to download metadata for repo 'beaker-SAP': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 23:53:04 rdma06 dnf[1171978]: beaker-SAP-debuginfo                            1.6 kB/s | 168  B     00:00
Apr 17 23:53:04 rdma06 dnf[1171978]: Errors during downloading metadata for repository 'beaker-SAP-debuginfo':
Apr 17 23:53:04 rdma06 dnf[1171978]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 23:53:04 rdma06 dnf[1171978]: Error: Failed to download metadata for repo 'beaker-SAP-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 23:53:04 rdma06 dnf[1171978]: beaker-SAPHANA                                  3.6 kB/s | 168  B     00:00
Apr 17 23:53:04 rdma06 dnf[1171978]: Errors during downloading metadata for repository 'beaker-SAPHANA':
Apr 17 23:53:04 rdma06 dnf[1171978]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 23:53:04 rdma06 dnf[1171978]: Error: Failed to download metadata for repo 'beaker-SAPHANA': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 23:53:04 rdma06 dnf[1171978]: beaker-SAPHANA-debuginfo                        4.5 kB/s | 168  B     00:00
Apr 17 23:53:04 rdma06 dnf[1171978]: Errors during downloading metadata for repository 'beaker-SAPHANA-debuginfo':
Apr 17 23:53:04 rdma06 dnf[1171978]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 23:53:04 rdma06 dnf[1171978]: Error: Failed to download metadata for repo 'beaker-SAPHANA-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 17 23:53:04 rdma06 dnf[1171978]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr 17 23:53:05 rdma06 dnf[1171978]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr 17 23:53:05 rdma06 dnf[1171978]: Ignoring repositories: beaker-AppStream, beaker-AppStream-debuginfo, beaker-BaseOS, beaker-BaseOS-debuginfo, beaker-CRB, beaker-CRB-debuginfo, beaker-HighAvailability, beaker-HighAvailability-debuginfo, beaker-NFV, beaker-NFV-debuginfo, beaker-RT, beaker-RT-debuginfo, beaker-ResilientStorage, beaker-ResilientStorage-debuginfo, beaker-SAP, beaker-SAP-debuginfo, beaker-SAPHANA, beaker-SAPHANA-debuginfo
Apr 17 23:53:05 rdma06 dnf[1171978]: Metadata cache created.
Apr 17 23:53:05 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 17 23:53:05 rdma06 systemd[1]: Finished dnf makecache.
Apr 17 23:53:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:53:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:53:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:53:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:53:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:53:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:53:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:53:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:53:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:53:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:53:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:53:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:53:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:53:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:53:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:53:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:53:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:53:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:53:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:53:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:53:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:53:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:53:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:53:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:53:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:53:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:53:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:53:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:53:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:53:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:53:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:53:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:53:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:53:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:53:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:53:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:53:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:53:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:53:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:53:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:53:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:53:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:53:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:53:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:53:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:53:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:53:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:53:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:53:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:53:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:53:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:53:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:53:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:53:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:53:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:53:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:53:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:53:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:53:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:53:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:53:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:53:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:53:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:53:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:53:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:53:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:53:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:53:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:53:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:53:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:53:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:53:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:53:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:53:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:53:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:53:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:53:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:53:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:53:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:53:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:53:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:53:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:53:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:53:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:53:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:53:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:53:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:53:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:53:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:53:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:53:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:53:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:53:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:53:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:53:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:53:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:53:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:53:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:53:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:53:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:53:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:53:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:53:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:53:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:53:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:53:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:53:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:53:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:53:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:53:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:53:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:53:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:53:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:53:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:53:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:53:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:53:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:53:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:53:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:53:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:53:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:53:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:53:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:53:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:53:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:53:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:53:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:53:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:53:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:53:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:53:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:53:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:53:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:53:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:53:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:53:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:53:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:53:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:53:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:53:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:53:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:53:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:53:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:53:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:53:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:53:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:53:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:53:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:53:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:53:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:53:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:53:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:53:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:53:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:53:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:53:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:53:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:53:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:53:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:53:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:53:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:53:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:53:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:53:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:53:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:53:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:53:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:53:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:53:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:53:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:53:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:53:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:53:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:53:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:53:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:53:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:53:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:53:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:53:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:53:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:53:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:53:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:53:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:53:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:53:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:53:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:53:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:53:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:53:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:53:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:53:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:53:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:53:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:53:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:53:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:53:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:53:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:53:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:54:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:54:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:54:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:54:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:54:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:54:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:54:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:54:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:54:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:54:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:54:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:54:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:54:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:54:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:54:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:54:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:54:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:54:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:54:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:54:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:54:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:54:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:54:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:54:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:54:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:54:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:54:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:54:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:54:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:54:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:54:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:54:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:54:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:54:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:54:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:54:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:54:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:54:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:54:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:54:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:54:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:54:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:54:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:54:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:54:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:54:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:54:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:54:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:54:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:54:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:54:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:54:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:54:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:54:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:54:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:54:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:54:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:54:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:54:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:54:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:54:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:54:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:54:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:54:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:54:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:54:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:54:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:54:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:54:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:54:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:54:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:54:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:54:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:54:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:54:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:54:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:54:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:54:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:54:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:54:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:54:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:54:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:54:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:54:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:54:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:54:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:54:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:54:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:54:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:54:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:54:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:54:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:54:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:54:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:54:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:54:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:54:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:54:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:54:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:54:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:54:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:54:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:54:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:54:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:54:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:54:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:54:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:54:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:54:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:54:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:54:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:54:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:54:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:54:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:54:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:54:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:54:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:54:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:54:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:54:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:54:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:54:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:54:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:54:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:54:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:54:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:54:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:54:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:54:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:54:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:54:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:54:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:54:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:54:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:54:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:54:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:54:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:54:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:54:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:54:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:54:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:54:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:54:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:54:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:54:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:54:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:54:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:54:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:54:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:54:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:54:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:54:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:54:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:54:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:54:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:54:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:54:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:54:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:54:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:54:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:54:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:54:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:54:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:54:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:54:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:54:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:54:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:54:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:54:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:54:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:54:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:54:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:54:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:54:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:54:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:54:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:54:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:54:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:54:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:54:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:54:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:54:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:54:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:54:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:54:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:54:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:54:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:54:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:54:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:54:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:54:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:54:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:54:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:54:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:54:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:54:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:54:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:54:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:54:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:54:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:54:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:54:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:54:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:54:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:54:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:54:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:54:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:54:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:54:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:54:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:54:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:54:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:54:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:54:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:54:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:54:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:55:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:55:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:55:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:55:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:55:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:55:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:55:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:55:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:55:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:55:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:55:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:55:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:55:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:55:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:55:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:55:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:55:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:55:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:55:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:55:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:55:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:55:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:55:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:55:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:55:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:55:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:55:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:55:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:55:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:55:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:55:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:55:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:55:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:55:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:55:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:55:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:55:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:55:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:55:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:55:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:55:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:55:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:55:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:55:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:55:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:55:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:55:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:55:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:55:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:55:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:55:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:55:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:55:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:55:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:55:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:55:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:55:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:55:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:55:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:55:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:55:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:55:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:55:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:55:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:55:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:55:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:55:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:55:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:55:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:55:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:55:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:55:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:55:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:55:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:55:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:55:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:55:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:55:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:55:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:55:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:55:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:55:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:55:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:55:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:55:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:55:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:55:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:55:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:55:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:55:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:55:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:55:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:55:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:55:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:55:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:55:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:55:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:55:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:55:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:55:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:55:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:55:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:55:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:55:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:55:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:55:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:55:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:55:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:55:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:55:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:55:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:55:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:55:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:55:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:55:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:55:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:55:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:55:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:55:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:55:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:55:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:55:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:55:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:55:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:55:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:55:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:55:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:55:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:55:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:55:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:55:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:55:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:55:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:55:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:55:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:55:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:55:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:55:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:55:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:55:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:55:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:55:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:55:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:55:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:55:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:55:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:55:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:55:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:55:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:55:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:55:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:55:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:55:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:55:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:55:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:55:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:55:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:55:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:55:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:55:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:55:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:55:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:55:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:55:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:55:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:55:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:55:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:55:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:55:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:55:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:55:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:55:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:55:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:55:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:55:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:55:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:55:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:55:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:55:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:55:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:55:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:55:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:55:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:55:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:55:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:55:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:55:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:55:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:55:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:55:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:55:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:55:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:55:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:55:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:55:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:55:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:55:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:55:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:55:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:55:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:55:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:55:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:55:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:55:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:55:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:55:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:55:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:55:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:55:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:55:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:55:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:55:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:55:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:55:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:55:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:55:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:56:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:56:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:56:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:56:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:56:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:56:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:56:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:56:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:56:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:56:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:56:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:56:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:56:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:56:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:56:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:56:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:56:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:56:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:56:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:56:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:56:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:56:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:56:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:56:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:56:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:56:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:56:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:56:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:56:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:56:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:56:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:56:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:56:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:56:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:56:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:56:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:56:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:56:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:56:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:56:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:56:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:56:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:56:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:56:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:56:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:56:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:56:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:56:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:56:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:56:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:56:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:56:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:56:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:56:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:56:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:56:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:56:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:56:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:56:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:56:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:56:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:56:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:56:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:56:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:56:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:56:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:56:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:56:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:56:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:56:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:56:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:56:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:56:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:56:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:56:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:56:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:56:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:56:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:56:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:56:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:56:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:56:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:56:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:56:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:56:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:56:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:56:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:56:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:56:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:56:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:56:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:56:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:56:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:56:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:56:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:56:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:56:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:56:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:56:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:56:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:56:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:56:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:56:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:56:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:56:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:56:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:56:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:56:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:56:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:56:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:56:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:56:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:56:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:56:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:56:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:56:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:56:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:56:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:56:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:56:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:56:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:56:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:56:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:56:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:56:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:56:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:56:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:56:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:56:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:56:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:56:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:56:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:56:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:56:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:56:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:56:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:56:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:56:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:56:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:56:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:56:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:56:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:56:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:56:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:56:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:56:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:56:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:56:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:56:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:56:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:56:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:56:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:56:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:56:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:56:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:56:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:56:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:56:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:56:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:56:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:56:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:56:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:56:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:56:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:56:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:56:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:56:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:56:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:56:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:56:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:56:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:56:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:56:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:56:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:56:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:56:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:56:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:56:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:56:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:56:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:56:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:56:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:56:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:56:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:56:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:56:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:56:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:56:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:56:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:56:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:56:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:56:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:56:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:56:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:56:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:56:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:56:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:56:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:56:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:56:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:56:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:56:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:56:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:56:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:56:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:56:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:56:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:56:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:56:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:56:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:56:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:56:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:56:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:56:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:56:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:56:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:57:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:57:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:57:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:57:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:57:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:57:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:57:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:57:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:57:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:57:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:57:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:57:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:57:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:57:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:57:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:57:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:57:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:57:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:57:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:57:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:57:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:57:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:57:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:57:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:57:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:57:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:57:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:57:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:57:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:57:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:57:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:57:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:57:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:57:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:57:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:57:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:57:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:57:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:57:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:57:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:57:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:57:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:57:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:57:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:57:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:57:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:57:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:57:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:57:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:57:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:57:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:57:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:57:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:57:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:57:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:57:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:57:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:57:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:57:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:57:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:57:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:57:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:57:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:57:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:57:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:57:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:57:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:57:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:57:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:57:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:57:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:57:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:57:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:57:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:57:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:57:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:57:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:57:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:57:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:57:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:57:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:57:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:57:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:57:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:57:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:57:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:57:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:57:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:57:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:57:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:57:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:57:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:57:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:57:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:57:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:57:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:57:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:57:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:57:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:57:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:57:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:57:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:57:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:57:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:57:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:57:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:57:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:57:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:57:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:57:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:57:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:57:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:57:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:57:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:57:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:57:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:57:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:57:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:57:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:57:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:57:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:57:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:57:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:57:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:57:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:57:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:57:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:57:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:57:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:57:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:57:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:57:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:57:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:57:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:57:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:57:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:57:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:57:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:57:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:57:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:57:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:57:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:57:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:57:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:57:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:57:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:57:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:57:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:57:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:57:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:57:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:57:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:57:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:57:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:57:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:57:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:57:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:57:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:57:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:57:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:57:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:57:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:57:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:57:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:57:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:57:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:57:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:57:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:57:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:57:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:57:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:57:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:57:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:57:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:57:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:57:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:57:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:57:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:57:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:57:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:57:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:57:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:57:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:57:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:57:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:57:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:57:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:57:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:57:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:57:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:57:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:57:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:57:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:57:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:57:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:57:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:57:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:57:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:57:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:57:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:57:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:57:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:57:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:57:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:57:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:57:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:57:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:57:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:57:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:57:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:57:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:57:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:57:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:57:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:57:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:57:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:58:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:58:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:58:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:58:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:58:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:58:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:58:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:58:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:58:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:58:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:58:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:58:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:58:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:58:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:58:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:58:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:58:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:58:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:58:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:58:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:58:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:58:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:58:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:58:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:58:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:58:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:58:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:58:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:58:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:58:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:58:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:58:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:58:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:58:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:58:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:58:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:58:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:58:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:58:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:58:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:58:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:58:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:58:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:58:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:58:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:58:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:58:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:58:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:58:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:58:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:58:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:58:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:58:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:58:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:58:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:58:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:58:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:58:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:58:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:58:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:58:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:58:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:58:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:58:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:58:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:58:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:58:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:58:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:58:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:58:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:58:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:58:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:58:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:58:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:58:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:58:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:58:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:58:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:58:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:58:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:58:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:58:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:58:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:58:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:58:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:58:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:58:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:58:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:58:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:58:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:58:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:58:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:58:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:58:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:58:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:58:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:58:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:58:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:58:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:58:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:58:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:58:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:58:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:58:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:58:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:58:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:58:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:58:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:58:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:58:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:58:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:58:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:58:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:58:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:58:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:58:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:58:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:58:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:58:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:58:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:58:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:58:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:58:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:58:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:58:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:58:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:58:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:58:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:58:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:58:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:58:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:58:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:58:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:58:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:58:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:58:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:58:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:58:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:58:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:58:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:58:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:58:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:58:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:58:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:58:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:58:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:58:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:58:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:58:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:58:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:58:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:58:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:58:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:58:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:58:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:58:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:58:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:58:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:58:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 17 23:58:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 17 23:58:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 17 23:58:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 17 23:58:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 17 23:58:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:58:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:58:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:58:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 17 23:58:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 17 23:58:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 17 23:58:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 17 23:58:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 17 23:58:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 17 23:58:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 17 23:58:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 17 23:58:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 17 23:58:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 17 23:58:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:00:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:00:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:00:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:00:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:00:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:00:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:00:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:00:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:00:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:00:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:00:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:00:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:00:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:00:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:00:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:00:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:00:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:00:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:00:54 rdma06 rsyslogd[1041]: imjournal: 5168 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 00:00:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:00:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:00:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:00:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:00:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:00:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:00:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:00:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:00:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:00:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:00:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:00:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:00:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:00:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:00:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:00:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:00:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:00:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:01:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:01:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:01:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:01:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:01:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:01:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:01:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:01:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:01:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:01:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:01:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:01:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:01:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:01:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:01:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:01:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:01:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:01:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:01:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:01:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:01:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:01:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:01:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:01:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:01:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:01:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:01:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:01:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:01:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:01:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:01:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:01:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:01:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:01:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:01:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:01:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:01:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:01:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:01:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:01:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:01:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:01:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:01:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:01:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:01:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:01:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:01:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:01:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:01:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:01:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:01:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:01:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:01:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:01:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:01:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:01:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:01:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:01:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:01:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:01:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:01:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:01:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:01:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:01:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:01:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:01:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:01:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:01:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:01:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:01:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:01:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:01:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:01:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:01:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:01:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:01:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:01:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:01:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:01:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:01:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:01:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:01:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:01:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:01:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:01:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:01:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:01:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:01:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:01:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:01:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:01:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:01:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:01:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:01:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:01:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:01:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:01:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:01:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:01:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:01:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:01:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:01:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:01:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:01:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:01:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:01:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:01:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:01:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:01:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:01:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:01:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:01:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:01:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:01:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:01:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:01:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:01:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:01:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:01:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:01:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:01:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:01:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:01:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:01:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:01:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:01:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:01:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:01:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:01:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:01:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:01:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:01:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:01:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:01:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:01:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:01:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:01:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:01:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:01:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:01:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:01:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:01:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:01:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:01:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:01:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:01:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:01:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:01:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:01:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:01:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:01:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:01:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:01:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:01:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:01:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:01:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:01:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:01:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:01:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:01:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:01:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:01:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:01:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:01:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:01:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:01:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:01:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:01:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:01:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:01:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:01:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:01:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:01:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:01:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:01:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:01:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:01:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:01:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:01:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:01:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:01:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:01:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:01:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:01:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:01:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:01:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:01:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:01:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:01:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:01:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:01:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:01:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:01:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:01:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:01:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:01:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:01:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:01:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:01:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:01:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:01:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:01:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:01:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:01:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:01:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:01:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:01:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:01:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:01:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:01:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:01:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:01:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:01:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:01:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:01:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:01:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:02:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:02:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:02:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:02:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:02:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:02:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:02:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:02:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:02:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:02:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:02:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:02:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:02:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:02:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:02:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:02:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:02:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:02:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:02:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:02:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:02:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:02:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:02:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:02:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:02:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:02:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:02:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:02:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:02:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:02:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:02:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:02:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:02:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:02:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:02:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:02:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:02:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:02:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:02:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:02:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:02:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:02:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:02:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:02:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:02:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:02:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:02:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:02:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:02:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:02:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:02:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:02:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:02:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:02:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:02:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:02:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:02:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:02:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:02:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:02:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:02:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:02:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:02:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:02:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:02:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:02:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:02:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:02:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:02:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:02:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:02:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:02:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:02:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:02:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:02:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:02:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:02:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:02:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:02:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:02:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:02:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:02:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:02:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:02:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:02:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:02:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:02:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:02:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:02:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:02:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:02:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:02:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:02:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:02:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:02:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:02:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:02:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:02:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:02:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:02:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:02:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:02:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:02:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:02:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:02:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:02:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:02:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:02:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:02:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:02:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:02:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:02:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:02:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:02:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:02:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:02:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:02:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:02:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:02:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:02:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:02:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:02:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:02:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:02:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:02:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:02:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:02:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:02:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:02:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:02:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:02:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:02:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:02:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:02:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:02:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:02:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:02:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:02:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:02:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:02:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:02:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:02:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:02:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:02:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:02:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:02:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:02:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:02:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:02:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:02:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:02:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:02:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:02:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:02:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:02:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:02:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:02:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:02:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:02:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:02:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:02:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:02:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:02:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:02:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:02:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:02:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:02:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:02:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:02:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:02:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:02:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:02:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:02:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:02:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:02:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:02:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:02:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:02:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:02:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:02:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:02:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:02:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:02:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:02:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:02:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:02:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:02:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:02:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:02:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:02:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:02:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:02:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:02:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:02:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:02:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:02:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:02:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:02:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:02:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:02:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:02:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:02:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:02:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:02:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:02:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:02:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:02:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:02:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:02:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:02:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:02:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:02:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:02:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:02:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:02:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:02:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:03:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:03:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:03:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:03:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:03:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:03:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:03:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:03:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:03:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:03:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:03:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:03:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:03:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:03:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:03:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:03:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:03:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:03:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:03:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:03:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:03:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:03:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:03:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:03:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:03:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:03:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:03:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:03:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:03:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:03:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:03:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:03:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:03:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:03:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:03:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:03:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:03:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:03:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:03:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:03:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:03:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:03:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:03:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:03:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:03:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:03:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:03:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:03:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:03:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:03:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:03:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:03:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:03:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:03:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:03:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:03:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:03:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:03:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:03:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:03:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:03:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:03:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:03:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:03:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:03:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:03:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:03:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:03:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:03:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:03:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:03:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:03:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:03:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:03:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:03:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:03:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:03:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:03:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:03:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:03:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:03:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:03:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:03:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:03:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:03:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:03:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:03:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:03:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:03:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:03:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:03:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:03:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:03:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:03:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:03:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:03:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:03:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:03:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:03:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:03:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:03:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:03:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:03:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:03:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:03:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:03:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:03:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:03:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:03:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:03:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:03:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:03:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:03:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:03:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:03:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:03:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:03:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:03:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:03:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:03:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:03:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:03:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:03:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:03:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:03:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:03:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:03:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:03:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:03:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:03:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:03:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:03:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:03:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:03:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:03:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:03:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:03:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:03:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:03:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:03:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:03:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:03:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:03:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:03:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:03:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:03:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:03:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:03:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:03:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:03:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:03:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:03:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:03:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:03:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:03:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:03:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:03:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:03:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:03:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:03:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:03:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:03:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:03:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:03:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:03:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:03:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:03:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:03:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:03:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:03:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:03:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:03:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:03:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:03:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:03:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:03:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:03:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:03:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:03:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:03:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:03:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:03:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:03:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:03:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:03:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:03:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:03:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:03:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:03:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:03:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:03:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:03:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:03:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:03:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:03:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:03:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:03:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:03:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:03:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:03:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:03:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:03:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:03:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:03:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:03:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:03:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:03:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:03:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:03:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:03:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:03:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:03:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:03:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:03:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:03:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:03:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:04:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:04:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:04:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:04:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:04:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:04:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:04:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:04:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:04:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:04:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:04:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:04:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:04:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:04:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:04:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:04:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:04:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:04:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:04:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:04:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:04:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:04:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:04:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:04:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:04:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:04:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:04:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:04:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:04:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:04:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:04:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:04:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:04:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:04:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:04:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:04:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:04:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:04:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:04:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:04:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:04:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:04:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:04:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:04:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:04:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:04:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:04:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:04:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:04:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:04:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:04:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:04:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:04:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:04:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:04:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:04:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:04:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:04:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:04:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:04:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:04:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:04:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:04:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:04:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:04:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:04:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:04:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:04:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:04:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:04:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:04:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:04:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:04:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:04:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:04:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:04:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:04:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:04:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:04:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:04:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:04:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:04:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:04:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:04:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:04:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:04:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:04:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:04:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:04:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:04:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:04:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:04:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:04:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:04:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:04:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:04:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:04:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:04:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:04:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:04:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:04:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:04:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:04:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:04:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:04:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:04:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:04:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:04:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:04:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:04:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:04:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:04:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:04:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:04:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:04:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:04:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:04:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:04:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:04:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:04:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:04:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:04:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:04:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:04:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:04:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:04:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:04:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:04:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:04:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:04:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:04:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:04:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:04:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:04:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:04:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:04:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:04:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:04:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:04:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:04:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:04:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:04:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:04:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:04:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:04:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:04:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:04:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:04:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:04:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:04:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:04:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:04:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:04:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:04:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:04:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:04:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:04:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:04:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:04:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:04:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:04:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:04:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:04:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:04:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:04:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:04:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:04:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:04:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:04:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:04:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:04:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:04:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:04:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:04:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:04:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:04:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:04:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:04:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:04:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:04:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:04:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:04:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:04:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:04:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:04:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:04:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:04:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:04:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:04:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:04:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:04:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:04:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:04:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:04:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:04:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:04:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:04:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:04:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:04:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:04:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:04:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:04:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:04:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:04:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:04:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:04:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:04:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:04:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:04:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:04:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:04:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:04:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:04:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:04:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:04:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:04:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:05:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:05:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:05:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:05:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:05:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:05:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:05:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:05:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:05:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:05:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:05:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:05:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:05:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:05:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:05:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:05:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:05:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:05:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:05:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:05:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:05:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:05:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:05:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:05:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:05:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:05:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:05:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:05:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:05:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:05:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:05:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:05:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:05:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:05:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:05:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:05:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:05:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:05:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:05:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:05:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:05:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:05:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:05:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:05:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:05:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:05:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:05:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:05:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:05:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:05:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:05:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:05:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:05:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:05:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:05:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:05:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:05:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:05:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:05:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:05:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:05:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:05:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:05:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:05:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:05:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:05:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:05:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:05:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:05:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:05:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:05:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:05:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:05:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:05:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:05:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:05:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:05:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:05:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:05:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:05:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:05:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:05:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:05:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:05:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:05:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:05:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:05:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:05:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:05:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:05:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:05:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:05:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:05:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:05:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:05:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:05:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:05:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:05:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:05:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:05:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:05:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:05:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:05:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:05:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:05:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:05:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:05:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:05:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:05:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:05:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:05:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:05:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:05:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:05:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:05:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:05:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:05:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:05:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:05:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:05:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:05:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:05:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:05:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:05:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:05:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:05:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:05:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:05:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:05:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:05:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:05:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:05:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:05:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:05:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:05:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:05:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:05:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:05:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:05:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:05:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:05:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:05:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:05:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:05:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:05:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:05:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:05:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:05:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:05:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:05:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:05:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:05:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:05:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:05:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:05:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:05:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:05:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:05:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:05:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:05:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:05:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:05:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:05:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:05:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:05:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:05:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:05:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:05:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:05:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:05:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:05:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:05:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:05:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:05:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:05:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:05:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:05:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:05:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:05:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:05:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:05:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:05:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:05:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:05:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:05:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:05:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:05:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:05:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:05:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:05:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:05:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:05:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:05:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:05:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:05:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:05:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:05:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:05:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:05:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:05:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:05:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:05:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:05:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:05:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:05:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:05:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:05:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:05:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:05:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:05:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:05:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:05:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:05:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:05:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:05:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:05:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:06:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:06:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:06:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:06:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:06:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:06:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:06:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:06:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:06:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:06:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:06:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:06:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:06:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:06:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:06:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:06:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:06:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:06:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:06:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:06:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:06:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:06:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:06:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:06:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:06:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:06:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:06:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:06:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:06:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:06:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:06:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:06:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:06:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:06:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:06:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:06:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:06:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:06:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:06:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:06:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:06:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:06:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:06:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:06:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:06:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:06:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:06:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:06:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:06:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:06:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:06:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:06:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:06:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:06:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:06:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:06:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:06:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:06:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:06:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:06:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:06:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:06:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:06:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:06:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:06:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:06:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:06:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:06:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:06:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:06:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:06:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:06:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:06:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:06:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:06:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:06:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:06:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:06:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:06:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:06:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:06:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:06:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:06:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:06:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:06:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:06:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:06:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:06:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:06:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:06:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:06:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:06:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:06:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:06:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:06:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:06:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:06:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:06:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:06:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:06:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:06:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:06:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:06:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:06:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:06:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:06:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:06:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:06:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:06:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:06:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:06:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:06:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:06:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:06:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:06:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:06:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:06:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:06:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:06:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:06:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:06:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:06:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:06:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:06:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:06:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:06:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:06:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:06:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:06:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:06:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:06:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:06:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:06:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:06:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:06:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:06:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:06:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:06:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:06:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:06:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:06:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:06:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:06:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:06:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:06:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:06:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:06:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:06:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:06:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:06:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:06:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:06:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:06:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:06:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:06:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:06:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:06:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:06:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:06:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:06:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:06:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:06:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:06:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:06:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:06:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:06:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:06:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:06:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:06:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:06:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:06:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:06:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:06:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:06:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:06:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:06:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:06:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:06:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:06:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:06:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:06:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:06:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:06:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:06:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:06:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:06:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:06:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:06:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:06:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:06:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:06:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:06:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:06:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:06:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:06:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:06:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:06:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:06:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:06:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:06:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:06:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:06:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:06:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:06:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:06:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:06:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:06:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:06:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:06:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:06:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:06:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:06:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:06:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:06:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:06:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:06:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:07:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:07:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:07:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:07:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:07:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:07:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:07:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:07:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:07:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:07:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:07:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:07:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:07:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:07:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:07:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:07:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:07:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:07:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:07:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:07:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:07:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:07:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:07:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:07:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:07:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:07:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:07:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:07:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:07:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:07:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:07:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:07:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:07:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:07:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:07:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:07:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:07:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:07:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:07:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:07:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:07:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:07:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:07:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:07:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:07:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:07:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:07:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:07:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:07:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:07:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:07:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:07:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:07:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:07:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:07:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:07:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:07:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:07:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:07:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:07:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:07:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:07:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:07:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:07:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:07:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:07:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:07:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:07:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:07:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:07:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:07:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:07:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:07:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:07:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:07:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:07:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:07:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:07:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:07:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:07:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:07:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:07:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:07:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:07:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:07:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:07:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:07:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:07:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:07:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:07:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:07:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:07:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:07:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:07:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:07:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:07:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:07:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:07:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:07:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:07:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:07:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:07:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:07:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:07:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:07:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:07:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:07:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:07:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:07:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:07:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:07:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:07:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:07:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:07:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:07:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:07:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:07:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:07:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:07:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:07:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:07:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:07:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:07:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:07:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:07:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:07:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:07:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:07:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:07:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:07:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:07:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:07:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:07:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:07:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:07:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:07:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:07:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:07:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:07:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:07:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:07:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:07:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:07:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:07:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:07:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:07:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:07:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:07:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:07:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:07:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:07:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:07:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:07:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:07:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:07:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:07:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:07:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:07:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:07:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:07:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:07:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:07:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:07:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:07:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:07:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:07:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:07:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:07:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:07:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:07:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:07:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:07:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:07:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:07:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:07:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:07:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:07:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:07:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:07:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:07:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:07:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:07:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:07:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:07:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:07:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:07:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:07:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:07:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:07:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:07:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:07:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:07:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:07:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:07:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:07:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:07:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:07:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:07:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:07:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:07:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:07:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:07:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:07:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:07:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:07:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:07:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:07:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:07:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:07:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:07:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:07:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:07:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:07:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:07:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:07:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:07:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:08:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:08:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:08:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:08:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:08:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:08:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:08:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:08:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:08:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:08:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:08:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:08:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:08:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:08:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:08:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:08:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:08:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:08:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:08:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:08:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:08:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:08:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:08:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:08:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:08:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:08:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:08:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:08:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:08:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:08:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:08:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:08:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:08:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:08:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:08:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:08:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:08:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:08:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:08:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:08:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:08:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:08:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:08:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:08:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:08:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:08:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:08:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:08:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:08:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:08:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:08:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:08:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:08:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:08:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:08:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:08:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:08:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:08:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:08:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:08:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:08:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:08:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:08:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:08:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:08:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:08:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:08:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:08:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:08:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:08:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:08:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:08:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:08:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:08:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:08:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:08:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:08:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:08:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:08:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:08:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:08:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:08:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:08:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:08:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:08:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:08:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:08:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:08:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:08:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:08:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:08:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:08:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:08:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:08:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:08:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:08:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:08:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:08:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:08:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:08:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:08:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:08:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:08:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:08:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:08:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:08:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:08:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:08:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:08:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:08:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:08:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:08:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:08:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:08:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:08:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:08:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:08:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:08:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:08:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:08:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:08:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:08:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:08:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:08:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:08:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:08:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:08:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:08:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:08:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:08:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:08:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:08:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:08:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:08:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:08:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:08:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:08:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:08:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:08:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:08:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:08:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:08:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:08:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:08:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:08:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:08:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:08:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:08:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:08:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:08:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:08:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:08:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:08:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:08:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:08:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:08:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:08:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:08:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:08:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:08:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:08:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:08:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:08:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:08:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:08:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:08:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:08:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:08:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:08:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:08:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:08:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:08:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:08:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:08:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:08:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:08:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:08:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:08:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:08:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:08:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:10:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:10:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:10:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:10:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:10:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:10:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:10:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:10:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:10:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:10:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:10:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:10:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:10:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:10:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:10:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:10:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:10:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:10:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:10:59 rdma06 rsyslogd[1041]: imjournal: 5297 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 00:11:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:11:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:11:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:11:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:11:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:11:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:11:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:11:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:11:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:11:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:11:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:11:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:11:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:11:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:11:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:11:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:11:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:11:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:11:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:11:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:11:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:11:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:11:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:11:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:11:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:11:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:11:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:11:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:11:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:11:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:11:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:11:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:11:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:11:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:11:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:11:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:11:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:11:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:11:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:11:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:11:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:11:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:11:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:11:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:11:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:11:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:11:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:11:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:11:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:11:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:11:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:11:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:11:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:11:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:11:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:11:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:11:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:11:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:11:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:11:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:11:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:11:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:11:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:11:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:11:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:11:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:11:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:11:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:11:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:11:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:11:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:11:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:11:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:11:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:11:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:11:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:11:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:11:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:11:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:11:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:11:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:11:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:11:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:11:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:11:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:11:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:11:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:11:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:11:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:11:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:11:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:11:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:11:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:11:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:11:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:11:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:11:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:11:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:11:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:11:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:11:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:11:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:11:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:11:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:11:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:11:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:11:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:11:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:11:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:11:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:11:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:11:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:11:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:11:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:11:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:11:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:11:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:11:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:11:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:11:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:11:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:11:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:11:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:11:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:11:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:11:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:11:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:11:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:11:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:11:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:11:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:11:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:11:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:11:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:11:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:11:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:11:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:11:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:11:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:11:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:11:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:11:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:11:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:11:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:11:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:11:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:11:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:11:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:11:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:11:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:11:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:11:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:11:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:11:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:11:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:11:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:11:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:11:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:11:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:11:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:11:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:11:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:11:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:11:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:11:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:11:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:11:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:11:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:11:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:11:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:11:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:11:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:11:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:11:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:11:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:11:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:11:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:11:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:11:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:11:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:11:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:11:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:11:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:11:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:11:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:11:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:11:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:11:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:11:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:11:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:11:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:11:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:11:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:11:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:11:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:11:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:11:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:11:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:11:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:11:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:11:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:11:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:11:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:11:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:11:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:11:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:11:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:11:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:11:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:11:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:11:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:11:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:11:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:11:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:11:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:11:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:12:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:12:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:12:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:12:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:12:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:12:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:12:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:12:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:12:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:12:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:12:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:12:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:12:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:12:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:12:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:12:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:12:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:12:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:12:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:12:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:12:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:12:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:12:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:12:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:12:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:12:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:12:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:12:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:12:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:12:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:12:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:12:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:12:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:12:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:12:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:12:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:12:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:12:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:12:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:12:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:12:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:12:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:12:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:12:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:12:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:12:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:12:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:12:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:12:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:12:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:12:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:12:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:12:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:12:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:12:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:12:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:12:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:12:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:12:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:12:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:12:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:12:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:12:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:12:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:12:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:12:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:12:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:12:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:12:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:12:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:12:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:12:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:12:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:12:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:12:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:12:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:12:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:12:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:12:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:12:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:12:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:12:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:12:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:12:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:12:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:12:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:12:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:12:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:12:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:12:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:12:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:12:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:12:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:12:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:12:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:12:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:12:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:12:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:12:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:12:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:12:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:12:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:12:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:12:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:12:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:12:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:12:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:12:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:12:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:12:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:12:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:12:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:12:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:12:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:12:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:12:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:12:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:12:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:12:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:12:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:12:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:12:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:12:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:12:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:12:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:12:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:12:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:12:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:12:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:12:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:12:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:12:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:12:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:12:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:12:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:12:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:12:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:12:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:12:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:12:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:12:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:12:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:12:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:12:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:12:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:12:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:12:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:12:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:12:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:12:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:12:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:12:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:12:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:12:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:12:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:12:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:12:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:12:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:12:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:12:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:12:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:12:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:12:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:12:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:12:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:12:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:12:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:12:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:12:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:12:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:12:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:12:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:12:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:12:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:12:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:12:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:12:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:12:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:12:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:12:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:12:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:12:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:12:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:12:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:12:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:12:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:12:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:12:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:12:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:12:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:12:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:12:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:12:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:12:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:12:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:12:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:12:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:12:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:12:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:12:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:12:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:12:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:12:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:12:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:12:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:12:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:12:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:12:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:12:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:12:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:12:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:12:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:12:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:12:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:12:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:12:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:13:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:13:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:13:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:13:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:13:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:13:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:13:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:13:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:13:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:13:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:13:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:13:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:13:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:13:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:13:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:13:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:13:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:13:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:13:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:13:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:13:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:13:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:13:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:13:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:13:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:13:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:13:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:13:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:13:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:13:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:13:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:13:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:13:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:13:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:13:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:13:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:13:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:13:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:13:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:13:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:13:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:13:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:13:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:13:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:13:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:13:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:13:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:13:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:13:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:13:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:13:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:13:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:13:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:13:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:13:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:13:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:13:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:13:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:13:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:13:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:13:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:13:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:13:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:13:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:13:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:13:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:13:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:13:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:13:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:13:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:13:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:13:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:13:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:13:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:13:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:13:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:13:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:13:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:13:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:13:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:13:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:13:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:13:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:13:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:13:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:13:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:13:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:13:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:13:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:13:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:13:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:13:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:13:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:13:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:13:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:13:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:13:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:13:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:13:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:13:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:13:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:13:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:13:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:13:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:13:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:13:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:13:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:13:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:13:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:13:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:13:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:13:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:13:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:13:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:13:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:13:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:13:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:13:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:13:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:13:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:13:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:13:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:13:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:13:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:13:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:13:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:13:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:13:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:13:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:13:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:13:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:13:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:13:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:13:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:13:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:13:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:13:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:13:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:13:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:13:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:13:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:13:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:13:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:13:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:13:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:13:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:13:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:13:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:13:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:13:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:13:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:13:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:13:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:13:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:13:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:13:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:13:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:13:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:13:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:13:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:13:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:13:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:13:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:13:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:13:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:13:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:13:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:13:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:13:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:13:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:13:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:13:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:13:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:13:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:13:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:13:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:13:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:13:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:13:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:13:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:13:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:13:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:13:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:13:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:13:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:13:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:13:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:13:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:13:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:13:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:13:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:13:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:13:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:13:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:13:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:13:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:13:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:13:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:13:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:13:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:13:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:13:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:13:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:13:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:13:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:13:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:13:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:13:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:13:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:13:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:13:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:13:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:13:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:13:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:13:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:13:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:14:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:14:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:14:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:14:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:14:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:14:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:14:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:14:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:14:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:14:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:14:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:14:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:14:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:14:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:14:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:14:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:14:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:14:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:14:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:14:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:14:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:14:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:14:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:14:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:14:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:14:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:14:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:14:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:14:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:14:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:14:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:14:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:14:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:14:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:14:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:14:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:14:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:14:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:14:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:14:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:14:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:14:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:14:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:14:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:14:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:14:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:14:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:14:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:14:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:14:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:14:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:14:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:14:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:14:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:14:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:14:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:14:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:14:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:14:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:14:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:14:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:14:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:14:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:14:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:14:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:14:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:14:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:14:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:14:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:14:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:14:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:14:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:14:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:14:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:14:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:14:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:14:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:14:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:14:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:14:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:14:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:14:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:14:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:14:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:14:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:14:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:14:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:14:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:14:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:14:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:14:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:14:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:14:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:14:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:14:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:14:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:14:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:14:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:14:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:14:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:14:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:14:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:14:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:14:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:14:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:14:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:14:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:14:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:14:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:14:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:14:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:14:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:14:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:14:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:14:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:14:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:14:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:14:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:14:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:14:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:14:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:14:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:14:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:14:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:14:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:14:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:14:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:14:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:14:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:14:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:14:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:14:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:14:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:14:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:14:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:14:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:14:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:14:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:14:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:14:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:14:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:14:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:14:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:14:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:14:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:14:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:14:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:14:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:14:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:14:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:14:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:14:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:14:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:14:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:14:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:14:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:14:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:14:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:14:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:14:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:14:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:14:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:14:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:14:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:14:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:14:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:14:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:14:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:14:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:14:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:14:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:14:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:14:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:14:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:14:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:14:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:14:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:14:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:14:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:14:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:14:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:14:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:14:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:14:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:14:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:14:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:14:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:14:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:14:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:14:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:14:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:14:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:14:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:14:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:14:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:14:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:14:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:14:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:14:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:14:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:14:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:14:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:14:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:14:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:14:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:14:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:14:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:14:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:14:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:14:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:14:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:14:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:14:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:14:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:14:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:14:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:15:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:15:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:15:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:15:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:15:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:15:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:15:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:15:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:15:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:15:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:15:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:15:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:15:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:15:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:15:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:15:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:15:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:15:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:15:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:15:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:15:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:15:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:15:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:15:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:15:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:15:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:15:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:15:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:15:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:15:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:15:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:15:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:15:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:15:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:15:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:15:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:15:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:15:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:15:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:15:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:15:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:15:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:15:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:15:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:15:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:15:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:15:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:15:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:15:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:15:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:15:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:15:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:15:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:15:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:15:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:15:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:15:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:15:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:15:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:15:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:15:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:15:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:15:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:15:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:15:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:15:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:15:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:15:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:15:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:15:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:15:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:15:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:15:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:15:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:15:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:15:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:15:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:15:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:15:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:15:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:15:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:15:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:15:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:15:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:15:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:15:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:15:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:15:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:15:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:15:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:15:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:15:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:15:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:15:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:15:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:15:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:15:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:15:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:15:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:15:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:15:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:15:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:15:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:15:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:15:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:15:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:15:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:15:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:15:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:15:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:15:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:15:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:15:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:15:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:15:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:15:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:15:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:15:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:15:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:15:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:15:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:15:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:15:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:15:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:15:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:15:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:15:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:15:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:15:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:15:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:15:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:15:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:15:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:15:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:15:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:15:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:15:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:15:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:15:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:15:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:15:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:15:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:15:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:15:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:15:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:15:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:15:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:15:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:15:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:15:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:15:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:15:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:15:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:15:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:15:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:15:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:15:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:15:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:15:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:15:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:15:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:15:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:15:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:15:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:15:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:15:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:15:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:15:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:15:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:15:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:15:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:15:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:15:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:15:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:15:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:15:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:15:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:15:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:15:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:15:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:15:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:15:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:15:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:15:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:15:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:15:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:15:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:15:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:15:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:15:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:15:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:15:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:15:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:15:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:15:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:15:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:15:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:15:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:15:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:15:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:15:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:15:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:15:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:15:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:15:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:15:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:15:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:15:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:15:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:15:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:15:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:15:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:15:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:15:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:15:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:15:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:16:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:16:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:16:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:16:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:16:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:16:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:16:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:16:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:16:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:16:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:16:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:16:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:16:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:16:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:16:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:16:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:16:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:16:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:16:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:16:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:16:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:16:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:16:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:16:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:16:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:16:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:16:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:16:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:16:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:16:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:16:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:16:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:16:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:16:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:16:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:16:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:16:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:16:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:16:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:16:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:16:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:16:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:16:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:16:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:16:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:16:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:16:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:16:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:16:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:16:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:16:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:16:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:16:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:16:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:16:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:16:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:16:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:16:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:16:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:16:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:16:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:16:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:16:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:16:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:16:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:16:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:16:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:16:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:16:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:16:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:16:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:16:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:16:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:16:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:16:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:16:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:16:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:16:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:16:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:16:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:16:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:16:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:16:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:16:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:16:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:16:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:16:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:16:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:16:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:16:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:16:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:16:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:16:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:16:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:16:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:16:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:16:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:16:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:16:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:16:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:16:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:16:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:16:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:16:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:16:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:16:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:16:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:16:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:16:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:16:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:16:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:16:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:16:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:16:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:16:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:16:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:16:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:16:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:16:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:16:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:16:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:16:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:16:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:16:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:16:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:16:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:16:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:16:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:16:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:16:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:16:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:16:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:16:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:16:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:16:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:16:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:16:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:16:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:16:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:16:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:16:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:16:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:16:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:16:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:16:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:16:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:16:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:16:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:16:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:16:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:16:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:16:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:16:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:16:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:16:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:16:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:16:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:16:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:16:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:16:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:16:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:16:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:16:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:16:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:16:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:16:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:16:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:16:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:16:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:16:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:16:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:16:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:16:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:16:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:16:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:16:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:16:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:16:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:16:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:16:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:16:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:16:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:16:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:16:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:16:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:16:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:16:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:16:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:16:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:16:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:16:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:16:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:16:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:16:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:16:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:16:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:16:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:16:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:16:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:16:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:16:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:16:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:16:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:16:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:16:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:16:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:16:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:16:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:16:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:16:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:16:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:16:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:16:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:16:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:16:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:16:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:17:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:17:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:17:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:17:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:17:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:17:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:17:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:17:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:17:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:17:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:17:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:17:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:17:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:17:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:17:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:17:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:17:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:17:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:17:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:17:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:17:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:17:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:17:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:17:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:17:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:17:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:17:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:17:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:17:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:17:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:17:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:17:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:17:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:17:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:17:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:17:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:17:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:17:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:17:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:17:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:17:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:17:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:17:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:17:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:17:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:17:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:17:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:17:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:17:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:17:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:17:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:17:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:17:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:17:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:17:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:17:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:17:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:17:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:17:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:17:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:17:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:17:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:17:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:17:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:17:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:17:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:17:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:17:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:17:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:17:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:17:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:17:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:17:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:17:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:17:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:17:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:17:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:17:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:17:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:17:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:17:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:17:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:17:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:17:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:17:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:17:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:17:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:17:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:17:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:17:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:17:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:17:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:17:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:17:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:17:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:17:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:17:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:17:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:17:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:17:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:17:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:17:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:17:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:17:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:17:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:17:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:17:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:17:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:17:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:17:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:17:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:17:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:17:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:17:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:17:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:17:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:17:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:17:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:17:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:17:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:17:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:17:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:17:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:17:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:17:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:17:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:17:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:17:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:17:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:17:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:17:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:17:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:17:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:17:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:17:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:17:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:17:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:17:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:17:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:17:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:17:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:17:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:17:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:17:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:17:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:17:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:17:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:17:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:17:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:17:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:17:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:17:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:17:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:17:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:17:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:17:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:17:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:17:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:17:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:17:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:17:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:17:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:17:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:17:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:17:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:17:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:17:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:17:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:17:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:17:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:17:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:17:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:17:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:17:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:17:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:17:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:17:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:17:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:17:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:17:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:17:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:17:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:17:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:17:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:17:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:17:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:17:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:17:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:17:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:17:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:17:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:17:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:17:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:17:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:17:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:17:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:17:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:17:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:17:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:17:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:17:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:17:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:17:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:17:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:17:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:17:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:17:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:17:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:17:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:17:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:17:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:17:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:17:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:17:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:17:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:17:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:18:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:18:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:18:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:18:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:18:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:18:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:18:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:18:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:18:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:18:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:18:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:18:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:18:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:18:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:18:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:18:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:18:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:18:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:18:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:18:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:18:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:18:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:18:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:18:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:18:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:18:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:18:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:18:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:18:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:18:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:18:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:18:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:18:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:18:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:18:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:18:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:18:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:18:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:18:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:18:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:18:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:18:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:18:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:18:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:18:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:18:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:18:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:18:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:18:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:18:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:18:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:18:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:18:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:18:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:18:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:18:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:18:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:18:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:18:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:18:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:18:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:18:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:18:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:18:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:18:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:18:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:18:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:18:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:18:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:18:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:18:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:18:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:18:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:18:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:18:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:18:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:18:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:18:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:18:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:18:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:18:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:18:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:18:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:18:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:18:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:18:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:18:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:18:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:18:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:18:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:18:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:18:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:18:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:18:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:18:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:18:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:18:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:18:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:18:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:18:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:18:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:18:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:18:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:18:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:18:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:18:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:18:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:18:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:18:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:18:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:18:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:18:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:18:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:18:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:18:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:18:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:18:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:18:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:18:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:18:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:18:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:18:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:18:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:18:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:18:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:18:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:18:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:18:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:18:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:18:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:18:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:18:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:18:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:18:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:18:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:18:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:18:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:18:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:18:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:18:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:18:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:18:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:18:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:18:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:18:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:18:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:18:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:18:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:18:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:18:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:18:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:18:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:18:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:18:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:18:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:18:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:18:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:18:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:18:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:18:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:18:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:18:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:18:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:18:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:18:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:18:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:18:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:18:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:18:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:18:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:18:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:18:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:18:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:18:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:18:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:18:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:18:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:18:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:18:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:18:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:18:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:18:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:18:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:18:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:18:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:18:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:18:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:18:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:18:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:18:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:18:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:18:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:18:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:18:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:18:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:18:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:18:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:18:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:21:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:21:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:21:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:21:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:21:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:21:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:21:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:21:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:21:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:21:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:21:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:21:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:21:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:21:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:21:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:21:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:21:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:21:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:21:04 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 00:21:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:21:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:21:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:21:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:21:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:21:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:21:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:21:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:21:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:21:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:21:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:21:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:21:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:21:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:21:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:21:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:21:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:21:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:21:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:21:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:21:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:21:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:21:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:21:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:21:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:21:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:21:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:21:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:21:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:21:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:21:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:21:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:21:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:21:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:21:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:21:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:21:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:21:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:21:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:21:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:21:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:21:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:21:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:21:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:21:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:21:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:21:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:21:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:21:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:21:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:21:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:21:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:21:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:21:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:21:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:21:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:21:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:21:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:21:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:21:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:21:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:21:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:21:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:21:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:21:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:21:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:21:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:21:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:21:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:21:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:21:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:21:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:21:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:21:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:21:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:21:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:21:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:21:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:21:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:21:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:21:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:21:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:21:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:21:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:21:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:21:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:21:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:21:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:21:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:21:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:21:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:21:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:21:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:21:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:21:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:21:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:21:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:21:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:22:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:22:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:22:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:22:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:22:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:22:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:22:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:22:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:22:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:22:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:22:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:22:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:22:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:22:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:22:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:22:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:22:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:22:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:22:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:22:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:22:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:22:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:22:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:22:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:22:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:22:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:22:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:22:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:22:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:22:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:22:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:22:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:22:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:22:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:22:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:22:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:22:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:22:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:22:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:22:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:22:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:22:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:22:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:22:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:22:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:22:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:22:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:22:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:22:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:22:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:22:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:22:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:22:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:22:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:22:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:22:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:22:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:22:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:22:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:22:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:22:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:22:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:22:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:22:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:22:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:22:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:22:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:22:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:22:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:22:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:22:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:22:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:22:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:22:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:22:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:22:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:22:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:22:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:22:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:22:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:22:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:22:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:22:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:22:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:22:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:22:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:22:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:22:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:22:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:22:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:22:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:22:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:22:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:22:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:22:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:22:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:22:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:22:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:22:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:22:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:22:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:22:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:22:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:22:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:22:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:22:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:22:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:22:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:22:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:22:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:22:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:22:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:22:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:22:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:22:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:22:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:22:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:22:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:22:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:22:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:22:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:22:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:22:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:22:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:22:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:22:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:22:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:22:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:22:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:22:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:22:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:22:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:22:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:22:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:22:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:22:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:22:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:22:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:22:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:22:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:22:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:22:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:22:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:22:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:22:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:22:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:22:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:22:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:22:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:22:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:22:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:22:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:22:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:22:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:22:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:22:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:22:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:22:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:22:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:22:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:22:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:22:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:22:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:22:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:22:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:22:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:22:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:22:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:22:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:22:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:22:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:22:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:22:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:22:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:22:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:22:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:22:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:22:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:22:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:22:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:22:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:22:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:22:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:22:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:22:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:22:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:22:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:22:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:22:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:22:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:22:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:22:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:22:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:22:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:22:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:22:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:22:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:22:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:22:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:22:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:22:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:22:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:22:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:22:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:22:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:22:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:22:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:22:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:22:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:22:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:22:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:22:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:22:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:22:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:22:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:22:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:23:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:23:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:23:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:23:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:23:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:23:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:23:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:23:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:23:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:23:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:23:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:23:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:23:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:23:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:23:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:23:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:23:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:23:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:23:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:23:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:23:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:23:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:23:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:23:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:23:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:23:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:23:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:23:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:23:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:23:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:23:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:23:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:23:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:23:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:23:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:23:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:23:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:23:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:23:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:23:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:23:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:23:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:23:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:23:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:23:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:23:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:23:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:23:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:23:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:23:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:23:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:23:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:23:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:23:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:23:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:23:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:23:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:23:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:23:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:23:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:23:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:23:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:23:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:23:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:23:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:23:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:23:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:23:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:23:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:23:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:23:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:23:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:23:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:23:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:23:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:23:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:23:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:23:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:23:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:23:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:23:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:23:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:23:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:23:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:23:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:23:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:23:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:23:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:23:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:23:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:23:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:23:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:23:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:23:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:23:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:23:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:23:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:23:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:23:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:23:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:23:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:23:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:23:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:23:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:23:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:23:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:23:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:23:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:23:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:23:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:23:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:23:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:23:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:23:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:23:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:23:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:23:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:23:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:23:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:23:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:23:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:23:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:23:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:23:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:23:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:23:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:23:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:23:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:23:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:23:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:23:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:23:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:23:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:23:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:23:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:23:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:23:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:23:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:23:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:23:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:23:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:23:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:23:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:23:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:23:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:23:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:23:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:23:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:23:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:23:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:23:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:23:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:23:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:23:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:23:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:23:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:23:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:23:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:23:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:23:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:23:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:23:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:23:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:23:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:23:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:23:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:23:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:23:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:23:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:23:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:23:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:23:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:23:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:23:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:23:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:23:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:23:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:23:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:23:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:23:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:23:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:23:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:23:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:23:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:23:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:23:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:23:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:23:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:23:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:23:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:23:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:23:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:23:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:23:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:23:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:23:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:23:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:23:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:23:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:23:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:23:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:23:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:23:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:23:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:23:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:23:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:23:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:23:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:23:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:23:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:23:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:23:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:23:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:23:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:23:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:23:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:24:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:24:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:24:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:24:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:24:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:24:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:24:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:24:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:24:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:24:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:24:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:24:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:24:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:24:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:24:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:24:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:24:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:24:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:24:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:24:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:24:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:24:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:24:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:24:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:24:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:24:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:24:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:24:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:24:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:24:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:24:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:24:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:24:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:24:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:24:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:24:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:24:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:24:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:24:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:24:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:24:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:24:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:24:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:24:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:24:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:24:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:24:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:24:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:24:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:24:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:24:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:24:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:24:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:24:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:24:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:24:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:24:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:24:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:24:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:24:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:24:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:24:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:24:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:24:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:24:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:24:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:24:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:24:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:24:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:24:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:24:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:24:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:24:19 rdma06 systemd-journald[790]: Data hash table of /run/log/journal/95379d2f723847b68d9e976dee8967a9/system.journal has a fill level at 75.0 (163750 of 218332 items, 75497472 file size, 461 bytes per hash table item), suggesting rotation.
Apr 18 00:24:19 rdma06 systemd-journald[790]: /run/log/journal/95379d2f723847b68d9e976dee8967a9/system.journal: Journal header limits reached or header out-of-date, rotating.
Apr 18 00:24:19 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 18 00:24:19 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 18 00:24:19 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 18 00:24:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:24:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:24:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:24:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:24:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:24:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:24:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:24:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:24:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:24:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:24:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:24:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:24:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:24:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:24:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:24:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:24:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:24:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:24:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:24:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:24:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:24:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:24:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:24:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:24:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:24:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:24:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:24:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:24:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:24:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:24:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:24:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:24:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:24:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:24:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:24:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:24:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:24:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:24:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:24:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:24:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:24:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:24:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:24:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:24:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:24:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:24:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:24:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:24:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:24:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:24:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:24:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:24:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:24:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:24:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:24:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:24:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:24:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:24:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:24:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:24:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:24:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:24:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:24:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:24:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:24:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:24:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:24:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:24:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:24:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:24:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:24:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:24:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:24:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:24:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:24:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:24:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:24:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:24:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:24:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:24:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:24:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:24:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:24:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:24:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:24:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:24:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:24:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:24:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:24:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:24:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:24:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:24:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:24:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:24:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:24:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:24:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:24:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:24:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:24:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:24:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:24:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:24:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:24:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:24:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:24:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:24:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:24:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:24:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:24:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:24:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:24:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:24:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:24:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:24:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:24:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:24:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:24:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:24:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:24:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:24:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:24:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:24:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:24:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:24:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:24:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:24:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:24:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:24:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:24:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:24:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:24:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:24:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:24:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:24:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:24:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:24:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:24:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:24:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:24:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:24:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:24:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:24:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:24:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:25:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:25:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:25:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:25:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:25:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:25:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:25:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:25:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:25:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:25:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:25:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:25:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:25:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:25:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:25:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:25:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:25:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:25:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:25:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:25:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:25:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:25:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:25:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:25:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:25:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:25:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:25:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:25:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:25:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:25:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:25:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:25:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:25:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:25:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:25:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:25:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:25:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:25:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:25:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:25:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:25:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:25:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:25:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:25:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:25:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:25:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:25:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:25:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:25:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:25:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:25:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:25:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:25:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:25:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:25:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:25:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:25:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:25:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:25:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:25:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:25:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:25:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:25:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:25:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:25:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:25:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:25:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:25:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:25:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:25:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:25:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:25:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:25:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:25:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:25:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:25:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:25:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:25:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:25:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:25:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:25:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:25:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:25:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:25:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:25:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:25:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:25:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:25:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:25:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:25:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:25:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:25:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:25:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:25:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:25:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:25:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:25:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:25:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:25:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:25:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:25:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:25:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:25:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:25:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:25:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:25:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:25:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:25:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:25:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:25:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:25:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:25:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:25:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:25:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:25:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:25:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:25:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:25:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:25:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:25:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:25:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:25:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:25:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:25:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:25:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:25:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:25:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:25:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:25:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:25:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:25:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:25:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:25:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:25:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:25:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:25:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:25:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:25:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:25:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:25:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:25:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:25:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:25:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:25:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:25:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:25:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:25:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:25:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:25:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:25:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:25:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:25:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:25:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:25:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:25:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:25:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:25:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:25:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:25:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:25:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:25:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:25:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:25:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:25:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:25:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:25:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:25:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:25:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:25:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:25:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:25:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:25:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:25:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:25:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:25:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:25:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:25:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:25:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:25:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:25:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:25:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:25:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:25:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:25:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:25:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:25:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:25:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:25:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:25:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:25:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:25:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:25:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:25:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:25:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:25:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:25:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:25:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:25:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:25:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:25:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:25:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:25:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:25:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:25:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:25:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:25:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:25:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:25:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:25:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:25:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:25:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:25:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:25:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:25:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:25:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:25:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:26:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:26:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:26:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:26:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:26:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:26:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:26:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:26:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:26:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:26:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:26:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:26:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:26:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:26:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:26:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:26:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:26:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:26:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:26:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:26:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:26:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:26:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:26:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:26:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:26:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:26:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:26:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:26:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:26:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:26:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:26:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:26:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:26:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:26:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:26:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:26:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:26:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:26:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:26:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:26:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:26:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:26:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:26:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:26:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:26:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:26:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:26:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:26:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:26:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:26:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:26:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:26:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:26:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:26:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:26:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:26:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:26:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:26:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:26:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:26:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:26:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:26:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:26:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:26:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:26:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:26:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:26:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:26:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:26:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:26:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:26:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:26:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:26:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:26:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:26:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:26:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:26:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:26:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:26:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:26:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:26:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:26:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:26:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:26:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:26:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:26:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:26:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:26:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:26:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:26:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:26:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:26:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:26:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:26:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:26:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:26:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:26:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:26:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:26:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:26:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:26:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:26:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:26:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:26:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:26:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:26:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:26:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:26:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:26:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:26:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:26:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:26:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:26:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:26:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:26:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:26:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:26:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:26:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:26:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:26:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:26:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:26:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:26:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:26:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:26:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:26:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:26:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:26:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:26:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:26:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:26:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:26:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:26:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:26:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:26:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:26:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:26:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:26:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:26:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:26:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:26:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:26:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:26:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:26:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:26:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:26:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:26:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:26:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:26:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:26:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:26:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:26:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:26:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:26:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:26:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:26:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:26:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:26:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:26:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:26:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:26:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:26:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:26:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:26:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:26:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:26:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:26:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:26:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:26:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:26:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:26:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:26:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:26:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:26:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:26:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:26:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:26:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:26:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:26:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:26:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:26:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:26:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:26:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:26:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:26:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:26:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:26:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:26:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:26:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:26:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:26:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:26:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:26:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:26:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:26:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:26:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:26:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:26:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:26:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:26:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:26:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:26:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:26:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:26:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:26:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:26:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:26:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:26:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:26:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:26:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:26:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:26:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:26:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:26:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:26:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:26:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:27:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:27:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:27:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:27:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:27:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:27:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:27:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:27:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:27:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:27:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:27:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:27:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:27:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:27:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:27:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:27:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:27:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:27:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:27:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:27:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:27:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:27:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:27:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:27:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:27:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:27:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:27:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:27:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:27:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:27:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:27:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:27:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:27:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:27:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:27:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:27:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:27:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:27:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:27:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:27:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:27:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:27:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:27:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:27:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:27:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:27:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:27:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:27:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:27:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:27:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:27:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:27:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:27:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:27:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:27:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:27:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:27:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:27:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:27:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:27:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:27:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:27:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:27:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:27:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:27:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:27:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:27:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:27:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:27:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:27:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:27:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:27:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:27:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:27:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:27:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:27:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:27:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:27:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:27:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:27:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:27:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:27:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:27:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:27:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:27:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:27:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:27:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:27:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:27:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:27:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:27:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:27:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:27:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:27:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:27:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:27:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:27:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:27:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:27:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:27:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:27:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:27:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:27:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:27:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:27:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:27:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:27:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:27:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:27:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:27:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:27:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:27:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:27:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:27:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:27:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:27:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:27:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:27:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:27:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:27:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:27:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:27:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:27:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:27:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:27:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:27:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:27:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:27:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:27:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:27:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:27:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:27:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:27:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:27:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:27:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:27:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:27:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:27:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:27:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:27:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:27:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:27:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:27:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:27:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:27:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:27:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:27:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:27:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:27:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:27:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:27:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:27:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:27:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:27:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:27:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:27:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:27:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:27:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:27:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:27:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:27:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:27:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:27:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:27:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:27:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:27:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:27:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:27:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:27:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:27:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:27:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:27:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:27:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:27:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:27:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:27:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:27:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:27:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:27:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:27:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:27:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:27:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:27:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:27:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:27:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:27:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:27:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:27:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:27:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:27:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:27:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:27:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:27:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:27:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:27:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:27:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:27:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:27:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:27:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:27:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:27:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:27:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:27:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:27:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:27:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:27:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:27:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:27:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:27:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:27:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:27:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:27:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:27:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:27:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:27:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:27:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:28:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:28:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:28:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:28:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:28:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:28:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:28:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:28:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:28:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:28:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:28:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:28:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:28:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:28:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:28:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:28:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:28:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:28:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:28:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:28:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:28:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:28:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:28:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:28:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:28:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:28:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:28:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:28:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:28:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:28:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:28:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:28:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:28:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:28:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:28:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:28:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:28:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:28:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:28:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:28:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:28:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:28:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:28:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:28:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:28:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:28:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:28:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:28:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:28:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:28:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:28:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:28:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:28:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:28:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:28:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:28:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:28:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:28:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:28:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:28:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:28:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:28:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:28:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:28:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:28:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:28:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:28:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:28:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:28:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:28:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:28:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:28:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:28:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:28:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:28:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:28:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:28:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:28:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:28:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:28:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:28:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:28:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:28:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:28:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:28:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:28:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:28:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:28:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:28:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:28:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:28:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:28:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:28:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:28:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:28:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:28:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:28:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:28:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:28:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:28:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:28:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:28:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:28:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:28:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:28:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:28:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:28:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:28:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:28:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:28:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:28:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:28:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:28:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:28:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:28:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:28:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:28:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:28:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:28:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:28:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:28:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:28:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:28:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:28:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:28:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:28:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:28:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:28:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:28:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:28:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:28:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:28:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:28:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:28:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:28:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:28:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:28:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:28:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:28:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:28:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:28:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:28:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:28:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:28:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:28:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:28:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:28:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:28:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:28:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:28:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:28:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:28:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:28:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:28:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:28:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:28:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:28:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:28:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:28:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:28:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:28:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:28:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:28:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:28:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:28:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:28:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:28:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:28:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:28:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:28:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:28:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:28:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:28:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:28:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:28:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:28:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:28:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:28:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:28:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:28:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:28:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:28:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:28:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:28:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:28:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:28:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:28:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:28:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:28:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:28:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:28:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:28:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:28:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:28:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:28:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:28:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:28:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:28:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:28:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:28:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:28:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:28:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:28:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:28:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:28:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:28:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:28:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:28:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:28:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:28:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:28:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:28:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:28:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:28:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:28:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:28:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:31:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:31:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:31:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:31:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:31:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:31:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:31:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:31:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:31:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:31:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:31:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:31:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:31:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:31:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:31:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:31:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:31:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:31:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:31:05 rdma06 rsyslogd[1041]: imjournal: 5087 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 00:31:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:31:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:31:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:31:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:31:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:31:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:31:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:31:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:31:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:31:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:31:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:31:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:31:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:31:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:31:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:31:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:31:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:31:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:31:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:31:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:31:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:31:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:31:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:31:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:31:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:31:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:31:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:31:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:31:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:31:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:31:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:31:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:31:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:31:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:31:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:31:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:31:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:31:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:31:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:31:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:31:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:31:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:31:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:31:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:31:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:31:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:31:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:31:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:31:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:31:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:31:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:31:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:31:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:31:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:31:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:31:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:31:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:31:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:31:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:31:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:31:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:31:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:31:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:31:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:31:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:31:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:31:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:31:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:31:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:31:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:31:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:31:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:31:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:31:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:31:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:31:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:31:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:31:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:31:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:31:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:31:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:31:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:31:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:31:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:31:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:31:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:31:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:31:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:31:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:31:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:31:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:31:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:31:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:31:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:31:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:31:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:31:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:31:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:31:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:31:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:31:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:31:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:31:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:31:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:31:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:31:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:31:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:31:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:31:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:31:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:31:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:31:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:31:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:31:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:31:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:31:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:31:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:31:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:31:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:31:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:31:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:31:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:31:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:31:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:31:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:31:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:31:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:31:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:31:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:31:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:31:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:31:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:31:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:31:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:31:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:31:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:31:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:31:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:31:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:31:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:31:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:31:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:31:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:31:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:31:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:31:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:31:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:31:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:31:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:31:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:31:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:31:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:31:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:31:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:31:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:31:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:31:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:31:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:31:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:31:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:31:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:31:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:31:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:31:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:31:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:31:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:31:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:31:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:31:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:31:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:31:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:31:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:31:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:31:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:31:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:31:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:31:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:31:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:31:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:31:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:31:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:31:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:31:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:31:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:31:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:31:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:31:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:31:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:32:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:32:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:32:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:32:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:32:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:32:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:32:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:32:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:32:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:32:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:32:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:32:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:32:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:32:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:32:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:32:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:32:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:32:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:32:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:32:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:32:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:32:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:32:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:32:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:32:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:32:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:32:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:32:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:32:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:32:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:32:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:32:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:32:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:32:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:32:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:32:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:32:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:32:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:32:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:32:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:32:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:32:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:32:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:32:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:32:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:32:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:32:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:32:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:32:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:32:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:32:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:32:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:32:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:32:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:32:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:32:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:32:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:32:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:32:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:32:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:32:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:32:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:32:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:32:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:32:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:32:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:32:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:32:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:32:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:32:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:32:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:32:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:32:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:32:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:32:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:32:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:32:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:32:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:32:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:32:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:32:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:32:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:32:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:32:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:32:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:32:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:32:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:32:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:32:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:32:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:32:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:32:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:32:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:32:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:32:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:32:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:32:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:32:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:32:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:32:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:32:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:32:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:32:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:32:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:32:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:32:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:32:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:32:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:32:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:32:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:32:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:32:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:32:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:32:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:32:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:32:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:32:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:32:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:32:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:32:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:32:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:32:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:32:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:32:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:32:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:32:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:32:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:32:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:32:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:32:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:32:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:32:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:32:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:32:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:32:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:32:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:32:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:32:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:32:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:32:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:32:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:32:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:32:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:32:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:32:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:32:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:32:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:32:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:32:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:32:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:32:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:32:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:32:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:32:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:32:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:32:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:32:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:32:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:32:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:32:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:32:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:32:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:32:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:32:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:32:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:32:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:32:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:32:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:32:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:32:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:32:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:32:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:32:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:32:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:32:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:32:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:32:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:32:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:32:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:32:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:32:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:32:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:32:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:32:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:32:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:32:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:32:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:32:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:32:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:32:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:32:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:32:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:32:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:32:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:32:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:32:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:32:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:32:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:32:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:32:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:32:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:32:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:32:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:32:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:32:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:32:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:32:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:32:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:32:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:32:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:32:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:32:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:32:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:32:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:32:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:32:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:33:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:33:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:33:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:33:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:33:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:33:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:33:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:33:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:33:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:33:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:33:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:33:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:33:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:33:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:33:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:33:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:33:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:33:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:33:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:33:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:33:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:33:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:33:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:33:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:33:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:33:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:33:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:33:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:33:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:33:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:33:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:33:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:33:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:33:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:33:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:33:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:33:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:33:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:33:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:33:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:33:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:33:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:33:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:33:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:33:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:33:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:33:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:33:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:33:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:33:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:33:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:33:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:33:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:33:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:33:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:33:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:33:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:33:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:33:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:33:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:33:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:33:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:33:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:33:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:33:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:33:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:33:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:33:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:33:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:33:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:33:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:33:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:33:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:33:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:33:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:33:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:33:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:33:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:33:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:33:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:33:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:33:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:33:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:33:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:33:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:33:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:33:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:33:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:33:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:33:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:33:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:33:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:33:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:33:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:33:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:33:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:33:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:33:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:33:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:33:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:33:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:33:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:33:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:33:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:33:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:33:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:33:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:33:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:33:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:33:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:33:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:33:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:33:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:33:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:33:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:33:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:33:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:33:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:33:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:33:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:33:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:33:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:33:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:33:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:33:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:33:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:33:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:33:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:33:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:33:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:33:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:33:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:33:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:33:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:33:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:33:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:33:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:33:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:33:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:33:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:33:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:33:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:33:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:33:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:33:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:33:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:33:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:33:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:33:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:33:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:33:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:33:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:33:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:33:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:33:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:33:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:33:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:33:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:33:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:33:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:33:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:33:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:33:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:33:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:33:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:33:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:33:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:33:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:33:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:33:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:33:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:33:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:33:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:33:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:33:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:33:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:33:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:33:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:33:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:33:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:33:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:33:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:33:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:33:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:33:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:33:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:33:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:33:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:33:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:33:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:33:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:33:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:33:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:33:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:33:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:33:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:33:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:33:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:33:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:33:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:33:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:33:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:33:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:33:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:33:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:33:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:33:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:33:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:33:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:33:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:33:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:33:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:33:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:33:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:33:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:33:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:34:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:34:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:34:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:34:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:34:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:34:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:34:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:34:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:34:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:34:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:34:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:34:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:34:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:34:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:34:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:34:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:34:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:34:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:34:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:34:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:34:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:34:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:34:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:34:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:34:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:34:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:34:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:34:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:34:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:34:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:34:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:34:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:34:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:34:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:34:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:34:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:34:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:34:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:34:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:34:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:34:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:34:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:34:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:34:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:34:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:34:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:34:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:34:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:34:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:34:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:34:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:34:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:34:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:34:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:34:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:34:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:34:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:34:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:34:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:34:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:34:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:34:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:34:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:34:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:34:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:34:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:34:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:34:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:34:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:34:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:34:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:34:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:34:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:34:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:34:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:34:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:34:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:34:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:34:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:34:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:34:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:34:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:34:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:34:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:34:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:34:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:34:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:34:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:34:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:34:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:34:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:34:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:34:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:34:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:34:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:34:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:34:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:34:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:34:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:34:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:34:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:34:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:34:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:34:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:34:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:34:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:34:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:34:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:34:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:34:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:34:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:34:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:34:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:34:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:34:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:34:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:34:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:34:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:34:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:34:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:34:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:34:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:34:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:34:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:34:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:34:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:34:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:34:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:34:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:34:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:34:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:34:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:34:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:34:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:34:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:34:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:34:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:34:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:34:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:34:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:34:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:34:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:34:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:34:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:34:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:34:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:34:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:34:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:34:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:34:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:34:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:34:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:34:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:34:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:34:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:34:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:34:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:34:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:34:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:34:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:34:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:34:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:34:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:34:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:34:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:34:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:34:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:34:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:34:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:34:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:34:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:34:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:34:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:34:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:34:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:34:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:34:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:34:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:34:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:34:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:34:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:34:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:34:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:34:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:34:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:34:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:34:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:34:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:34:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:34:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:34:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:34:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:34:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:34:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:34:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:34:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:34:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:34:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:34:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:34:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:34:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:34:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:34:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:34:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:34:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:34:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:34:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:34:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:34:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:34:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:34:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:34:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:34:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:34:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:34:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:34:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:35:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:35:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:35:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:35:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:35:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:35:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:35:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:35:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:35:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:35:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:35:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:35:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:35:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:35:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:35:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:35:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:35:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:35:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:35:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:35:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:35:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:35:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:35:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:35:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:35:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:35:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:35:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:35:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:35:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:35:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:35:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:35:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:35:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:35:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:35:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:35:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:35:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:35:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:35:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:35:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:35:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:35:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:35:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:35:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:35:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:35:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:35:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:35:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:35:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:35:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:35:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:35:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:35:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:35:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:35:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:35:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:35:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:35:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:35:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:35:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:35:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:35:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:35:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:35:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:35:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:35:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:35:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:35:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:35:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:35:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:35:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:35:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:35:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:35:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:35:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:35:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:35:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:35:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:35:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:35:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:35:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:35:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:35:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:35:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:35:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:35:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:35:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:35:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:35:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:35:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:35:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:35:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:35:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:35:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:35:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:35:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:35:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:35:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:35:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:35:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:35:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:35:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:35:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:35:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:35:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:35:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:35:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:35:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:35:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:35:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:35:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:35:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:35:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:35:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:35:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:35:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:35:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:35:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:35:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:35:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:35:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:35:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:35:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:35:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:35:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:35:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:35:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:35:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:35:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:35:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:35:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:35:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:35:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:35:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:35:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:35:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:35:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:35:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:35:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:35:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:35:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:35:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:35:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:35:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:35:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:35:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:35:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:35:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:35:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:35:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:35:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:35:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:35:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:35:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:35:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:35:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:35:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:35:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:35:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:35:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:35:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:35:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:35:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:35:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:35:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:35:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:35:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:35:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:35:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:35:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:35:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:35:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:35:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:35:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:35:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:35:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:35:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:35:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:35:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:35:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:35:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:35:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:35:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:35:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:35:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:35:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:35:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:35:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:35:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:35:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:35:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:35:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:35:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:35:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:35:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:35:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:35:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:35:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:35:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:35:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:35:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:35:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:35:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:35:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:35:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:35:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:35:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:35:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:35:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:35:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:35:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:35:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:35:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:35:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:35:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:35:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:36:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:36:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:36:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:36:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:36:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:36:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:36:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:36:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:36:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:36:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:36:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:36:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:36:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:36:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:36:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:36:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:36:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:36:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:36:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:36:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:36:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:36:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:36:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:36:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:36:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:36:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:36:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:36:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:36:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:36:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:36:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:36:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:36:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:36:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:36:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:36:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:36:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:36:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:36:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:36:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:36:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:36:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:36:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:36:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:36:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:36:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:36:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:36:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:36:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:36:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:36:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:36:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:36:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:36:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:36:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:36:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:36:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:36:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:36:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:36:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:36:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:36:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:36:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:36:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:36:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:36:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:36:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:36:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:36:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:36:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:36:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:36:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:36:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:36:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:36:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:36:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:36:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:36:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:36:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:36:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:36:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:36:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:36:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:36:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:36:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:36:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:36:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:36:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:36:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:36:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:36:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:36:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:36:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:36:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:36:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:36:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:36:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:36:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:36:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:36:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:36:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:36:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:36:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:36:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:36:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:36:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:36:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:36:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:36:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:36:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:36:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:36:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:36:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:36:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:36:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:36:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:36:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:36:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:36:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:36:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:36:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:36:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:36:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:36:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:36:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:36:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:36:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:36:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:36:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:36:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:36:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:36:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:36:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:36:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:36:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:36:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:36:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:36:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:36:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:36:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:36:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:36:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:36:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:36:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:36:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:36:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:36:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:36:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:36:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:36:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:36:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:36:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:36:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:36:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:36:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:36:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:36:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:36:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:36:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:36:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:36:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:36:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:36:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:36:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:36:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:36:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:36:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:36:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:36:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:36:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:36:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:36:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:36:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:36:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:36:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:36:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:36:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:36:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:36:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:36:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:36:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:36:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:36:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:36:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:36:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:36:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:36:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:36:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:36:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:36:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:36:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:36:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:36:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:36:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:36:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:36:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:36:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:36:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:36:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:36:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:36:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:36:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:36:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:36:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:36:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:36:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:36:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:36:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:36:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:36:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:36:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:36:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:36:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:36:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:36:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:36:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:37:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:37:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:37:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:37:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:37:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:37:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:37:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:37:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:37:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:37:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:37:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:37:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:37:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:37:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:37:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:37:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:37:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:37:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:37:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:37:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:37:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:37:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:37:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:37:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:37:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:37:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:37:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:37:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:37:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:37:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:37:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:37:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:37:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:37:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:37:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:37:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:37:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:37:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:37:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:37:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:37:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:37:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:37:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:37:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:37:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:37:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:37:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:37:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:37:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:37:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:37:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:37:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:37:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:37:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:37:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:37:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:37:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:37:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:37:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:37:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:37:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:37:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:37:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:37:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:37:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:37:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:37:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:37:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:37:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:37:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:37:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:37:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:37:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:37:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:37:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:37:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:37:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:37:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:37:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:37:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:37:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:37:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:37:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:37:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:37:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:37:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:37:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:37:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:37:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:37:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:37:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:37:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:37:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:37:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:37:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:37:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:37:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:37:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:37:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:37:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:37:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:37:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:37:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:37:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:37:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:37:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:37:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:37:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:37:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:37:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:37:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:37:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:37:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:37:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:37:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:37:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:37:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:37:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:37:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:37:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:37:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:37:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:37:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:37:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:37:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:37:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:37:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:37:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:37:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:37:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:37:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:37:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:37:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:37:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:37:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:37:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:37:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:37:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:37:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:37:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:37:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:37:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:37:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:37:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:37:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:37:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:37:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:37:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:37:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:37:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:37:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:37:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:37:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:37:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:37:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:37:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:37:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:37:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:37:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:37:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:37:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:37:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:37:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:37:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:37:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:37:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:37:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:37:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:37:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:37:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:37:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:37:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:37:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:37:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:37:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:37:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:37:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:37:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:37:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:37:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:37:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:37:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:37:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:37:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:37:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:37:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:37:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:37:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:37:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:37:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:37:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:37:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:37:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:37:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:37:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:37:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:37:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:37:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:37:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:37:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:37:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:37:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:37:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:37:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:37:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:37:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:37:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:37:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:37:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:37:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:37:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:37:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:37:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:37:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:37:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:37:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:38:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:38:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:38:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:38:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:38:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:38:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:38:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:38:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:38:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:38:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:38:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:38:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:38:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:38:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:38:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:38:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:38:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:38:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:38:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:38:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:38:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:38:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:38:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:38:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:38:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:38:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:38:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:38:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:38:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:38:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:38:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:38:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:38:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:38:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:38:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:38:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:38:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:38:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:38:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:38:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:38:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:38:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:38:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:38:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:38:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:38:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:38:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:38:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:38:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:38:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:38:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:38:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:38:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:38:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:38:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:38:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:38:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:38:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:38:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:38:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:38:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:38:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:38:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:38:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:38:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:38:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:38:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:38:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:38:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:38:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:38:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:38:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:38:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:38:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:38:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:38:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:38:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:38:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:38:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:38:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:38:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:38:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:38:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:38:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:38:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:38:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:38:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:38:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:38:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:38:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:38:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:38:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:38:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:38:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:38:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:38:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:38:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:38:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:38:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:38:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:38:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:38:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:38:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:38:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:38:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:38:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:38:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:38:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:38:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:38:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:38:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:38:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:38:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:38:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:38:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:38:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:38:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:38:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:38:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:38:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:38:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:38:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:38:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:38:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:38:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:38:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:38:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:38:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:38:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:38:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:38:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:38:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:38:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:38:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:38:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:38:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:38:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:38:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:38:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:38:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:38:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:38:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:38:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:38:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:38:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:38:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:38:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:38:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:38:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:38:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:38:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:38:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:38:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:38:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:38:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:38:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:38:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:38:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:38:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:38:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:38:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:38:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:38:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:38:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:38:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:38:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:38:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:38:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:38:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:38:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:38:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:38:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:38:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:38:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:38:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:38:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:38:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:38:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:38:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:38:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:38:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:38:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:38:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:38:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:38:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:38:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:38:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:38:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:38:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:38:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:38:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:38:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:38:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:38:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:38:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:38:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:38:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:38:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:38:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:38:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:38:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:38:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:38:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:38:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:38:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:38:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:38:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:38:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:38:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:38:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:38:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:38:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:38:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:38:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:38:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:38:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:41:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:41:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:41:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:41:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:41:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:41:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:41:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:41:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:41:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:41:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:41:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:41:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:41:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:41:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:41:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:41:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:41:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:41:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:41:10 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 00:41:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:41:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:41:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:41:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:41:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:41:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:41:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:41:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:41:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:41:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:41:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:41:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:41:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:41:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:41:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:41:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:41:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:41:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:41:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:41:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:41:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:41:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:41:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:41:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:41:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:41:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:41:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:41:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:41:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:41:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:41:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:41:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:41:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:41:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:41:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:41:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:41:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:41:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:41:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:41:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:41:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:41:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:41:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:41:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:41:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:41:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:41:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:41:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:41:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:41:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:41:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:41:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:41:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:41:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:41:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:41:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:41:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:41:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:41:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:41:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:41:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:41:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:41:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:41:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:41:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:41:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:41:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:41:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:41:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:41:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:41:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:41:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:41:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:41:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:41:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:41:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:41:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:41:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:41:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:41:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:41:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:41:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:41:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:41:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:41:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:41:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:41:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:41:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:41:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:41:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:41:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:41:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:41:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:41:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:41:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:41:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:41:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:41:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:41:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:41:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:41:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:41:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:41:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:41:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:41:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:41:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:41:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:41:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:41:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:41:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:41:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:41:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:41:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:41:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:41:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:41:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:41:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:41:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:41:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:41:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:41:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:41:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:41:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:41:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:41:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:41:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:41:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:41:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:41:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:41:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:41:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:41:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:41:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:41:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:41:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:41:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:41:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:41:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:41:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:41:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:41:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:41:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:41:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:41:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:41:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:41:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:41:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:41:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:41:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:41:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:41:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:41:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:41:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:41:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:41:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:41:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:41:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:41:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:41:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:41:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:41:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:41:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:41:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:41:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:41:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:41:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:41:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:41:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:41:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:41:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:41:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:41:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:41:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:41:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:41:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:41:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:41:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:42:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:42:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:42:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:42:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:42:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:42:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:42:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:42:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:42:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:42:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:42:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:42:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:42:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:42:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:42:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:42:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:42:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:42:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:42:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:42:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:42:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:42:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:42:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:42:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:42:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:42:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:42:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:42:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:42:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:42:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:42:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:42:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:42:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:42:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:42:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:42:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:42:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:42:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:42:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:42:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:42:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:42:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:42:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:42:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:42:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:42:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:42:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:42:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:42:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:42:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:42:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:42:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:42:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:42:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:42:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:42:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:42:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:42:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:42:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:42:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:42:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:42:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:42:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:42:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:42:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:42:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:42:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:42:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:42:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:42:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:42:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:42:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:42:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:42:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:42:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:42:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:42:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:42:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:42:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:42:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:42:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:42:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:42:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:42:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:42:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:42:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:42:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:42:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:42:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:42:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:42:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:42:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:42:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:42:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:42:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:42:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:42:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:42:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:42:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:42:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:42:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:42:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:42:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:42:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:42:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:42:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:42:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:42:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:42:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:42:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:42:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:42:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:42:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:42:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:42:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:42:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:42:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:42:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:42:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:42:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:42:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:42:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:42:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:42:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:42:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:42:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:42:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:42:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:42:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:42:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:42:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:42:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:42:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:42:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:42:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:42:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:42:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:42:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:42:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:42:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:42:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:42:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:42:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:42:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:42:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:42:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:42:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:42:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:42:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:42:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:42:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:42:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:42:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:42:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:42:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:42:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:42:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:42:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:42:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:42:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:42:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:42:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:42:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:42:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:42:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:42:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:42:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:42:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:42:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:42:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:42:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:42:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:42:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:42:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:42:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:42:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:42:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:42:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:42:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:42:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:42:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:42:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:42:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:42:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:42:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:42:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:42:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:42:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:42:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:42:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:42:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:42:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:42:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:42:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:42:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:42:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:42:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:42:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:42:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:42:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:42:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:43:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:43:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:43:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:43:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:43:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:43:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:43:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:43:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:43:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:43:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:43:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:43:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:43:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:43:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:43:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:43:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:43:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:43:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:43:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:43:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:43:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:43:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:43:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:43:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:43:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:43:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:43:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:43:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:43:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:43:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:43:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:43:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:43:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:43:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:43:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:43:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:43:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:43:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:43:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:43:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:43:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:43:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:43:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:43:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:43:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:43:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:43:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:43:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:43:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:43:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:43:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:43:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:43:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:43:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:43:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:43:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:43:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:43:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:43:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:43:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:43:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:43:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:43:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:43:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:43:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:43:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:43:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:43:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:43:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:43:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:43:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:43:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:43:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:43:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:43:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:43:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:43:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:43:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:43:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:43:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:43:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:43:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:43:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:43:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:43:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:43:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:43:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:43:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:43:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:43:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:43:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:43:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:43:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:43:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:43:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:43:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:43:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:43:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:43:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:43:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:43:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:43:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:43:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:43:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:43:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:43:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:43:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:43:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:43:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:43:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:43:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:43:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:43:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:43:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:43:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:43:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:43:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:43:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:43:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:43:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:43:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:43:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:43:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:43:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:43:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:43:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:43:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:43:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:43:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:43:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:43:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:43:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:43:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:43:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:43:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:43:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:43:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:43:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:43:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:43:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:43:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:43:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:43:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:43:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:43:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:43:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:43:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:43:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:43:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:43:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:43:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:43:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:43:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:43:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:43:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:43:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:43:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:43:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:43:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:43:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:43:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:43:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:43:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:43:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:43:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:43:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:43:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:43:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:43:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:43:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:43:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:43:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:43:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:43:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:43:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:43:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:43:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:43:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:43:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:43:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:43:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:43:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:43:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:43:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:43:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:43:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:43:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:43:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:43:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:43:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:43:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:43:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:43:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:43:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:43:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:43:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:43:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:43:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:43:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:43:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:43:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:43:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:43:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:43:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:43:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:43:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:43:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:43:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:43:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:43:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:43:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:43:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:43:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:43:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:43:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:43:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:44:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:44:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:44:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:44:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:44:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:44:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:44:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:44:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:44:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:44:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:44:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:44:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:44:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:44:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:44:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:44:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:44:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:44:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:44:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:44:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:44:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:44:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:44:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:44:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:44:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:44:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:44:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:44:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:44:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:44:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:44:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:44:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:44:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:44:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:44:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:44:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:44:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:44:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:44:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:44:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:44:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:44:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:44:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:44:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:44:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:44:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:44:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:44:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:44:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:44:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:44:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:44:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:44:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:44:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:44:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:44:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:44:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:44:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:44:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:44:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:44:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:44:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:44:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:44:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:44:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:44:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:44:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:44:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:44:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:44:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:44:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:44:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:44:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:44:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:44:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:44:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:44:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:44:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:44:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:44:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:44:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:44:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:44:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:44:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:44:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:44:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:44:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:44:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:44:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:44:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:44:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:44:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:44:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:44:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:44:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:44:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:44:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:44:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:44:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:44:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:44:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:44:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:44:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:44:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:44:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:44:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:44:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:44:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:44:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:44:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:44:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:44:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:44:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:44:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:44:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:44:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:44:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:44:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:44:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:44:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:44:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:44:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:44:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:44:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:44:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:44:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:44:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:44:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:44:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:44:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:44:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:44:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:44:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:44:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:44:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:44:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:44:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:44:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:44:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:44:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:44:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:44:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:44:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:44:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:44:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:44:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:44:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:44:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:44:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:44:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:44:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:44:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:44:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:44:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:44:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:44:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:44:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:44:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:44:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:44:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:44:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:44:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:44:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:44:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:44:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:44:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:44:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:44:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:44:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:44:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:44:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:44:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:44:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:44:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:44:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:44:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:44:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:44:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:44:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:44:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:44:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:44:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:44:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:44:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:44:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:44:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:44:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:44:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:44:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:44:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:44:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:44:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:44:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:44:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:44:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:44:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:44:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:44:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:44:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:44:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:44:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:44:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:44:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:44:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:44:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:44:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:44:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:44:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:44:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:44:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:44:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:44:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:44:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:44:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:44:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:44:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:45:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:45:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:45:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:45:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:45:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:45:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:45:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:45:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:45:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:45:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:45:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:45:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:45:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:45:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:45:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:45:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:45:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:45:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:45:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:45:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:45:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:45:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:45:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:45:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:45:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:45:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:45:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:45:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:45:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:45:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:45:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:45:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:45:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:45:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:45:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:45:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:45:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:45:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:45:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:45:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:45:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:45:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:45:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:45:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:45:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:45:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:45:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:45:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:45:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:45:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:45:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:45:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:45:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:45:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:45:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:45:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:45:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:45:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:45:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:45:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:45:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:45:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:45:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:45:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:45:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:45:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:45:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:45:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:45:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:45:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:45:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:45:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:45:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:45:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:45:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:45:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:45:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:45:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:45:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:45:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:45:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:45:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:45:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:45:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:45:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:45:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:45:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:45:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:45:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:45:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:45:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:45:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:45:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:45:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:45:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:45:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:45:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:45:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:45:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:45:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:45:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:45:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:45:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:45:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:45:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:45:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:45:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:45:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:45:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:45:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:45:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:45:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:45:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:45:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:45:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:45:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:45:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:45:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:45:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:45:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:45:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:45:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:45:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:45:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:45:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:45:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:45:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:45:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:45:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:45:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:45:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:45:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:45:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:45:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:45:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:45:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:45:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:45:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:45:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:45:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:45:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:45:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:45:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:45:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:45:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:45:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:45:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:45:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:45:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:45:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:45:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:45:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:45:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:45:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:45:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:45:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:45:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:45:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:45:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:45:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:45:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:45:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:45:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:45:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:45:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:45:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:45:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:45:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:45:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:45:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:45:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:45:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:45:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:45:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:45:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:45:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:45:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:45:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:45:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:45:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:45:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:45:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:45:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:45:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:45:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:45:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:45:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:45:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:45:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:45:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:45:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:45:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:45:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:45:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:45:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:45:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:45:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:45:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:45:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:45:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:45:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:45:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:45:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:45:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:45:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:45:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:45:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:45:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:45:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:45:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:45:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:45:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:45:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:45:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:45:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:45:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:46:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:46:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:46:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:46:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:46:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:46:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:46:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:46:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:46:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:46:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:46:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:46:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:46:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:46:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:46:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:46:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:46:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:46:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:46:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:46:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:46:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:46:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:46:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:46:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:46:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:46:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:46:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:46:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:46:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:46:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:46:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:46:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:46:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:46:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:46:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:46:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:46:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:46:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:46:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:46:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:46:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:46:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:46:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:46:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:46:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:46:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:46:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:46:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:46:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:46:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:46:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:46:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:46:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:46:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:46:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:46:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:46:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:46:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:46:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:46:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:46:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:46:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:46:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:46:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:46:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:46:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:46:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:46:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:46:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:46:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:46:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:46:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:46:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:46:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:46:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:46:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:46:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:46:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:46:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:46:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:46:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:46:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:46:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:46:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:46:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:46:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:46:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:46:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:46:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:46:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:46:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:46:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:46:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:46:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:46:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:46:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:46:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:46:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:46:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:46:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:46:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:46:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:46:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:46:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:46:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:46:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:46:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:46:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:46:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:46:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:46:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:46:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:46:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:46:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:46:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:46:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:46:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:46:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:46:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:46:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:46:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:46:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:46:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:46:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:46:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:46:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:46:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:46:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:46:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:46:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:46:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:46:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:46:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:46:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:46:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:46:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:46:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:46:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:46:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:46:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:46:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:46:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:46:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:46:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:46:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:46:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:46:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:46:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:46:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:46:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:46:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:46:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:46:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:46:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:46:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:46:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:46:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:46:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:46:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:46:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:46:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:46:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:46:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:46:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:46:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:46:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:46:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:46:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:46:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:46:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:46:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:46:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:46:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:46:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:46:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:46:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:46:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:46:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:46:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:46:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:46:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:46:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:46:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:46:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:46:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:46:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:46:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:46:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:46:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:46:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:46:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:46:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:46:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:46:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:46:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:46:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:46:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:46:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:46:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:46:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:46:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:46:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:46:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:46:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:46:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:46:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:46:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:46:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:46:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:46:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:46:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:46:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:46:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:46:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:46:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:46:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:47:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:47:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:47:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:47:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:47:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:47:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:47:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:47:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:47:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:47:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:47:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:47:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:47:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:47:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:47:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:47:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:47:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:47:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:47:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:47:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:47:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:47:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:47:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:47:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:47:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:47:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:47:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:47:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:47:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:47:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:47:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:47:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:47:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:47:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:47:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:47:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:47:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:47:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:47:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:47:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:47:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:47:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:47:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:47:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:47:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:47:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:47:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:47:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:47:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:47:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:47:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:47:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:47:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:47:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:47:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:47:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:47:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:47:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:47:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:47:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:47:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:47:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:47:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:47:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:47:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:47:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:47:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:47:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:47:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:47:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:47:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:47:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:47:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:47:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:47:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:47:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:47:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:47:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:47:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:47:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:47:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:47:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:47:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:47:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:47:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:47:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:47:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:47:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:47:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:47:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:47:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:47:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:47:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:47:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:47:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:47:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:47:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:47:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:47:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:47:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:47:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:47:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:47:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:47:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:47:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:47:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:47:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:47:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:47:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:47:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:47:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:47:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:47:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:47:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:47:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:47:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:47:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:47:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:47:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:47:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:47:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:47:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:47:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:47:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:47:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:47:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:47:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:47:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:47:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:47:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:47:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:47:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:47:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:47:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:47:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:47:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:47:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:47:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:47:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:47:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:47:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:47:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:47:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:47:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:47:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:47:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:47:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:47:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:47:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:47:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:47:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:47:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:47:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:47:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:47:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:47:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:47:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:47:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:47:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:47:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:47:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:47:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:47:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:47:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:47:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:47:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:47:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:47:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:47:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:47:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:47:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:47:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:47:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:47:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:47:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:47:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:47:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:47:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:47:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:47:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:47:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:47:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:47:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:47:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:47:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:47:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:47:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:47:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:47:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:47:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:47:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:47:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:47:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:47:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:47:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:47:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:47:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:47:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:47:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:47:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:47:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:47:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:47:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:47:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:47:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:47:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:47:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:47:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:47:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:47:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:47:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:47:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:47:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:47:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:47:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:47:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:48:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:48:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:48:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:48:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:48:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:48:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:48:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:48:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:48:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:48:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:48:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:48:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:48:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:48:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:48:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:48:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:48:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:48:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:48:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:48:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:48:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:48:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:48:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:48:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:48:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:48:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:48:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:48:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:48:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:48:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:48:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:48:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:48:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:48:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:48:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:48:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:48:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:48:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:48:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:48:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:48:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:48:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:48:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:48:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:48:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:48:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:48:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:48:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:48:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:48:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:48:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:48:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:48:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:48:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:48:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:48:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:48:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:48:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:48:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:48:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:48:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:48:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:48:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:48:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:48:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:48:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:48:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:48:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:48:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:48:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:48:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:48:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:48:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:48:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:48:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:48:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:48:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:48:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:48:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:48:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:48:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:48:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:48:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:48:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:48:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:48:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:48:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:48:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:48:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:48:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:48:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:48:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:48:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:48:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:48:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:48:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:48:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:48:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:48:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:48:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:48:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:48:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:48:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:48:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:48:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:48:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:48:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:48:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:48:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:48:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:48:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:48:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:48:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:48:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:48:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:48:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:48:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:48:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:48:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:48:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:48:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:48:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:48:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:48:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:48:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:48:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:48:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:48:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:48:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:48:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:48:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:48:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:48:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:48:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:48:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:48:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:48:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:48:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:48:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:48:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:48:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:48:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:48:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:48:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:48:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:48:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:48:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:48:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:48:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:48:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:48:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:48:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:48:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:48:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:48:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:48:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:48:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:48:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:48:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:48:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:48:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:48:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:48:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:48:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:48:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:48:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:48:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:48:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:48:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:48:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:48:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:48:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:48:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:48:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:48:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:48:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:48:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:48:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:48:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:48:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:48:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:48:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:48:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:48:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:48:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:48:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:48:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:48:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:48:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:48:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:48:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:48:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:48:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:48:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:48:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:48:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:48:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:48:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:48:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:48:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:48:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:48:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:48:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:48:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:48:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:48:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:48:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:48:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:48:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:48:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:48:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:48:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:48:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:48:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:48:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:48:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:49:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:49:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:49:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:49:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:49:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:49:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:49:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:49:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:49:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:49:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:49:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:49:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:49:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:49:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:49:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:49:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:49:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:49:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:49:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:49:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:49:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:49:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:49:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:49:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:49:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:49:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:49:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:49:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:49:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:49:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:49:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:49:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:49:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:49:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:49:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:49:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:51:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:51:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:51:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:51:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:51:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:51:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:51:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:51:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:51:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:51:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:51:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:51:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:51:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:51:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:51:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:51:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:51:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:51:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:51:15 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 00:51:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:51:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:51:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:51:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:51:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:51:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:51:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:51:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:51:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:51:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:51:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:51:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:51:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:51:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:51:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:51:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:51:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:51:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:51:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:51:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:51:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:51:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:51:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:51:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:51:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:51:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:51:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:51:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:51:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:51:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:51:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:51:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:51:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:51:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:51:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:51:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:51:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:51:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:51:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:51:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:51:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:51:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:51:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:51:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:51:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:51:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:51:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:51:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:51:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:51:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:51:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:51:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:51:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:51:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:51:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:51:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:51:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:51:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:51:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:51:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:51:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:51:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:51:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:51:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:51:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:51:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:51:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:51:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:51:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:51:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:51:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:51:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:51:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:51:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:51:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:51:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:51:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:51:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:51:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:51:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:51:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:51:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:51:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:51:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:51:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:51:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:51:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:51:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:51:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:51:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:51:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:51:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:51:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:51:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:51:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:51:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:51:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:51:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:51:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:51:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:51:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:51:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:51:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:51:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:51:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:51:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:51:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:51:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:51:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:51:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:51:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:51:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:51:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:51:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:52:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:52:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:52:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:52:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:52:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:52:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:52:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:52:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:52:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:52:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:52:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:52:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:52:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:52:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:52:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:52:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:52:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:52:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:52:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:52:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:52:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:52:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:52:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:52:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:52:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:52:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:52:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:52:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:52:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:52:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:52:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:52:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:52:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:52:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:52:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:52:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:52:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:52:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:52:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:52:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:52:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:52:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:52:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:52:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:52:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:52:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:52:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:52:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:52:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:52:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:52:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:52:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:52:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:52:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:52:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:52:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:52:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:52:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:52:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:52:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:52:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:52:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:52:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:52:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:52:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:52:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:52:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:52:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:52:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:52:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:52:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:52:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:52:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:52:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:52:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:52:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:52:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:52:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:52:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:52:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:52:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:52:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:52:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:52:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:52:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:52:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:52:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:52:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:52:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:52:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:52:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:52:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:52:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:52:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:52:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:52:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:52:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:52:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:52:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:52:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:52:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:52:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:52:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:52:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:52:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:52:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:52:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:52:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:52:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:52:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:52:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:52:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:52:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:52:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:52:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:52:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:52:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:52:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:52:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:52:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:52:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:52:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:52:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:52:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:52:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:52:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:52:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:52:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:52:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:52:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:52:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:52:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:52:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:52:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:52:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:52:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:52:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:52:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:52:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:52:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:52:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:52:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:52:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:52:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:52:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:52:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:52:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:52:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:52:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:52:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:52:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:52:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:52:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:52:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:52:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:52:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:52:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:52:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:52:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:52:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:52:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:52:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:52:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:52:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:52:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:52:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:52:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:52:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:52:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:52:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:52:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:52:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:52:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:52:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:52:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:52:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:52:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:52:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:52:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:52:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:52:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:52:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:52:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:52:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:52:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:52:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:52:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:52:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:52:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:52:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:52:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:52:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:52:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:52:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:52:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:52:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:52:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:52:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:52:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:52:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:52:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:52:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:52:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:52:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:52:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:52:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:52:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:52:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:52:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:52:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:52:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:52:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:52:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:52:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:52:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:52:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:53:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:53:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:53:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:53:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:53:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:53:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:53:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:53:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:53:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:53:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:53:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:53:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:53:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:53:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:53:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:53:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:53:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:53:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:53:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:53:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:53:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:53:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:53:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:53:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:53:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:53:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:53:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:53:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:53:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:53:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:53:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:53:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:53:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:53:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:53:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:53:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:53:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:53:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:53:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:53:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:53:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:53:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:53:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:53:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:53:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:53:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:53:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:53:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:53:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:53:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:53:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:53:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:53:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:53:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:53:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:53:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:53:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:53:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:53:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:53:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:53:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:53:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:53:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:53:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:53:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:53:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:53:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:53:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:53:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:53:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:53:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:53:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:53:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:53:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:53:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:53:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:53:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:53:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:53:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:53:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:53:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:53:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:53:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:53:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:53:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:53:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:53:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:53:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:53:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:53:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:53:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:53:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:53:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:53:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:53:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:53:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:53:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:53:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:53:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:53:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:53:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:53:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:53:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:53:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:53:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:53:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:53:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:53:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:53:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:53:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:53:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:53:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:53:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:53:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:53:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:53:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:53:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:53:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:53:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:53:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:53:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:53:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:53:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:53:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:53:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:53:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:53:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:53:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:53:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:53:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:53:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:53:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:53:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:53:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:53:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:53:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:53:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:53:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:53:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:53:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:53:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:53:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:53:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:53:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:53:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:53:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:53:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:53:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:53:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:53:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:53:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:53:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:53:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:53:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:53:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:53:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:53:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:53:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:53:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:53:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:53:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:53:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:53:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:53:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:53:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:53:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:53:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:53:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:53:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:53:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:53:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:53:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:53:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:53:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:53:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:53:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:53:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:53:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:53:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:53:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:53:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:53:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:53:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:53:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:53:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:53:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:53:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:53:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:53:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:53:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:53:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:53:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:53:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:53:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:53:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:53:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:53:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:53:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:53:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:53:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:53:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:53:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:53:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:53:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:53:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:53:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:53:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:53:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:53:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:53:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:53:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:53:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:53:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:53:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:53:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:53:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:54:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:54:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:54:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:54:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:54:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:54:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:54:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:54:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:54:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:54:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:54:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:54:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:54:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:54:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:54:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:54:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:54:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:54:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:54:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:54:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:54:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:54:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:54:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:54:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:54:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:54:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:54:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:54:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:54:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:54:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:54:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:54:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:54:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:54:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:54:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:54:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:54:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:54:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:54:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:54:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:54:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:54:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:54:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:54:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:54:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:54:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:54:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:54:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:54:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:54:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:54:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:54:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:54:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:54:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:54:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:54:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:54:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:54:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:54:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:54:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:54:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:54:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:54:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:54:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:54:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:54:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:54:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:54:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:54:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:54:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:54:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:54:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:54:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:54:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:54:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:54:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:54:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:54:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:54:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:54:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:54:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:54:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:54:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:54:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:54:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:54:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:54:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:54:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:54:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:54:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:54:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:54:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:54:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:54:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:54:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:54:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:54:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:54:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:54:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:54:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:54:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:54:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:54:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:54:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:54:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:54:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:54:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:54:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:54:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:54:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:54:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:54:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:54:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:54:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:54:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:54:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:54:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:54:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:54:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:54:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:54:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:54:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:54:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:54:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:54:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:54:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:54:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:54:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:54:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:54:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:54:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:54:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:54:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:54:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:54:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:54:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:54:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:54:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:54:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:54:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:54:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:54:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:54:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:54:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:54:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:54:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:54:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:54:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:54:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:54:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:54:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:54:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:54:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:54:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:54:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:54:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:54:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:54:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:54:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:54:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:54:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:54:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:54:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:54:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:54:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:54:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:54:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:54:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:54:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:54:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:54:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:54:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:54:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:54:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:54:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:54:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:54:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:54:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:54:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:54:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:54:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:54:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:54:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:54:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:54:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:54:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:54:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:54:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:54:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:54:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:54:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:54:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:54:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:54:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:54:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:54:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:54:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:54:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:54:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:54:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:54:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:54:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:54:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:54:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:54:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:54:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:54:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:54:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:54:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:54:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:54:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:54:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:54:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:54:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:54:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:54:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:55:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:55:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:55:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:55:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:55:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:55:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:55:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:55:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:55:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:55:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:55:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:55:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:55:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:55:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:55:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:55:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:55:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:55:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:55:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:55:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:55:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:55:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:55:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:55:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:55:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:55:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:55:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:55:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:55:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:55:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:55:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:55:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:55:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:55:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:55:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:55:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:55:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:55:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:55:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:55:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:55:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:55:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:55:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:55:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:55:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:55:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:55:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:55:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:55:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:55:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:55:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:55:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:55:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:55:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:55:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:55:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:55:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:55:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:55:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:55:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:55:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:55:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:55:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:55:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:55:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:55:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:55:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:55:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:55:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:55:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:55:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:55:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:55:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:55:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:55:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:55:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:55:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:55:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:55:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:55:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:55:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:55:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:55:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:55:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:55:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:55:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:55:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:55:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:55:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:55:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:55:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:55:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:55:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:55:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:55:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:55:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:55:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:55:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:55:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:55:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:55:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:55:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:55:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:55:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:55:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:55:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:55:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:55:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:55:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:55:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:55:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:55:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:55:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:55:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:55:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:55:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:55:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:55:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:55:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:55:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:55:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:55:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:55:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:55:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:55:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:55:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:55:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:55:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:55:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:55:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:55:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:55:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:55:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:55:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:55:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:55:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:55:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:55:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:55:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:55:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:55:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:55:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:55:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:55:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:55:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:55:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:55:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:55:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:55:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:55:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:55:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:55:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:55:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:55:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:55:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:55:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:55:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:55:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:55:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:55:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:55:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:55:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:55:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:55:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:55:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:55:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:55:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:55:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:55:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:55:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:55:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:55:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:55:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:55:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:55:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:55:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:55:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:55:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:55:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:55:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:55:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:55:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:55:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:55:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:55:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:55:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:55:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:55:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:55:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:55:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:55:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:55:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:55:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:55:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:55:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:55:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:55:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:55:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:55:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:55:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:55:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:55:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:55:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:55:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:55:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:55:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:55:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:55:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:55:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:55:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:55:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:55:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:55:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:55:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:55:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:55:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:56:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:56:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:56:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:56:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:56:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:56:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:56:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:56:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:56:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:56:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:56:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:56:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:56:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:56:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:56:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:56:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:56:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:56:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:56:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:56:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:56:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:56:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:56:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:56:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:56:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:56:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:56:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:56:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:56:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:56:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:56:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:56:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:56:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:56:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:56:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:56:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:56:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:56:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:56:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:56:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:56:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:56:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:56:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:56:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:56:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:56:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:56:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:56:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:56:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:56:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:56:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:56:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:56:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:56:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:56:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:56:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:56:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:56:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:56:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:56:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:56:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:56:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:56:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:56:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:56:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:56:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:56:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:56:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:56:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:56:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:56:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:56:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:56:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:56:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:56:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:56:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:56:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:56:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:56:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:56:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:56:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:56:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:56:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:56:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:56:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:56:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:56:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:56:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:56:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:56:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:56:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:56:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:56:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:56:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:56:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:56:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:56:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:56:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:56:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:56:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:56:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:56:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:56:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:56:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:56:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:56:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:56:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:56:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:56:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:56:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:56:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:56:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:56:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:56:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:56:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:56:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:56:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:56:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:56:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:56:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:56:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:56:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:56:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:56:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:56:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:56:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:56:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:56:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:56:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:56:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:56:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:56:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:56:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:56:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:56:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:56:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:56:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:56:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:56:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:56:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:56:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:56:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:56:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:56:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:56:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:56:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:56:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:56:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:56:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:56:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:56:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:56:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:56:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:56:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:56:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:56:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:56:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:56:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:56:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:56:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:56:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:56:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:56:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:56:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:56:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:56:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:56:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:56:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:56:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:56:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:56:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:56:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:56:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:56:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:56:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:56:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:56:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:56:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:56:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:56:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:56:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:56:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:56:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:56:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:56:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:56:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:56:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:56:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:56:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:56:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:56:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:56:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:56:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:56:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:56:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:56:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:56:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:56:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:56:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:56:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:56:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:56:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:56:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:56:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:56:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:56:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:56:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:56:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:56:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:56:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:56:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:56:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:56:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:56:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:56:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:56:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:57:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:57:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:57:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:57:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:57:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:57:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:57:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:57:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:57:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:57:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:57:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:57:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:57:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:57:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:57:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:57:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:57:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:57:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:57:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:57:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:57:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:57:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:57:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:57:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:57:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:57:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:57:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:57:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:57:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:57:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:57:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:57:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:57:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:57:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:57:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:57:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:57:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:57:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:57:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:57:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:57:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:57:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:57:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:57:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:57:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:57:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:57:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:57:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:57:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:57:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:57:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:57:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:57:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:57:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:57:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:57:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:57:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:57:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:57:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:57:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:57:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:57:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:57:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:57:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:57:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:57:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:57:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:57:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:57:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:57:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:57:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:57:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:57:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:57:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:57:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:57:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:57:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:57:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:57:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:57:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:57:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:57:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:57:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:57:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:57:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:57:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:57:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:57:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:57:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:57:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:57:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:57:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:57:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:57:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:57:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:57:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:57:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:57:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:57:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:57:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:57:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:57:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:57:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:57:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:57:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:57:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:57:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:57:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:57:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:57:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:57:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:57:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:57:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:57:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:57:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:57:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:57:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:57:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:57:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:57:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:57:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:57:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:57:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:57:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:57:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:57:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:57:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:57:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:57:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:57:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:57:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:57:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:57:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:57:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:57:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:57:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:57:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:57:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:57:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:57:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:57:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:57:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:57:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:57:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:57:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:57:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:57:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:57:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:57:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:57:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:57:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:57:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:57:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:57:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:57:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:57:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:57:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:57:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:57:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:57:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:57:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:57:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:57:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:57:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:57:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:57:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:57:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:57:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:57:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:57:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:57:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:57:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:57:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:57:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:57:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:57:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:57:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:57:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:57:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:57:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:57:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:57:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:57:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:57:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:57:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:57:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:57:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:57:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:57:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:57:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:57:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:57:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:57:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:57:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:57:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:57:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:57:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:57:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:57:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:57:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:57:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:57:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:57:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:57:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:57:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:57:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:57:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:57:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:57:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:57:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:57:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:57:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:57:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:57:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:57:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:57:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:58:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:58:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:58:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:58:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:58:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:58:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:58:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:58:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:58:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:58:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:58:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:58:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:58:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:58:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:58:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:58:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:58:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:58:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:58:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:58:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:58:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:58:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:58:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:58:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:58:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:58:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:58:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:58:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:58:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:58:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:58:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:58:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:58:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:58:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:58:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:58:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:58:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:58:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:58:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:58:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:58:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:58:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:58:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:58:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:58:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:58:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:58:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:58:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:58:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:58:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:58:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:58:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:58:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:58:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:58:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:58:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:58:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:58:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:58:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:58:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:58:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:58:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:58:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:58:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:58:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:58:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:58:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:58:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:58:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:58:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:58:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:58:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:58:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:58:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:58:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:58:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:58:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:58:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:58:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:58:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:58:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:58:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:58:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:58:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:58:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:58:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:58:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:58:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:58:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:58:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:58:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:58:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:58:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:58:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:58:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:58:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:58:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:58:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:58:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:58:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:58:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:58:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:58:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:58:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:58:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:58:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:58:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:58:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:58:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:58:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:58:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:58:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:58:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:58:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:58:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:58:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:58:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:58:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:58:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:58:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:58:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:58:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:58:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:58:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:58:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:58:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:58:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:58:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:58:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:58:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:58:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:58:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:58:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:58:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:58:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:58:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:58:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:58:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:58:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:58:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:58:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:58:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:58:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:58:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:58:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:58:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:58:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:58:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:58:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:58:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:58:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:58:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:58:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:58:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:58:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:58:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:58:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:58:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:58:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:58:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:58:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:58:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:58:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:58:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:58:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:58:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:58:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:58:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:58:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:58:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:58:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:58:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:58:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:58:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:58:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:58:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:58:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:58:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:58:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:58:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:58:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:58:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:58:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:58:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:58:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:58:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:58:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:58:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:58:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:58:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:58:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:58:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:58:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:58:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:58:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:58:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:58:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:58:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:58:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:58:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:58:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:58:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:58:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:58:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:58:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:58:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:58:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:58:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:58:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:58:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:58:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:58:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:58:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:58:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:58:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:58:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:59:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:59:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:59:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:59:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:59:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:59:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:59:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:59:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:59:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:59:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:59:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:59:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:59:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:59:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:59:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:59:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:59:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:59:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:59:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:59:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:59:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:59:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:59:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:59:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:59:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:59:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:59:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:59:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:59:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:59:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:59:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:59:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:59:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:59:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:59:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:59:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 00:59:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 00:59:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:59:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:59:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:59:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 00:59:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 00:59:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 00:59:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 00:59:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 00:59:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 00:59:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 00:59:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 00:59:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 00:59:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 00:59:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 00:59:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 00:59:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 00:59:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:01:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:01:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:01:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:01:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:01:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:01:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:01:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:01:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:01:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:01:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:01:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:01:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:01:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:01:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:01:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:01:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:01:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:01:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:01:16 rdma06 rsyslogd[1041]: imjournal: 5088 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 01:01:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:01:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:01:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:01:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:01:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:01:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:01:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:01:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:01:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:01:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:01:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:01:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:01:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:01:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:01:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:01:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:01:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:01:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:01:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:01:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:01:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:01:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:01:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:01:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:01:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:01:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:01:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:01:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:01:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:01:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:01:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:01:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:01:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:01:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:01:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:01:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:01:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:01:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:01:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:01:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:01:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:01:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:01:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:01:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:01:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:01:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:01:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:01:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:01:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:01:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:01:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:01:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:01:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:01:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:01:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:01:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:01:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:01:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:01:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:01:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:01:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:01:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:01:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:01:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:01:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:01:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:01:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:01:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:01:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:01:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:01:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:01:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:01:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:01:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:01:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:01:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:01:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:01:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:01:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:01:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:01:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:01:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:01:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:01:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:02:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:02:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:02:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:02:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:02:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:02:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:02:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:02:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:02:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:02:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:02:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:02:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:02:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:02:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:02:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:02:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:02:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:02:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:02:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:02:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:02:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:02:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:02:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:02:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:02:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:02:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:02:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:02:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:02:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:02:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:02:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:02:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:02:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:02:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:02:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:02:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:02:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:02:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:02:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:02:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:02:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:02:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:02:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:02:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:02:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:02:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:02:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:02:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:02:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:02:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:02:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:02:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:02:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:02:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:02:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:02:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:02:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:02:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:02:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:02:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:02:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:02:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:02:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:02:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:02:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:02:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:02:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:02:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:02:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:02:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:02:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:02:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:02:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:02:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:02:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:02:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:02:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:02:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:02:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:02:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:02:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:02:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:02:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:02:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:02:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:02:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:02:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:02:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:02:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:02:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:02:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:02:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:02:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:02:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:02:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:02:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:02:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:02:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:02:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:02:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:02:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:02:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:02:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:02:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:02:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:02:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:02:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:02:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:02:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:02:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:02:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:02:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:02:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:02:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:02:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:02:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:02:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:02:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:02:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:02:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:02:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:02:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:02:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:02:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:02:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:02:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:02:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:02:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:02:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:02:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:02:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:02:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:02:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:02:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:02:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:02:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:02:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:02:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:02:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:02:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:02:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:02:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:02:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:02:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:02:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:02:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:02:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:02:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:02:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:02:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:02:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:02:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:02:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:02:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:02:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:02:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:02:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:02:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:02:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:02:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:02:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:02:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:02:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:02:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:02:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:02:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:02:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:02:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:02:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:02:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:02:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:02:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:02:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:02:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:02:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:02:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:02:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:02:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:02:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:02:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:02:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:02:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:02:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:02:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:02:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:02:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:02:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:02:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:02:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:02:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:02:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:02:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:02:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:02:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:02:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:02:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:02:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:02:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:02:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:02:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:02:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:02:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:02:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:02:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:02:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:02:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:02:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:02:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:02:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:02:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:02:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:02:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:02:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:02:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:02:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:02:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:03:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:03:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:03:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:03:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:03:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:03:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:03:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:03:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:03:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:03:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:03:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:03:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:03:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:03:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:03:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:03:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:03:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:03:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:03:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:03:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:03:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:03:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:03:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:03:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:03:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:03:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:03:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:03:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:03:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:03:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:03:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:03:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:03:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:03:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:03:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:03:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:03:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:03:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:03:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:03:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:03:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:03:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:03:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:03:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:03:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:03:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:03:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:03:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:03:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:03:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:03:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:03:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:03:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:03:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:03:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:03:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:03:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:03:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:03:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:03:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:03:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:03:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:03:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:03:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:03:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:03:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:03:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:03:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:03:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:03:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:03:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:03:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:03:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:03:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:03:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:03:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:03:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:03:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:03:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:03:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:03:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:03:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:03:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:03:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:03:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:03:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:03:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:03:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:03:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:03:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:03:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:03:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:03:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:03:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:03:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:03:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:03:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:03:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:03:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:03:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:03:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:03:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:03:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:03:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:03:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:03:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:03:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:03:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:03:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:03:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:03:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:03:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:03:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:03:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:03:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:03:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:03:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:03:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:03:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:03:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:03:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:03:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:03:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:03:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:03:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:03:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:03:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:03:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:03:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:03:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:03:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:03:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:03:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:03:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:03:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:03:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:03:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:03:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:03:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:03:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:03:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:03:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:03:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:03:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:03:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:03:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:03:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:03:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:03:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:03:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:03:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:03:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:03:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:03:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:03:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:03:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:03:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:03:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:03:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:03:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:03:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:03:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:03:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:03:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:03:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:03:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:03:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:03:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:03:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:03:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:03:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:03:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:03:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:03:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:03:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:03:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:03:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:03:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:03:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:03:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:03:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:03:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:03:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:03:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:03:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:03:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:03:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:03:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:03:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:03:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:03:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:03:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:03:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:03:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:03:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:03:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:03:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:03:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:03:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:03:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:03:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:03:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:03:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:03:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:03:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:03:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:03:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:03:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:03:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:03:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:03:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:03:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:03:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:03:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:03:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:03:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:04:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:04:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:04:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:04:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:04:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:04:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:04:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:04:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:04:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:04:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:04:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:04:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:04:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:04:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:04:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:04:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:04:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:04:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:04:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:04:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:04:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:04:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:04:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:04:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:04:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:04:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:04:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:04:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:04:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:04:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:04:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:04:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:04:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:04:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:04:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:04:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:04:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:04:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:04:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:04:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:04:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:04:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:04:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:04:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:04:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:04:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:04:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:04:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:04:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:04:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:04:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:04:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:04:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:04:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:04:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:04:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:04:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:04:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:04:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:04:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:04:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:04:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:04:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:04:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:04:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:04:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:04:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:04:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:04:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:04:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:04:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:04:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:04:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:04:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:04:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:04:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:04:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:04:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:04:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:04:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:04:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:04:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:04:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:04:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:04:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:04:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:04:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:04:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:04:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:04:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:04:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:04:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:04:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:04:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:04:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:04:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:04:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:04:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:04:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:04:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:04:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:04:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:04:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:04:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:04:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:04:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:04:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:04:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:04:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:04:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:04:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:04:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:04:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:04:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:04:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:04:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:04:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:04:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:04:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:04:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:04:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:04:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:04:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:04:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:04:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:04:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:04:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:04:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:04:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:04:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:04:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:04:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:04:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:04:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:04:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:04:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:04:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:04:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:04:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:04:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:04:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:04:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:04:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:04:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:04:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:04:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:04:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:04:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:04:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:04:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:04:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:04:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:04:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:04:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:04:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:04:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:04:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:04:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:04:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:04:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:04:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:04:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:04:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:04:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:04:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:04:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:04:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:04:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:04:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:04:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:04:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:04:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:04:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:04:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:04:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:04:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:04:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:04:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:04:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:04:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:04:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:04:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:04:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:04:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:04:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:04:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:04:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:04:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:04:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:04:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:04:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:04:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:04:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:04:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:04:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:04:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:04:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:04:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:04:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:04:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:04:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:04:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:04:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:04:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:04:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:04:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:04:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:04:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:04:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:04:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:04:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:04:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:04:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:04:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:04:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:04:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:05:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:05:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:05:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:05:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:05:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:05:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:05:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:05:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:05:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:05:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:05:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:05:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:05:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:05:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:05:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:05:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:05:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:05:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:05:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:05:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:05:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:05:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:05:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:05:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:05:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:05:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:05:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:05:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:05:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:05:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:05:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:05:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:05:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:05:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:05:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:05:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:05:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:05:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:05:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:05:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:05:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:05:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:05:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:05:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:05:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:05:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:05:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:05:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:05:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:05:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:05:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:05:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:05:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:05:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:05:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:05:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:05:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:05:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:05:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:05:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:05:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:05:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:05:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:05:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:05:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:05:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:05:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:05:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:05:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:05:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:05:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:05:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:05:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:05:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:05:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:05:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:05:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:05:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:05:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:05:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:05:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:05:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:05:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:05:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:05:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:05:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:05:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:05:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:05:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:05:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:05:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:05:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:05:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:05:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:05:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:05:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:05:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:05:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:05:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:05:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:05:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:05:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:05:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:05:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:05:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:05:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:05:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:05:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:05:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:05:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:05:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:05:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:05:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:05:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:05:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:05:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:05:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:05:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:05:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:05:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:05:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:05:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:05:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:05:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:05:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:05:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:05:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:05:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:05:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:05:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:05:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:05:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:05:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:05:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:05:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:05:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:05:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:05:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:05:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:05:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:05:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:05:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:05:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:05:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:05:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:05:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:05:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:05:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:05:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:05:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:05:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:05:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:05:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:05:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:05:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:05:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:05:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:05:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:05:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:05:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:05:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:05:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:05:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:05:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:05:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:05:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:05:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:05:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:05:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:05:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:05:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:05:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:05:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:05:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:05:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:05:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:05:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:05:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:05:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:05:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:05:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:05:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:05:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:05:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:05:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:05:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:05:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:05:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:05:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:05:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:05:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:05:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:05:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:05:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:05:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:05:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:05:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:05:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:05:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:05:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:05:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:05:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:05:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:05:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:05:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:05:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:05:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:05:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:05:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:05:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:05:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:05:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:05:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:05:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:05:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:05:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:06:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:06:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:06:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:06:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:06:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:06:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:06:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:06:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:06:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:06:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:06:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:06:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:06:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:06:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:06:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:06:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:06:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:06:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:06:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:06:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:06:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:06:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:06:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:06:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:06:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:06:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:06:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:06:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:06:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:06:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:06:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:06:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:06:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:06:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:06:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:06:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:06:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:06:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:06:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:06:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:06:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:06:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:06:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:06:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:06:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:06:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:06:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:06:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:06:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:06:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:06:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:06:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:06:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:06:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:06:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:06:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:06:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:06:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:06:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:06:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:06:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:06:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:06:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:06:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:06:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:06:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:06:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:06:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:06:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:06:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:06:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:06:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:06:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:06:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:06:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:06:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:06:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:06:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:06:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:06:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:06:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:06:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:06:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:06:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:06:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:06:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:06:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:06:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:06:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:06:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:06:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:06:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:06:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:06:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:06:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:06:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:06:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:06:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:06:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:06:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:06:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:06:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:06:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:06:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:06:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:06:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:06:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:06:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:06:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:06:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:06:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:06:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:06:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:06:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:06:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:06:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:06:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:06:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:06:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:06:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:06:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:06:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:06:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:06:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:06:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:06:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:06:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:06:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:06:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:06:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:06:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:06:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:06:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:06:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:06:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:06:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:06:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:06:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:06:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:06:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:06:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:06:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:06:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:06:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:06:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:06:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:06:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:06:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:06:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:06:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:06:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:06:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:06:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:06:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:06:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:06:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:06:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:06:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:06:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:06:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:06:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:06:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:06:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:06:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:06:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:06:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:06:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:06:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:06:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:06:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:06:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:06:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:06:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:06:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:06:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:06:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:06:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:06:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:06:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:06:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:06:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:06:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:06:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:06:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:06:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:06:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:06:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:06:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:06:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:06:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:06:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:06:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:06:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:06:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:06:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:06:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:06:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:06:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:06:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:06:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:06:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:06:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:06:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:06:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:06:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:06:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:06:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:06:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:06:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:06:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:06:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:06:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:06:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:06:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:06:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:06:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:07:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:07:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:07:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:07:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:07:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:07:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:07:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:07:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:07:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:07:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:07:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:07:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:07:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:07:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:07:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:07:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:07:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:07:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:07:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:07:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:07:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:07:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:07:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:07:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:07:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:07:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:07:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:07:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:07:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:07:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:07:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:07:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:07:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:07:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:07:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:07:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:07:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:07:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:07:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:07:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:07:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:07:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:07:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:07:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:07:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:07:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:07:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:07:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:07:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:07:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:07:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:07:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:07:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:07:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:07:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:07:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:07:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:07:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:07:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:07:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:07:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:07:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:07:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:07:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:07:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:07:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:07:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:07:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:07:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:07:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:07:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:07:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:07:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:07:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:07:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:07:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:07:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:07:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:07:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:07:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:07:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:07:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:07:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:07:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:07:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:07:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:07:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:07:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:07:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:07:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:07:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:07:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:07:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:07:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:07:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:07:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:07:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:07:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:07:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:07:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:07:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:07:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:07:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:07:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:07:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:07:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:07:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:07:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:07:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:07:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:07:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:07:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:07:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:07:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:07:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:07:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:07:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:07:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:07:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:07:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:07:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:07:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:07:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:07:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:07:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:07:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:07:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:07:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:07:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:07:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:07:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:07:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:07:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:07:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:07:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:07:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:07:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:07:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:07:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:07:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:07:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:07:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:07:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:07:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:07:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:07:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:07:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:07:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:07:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:07:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:07:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:07:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:07:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:07:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:07:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:07:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:07:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:07:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:07:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:07:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:07:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:07:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:07:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:07:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:07:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:07:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:07:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:07:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:07:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:07:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:07:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:07:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:07:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:07:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:07:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:07:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:07:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:07:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:07:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:07:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:07:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:07:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:07:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:07:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:07:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:07:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:07:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:07:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:07:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:07:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:07:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:07:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:07:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:07:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:07:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:07:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:07:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:07:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:07:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:07:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:07:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:07:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:07:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:07:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:07:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:07:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:07:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:07:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:07:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:07:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:07:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:07:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:07:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:07:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:07:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:07:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:08:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:08:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:08:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:08:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:08:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:08:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:08:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:08:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:08:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:08:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:08:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:08:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:08:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:08:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:08:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:08:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:08:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:08:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:08:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:08:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:08:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:08:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:08:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:08:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:08:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:08:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:08:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:08:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:08:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:08:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:08:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:08:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:08:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:08:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:08:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:08:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:08:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:08:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:08:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:08:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:08:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:08:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:08:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:08:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:08:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:08:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:08:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:08:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:08:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:08:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:08:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:08:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:08:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:08:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:08:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:08:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:08:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:08:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:08:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:08:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:08:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:08:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:08:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:08:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:08:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:08:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:08:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:08:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:08:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:08:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:08:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:08:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:08:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:08:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:08:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:08:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:08:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:08:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:08:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:08:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:08:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:08:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:08:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:08:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:08:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:08:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:08:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:08:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:08:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:08:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:08:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:08:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:08:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:08:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:08:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:08:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:08:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:08:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:08:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:08:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:08:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:08:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:08:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:08:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:08:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:08:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:08:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:08:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:08:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:08:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:08:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:08:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:08:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:08:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:08:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:08:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:08:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:08:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:08:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:08:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:08:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:08:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:08:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:08:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:08:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:08:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:08:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:08:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:08:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:08:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:08:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:08:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:08:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:08:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:08:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:08:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:08:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:08:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:08:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:08:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:08:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:08:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:08:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:08:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:08:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:08:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:08:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:08:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:08:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:08:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:08:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:08:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:08:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:08:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:08:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:08:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:08:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:08:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:08:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:08:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:08:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:08:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:08:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:08:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:08:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:08:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:08:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:08:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:08:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:08:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:08:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:08:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:08:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:08:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:08:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:08:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:08:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:08:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:08:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:08:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:08:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:08:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:08:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:08:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:08:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:08:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:08:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:08:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:08:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:08:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:08:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:08:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:08:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:08:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:08:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:08:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:08:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:08:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:08:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:08:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:08:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:08:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:08:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:08:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:08:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:08:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:08:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:08:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:08:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:08:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:08:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:08:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:08:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:08:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:08:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:08:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:09:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:09:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:09:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:09:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:09:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:09:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:09:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:09:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:09:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:09:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:09:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:09:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:09:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:09:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:09:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:09:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:09:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:09:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:09:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:09:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:09:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:09:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:09:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:09:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:09:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:09:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:09:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:09:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:09:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:09:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:09:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:09:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:09:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:09:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:09:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:09:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:09:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:09:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:09:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:09:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:09:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:09:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:09:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:09:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:09:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:09:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:09:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:09:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:09:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:09:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:09:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:09:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:09:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:09:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:11:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:11:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:11:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:11:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:11:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:11:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:11:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:11:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:11:21 rdma06 rsyslogd[1041]: imjournal: 5298 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 01:11:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:11:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:11:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:11:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:11:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:11:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:11:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:11:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:11:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:11:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:11:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:11:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:11:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:11:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:11:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:11:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:11:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:11:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:11:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:11:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:11:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:11:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:11:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:11:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:11:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:11:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:11:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:11:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:11:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:11:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:11:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:11:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:11:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:11:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:11:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:11:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:11:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:11:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:11:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:11:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:11:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:11:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:11:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:11:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:11:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:11:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:11:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:11:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:11:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:11:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:11:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:11:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:11:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:11:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:11:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:11:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:11:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:11:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:11:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:11:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:11:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:11:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:11:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:11:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:11:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:11:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:11:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:11:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:11:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:11:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:11:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:11:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:11:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:11:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:11:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:11:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:11:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:11:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:11:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:11:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:11:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:11:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:11:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:11:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:11:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:11:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:11:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:11:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:11:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:11:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:11:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:11:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:11:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:11:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:11:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:11:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:11:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:11:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:11:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:11:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:11:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:11:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:11:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:11:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:11:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:11:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:11:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:11:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:11:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:11:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:11:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:11:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:11:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:11:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:11:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:11:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:11:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:11:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:12:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:12:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:12:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:12:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:12:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:12:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:12:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:12:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:12:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:12:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:12:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:12:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:12:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:12:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:12:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:12:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:12:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:12:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:12:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:12:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:12:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:12:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:12:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:12:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:12:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:12:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:12:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:12:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:12:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:12:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:12:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:12:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:12:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:12:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:12:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:12:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:12:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:12:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:12:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:12:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:12:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:12:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:12:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:12:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:12:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:12:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:12:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:12:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:12:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:12:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:12:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:12:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:12:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:12:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:12:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:12:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:12:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:12:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:12:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:12:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:12:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:12:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:12:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:12:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:12:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:12:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:12:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:12:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:12:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:12:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:12:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:12:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:12:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:12:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:12:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:12:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:12:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:12:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:12:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:12:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:12:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:12:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:12:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:12:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:12:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:12:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:12:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:12:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:12:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:12:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:12:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:12:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:12:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:12:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:12:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:12:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:12:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:12:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:12:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:12:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:12:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:12:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:12:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:12:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:12:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:12:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:12:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:12:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:12:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:12:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:12:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:12:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:12:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:12:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:12:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:12:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:12:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:12:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:12:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:12:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:12:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:12:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:12:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:12:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:12:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:12:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:12:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:12:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:12:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:12:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:12:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:12:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:12:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:12:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:12:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:12:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:12:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:12:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:12:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:12:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:12:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:12:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:12:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:12:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:12:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:12:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:12:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:12:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:12:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:12:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:12:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:12:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:12:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:12:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:12:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:12:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:12:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:12:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:12:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:12:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:12:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:12:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:12:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:12:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:12:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:12:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:12:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:12:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:12:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:12:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:12:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:12:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:12:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:12:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:12:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:12:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:12:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:12:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:12:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:12:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:12:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:12:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:12:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:12:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:12:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:12:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:12:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:12:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:12:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:12:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:12:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:12:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:12:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:12:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:12:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:12:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:12:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:12:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:12:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:12:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:12:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:12:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:12:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:12:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:12:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:12:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:12:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:12:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:12:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:12:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:12:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:12:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:12:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:12:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:12:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:12:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:13:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:13:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:13:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:13:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:13:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:13:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:13:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:13:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:13:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:13:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:13:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:13:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:13:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:13:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:13:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:13:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:13:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:13:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:13:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:13:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:13:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:13:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:13:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:13:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:13:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:13:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:13:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:13:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:13:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:13:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:13:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:13:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:13:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:13:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:13:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:13:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:13:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:13:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:13:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:13:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:13:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:13:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:13:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:13:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:13:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:13:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:13:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:13:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:13:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:13:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:13:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:13:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:13:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:13:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:13:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:13:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:13:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:13:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:13:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:13:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:13:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:13:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:13:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:13:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:13:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:13:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:13:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:13:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:13:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:13:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:13:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:13:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:13:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:13:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:13:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:13:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:13:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:13:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:13:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:13:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:13:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:13:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:13:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:13:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:13:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:13:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:13:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:13:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:13:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:13:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:13:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:13:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:13:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:13:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:13:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:13:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:13:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:13:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:13:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:13:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:13:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:13:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:13:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:13:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:13:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:13:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:13:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:13:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:13:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:13:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:13:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:13:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:13:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:13:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:13:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:13:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:13:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:13:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:13:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:13:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:13:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:13:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:13:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:13:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:13:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:13:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:13:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:13:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:13:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:13:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:13:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:13:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:13:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:13:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:13:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:13:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:13:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:13:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:13:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:13:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:13:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:13:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:13:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:13:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:13:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:13:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:13:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:13:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:13:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:13:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:13:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:13:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:13:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:13:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:13:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:13:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:13:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:13:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:13:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:13:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:13:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:13:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:13:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:13:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:13:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:13:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:13:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:13:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:13:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:13:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:13:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:13:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:13:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:13:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:13:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:13:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:13:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:13:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:13:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:13:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:13:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:13:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:13:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:13:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:13:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:13:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:13:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:13:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:13:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:13:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:13:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:13:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:13:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:13:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:13:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:13:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:13:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:13:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:13:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:13:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:13:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:13:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:13:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:13:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:13:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:13:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:13:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:13:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:13:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:13:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:13:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:13:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:13:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:13:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:13:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:13:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:14:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:14:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:14:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:14:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:14:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:14:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:14:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:14:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:14:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:14:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:14:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:14:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:14:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:14:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:14:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:14:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:14:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:14:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:14:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:14:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:14:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:14:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:14:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:14:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:14:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:14:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:14:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:14:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:14:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:14:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:14:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:14:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:14:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:14:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:14:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:14:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:14:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:14:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:14:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:14:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:14:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:14:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:14:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:14:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:14:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:14:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:14:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:14:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:14:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:14:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:14:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:14:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:14:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:14:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:14:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:14:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:14:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:14:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:14:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:14:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:14:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:14:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:14:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:14:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:14:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:14:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:14:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:14:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:14:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:14:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:14:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:14:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:14:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:14:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:14:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:14:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:14:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:14:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:14:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:14:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:14:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:14:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:14:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:14:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:14:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:14:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:14:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:14:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:14:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:14:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:14:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:14:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:14:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:14:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:14:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:14:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:14:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:14:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:14:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:14:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:14:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:14:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:14:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:14:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:14:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:14:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:14:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:14:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:14:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:14:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:14:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:14:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:14:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:14:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:14:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:14:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:14:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:14:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:14:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:14:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:14:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:14:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:14:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:14:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:14:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:14:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:14:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:14:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:14:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:14:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:14:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:14:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:14:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:14:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:14:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:14:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:14:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:14:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:14:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:14:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:14:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:14:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:14:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:14:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:14:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:14:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:14:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:14:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:14:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:14:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:14:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:14:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:14:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:14:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:14:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:14:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:14:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:14:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:14:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:14:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:14:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:14:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:14:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:14:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:14:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:14:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:14:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:14:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:14:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:14:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:14:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:14:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:14:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:14:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:14:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:14:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:14:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:14:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:14:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:14:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:14:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:14:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:14:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:14:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:14:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:14:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:14:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:14:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:14:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:14:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:14:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:14:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:14:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:14:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:14:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:14:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:14:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:14:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:14:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:14:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:14:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:14:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:14:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:14:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:14:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:14:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:14:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:14:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:14:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:14:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:14:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:14:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:14:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:14:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:14:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:14:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:15:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:15:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:15:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:15:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:15:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:15:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:15:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:15:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:15:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:15:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:15:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:15:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:15:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:15:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:15:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:15:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:15:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:15:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:15:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:15:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:15:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:15:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:15:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:15:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:15:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:15:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:15:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:15:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:15:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:15:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:15:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:15:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:15:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:15:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:15:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:15:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:15:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:15:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:15:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:15:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:15:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:15:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:15:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:15:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:15:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:15:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:15:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:15:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:15:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:15:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:15:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:15:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:15:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:15:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:15:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:15:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:15:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:15:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:15:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:15:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:15:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:15:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:15:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:15:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:15:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:15:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:15:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:15:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:15:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:15:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:15:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:15:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:15:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:15:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:15:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:15:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:15:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:15:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:15:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:15:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:15:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:15:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:15:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:15:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:15:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:15:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:15:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:15:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:15:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:15:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:15:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:15:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:15:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:15:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:15:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:15:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:15:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:15:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:15:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:15:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:15:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:15:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:15:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:15:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:15:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:15:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:15:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:15:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:15:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:15:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:15:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:15:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:15:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:15:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:15:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:15:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:15:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:15:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:15:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:15:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:15:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:15:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:15:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:15:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:15:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:15:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:15:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:15:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:15:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:15:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:15:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:15:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:15:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:15:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:15:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:15:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:15:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:15:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:15:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:15:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:15:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:15:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:15:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:15:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:15:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:15:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:15:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:15:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:15:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:15:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:15:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:15:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:15:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:15:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:15:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:15:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:15:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:15:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:15:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:15:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:15:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:15:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:15:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:15:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:15:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:15:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:15:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:15:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:15:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:15:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:15:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:15:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:15:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:15:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:15:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:15:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:15:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:15:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:15:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:15:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:15:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:15:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:15:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:15:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:15:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:15:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:15:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:15:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:15:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:15:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:15:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:15:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:15:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:15:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:15:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:15:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:15:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:15:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:15:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:15:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:15:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:15:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:15:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:15:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:15:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:15:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:15:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:15:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:15:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:15:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:15:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:15:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:15:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:15:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:15:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:15:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:16:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:16:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:16:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:16:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:16:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:16:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:16:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:16:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:16:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:16:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:16:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:16:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:16:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:16:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:16:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:16:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:16:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:16:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:16:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:16:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:16:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:16:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:16:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:16:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:16:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:16:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:16:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:16:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:16:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:16:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:16:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:16:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:16:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:16:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:16:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:16:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:16:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:16:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:16:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:16:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:16:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:16:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:16:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:16:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:16:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:16:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:16:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:16:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:16:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:16:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:16:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:16:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:16:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:16:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:16:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:16:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:16:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:16:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:16:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:16:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:16:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:16:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:16:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:16:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:16:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:16:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:16:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:16:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:16:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:16:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:16:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:16:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:16:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:16:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:16:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:16:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:16:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:16:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:16:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:16:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:16:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:16:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:16:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:16:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:16:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:16:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:16:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:16:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:16:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:16:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:16:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:16:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:16:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:16:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:16:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:16:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:16:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:16:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:16:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:16:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:16:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:16:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:16:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:16:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:16:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:16:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:16:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:16:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:16:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:16:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:16:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:16:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:16:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:16:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:16:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:16:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:16:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:16:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:16:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:16:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:16:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:16:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:16:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:16:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:16:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:16:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:16:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:16:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:16:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:16:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:16:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:16:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:16:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:16:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:16:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:16:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:16:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:16:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:16:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:16:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:16:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:16:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:16:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:16:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:16:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:16:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:16:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:16:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:16:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:16:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:16:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:16:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:16:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:16:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:16:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:16:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:16:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:16:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:16:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:16:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:16:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:16:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:16:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:16:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:16:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:16:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:16:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:16:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:16:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:16:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:16:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:16:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:16:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:16:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:16:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:16:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:16:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:16:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:16:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:16:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:16:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:16:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:16:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:16:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:16:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:16:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:16:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:16:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:16:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:16:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:16:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:16:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:16:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:16:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:16:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:16:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:16:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:16:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:16:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:16:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:16:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:16:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:16:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:16:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:16:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:16:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:16:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:16:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:16:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:16:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:16:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:16:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:16:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:16:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:16:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:16:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:17:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:17:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:17:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:17:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:17:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:17:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:17:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:17:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:17:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:17:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:17:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:17:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:17:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:17:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:17:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:17:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:17:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:17:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:17:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:17:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:17:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:17:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:17:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:17:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:17:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:17:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:17:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:17:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:17:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:17:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:17:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:17:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:17:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:17:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:17:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:17:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:17:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:17:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:17:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:17:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:17:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:17:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:17:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:17:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:17:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:17:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:17:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:17:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:17:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:17:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:17:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:17:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:17:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:17:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:17:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:17:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:17:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:17:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:17:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:17:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:17:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:17:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:17:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:17:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:17:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:17:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:17:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:17:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:17:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:17:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:17:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:17:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:17:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:17:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:17:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:17:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:17:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:17:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:17:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:17:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:17:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:17:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:17:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:17:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:17:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:17:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:17:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:17:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:17:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:17:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:17:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:17:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:17:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:17:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:17:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:17:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:17:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:17:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:17:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:17:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:17:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:17:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:17:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:17:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:17:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:17:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:17:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:17:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:17:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:17:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:17:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:17:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:17:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:17:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:17:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:17:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:17:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:17:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:17:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:17:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:17:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:17:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:17:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:17:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:17:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:17:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:17:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:17:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:17:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:17:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:17:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:17:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:17:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:17:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:17:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:17:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:17:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:17:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:17:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:17:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:17:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:17:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:17:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:17:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:17:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:17:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:17:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:17:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:17:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:17:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:17:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:17:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:17:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:17:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:17:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:17:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:17:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:17:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:17:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:17:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:17:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:17:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:17:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:17:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:17:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:17:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:17:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:17:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:17:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:17:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:17:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:17:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:17:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:17:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:17:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:17:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:17:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:17:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:17:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:17:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:17:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:17:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:17:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:17:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:17:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:17:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:17:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:17:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:17:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:17:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:17:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:17:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:17:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:17:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:17:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:17:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:17:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:17:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:17:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:17:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:17:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:17:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:17:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:17:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:17:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:17:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:17:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:17:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:17:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:17:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:17:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:17:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:17:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:17:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:17:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:17:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:18:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:18:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:18:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:18:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:18:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:18:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:18:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:18:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:18:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:18:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:18:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:18:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:18:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:18:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:18:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:18:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:18:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:18:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:18:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:18:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:18:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:18:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:18:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:18:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:18:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:18:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:18:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:18:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:18:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:18:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:18:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:18:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:18:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:18:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:18:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:18:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:18:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:18:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:18:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:18:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:18:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:18:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:18:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:18:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:18:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:18:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:18:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:18:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:18:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:18:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:18:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:18:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:18:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:18:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:18:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:18:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:18:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:18:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:18:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:18:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:18:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:18:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:18:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:18:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:18:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:18:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:18:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:18:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:18:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:18:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:18:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:18:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:18:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:18:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:18:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:18:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:18:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:18:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:18:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:18:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:18:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:18:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:18:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:18:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:18:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:18:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:18:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:18:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:18:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:18:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:18:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:18:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:18:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:18:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:18:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:18:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:18:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:18:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:18:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:18:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:18:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:18:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:18:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:18:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:18:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:18:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:18:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:18:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:18:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:18:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:18:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:18:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:18:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:18:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:18:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:18:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:18:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:18:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:18:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:18:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:18:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:18:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:18:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:18:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:18:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:18:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:18:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:18:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:18:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:18:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:18:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:18:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:18:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:18:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:18:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:18:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:18:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:18:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:18:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:18:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:18:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:18:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:18:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:18:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:18:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:18:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:18:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:18:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:18:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:18:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:18:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:18:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:18:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:18:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:18:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:18:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:18:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:18:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:18:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:18:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:18:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:18:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:18:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:18:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:18:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:18:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:18:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:18:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:18:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:18:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:18:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:18:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:18:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:18:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:18:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:18:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:18:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:18:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:18:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:18:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:18:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:18:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:18:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:18:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:18:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:18:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:18:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:18:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:18:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:18:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:18:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:18:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:18:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:18:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:18:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:18:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:18:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:18:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:18:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:18:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:18:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:18:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:18:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:18:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:18:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:18:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:18:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:18:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:18:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:18:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:18:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:18:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:18:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:18:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:18:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:18:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:19:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:19:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:19:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:19:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:19:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:19:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:19:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:19:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:19:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:19:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:19:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:19:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:19:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:19:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:19:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:19:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:19:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:19:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:19:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:19:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:19:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:19:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:19:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:19:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:19:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:19:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:19:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:19:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:19:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:19:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:19:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:19:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:19:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:19:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:19:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:19:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:19:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:19:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:19:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:19:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:19:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:19:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:19:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:19:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:19:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:19:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:19:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:19:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:19:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:19:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:19:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:19:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:19:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:19:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:19:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:19:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:19:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:19:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:19:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:19:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:19:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:19:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:19:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:19:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:19:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:19:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:19:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:19:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:19:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:19:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:19:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:19:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:21:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:21:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:21:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:21:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:21:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:21:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:21:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:21:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:21:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:21:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:21:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:21:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:21:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:21:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:21:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:21:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:21:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:21:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:21:26 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 01:21:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:21:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:21:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:21:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:21:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:21:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:21:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:21:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:21:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:21:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:21:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:21:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:21:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:21:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:21:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:21:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:21:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:21:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:21:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:21:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:21:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:21:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:21:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:21:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:21:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:21:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:21:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:21:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:21:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:21:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:21:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:21:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:21:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:21:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:21:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:21:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:21:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:21:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:21:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:21:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:21:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:21:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:21:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:21:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:21:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:21:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:21:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:21:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:21:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:21:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:21:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:21:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:21:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:21:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:21:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:21:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:21:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:21:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:21:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:21:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:21:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:21:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:21:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:21:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:21:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:21:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:21:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:21:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:21:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:21:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:21:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:21:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:21:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:21:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:21:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:21:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:21:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:21:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:21:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:21:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:21:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:21:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:21:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:21:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:21:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:21:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:21:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:21:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:21:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:21:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:21:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:21:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:21:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:21:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:21:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:21:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:21:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:21:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:21:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:21:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:21:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:21:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:21:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:21:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:21:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:21:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:21:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:21:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:22:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:22:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:22:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:22:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:22:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:22:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:22:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:22:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:22:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:22:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:22:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:22:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:22:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:22:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:22:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:22:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:22:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:22:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:22:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:22:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:22:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:22:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:22:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:22:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:22:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:22:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:22:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:22:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:22:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:22:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:22:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:22:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:22:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:22:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:22:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:22:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:22:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:22:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:22:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:22:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:22:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:22:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:22:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:22:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:22:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:22:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:22:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:22:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:22:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:22:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:22:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:22:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:22:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:22:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:22:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:22:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:22:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:22:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:22:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:22:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:22:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:22:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:22:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:22:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:22:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:22:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:22:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:22:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:22:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:22:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:22:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:22:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:22:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:22:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:22:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:22:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:22:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:22:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:22:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:22:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:22:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:22:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:22:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:22:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:22:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:22:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:22:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:22:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:22:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:22:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:22:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:22:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:22:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:22:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:22:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:22:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:22:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:22:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:22:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:22:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:22:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:22:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:22:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:22:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:22:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:22:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:22:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:22:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:22:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:22:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:22:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:22:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:22:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:22:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:22:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:22:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:22:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:22:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:22:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:22:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:22:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:22:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:22:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:22:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:22:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:22:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:22:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:22:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:22:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:22:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:22:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:22:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:22:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:22:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:22:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:22:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:22:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:22:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:22:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:22:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:22:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:22:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:22:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:22:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:22:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:22:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:22:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:22:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:22:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:22:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:22:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:22:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:22:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:22:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:22:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:22:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:22:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:22:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:22:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:22:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:22:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:22:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:22:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:22:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:22:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:22:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:22:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:22:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:22:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:22:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:22:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:22:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:22:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:22:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:22:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:22:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:22:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:22:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:22:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:22:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:22:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:22:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:22:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:22:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:22:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:22:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:22:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:22:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:22:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:22:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:22:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:22:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:22:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:22:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:22:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:22:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:22:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:22:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:22:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:22:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:22:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:22:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:22:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:22:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:22:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:22:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:22:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:22:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:22:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:22:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:22:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:22:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:22:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:22:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:22:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:22:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:23:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:23:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:23:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:23:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:23:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:23:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:23:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:23:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:23:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:23:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:23:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:23:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:23:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:23:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:23:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:23:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:23:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:23:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:23:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:23:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:23:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:23:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:23:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:23:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:23:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:23:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:23:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:23:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:23:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:23:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:23:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:23:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:23:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:23:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:23:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:23:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:23:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:23:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:23:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:23:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:23:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:23:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:23:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:23:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:23:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:23:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:23:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:23:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:23:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:23:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:23:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:23:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:23:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:23:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:23:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:23:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:23:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:23:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:23:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:23:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:23:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:23:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:23:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:23:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:23:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:23:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:23:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:23:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:23:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:23:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:23:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:23:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:23:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:23:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:23:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:23:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:23:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:23:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:23:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:23:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:23:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:23:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:23:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:23:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:23:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:23:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:23:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:23:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:23:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:23:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:23:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:23:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:23:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:23:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:23:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:23:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:23:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:23:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:23:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:23:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:23:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:23:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:23:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:23:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:23:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:23:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:23:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:23:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:23:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:23:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:23:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:23:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:23:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:23:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:23:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:23:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:23:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:23:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:23:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:23:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:23:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:23:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:23:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:23:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:23:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:23:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:23:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:23:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:23:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:23:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:23:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:23:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:23:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:23:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:23:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:23:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:23:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:23:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:23:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:23:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:23:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:23:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:23:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:23:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:23:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:23:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:23:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:23:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:23:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:23:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:23:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:23:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:23:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:23:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:23:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:23:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:23:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:23:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:23:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:23:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:23:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:23:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:23:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:23:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:23:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:23:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:23:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:23:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:23:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:23:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:23:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:23:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:23:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:23:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:23:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:23:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:23:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:23:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:23:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:23:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:23:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:23:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:23:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:23:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:23:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:23:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:23:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:23:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:23:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:23:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:23:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:23:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:23:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:23:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:23:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:23:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:23:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:23:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:23:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:23:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:23:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:23:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:23:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:23:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:23:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:23:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:23:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:23:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:23:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:23:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:23:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:23:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:23:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:23:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:23:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:23:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:24:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:24:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:24:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:24:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:24:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:24:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:24:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:24:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:24:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:24:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:24:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:24:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:24:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:24:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:24:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:24:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:24:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:24:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:24:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:24:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:24:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:24:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:24:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:24:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:24:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:24:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:24:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:24:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:24:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:24:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:24:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:24:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:24:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:24:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:24:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:24:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:24:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:24:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:24:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:24:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:24:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:24:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:24:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:24:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:24:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:24:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:24:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:24:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:24:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:24:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:24:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:24:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:24:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:24:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:24:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:24:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:24:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:24:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:24:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:24:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:24:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:24:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:24:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:24:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:24:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:24:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:24:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:24:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:24:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:24:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:24:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:24:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:24:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:24:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:24:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:24:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:24:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:24:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:24:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:24:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:24:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:24:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:24:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:24:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:24:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:24:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:24:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:24:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:24:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:24:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:24:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:24:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:24:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:24:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:24:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:24:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:24:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:24:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:24:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:24:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:24:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:24:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:24:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:24:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:24:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:24:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:24:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:24:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:24:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:24:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:24:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:24:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:24:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:24:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:24:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:24:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:24:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:24:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:24:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:24:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:24:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:24:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:24:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:24:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:24:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:24:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:24:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:24:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:24:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:24:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:24:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:24:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:24:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:24:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:24:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:24:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:24:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:24:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:24:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:24:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:24:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:24:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:24:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:24:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:24:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:24:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:24:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:24:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:24:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:24:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:24:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:24:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:24:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:24:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:24:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:24:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:24:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:24:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:24:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:24:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:24:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:24:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:24:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:24:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:24:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:24:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:24:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:24:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:24:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:24:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:24:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:24:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:24:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:24:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:24:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:24:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:24:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:24:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:24:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:24:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:24:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:24:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:24:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:24:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:24:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:24:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:24:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:24:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:24:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:24:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:24:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:24:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:24:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:24:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:24:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:24:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:24:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:24:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:24:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:24:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:24:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:24:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:24:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:24:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:24:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:24:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:24:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:24:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:24:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:24:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:24:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:24:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:24:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:24:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:24:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:24:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:25:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:25:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:25:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:25:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:25:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:25:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:25:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:25:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:25:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:25:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:25:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:25:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:25:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:25:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:25:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:25:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:25:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:25:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:25:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:25:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:25:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:25:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:25:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:25:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:25:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:25:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:25:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:25:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:25:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:25:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:25:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:25:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:25:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:25:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:25:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:25:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:25:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:25:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:25:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:25:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:25:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:25:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:25:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:25:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:25:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:25:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:25:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:25:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:25:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:25:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:25:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:25:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:25:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:25:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:25:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:25:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:25:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:25:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:25:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:25:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:25:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:25:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:25:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:25:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:25:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:25:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:25:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:25:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:25:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:25:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:25:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:25:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:25:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:25:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:25:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:25:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:25:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:25:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:25:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:25:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:25:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:25:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:25:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:25:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:25:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:25:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:25:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:25:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:25:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:25:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:25:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:25:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:25:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:25:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:25:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:25:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:25:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:25:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:25:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:25:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:25:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:25:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:25:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:25:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:25:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:25:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:25:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:25:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:25:26 rdma06 systemd-journald[790]: Data hash table of /run/log/journal/95379d2f723847b68d9e976dee8967a9/system.journal has a fill level at 75.0 (163750 of 218332 items, 75497472 file size, 461 bytes per hash table item), suggesting rotation.
Apr 18 01:25:26 rdma06 systemd-journald[790]: /run/log/journal/95379d2f723847b68d9e976dee8967a9/system.journal: Journal header limits reached or header out-of-date, rotating.
Apr 18 01:25:26 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 18 01:25:26 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 18 01:25:26 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 18 01:25:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:25:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:25:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:25:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:25:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:25:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:25:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:25:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:25:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:25:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:25:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:25:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:25:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:25:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:25:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:25:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:25:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:25:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:25:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:25:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:25:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:25:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:25:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:25:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:25:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:25:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:25:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:25:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:25:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:25:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:25:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:25:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:25:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:25:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:25:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:25:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:25:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:25:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:25:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:25:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:25:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:25:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:25:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:25:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:25:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:25:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:25:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:25:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:25:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:25:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:25:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:25:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:25:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:25:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:25:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:25:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:25:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:25:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:25:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:25:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:25:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:25:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:25:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:25:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:25:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:25:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:25:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:25:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:25:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:25:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:25:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:25:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:25:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:25:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:25:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:25:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:25:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:25:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:25:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:25:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:25:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:25:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:25:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:25:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:25:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:25:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:25:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:25:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:25:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:25:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:25:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:25:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:25:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:25:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:25:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:25:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:25:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:25:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:25:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:25:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:25:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:25:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:25:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:25:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:25:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:25:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:25:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:25:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:26:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:26:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:26:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:26:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:26:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:26:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:26:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:26:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:26:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:26:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:26:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:26:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:26:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:26:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:26:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:26:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:26:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:26:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:26:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:26:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:26:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:26:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:26:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:26:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:26:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:26:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:26:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:26:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:26:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:26:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:26:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:26:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:26:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:26:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:26:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:26:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:26:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:26:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:26:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:26:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:26:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:26:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:26:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:26:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:26:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:26:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:26:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:26:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:26:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:26:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:26:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:26:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:26:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:26:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:26:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:26:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:26:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:26:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:26:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:26:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:26:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:26:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:26:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:26:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:26:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:26:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:26:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:26:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:26:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:26:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:26:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:26:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:26:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:26:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:26:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:26:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:26:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:26:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:26:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:26:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:26:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:26:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:26:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:26:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:26:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:26:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:26:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:26:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:26:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:26:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:26:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:26:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:26:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:26:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:26:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:26:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:26:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:26:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:26:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:26:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:26:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:26:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:26:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:26:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:26:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:26:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:26:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:26:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:26:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:26:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:26:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:26:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:26:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:26:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:26:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:26:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:26:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:26:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:26:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:26:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:26:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:26:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:26:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:26:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:26:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:26:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:26:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:26:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:26:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:26:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:26:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:26:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:26:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:26:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:26:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:26:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:26:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:26:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:26:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:26:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:26:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:26:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:26:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:26:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:26:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:26:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:26:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:26:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:26:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:26:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:26:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:26:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:26:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:26:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:26:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:26:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:26:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:26:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:26:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:26:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:26:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:26:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:26:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:26:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:26:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:26:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:26:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:26:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:26:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:26:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:26:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:26:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:26:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:26:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:26:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:26:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:26:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:26:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:26:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:26:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:26:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:26:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:26:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:26:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:26:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:26:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:26:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:26:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:26:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:26:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:26:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:26:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:26:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:26:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:26:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:26:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:26:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:26:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:26:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:26:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:26:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:26:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:26:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:26:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:26:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:26:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:26:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:26:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:26:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:26:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:26:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:26:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:26:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:26:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:26:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:26:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:27:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:27:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:27:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:27:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:27:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:27:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:27:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:27:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:27:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:27:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:27:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:27:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:27:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:27:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:27:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:27:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:27:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:27:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:27:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:27:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:27:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:27:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:27:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:27:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:27:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:27:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:27:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:27:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:27:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:27:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:27:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:27:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:27:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:27:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:27:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:27:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:27:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:27:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:27:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:27:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:27:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:27:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:27:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:27:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:27:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:27:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:27:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:27:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:27:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:27:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:27:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:27:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:27:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:27:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:27:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:27:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:27:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:27:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:27:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:27:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:27:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:27:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:27:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:27:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:27:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:27:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:27:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:27:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:27:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:27:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:27:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:27:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:27:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:27:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:27:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:27:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:27:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:27:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:27:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:27:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:27:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:27:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:27:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:27:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:27:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:27:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:27:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:27:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:27:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:27:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:27:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:27:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:27:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:27:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:27:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:27:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:27:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:27:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:27:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:27:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:27:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:27:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:27:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:27:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:27:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:27:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:27:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:27:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:27:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:27:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:27:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:27:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:27:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:27:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:27:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:27:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:27:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:27:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:27:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:27:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:27:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:27:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:27:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:27:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:27:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:27:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:27:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:27:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:27:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:27:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:27:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:27:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:27:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:27:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:27:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:27:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:27:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:27:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:27:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:27:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:27:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:27:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:27:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:27:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:27:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:27:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:27:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:27:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:27:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:27:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:27:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:27:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:27:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:27:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:27:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:27:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:27:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:27:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:27:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:27:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:27:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:27:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:27:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:27:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:27:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:27:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:27:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:27:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:27:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:27:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:27:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:27:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:27:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:27:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:27:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:27:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:27:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:27:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:27:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:27:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:27:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:27:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:27:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:27:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:27:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:27:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:27:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:27:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:27:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:27:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:27:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:27:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:27:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:27:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:27:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:27:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:27:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:27:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:27:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:27:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:27:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:27:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:27:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:27:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:27:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:27:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:27:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:27:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:27:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:27:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:27:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:27:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:27:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:27:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:27:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:27:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:28:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:28:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:28:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:28:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:28:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:28:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:28:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:28:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:28:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:28:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:28:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:28:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:28:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:28:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:28:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:28:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:28:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:28:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:28:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:28:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:28:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:28:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:28:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:28:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:28:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:28:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:28:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:28:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:28:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:28:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:28:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:28:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:28:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:28:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:28:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:28:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:28:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:28:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:28:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:28:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:28:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:28:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:28:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:28:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:28:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:28:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:28:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:28:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:28:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:28:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:28:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:28:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:28:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:28:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:28:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:28:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:28:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:28:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:28:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:28:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:28:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:28:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:28:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:28:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:28:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:28:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:28:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:28:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:28:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:28:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:28:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:28:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:28:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:28:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:28:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:28:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:28:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:28:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:28:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:28:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:28:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:28:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:28:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:28:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:28:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:28:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:28:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:28:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:28:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:28:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:28:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:28:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:28:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:28:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:28:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:28:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:28:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:28:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:28:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:28:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:28:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:28:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:28:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:28:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:28:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:28:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:28:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:28:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:28:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:28:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:28:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:28:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:28:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:28:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:28:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:28:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:28:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:28:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:28:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:28:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:28:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:28:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:28:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:28:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:28:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:28:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:28:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:28:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:28:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:28:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:28:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:28:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:28:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:28:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:28:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:28:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:28:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:28:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:28:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:28:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:28:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:28:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:28:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:28:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:28:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:28:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:28:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:28:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:28:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:28:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:28:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:28:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:28:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:28:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:28:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:28:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:28:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:28:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:28:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:28:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:28:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:28:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:28:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:28:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:28:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:28:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:28:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:28:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:28:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:28:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:28:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:28:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:28:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:28:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:28:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:28:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:28:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:28:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:28:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:28:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:28:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:28:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:28:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:28:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:28:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:28:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:28:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:28:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:28:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:28:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:28:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:28:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:28:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:28:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:28:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:28:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:28:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:28:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:28:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:28:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:28:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:28:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:28:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:28:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:28:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:28:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:28:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:28:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:28:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:28:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:28:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:28:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:28:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:28:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:28:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:28:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:29:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:29:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:29:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:29:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:29:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:29:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:29:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:29:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:29:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:29:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:29:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:29:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:29:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:29:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:29:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:29:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:29:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:29:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:29:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:29:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:29:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:29:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:29:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:29:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:29:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:29:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:29:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:29:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:29:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:29:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:29:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:29:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:29:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:29:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:29:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:29:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:29:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:29:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:29:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:29:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:29:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:29:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:29:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:29:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:29:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:29:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:29:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:29:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:29:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:29:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:29:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:29:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:29:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:29:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:29:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:29:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:29:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:29:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:29:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:29:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:29:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:29:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:29:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:29:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:29:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:29:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:29:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:29:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:29:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:29:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:29:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:29:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:29:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:29:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:29:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:29:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:29:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:29:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:29:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:29:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:29:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:29:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:29:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:29:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:29:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:29:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:29:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:29:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:29:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:29:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:31:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:31:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:31:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:31:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:31:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:31:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:31:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:31:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:31:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:31:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:31:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:31:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:31:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:31:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:31:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:31:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:31:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:31:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:31:27 rdma06 rsyslogd[1041]: imjournal: 5087 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 01:31:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:31:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:31:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:31:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:31:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:31:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:31:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:31:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:31:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:31:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:31:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:31:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:31:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:31:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:31:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:31:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:31:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:31:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:31:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:31:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:31:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:31:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:31:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:31:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:31:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:31:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:31:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:31:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:31:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:31:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:31:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:31:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:31:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:31:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:31:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:31:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:31:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:31:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:31:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:31:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:31:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:31:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:31:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:31:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:31:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:31:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:31:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:31:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:31:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:31:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:31:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:31:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:31:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:31:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:31:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:31:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:31:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:31:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:32:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:32:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:32:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:32:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:32:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:32:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:32:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:32:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:32:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:32:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:32:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:32:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:32:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:32:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:32:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:32:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:32:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:32:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:32:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:32:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:32:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:32:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:32:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:32:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:32:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:32:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:32:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:32:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:32:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:32:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:32:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:32:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:32:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:32:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:32:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:32:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:32:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:32:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:32:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:32:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:32:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:32:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:32:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:32:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:32:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:32:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:32:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:32:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:32:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:32:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:32:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:32:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:32:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:32:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:32:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:32:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:32:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:32:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:32:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:32:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:32:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:32:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:32:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:32:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:32:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:32:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:32:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:32:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:32:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:32:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:32:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:32:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:32:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:32:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:32:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:32:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:32:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:32:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:32:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:32:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:32:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:32:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:32:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:32:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:32:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:32:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:32:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:32:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:32:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:32:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:32:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:32:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:32:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:32:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:32:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:32:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:32:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:32:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:32:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:32:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:32:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:32:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:32:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:32:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:32:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:32:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:32:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:32:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:32:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:32:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:32:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:32:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:32:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:32:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:32:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:32:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:32:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:32:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:32:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:32:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:32:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:32:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:32:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:32:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:32:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:32:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:32:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:32:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:32:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:32:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:32:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:32:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:32:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:32:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:32:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:32:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:32:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:32:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:32:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:32:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:32:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:32:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:32:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:32:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:32:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:32:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:32:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:32:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:32:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:32:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:32:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:32:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:32:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:32:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:32:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:32:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:32:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:32:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:32:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:32:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:32:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:32:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:32:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:32:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:32:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:32:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:32:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:32:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:32:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:32:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:32:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:32:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:32:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:32:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:32:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:32:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:32:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:32:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:32:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:32:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:32:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:32:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:32:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:32:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:32:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:32:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:32:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:32:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:32:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:32:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:32:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:32:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:32:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:32:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:32:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:32:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:32:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:32:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:32:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:32:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:32:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:32:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:32:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:32:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:32:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:32:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:32:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:32:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:32:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:32:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:32:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:32:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:32:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:32:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:32:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:32:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:33:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:33:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:33:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:33:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:33:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:33:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:33:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:33:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:33:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:33:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:33:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:33:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:33:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:33:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:33:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:33:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:33:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:33:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:33:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:33:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:33:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:33:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:33:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:33:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:33:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:33:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:33:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:33:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:33:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:33:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:33:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:33:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:33:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:33:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:33:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:33:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:33:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:33:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:33:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:33:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:33:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:33:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:33:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:33:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:33:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:33:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:33:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:33:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:33:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:33:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:33:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:33:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:33:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:33:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:33:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:33:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:33:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:33:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:33:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:33:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:33:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:33:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:33:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:33:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:33:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:33:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:33:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:33:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:33:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:33:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:33:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:33:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:33:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:33:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:33:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:33:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:33:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:33:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:33:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:33:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:33:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:33:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:33:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:33:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:33:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:33:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:33:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:33:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:33:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:33:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:33:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:33:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:33:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:33:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:33:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:33:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:33:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:33:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:33:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:33:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:33:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:33:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:33:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:33:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:33:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:33:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:33:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:33:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:33:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:33:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:33:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:33:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:33:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:33:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:33:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:33:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:33:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:33:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:33:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:33:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:33:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:33:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:33:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:33:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:33:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:33:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:33:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:33:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:33:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:33:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:33:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:33:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:33:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:33:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:33:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:33:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:33:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:33:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:33:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:33:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:33:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:33:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:33:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:33:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:33:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:33:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:33:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:33:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:33:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:33:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:33:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:33:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:33:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:33:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:33:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:33:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:33:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:33:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:33:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:33:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:33:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:33:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:33:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:33:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:33:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:33:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:33:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:33:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:33:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:33:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:33:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:33:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:33:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:33:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:33:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:33:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:33:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:33:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:33:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:33:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:33:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:33:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:33:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:33:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:33:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:33:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:33:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:33:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:33:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:33:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:33:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:33:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:33:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:33:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:33:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:33:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:33:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:33:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:33:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:33:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:33:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:33:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:33:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:33:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:33:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:33:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:33:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:33:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:33:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:33:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:33:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:33:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:33:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:33:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:33:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:33:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:34:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:34:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:34:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:34:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:34:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:34:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:34:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:34:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:34:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:34:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:34:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:34:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:34:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:34:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:34:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:34:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:34:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:34:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:34:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:34:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:34:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:34:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:34:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:34:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:34:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:34:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:34:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:34:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:34:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:34:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:34:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:34:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:34:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:34:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:34:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:34:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:34:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:34:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:34:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:34:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:34:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:34:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:34:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:34:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:34:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:34:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:34:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:34:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:34:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:34:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:34:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:34:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:34:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:34:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:34:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:34:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:34:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:34:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:34:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:34:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:34:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:34:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:34:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:34:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:34:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:34:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:34:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:34:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:34:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:34:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:34:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:34:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:34:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:34:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:34:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:34:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:34:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:34:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:34:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:34:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:34:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:34:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:34:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:34:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:34:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:34:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:34:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:34:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:34:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:34:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:34:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:34:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:34:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:34:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:34:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:34:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:34:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:34:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:34:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:34:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:34:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:34:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:34:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:34:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:34:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:34:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:34:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:34:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:34:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:34:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:34:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:34:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:34:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:34:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:34:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:34:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:34:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:34:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:34:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:34:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:34:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:34:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:34:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:34:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:34:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:34:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:34:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:34:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:34:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:34:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:34:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:34:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:34:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:34:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:34:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:34:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:34:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:34:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:34:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:34:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:34:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:34:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:34:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:34:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:34:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:34:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:34:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:34:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:34:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:34:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:34:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:34:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:34:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:34:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:34:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:34:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:34:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:34:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:34:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:34:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:34:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:34:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:34:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:34:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:34:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:34:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:34:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:34:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:34:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:34:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:34:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:34:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:34:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:34:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:34:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:34:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:34:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:34:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:34:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:34:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:34:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:34:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:34:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:34:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:34:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:34:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:34:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:34:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:34:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:34:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:34:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:34:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:34:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:34:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:34:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:34:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:34:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:34:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:34:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:34:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:34:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:34:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:34:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:34:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:34:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:34:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:34:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:34:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:34:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:34:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:34:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:34:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:34:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:34:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:34:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:34:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:35:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:35:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:35:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:35:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:35:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:35:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:35:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:35:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:35:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:35:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:35:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:35:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:35:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:35:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:35:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:35:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:35:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:35:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:35:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:35:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:35:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:35:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:35:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:35:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:35:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:35:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:35:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:35:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:35:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:35:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:35:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:35:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:35:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:35:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:35:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:35:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:35:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:35:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:35:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:35:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:35:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:35:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:35:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:35:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:35:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:35:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:35:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:35:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:35:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:35:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:35:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:35:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:35:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:35:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:35:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:35:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:35:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:35:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:35:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:35:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:35:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:35:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:35:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:35:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:35:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:35:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:35:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:35:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:35:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:35:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:35:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:35:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:35:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:35:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:35:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:35:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:35:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:35:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:35:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:35:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:35:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:35:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:35:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:35:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:35:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:35:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:35:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:35:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:35:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:35:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:35:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:35:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:35:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:35:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:35:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:35:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:35:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:35:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:35:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:35:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:35:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:35:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:35:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:35:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:35:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:35:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:35:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:35:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:35:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:35:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:35:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:35:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:35:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:35:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:35:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:35:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:35:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:35:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:35:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:35:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:35:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:35:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:35:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:35:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:35:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:35:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:35:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:35:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:35:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:35:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:35:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:35:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:35:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:35:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:35:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:35:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:35:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:35:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:35:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:35:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:35:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:35:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:35:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:35:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:35:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:35:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:35:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:35:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:35:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:35:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:35:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:35:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:35:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:35:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:35:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:35:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:35:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:35:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:35:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:35:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:35:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:35:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:35:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:35:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:35:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:35:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:35:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:35:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:35:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:35:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:35:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:35:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:35:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:35:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:35:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:35:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:35:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:35:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:35:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:35:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:35:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:35:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:35:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:35:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:35:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:35:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:35:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:35:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:35:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:35:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:35:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:35:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:35:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:35:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:35:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:35:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:35:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:35:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:35:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:35:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:35:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:35:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:35:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:35:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:35:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:35:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:35:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:35:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:35:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:35:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:35:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:35:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:35:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:35:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:35:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:35:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:36:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:36:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:36:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:36:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:36:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:36:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:36:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:36:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:36:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:36:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:36:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:36:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:36:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:36:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:36:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:36:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:36:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:36:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:36:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:36:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:36:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:36:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:36:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:36:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:36:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:36:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:36:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:36:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:36:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:36:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:36:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:36:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:36:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:36:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:36:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:36:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:36:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:36:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:36:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:36:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:36:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:36:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:36:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:36:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:36:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:36:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:36:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:36:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:36:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:36:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:36:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:36:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:36:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:36:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:36:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:36:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:36:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:36:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:36:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:36:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:36:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:36:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:36:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:36:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:36:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:36:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:36:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:36:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:36:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:36:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:36:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:36:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:36:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:36:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:36:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:36:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:36:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:36:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:36:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:36:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:36:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:36:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:36:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:36:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:36:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:36:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:36:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:36:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:36:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:36:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:36:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:36:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:36:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:36:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:36:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:36:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:36:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:36:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:36:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:36:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:36:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:36:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:36:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:36:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:36:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:36:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:36:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:36:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:36:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:36:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:36:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:36:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:36:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:36:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:36:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:36:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:36:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:36:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:36:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:36:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:36:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:36:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:36:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:36:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:36:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:36:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:36:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:36:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:36:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:36:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:36:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:36:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:36:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:36:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:36:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:36:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:36:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:36:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:36:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:36:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:36:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:36:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:36:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:36:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:36:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:36:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:36:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:36:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:36:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:36:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:36:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:36:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:36:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:36:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:36:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:36:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:36:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:36:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:36:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:36:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:36:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:36:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:36:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:36:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:36:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:36:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:36:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:36:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:36:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:36:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:36:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:36:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:36:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:36:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:36:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:36:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:36:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:36:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:36:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:36:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:36:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:36:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:36:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:36:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:36:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:36:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:36:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:36:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:36:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:36:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:36:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:36:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:36:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:36:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:36:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:36:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:36:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:36:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:36:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:36:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:36:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:36:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:36:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:36:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:36:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:36:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:36:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:36:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:36:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:36:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:36:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:36:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:36:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:36:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:36:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:36:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:37:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:37:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:37:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:37:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:37:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:37:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:37:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:37:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:37:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:37:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:37:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:37:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:37:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:37:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:37:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:37:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:37:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:37:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:37:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:37:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:37:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:37:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:37:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:37:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:37:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:37:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:37:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:37:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:37:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:37:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:37:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:37:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:37:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:37:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:37:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:37:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:37:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:37:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:37:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:37:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:37:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:37:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:37:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:37:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:37:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:37:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:37:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:37:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:37:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:37:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:37:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:37:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:37:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:37:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:37:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:37:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:37:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:37:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:37:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:37:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:37:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:37:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:37:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:37:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:37:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:37:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:37:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:37:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:37:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:37:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:37:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:37:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:37:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:37:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:37:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:37:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:37:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:37:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:37:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:37:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:37:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:37:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:37:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:37:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:37:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:37:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:37:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:37:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:37:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:37:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:37:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:37:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:37:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:37:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:37:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:37:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:37:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:37:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:37:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:37:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:37:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:37:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:37:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:37:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:37:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:37:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:37:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:37:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:37:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:37:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:37:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:37:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:37:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:37:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:37:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:37:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:37:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:37:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:37:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:37:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:37:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:37:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:37:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:37:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:37:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:37:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:37:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:37:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:37:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:37:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:37:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:37:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:37:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:37:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:37:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:37:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:37:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:37:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:37:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:37:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:37:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:37:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:37:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:37:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:37:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:37:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:37:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:37:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:37:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:37:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:37:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:37:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:37:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:37:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:37:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:37:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:37:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:37:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:37:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:37:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:37:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:37:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:37:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:37:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:37:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:37:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:37:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:37:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:37:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:37:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:37:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:37:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:37:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:37:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:37:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:37:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:37:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:37:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:37:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:37:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:37:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:37:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:37:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:37:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:37:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:37:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:37:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:37:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:37:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:37:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:37:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:37:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:37:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:37:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:37:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:37:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:37:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:37:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:37:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:37:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:37:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:37:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:37:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:37:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:37:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:37:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:37:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:37:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:37:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:37:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:37:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:37:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:37:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:37:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:37:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:37:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:38:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:38:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:38:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:38:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:38:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:38:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:38:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:38:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:38:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:38:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:38:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:38:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:38:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:38:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:38:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:38:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:38:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:38:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:38:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:38:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:38:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:38:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:38:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:38:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:38:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:38:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:38:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:38:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:38:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:38:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:38:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:38:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:38:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:38:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:38:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:38:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:38:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:38:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:38:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:38:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:38:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:38:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:38:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:38:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:38:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:38:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:38:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:38:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:38:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:38:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:38:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:38:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:38:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:38:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:38:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:38:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:38:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:38:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:38:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:38:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:38:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:38:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:38:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:38:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:38:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:38:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:38:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:38:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:38:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:38:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:38:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:38:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:38:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:38:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:38:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:38:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:38:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:38:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:38:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:38:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:38:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:38:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:38:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:38:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:38:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:38:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:38:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:38:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:38:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:38:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:38:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:38:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:38:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:38:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:38:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:38:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:38:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:38:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:38:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:38:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:38:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:38:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:38:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:38:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:38:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:38:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:38:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:38:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:38:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:38:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:38:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:38:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:38:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:38:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:38:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:38:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:38:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:38:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:38:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:38:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:38:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:38:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:38:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:38:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:38:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:38:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:38:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:38:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:38:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:38:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:38:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:38:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:38:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:38:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:38:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:38:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:38:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:38:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:38:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:38:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:38:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:38:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:38:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:38:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:38:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:38:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:38:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:38:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:38:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:38:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:38:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:38:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:38:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:38:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:38:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:38:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:38:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:38:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:38:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:38:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:38:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:38:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:38:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:38:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:38:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:38:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:38:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:38:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:38:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:38:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:38:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:38:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:38:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:38:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:38:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:38:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:38:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:38:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:38:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:38:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:38:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:38:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:38:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:38:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:38:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:38:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:38:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:38:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:38:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:38:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:38:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:38:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:38:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:38:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:38:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:38:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:38:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:38:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:38:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:38:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:38:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:38:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:38:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:38:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:38:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:38:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:38:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:38:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:38:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:38:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:38:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:38:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:38:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:38:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:38:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:38:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:39:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:39:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:39:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:39:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:39:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:39:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:39:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:39:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:39:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:39:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:39:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:39:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:39:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:39:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:39:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:39:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:39:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:39:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:39:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:39:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:39:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:39:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:39:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:39:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:39:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:39:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:39:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:39:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:39:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:39:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:39:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:39:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:39:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:39:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:39:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:39:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:39:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:39:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:39:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:39:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:39:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:39:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:39:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:39:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:39:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:39:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:39:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:39:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:39:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:39:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:39:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:39:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:39:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:39:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:39:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:39:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:39:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:39:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:39:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:39:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:39:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:39:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:39:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:39:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:39:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:39:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:39:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:39:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:39:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:39:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:39:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:39:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:39:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:39:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:39:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:39:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:39:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:39:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:39:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:39:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:39:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:39:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:39:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:39:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:39:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:39:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:39:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:39:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:39:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:39:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:41:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:41:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:41:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:41:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:41:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:41:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:41:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:41:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:41:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:41:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:41:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:41:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:41:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:41:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:41:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:41:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:41:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:41:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:41:32 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 01:41:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:41:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:41:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:41:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:41:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:41:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:41:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:41:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:41:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:41:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:41:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:41:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:41:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:41:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:41:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:41:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:41:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:41:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:41:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:41:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:41:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:41:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:41:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:41:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:41:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:41:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:41:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:41:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:41:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:41:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:41:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:41:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:41:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:41:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:41:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:41:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:41:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:41:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:41:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:41:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:41:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:41:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:41:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:41:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:41:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:41:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:41:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:41:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:41:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:41:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:41:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:41:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:41:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:41:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:41:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:41:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:41:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:41:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:41:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:41:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:41:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:41:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:41:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:41:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:41:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:41:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:41:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:41:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:41:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:41:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:41:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:41:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:41:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:41:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:41:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:41:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:41:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:41:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:41:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:41:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:41:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:41:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:41:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:41:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:41:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:41:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:41:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:41:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:41:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:41:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:42:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:42:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:42:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:42:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:42:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:42:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:42:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:42:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:42:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:42:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:42:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:42:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:42:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:42:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:42:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:42:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:42:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:42:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:42:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:42:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:42:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:42:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:42:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:42:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:42:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:42:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:42:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:42:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:42:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:42:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:42:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:42:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:42:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:42:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:42:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:42:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:42:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:42:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:42:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:42:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:42:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:42:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:42:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:42:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:42:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:42:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:42:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:42:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:42:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:42:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:42:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:42:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:42:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:42:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:42:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:42:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:42:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:42:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:42:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:42:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:42:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:42:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:42:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:42:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:42:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:42:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:42:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:42:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:42:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:42:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:42:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:42:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:42:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:42:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:42:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:42:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:42:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:42:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:42:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:42:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:42:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:42:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:42:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:42:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:42:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:42:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:42:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:42:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:42:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:42:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:42:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:42:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:42:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:42:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:42:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:42:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:42:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:42:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:42:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:42:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:42:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:42:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:42:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:42:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:42:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:42:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:42:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:42:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:42:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:42:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:42:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:42:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:42:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:42:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:42:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:42:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:42:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:42:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:42:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:42:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:42:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:42:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:42:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:42:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:42:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:42:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:42:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:42:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:42:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:42:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:42:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:42:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:42:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:42:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:42:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:42:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:42:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:42:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:42:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:42:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:42:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:42:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:42:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:42:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:42:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:42:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:42:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:42:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:42:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:42:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:42:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:42:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:42:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:42:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:42:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:42:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:42:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:42:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:42:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:42:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:42:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:42:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:42:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:42:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:42:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:42:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:42:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:42:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:42:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:42:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:42:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:42:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:42:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:42:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:42:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:42:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:42:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:42:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:42:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:42:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:42:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:42:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:42:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:42:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:42:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:42:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:42:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:42:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:42:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:42:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:42:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:42:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:42:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:42:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:42:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:42:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:42:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:42:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:42:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:42:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:42:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:42:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:42:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:42:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:42:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:42:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:42:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:42:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:42:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:42:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:42:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:42:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:42:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:42:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:42:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:42:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:43:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:43:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:43:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:43:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:43:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:43:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:43:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:43:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:43:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:43:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:43:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:43:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:43:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:43:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:43:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:43:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:43:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:43:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:43:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:43:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:43:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:43:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:43:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:43:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:43:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:43:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:43:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:43:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:43:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:43:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:43:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:43:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:43:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:43:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:43:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:43:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:43:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:43:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:43:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:43:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:43:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:43:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:43:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:43:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:43:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:43:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:43:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:43:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:43:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:43:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:43:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:43:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:43:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:43:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:43:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:43:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:43:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:43:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:43:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:43:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:43:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:43:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:43:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:43:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:43:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:43:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:43:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:43:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:43:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:43:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:43:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:43:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:43:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:43:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:43:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:43:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:43:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:43:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:43:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:43:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:43:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:43:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:43:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:43:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:43:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:43:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:43:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:43:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:43:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:43:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:43:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:43:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:43:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:43:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:43:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:43:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:43:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:43:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:43:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:43:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:43:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:43:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:43:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:43:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:43:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:43:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:43:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:43:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:43:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:43:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:43:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:43:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:43:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:43:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:43:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:43:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:43:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:43:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:43:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:43:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:43:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:43:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:43:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:43:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:43:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:43:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:43:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:43:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:43:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:43:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:43:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:43:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:43:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:43:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:43:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:43:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:43:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:43:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:43:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:43:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:43:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:43:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:43:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:43:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:43:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:43:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:43:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:43:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:43:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:43:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:43:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:43:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:43:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:43:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:43:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:43:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:43:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:43:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:43:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:43:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:43:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:43:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:43:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:43:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:43:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:43:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:43:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:43:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:43:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:43:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:43:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:43:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:43:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:43:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:43:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:43:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:43:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:43:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:43:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:43:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:43:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:43:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:43:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:43:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:43:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:43:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:43:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:43:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:43:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:43:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:43:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:43:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:43:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:43:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:43:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:43:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:43:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:43:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:43:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:43:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:43:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:43:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:43:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:43:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:43:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:43:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:43:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:43:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:43:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:43:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:43:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:43:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:43:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:43:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:43:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:43:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:44:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:44:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:44:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:44:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:44:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:44:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:44:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:44:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:44:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:44:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:44:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:44:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:44:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:44:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:44:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:44:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:44:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:44:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:44:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:44:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:44:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:44:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:44:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:44:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:44:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:44:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:44:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:44:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:44:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:44:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:44:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:44:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:44:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:44:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:44:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:44:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:44:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:44:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:44:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:44:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:44:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:44:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:44:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:44:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:44:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:44:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:44:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:44:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:44:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:44:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:44:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:44:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:44:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:44:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:44:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:44:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:44:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:44:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:44:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:44:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:44:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:44:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:44:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:44:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:44:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:44:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:44:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:44:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:44:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:44:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:44:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:44:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:44:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:44:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:44:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:44:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:44:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:44:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:44:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:44:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:44:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:44:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:44:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:44:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:44:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:44:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:44:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:44:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:44:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:44:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:44:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:44:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:44:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:44:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:44:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:44:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:44:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:44:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:44:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:44:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:44:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:44:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:44:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:44:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:44:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:44:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:44:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:44:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:44:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:44:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:44:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:44:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:44:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:44:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:44:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:44:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:44:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:44:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:44:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:44:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:44:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:44:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:44:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:44:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:44:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:44:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:44:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:44:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:44:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:44:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:44:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:44:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:44:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:44:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:44:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:44:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:44:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:44:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:44:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:44:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:44:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:44:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:44:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:44:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:44:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:44:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:44:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:44:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:44:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:44:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:44:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:44:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:44:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:44:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:44:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:44:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:44:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:44:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:44:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:44:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:44:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:44:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:44:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:44:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:44:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:44:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:44:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:44:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:44:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:44:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:44:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:44:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:44:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:44:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:44:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:44:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:44:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:44:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:44:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:44:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:44:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:44:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:44:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:44:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:44:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:44:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:44:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:44:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:44:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:44:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:44:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:44:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:44:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:44:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:44:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:44:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:44:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:44:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:44:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:44:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:44:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:44:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:44:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:44:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:44:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:44:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:44:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:44:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:44:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:44:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:44:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:44:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:44:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:44:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:44:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:44:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:45:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:45:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:45:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:45:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:45:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:45:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:45:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:45:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:45:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:45:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:45:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:45:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:45:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:45:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:45:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:45:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:45:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:45:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:45:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:45:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:45:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:45:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:45:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:45:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:45:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:45:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:45:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:45:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:45:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:45:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:45:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:45:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:45:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:45:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:45:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:45:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:45:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:45:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:45:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:45:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:45:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:45:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:45:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:45:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:45:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:45:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:45:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:45:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:45:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:45:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:45:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:45:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:45:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:45:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:45:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:45:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:45:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:45:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:45:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:45:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:45:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:45:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:45:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:45:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:45:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:45:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:45:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:45:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:45:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:45:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:45:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:45:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:45:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:45:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:45:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:45:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:45:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:45:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:45:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:45:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:45:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:45:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:45:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:45:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:45:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:45:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:45:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:45:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:45:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:45:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:45:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:45:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:45:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:45:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:45:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:45:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:45:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:45:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:45:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:45:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:45:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:45:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:45:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:45:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:45:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:45:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:45:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:45:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:45:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:45:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:45:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:45:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:45:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:45:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:45:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:45:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:45:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:45:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:45:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:45:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:45:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:45:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:45:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:45:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:45:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:45:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:45:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:45:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:45:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:45:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:45:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:45:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:45:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:45:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:45:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:45:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:45:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:45:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:45:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:45:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:45:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:45:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:45:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:45:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:45:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:45:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:45:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:45:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:45:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:45:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:45:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:45:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:45:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:45:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:45:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:45:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:45:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:45:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:45:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:45:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:45:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:45:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:45:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:45:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:45:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:45:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:45:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:45:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:45:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:45:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:45:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:45:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:45:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:45:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:45:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:45:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:45:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:45:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:45:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:45:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:45:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:45:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:45:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:45:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:45:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:45:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:45:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:45:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:45:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:45:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:45:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:45:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:45:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:45:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:45:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:45:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:45:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:45:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:45:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:45:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:45:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:45:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:45:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:45:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:45:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:45:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:45:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:45:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:45:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:45:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:45:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:45:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:45:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:45:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:45:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:45:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:46:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:46:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:46:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:46:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:46:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:46:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:46:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:46:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:46:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:46:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:46:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:46:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:46:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:46:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:46:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:46:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:46:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:46:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:46:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:46:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:46:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:46:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:46:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:46:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:46:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:46:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:46:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:46:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:46:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:46:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:46:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:46:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:46:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:46:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:46:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:46:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:46:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:46:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:46:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:46:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:46:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:46:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:46:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:46:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:46:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:46:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:46:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:46:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:46:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:46:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:46:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:46:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:46:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:46:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:46:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:46:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:46:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:46:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:46:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:46:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:46:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:46:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:46:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:46:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:46:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:46:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:46:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:46:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:46:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:46:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:46:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:46:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:46:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:46:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:46:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:46:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:46:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:46:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:46:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:46:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:46:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:46:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:46:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:46:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:46:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:46:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:46:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:46:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:46:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:46:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:46:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:46:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:46:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:46:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:46:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:46:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:46:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:46:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:46:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:46:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:46:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:46:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:46:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:46:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:46:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:46:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:46:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:46:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:46:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:46:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:46:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:46:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:46:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:46:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:46:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:46:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:46:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:46:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:46:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:46:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:46:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:46:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:46:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:46:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:46:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:46:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:46:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:46:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:46:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:46:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:46:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:46:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:46:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:46:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:46:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:46:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:46:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:46:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:46:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:46:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:46:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:46:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:46:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:46:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:46:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:46:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:46:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:46:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:46:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:46:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:46:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:46:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:46:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:46:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:46:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:46:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:46:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:46:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:46:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:46:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:46:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:46:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:46:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:46:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:46:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:46:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:46:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:46:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:46:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:46:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:46:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:46:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:46:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:46:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:46:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:46:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:46:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:46:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:46:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:46:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:46:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:46:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:46:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:46:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:46:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:46:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:46:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:46:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:46:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:46:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:46:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:46:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:46:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:46:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:46:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:46:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:46:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:46:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:46:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:46:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:46:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:46:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:46:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:46:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:46:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:46:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:46:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:46:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:46:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:46:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:46:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:46:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:46:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:46:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:46:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:46:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:47:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:47:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:47:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:47:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:47:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:47:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:47:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:47:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:47:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:47:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:47:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:47:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:47:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:47:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:47:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:47:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:47:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:47:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:47:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:47:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:47:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:47:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:47:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:47:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:47:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:47:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:47:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:47:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:47:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:47:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:47:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:47:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:47:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:47:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:47:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:47:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:47:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:47:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:47:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:47:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:47:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:47:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:47:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:47:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:47:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:47:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:47:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:47:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:47:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:47:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:47:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:47:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:47:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:47:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:47:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:47:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:47:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:47:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:47:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:47:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:47:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:47:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:47:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:47:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:47:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:47:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:47:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:47:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:47:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:47:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:47:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:47:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:47:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:47:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:47:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:47:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:47:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:47:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:47:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:47:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:47:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:47:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:47:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:47:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:47:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:47:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:47:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:47:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:47:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:47:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:47:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:47:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:47:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:47:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:47:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:47:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:47:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:47:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:47:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:47:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:47:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:47:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:47:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:47:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:47:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:47:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:47:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:47:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:47:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:47:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:47:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:47:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:47:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:47:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:47:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:47:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:47:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:47:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:47:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:47:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:47:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:47:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:47:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:47:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:47:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:47:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:47:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:47:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:47:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:47:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:47:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:47:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:47:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:47:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:47:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:47:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:47:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:47:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:47:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:47:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:47:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:47:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:47:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:47:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:47:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:47:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:47:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:47:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:47:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:47:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:47:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:47:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:47:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:47:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:47:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:47:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:47:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:47:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:47:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:47:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:47:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:47:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:47:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:47:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:47:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:47:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:47:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:47:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:47:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:47:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:47:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:47:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:47:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:47:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:47:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:47:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:47:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:47:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:47:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:47:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:47:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:47:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:47:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:47:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:47:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:47:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:47:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:47:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:47:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:47:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:47:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:47:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:47:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:47:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:47:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:47:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:47:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:47:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:47:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:47:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:47:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:47:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:47:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:47:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:47:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:47:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:47:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:47:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:47:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:47:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:47:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:47:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:47:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:47:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:47:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:47:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:48:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:48:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:48:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:48:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:48:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:48:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:48:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:48:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:48:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:48:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:48:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:48:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:48:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:48:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:48:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:48:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:48:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:48:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:48:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:48:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:48:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:48:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:48:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:48:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:48:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:48:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:48:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:48:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:48:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:48:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:48:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:48:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:48:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:48:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:48:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:48:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:48:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:48:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:48:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:48:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:48:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:48:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:48:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:48:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:48:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:48:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:48:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:48:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:48:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:48:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:48:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:48:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:48:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:48:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:48:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:48:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:48:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:48:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:48:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:48:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:48:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:48:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:48:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:48:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:48:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:48:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:48:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:48:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:48:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:48:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:48:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:48:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:48:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:48:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:48:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:48:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:48:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:48:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:48:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:48:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:48:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:48:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:48:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:48:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:48:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:48:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:48:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:48:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:48:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:48:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:48:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:48:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:48:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:48:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:48:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:48:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:48:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:48:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:48:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:48:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:48:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:48:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:48:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:48:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:48:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:48:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:48:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:48:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:48:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:48:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:48:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:48:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:48:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:48:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:48:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:48:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:48:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:48:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:48:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:48:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:48:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:48:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:48:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:48:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:48:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:48:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:48:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:48:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:48:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:48:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:48:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:48:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:48:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:48:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:48:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:48:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:48:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:48:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:48:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:48:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:48:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:48:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:48:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:48:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:48:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:48:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:48:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:48:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:48:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:48:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:48:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:48:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:48:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:48:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:48:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:48:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:48:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:48:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:48:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:48:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:48:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:48:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:48:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:48:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:48:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:48:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:48:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:48:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:48:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:48:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:48:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:48:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:48:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:48:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:48:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:48:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:48:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:48:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:48:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:48:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:48:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:48:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:48:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:48:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:48:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:48:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:48:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:48:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:48:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:48:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:48:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:48:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:48:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:48:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:48:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:48:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:48:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:48:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:48:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:48:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:48:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:48:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:48:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:48:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:48:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:48:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:48:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:48:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:48:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:48:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:48:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:48:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:48:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:48:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:48:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:48:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:49:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:49:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:49:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:49:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:49:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:49:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:49:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:49:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:49:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:49:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:49:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:49:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:49:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:49:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:49:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:49:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:49:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:49:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:49:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:49:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:49:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:49:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:49:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:49:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:49:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:49:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:49:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:49:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:49:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:49:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:49:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:49:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:49:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:49:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:49:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:49:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:49:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:49:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:49:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:49:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:49:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:49:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:49:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:49:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:49:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:49:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:49:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:49:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:49:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:49:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:49:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:49:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:49:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:49:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:49:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:49:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:49:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:49:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:49:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:49:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:49:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:49:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:49:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:49:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:49:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:49:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:49:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:49:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:49:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:49:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:49:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:49:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:49:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:49:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:49:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:49:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:49:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:49:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:49:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:49:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:49:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:49:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:49:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:49:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:49:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:49:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:49:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:49:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:49:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:49:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:49:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:49:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:49:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:49:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:49:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:49:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:49:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:49:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:49:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:49:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:49:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:49:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:49:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:49:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:49:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:49:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:49:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:49:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:51:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:51:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:51:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:51:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:51:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:51:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:51:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:51:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:51:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:51:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:51:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:51:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:51:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:51:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:51:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:51:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:51:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:51:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:51:37 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 01:51:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:51:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:51:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:51:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:51:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:51:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:51:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:51:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:51:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:51:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:51:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:51:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:51:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:51:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:51:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:51:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:51:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:51:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:51:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:51:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:51:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:51:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:51:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:51:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:51:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:51:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:51:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:51:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:51:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:51:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:51:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:51:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:51:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:51:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:51:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:51:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:51:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:51:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:51:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:51:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:51:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:51:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:51:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:51:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:51:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:51:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:51:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:51:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:51:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:51:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:51:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:51:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:51:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:51:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:51:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:51:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:51:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:51:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:51:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:51:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:51:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:51:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:51:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:51:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:51:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:51:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:51:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:51:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:51:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:51:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:51:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:51:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:52:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:52:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:52:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:52:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:52:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:52:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:52:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:52:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:52:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:52:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:52:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:52:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:52:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:52:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:52:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:52:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:52:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:52:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:52:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:52:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:52:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:52:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:52:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:52:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:52:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:52:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:52:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:52:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:52:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:52:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:52:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:52:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:52:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:52:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:52:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:52:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:52:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:52:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:52:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:52:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:52:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:52:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:52:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:52:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:52:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:52:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:52:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:52:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:52:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:52:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:52:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:52:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:52:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:52:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:52:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:52:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:52:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:52:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:52:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:52:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:52:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:52:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:52:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:52:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:52:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:52:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:52:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:52:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:52:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:52:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:52:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:52:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:52:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:52:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:52:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:52:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:52:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:52:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:52:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:52:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:52:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:52:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:52:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:52:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:52:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:52:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:52:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:52:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:52:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:52:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:52:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:52:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:52:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:52:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:52:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:52:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:52:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:52:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:52:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:52:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:52:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:52:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:52:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:52:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:52:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:52:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:52:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:52:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:52:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:52:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:52:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:52:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:52:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:52:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:52:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:52:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:52:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:52:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:52:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:52:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:52:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:52:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:52:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:52:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:52:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:52:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:52:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:52:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:52:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:52:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:52:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:52:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:52:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:52:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:52:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:52:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:52:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:52:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:52:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:52:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:52:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:52:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:52:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:52:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:52:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:52:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:52:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:52:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:52:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:52:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:52:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:52:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:52:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:52:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:52:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:52:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:52:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:52:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:52:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:52:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:52:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:52:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:52:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:52:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:52:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:52:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:52:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:52:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:52:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:52:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:52:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:52:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:52:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:52:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:52:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:52:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:52:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:52:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:52:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:52:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:52:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:52:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:52:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:52:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:52:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:52:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:52:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:52:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:52:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:52:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:52:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:52:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:52:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:52:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:52:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:52:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:52:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:52:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:52:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:52:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:52:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:52:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:52:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:52:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:52:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:52:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:52:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:52:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:52:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:52:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:52:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:52:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:52:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:52:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:52:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:52:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:53:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:53:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:53:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:53:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:53:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:53:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:53:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:53:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:53:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:53:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:53:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:53:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:53:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:53:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:53:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:53:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:53:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:53:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:53:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:53:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:53:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:53:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:53:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:53:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:53:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:53:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:53:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:53:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:53:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:53:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:53:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:53:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:53:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:53:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:53:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:53:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:53:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:53:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:53:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:53:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:53:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:53:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:53:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:53:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:53:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:53:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:53:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:53:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:53:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:53:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:53:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:53:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:53:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:53:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:53:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:53:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:53:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:53:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:53:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:53:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:53:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:53:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:53:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:53:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:53:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:53:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:53:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:53:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:53:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:53:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:53:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:53:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:53:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:53:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:53:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:53:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:53:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:53:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:53:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:53:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:53:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:53:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:53:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:53:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:53:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:53:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:53:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:53:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:53:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:53:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:53:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:53:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:53:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:53:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:53:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:53:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:53:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:53:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:53:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:53:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:53:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:53:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:53:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:53:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:53:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:53:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:53:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:53:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:53:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:53:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:53:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:53:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:53:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:53:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:53:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:53:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:53:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:53:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:53:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:53:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:53:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:53:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:53:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:53:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:53:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:53:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:53:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:53:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:53:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:53:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:53:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:53:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:53:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:53:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:53:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:53:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:53:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:53:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:53:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:53:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:53:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:53:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:53:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:53:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:53:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:53:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:53:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:53:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:53:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:53:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:53:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:53:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:53:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:53:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:53:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:53:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:53:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:53:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:53:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:53:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:53:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:53:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:53:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:53:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:53:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:53:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:53:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:53:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:53:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:53:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:53:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:53:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:53:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:53:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:53:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:53:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:53:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:53:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:53:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:53:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:53:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:53:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:53:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:53:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:53:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:53:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:53:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:53:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:53:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:53:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:53:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:53:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:53:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:53:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:53:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:53:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:53:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:53:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:53:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:53:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:53:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:53:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:53:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:53:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:53:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:53:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:53:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:53:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:53:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:53:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:53:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:53:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:53:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:53:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:53:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:53:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:54:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:54:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:54:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:54:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:54:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:54:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:54:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:54:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:54:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:54:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:54:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:54:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:54:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:54:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:54:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:54:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:54:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:54:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:54:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:54:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:54:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:54:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:54:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:54:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:54:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:54:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:54:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:54:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:54:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:54:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:54:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:54:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:54:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:54:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:54:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:54:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:54:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:54:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:54:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:54:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:54:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:54:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:54:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:54:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:54:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:54:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:54:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:54:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:54:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:54:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:54:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:54:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:54:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:54:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:54:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:54:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:54:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:54:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:54:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:54:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:54:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:54:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:54:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:54:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:54:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:54:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:54:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:54:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:54:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:54:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:54:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:54:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:54:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:54:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:54:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:54:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:54:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:54:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:54:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:54:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:54:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:54:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:54:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:54:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:54:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:54:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:54:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:54:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:54:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:54:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:54:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:54:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:54:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:54:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:54:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:54:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:54:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:54:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:54:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:54:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:54:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:54:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:54:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:54:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:54:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:54:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:54:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:54:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:54:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:54:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:54:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:54:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:54:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:54:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:54:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:54:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:54:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:54:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:54:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:54:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:54:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:54:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:54:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:54:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:54:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:54:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:54:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:54:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:54:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:54:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:54:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:54:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:54:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:54:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:54:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:54:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:54:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:54:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:54:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:54:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:54:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:54:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:54:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:54:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:54:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:54:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:54:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:54:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:54:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:54:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:54:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:54:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:54:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:54:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:54:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:54:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:54:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:54:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:54:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:54:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:54:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:54:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:54:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:54:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:54:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:54:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:54:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:54:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:54:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:54:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:54:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:54:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:54:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:54:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:54:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:54:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:54:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:54:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:54:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:54:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:54:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:54:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:54:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:54:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:54:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:54:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:54:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:54:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:54:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:54:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:54:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:54:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:54:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:54:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:54:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:54:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:54:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:54:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:54:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:54:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:54:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:54:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:54:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:54:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:54:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:54:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:54:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:54:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:54:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:54:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:54:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:54:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:54:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:54:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:54:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:54:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:55:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:55:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:55:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:55:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:55:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:55:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:55:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:55:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:55:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:55:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:55:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:55:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:55:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:55:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:55:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:55:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:55:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:55:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:55:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:55:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:55:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:55:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:55:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:55:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:55:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:55:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:55:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:55:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:55:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:55:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:55:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:55:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:55:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:55:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:55:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:55:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:55:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:55:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:55:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:55:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:55:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:55:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:55:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:55:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:55:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:55:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:55:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:55:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:55:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:55:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:55:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:55:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:55:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:55:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:55:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:55:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:55:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:55:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:55:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:55:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:55:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:55:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:55:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:55:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:55:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:55:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:55:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:55:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:55:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:55:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:55:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:55:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:55:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:55:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:55:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:55:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:55:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:55:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:55:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:55:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:55:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:55:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:55:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:55:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:55:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:55:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:55:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:55:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:55:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:55:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:55:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:55:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:55:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:55:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:55:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:55:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:55:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:55:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:55:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:55:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:55:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:55:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:55:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:55:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:55:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:55:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:55:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:55:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:55:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:55:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:55:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:55:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:55:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:55:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:55:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:55:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:55:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:55:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:55:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:55:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:55:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:55:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:55:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:55:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:55:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:55:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:55:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:55:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:55:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:55:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:55:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:55:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:55:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:55:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:55:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:55:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:55:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:55:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:55:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:55:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:55:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:55:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:55:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:55:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:55:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:55:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:55:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:55:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:55:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:55:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:55:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:55:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:55:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:55:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:55:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:55:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:55:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:55:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:55:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:55:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:55:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:55:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:55:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:55:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:55:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:55:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:55:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:55:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:55:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:55:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:55:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:55:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:55:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:55:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:55:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:55:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:55:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:55:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:55:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:55:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:55:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:55:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:55:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:55:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:55:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:55:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:55:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:55:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:55:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:55:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:55:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:55:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:55:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:55:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:55:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:55:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:55:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:55:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:55:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:55:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:55:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:55:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:55:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:55:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:55:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:55:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:55:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:55:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:55:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:55:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:55:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:55:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:55:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:55:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:55:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:55:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:56:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:56:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:56:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:56:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:56:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:56:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:56:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:56:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:56:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:56:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:56:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:56:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:56:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:56:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:56:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:56:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:56:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:56:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:56:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:56:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:56:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:56:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:56:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:56:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:56:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:56:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:56:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:56:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:56:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:56:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:56:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:56:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:56:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:56:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:56:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:56:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:56:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:56:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:56:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:56:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:56:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:56:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:56:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:56:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:56:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:56:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:56:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:56:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:56:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:56:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:56:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:56:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:56:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:56:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:56:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:56:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:56:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:56:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:56:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:56:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:56:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:56:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:56:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:56:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:56:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:56:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:56:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:56:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:56:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:56:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:56:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:56:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:56:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:56:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:56:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:56:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:56:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:56:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:56:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:56:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:56:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:56:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:56:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:56:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:56:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:56:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:56:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:56:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:56:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:56:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:56:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:56:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:56:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:56:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:56:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:56:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:56:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:56:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:56:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:56:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:56:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:56:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:56:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:56:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:56:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:56:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:56:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:56:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:56:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:56:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:56:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:56:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:56:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:56:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:56:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:56:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:56:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:56:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:56:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:56:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:56:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:56:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:56:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:56:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:56:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:56:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:56:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:56:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:56:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:56:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:56:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:56:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:56:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:56:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:56:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:56:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:56:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:56:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:56:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:56:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:56:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:56:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:56:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:56:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:56:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:56:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:56:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:56:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:56:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:56:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:56:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:56:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:56:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:56:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:56:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:56:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:56:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:56:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:56:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:56:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:56:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:56:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:56:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:56:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:56:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:56:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:56:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:56:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:56:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:56:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:56:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:56:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:56:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:56:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:56:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:56:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:56:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:56:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:56:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:56:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:56:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:56:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:56:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:56:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:56:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:56:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:56:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:56:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:56:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:56:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:56:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:56:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:56:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:56:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:56:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:56:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:56:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:56:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:56:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:56:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:56:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:56:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:56:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:56:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:56:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:56:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:56:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:56:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:56:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:56:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:56:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:56:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:56:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:56:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:56:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:56:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:57:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:57:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:57:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:57:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:57:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:57:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:57:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:57:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:57:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:57:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:57:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:57:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:57:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:57:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:57:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:57:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:57:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:57:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:57:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:57:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:57:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:57:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:57:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:57:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:57:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:57:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:57:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:57:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:57:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:57:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:57:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:57:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:57:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:57:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:57:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:57:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:57:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:57:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:57:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:57:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:57:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:57:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:57:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:57:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:57:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:57:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:57:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:57:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:57:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:57:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:57:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:57:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:57:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:57:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:57:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:57:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:57:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:57:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:57:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:57:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:57:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:57:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:57:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:57:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:57:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:57:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:57:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:57:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:57:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:57:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:57:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:57:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:57:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:57:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:57:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:57:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:57:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:57:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:57:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:57:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:57:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:57:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:57:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:57:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:57:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:57:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:57:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:57:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:57:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:57:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:57:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:57:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:57:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:57:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:57:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:57:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:57:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:57:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:57:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:57:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:57:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:57:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:57:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:57:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:57:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:57:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:57:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:57:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:57:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:57:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:57:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:57:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:57:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:57:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:57:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:57:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:57:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:57:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:57:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:57:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:57:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:57:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:57:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:57:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:57:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:57:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:57:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:57:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:57:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:57:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:57:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:57:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:57:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:57:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:57:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:57:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:57:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:57:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:57:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:57:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:57:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:57:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:57:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:57:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:57:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:57:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:57:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:57:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:57:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:57:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:57:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:57:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:57:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:57:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:57:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:57:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:57:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:57:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:57:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:57:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:57:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:57:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:57:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:57:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:57:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:57:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:57:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:57:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:57:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:57:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:57:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:57:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:57:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:57:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:57:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:57:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:57:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:57:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:57:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:57:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:57:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:57:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:57:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:57:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:57:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:57:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:57:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:57:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:57:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:57:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:57:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:57:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:57:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:57:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:57:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:57:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:57:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:57:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:57:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:57:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:57:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:57:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:57:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:57:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:57:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:57:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:57:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:57:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:57:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:57:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:57:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:57:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:57:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:57:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:57:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:57:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:58:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:58:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:58:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:58:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:58:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:58:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:58:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:58:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:58:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:58:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:58:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:58:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:58:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:58:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:58:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:58:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:58:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:58:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:58:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:58:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:58:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:58:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:58:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:58:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:58:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:58:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:58:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:58:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:58:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:58:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:58:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:58:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:58:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:58:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:58:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:58:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:58:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:58:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:58:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:58:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:58:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:58:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:58:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:58:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:58:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:58:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:58:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:58:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:58:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:58:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:58:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:58:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:58:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:58:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:58:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:58:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:58:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:58:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:58:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:58:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:58:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:58:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:58:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:58:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:58:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:58:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:58:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:58:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:58:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:58:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:58:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:58:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:58:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:58:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:58:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:58:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:58:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:58:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:58:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:58:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:58:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:58:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:58:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:58:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:58:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:58:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:58:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:58:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:58:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:58:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:58:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:58:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:58:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:58:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:58:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:58:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:58:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:58:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:58:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:58:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:58:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:58:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:58:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:58:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:58:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:58:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:58:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:58:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:58:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:58:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:58:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:58:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:58:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:58:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:58:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:58:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:58:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:58:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:58:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:58:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:58:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:58:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:58:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:58:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:58:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:58:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:58:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:58:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:58:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:58:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:58:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:58:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:58:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:58:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:58:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:58:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:58:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:58:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:58:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:58:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:58:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:58:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:58:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:58:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:58:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:58:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:58:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:58:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:58:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:58:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:58:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:58:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:58:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:58:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:58:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:58:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:58:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:58:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:58:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:58:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:58:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:58:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:58:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:58:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:58:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:58:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:58:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:58:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:58:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:58:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:58:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:58:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:58:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:58:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:58:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:58:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:58:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:58:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:58:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:58:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:58:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:58:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:58:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:58:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:58:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:58:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:58:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:58:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:58:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:58:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:58:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:58:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:58:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:58:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:58:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:58:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:58:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:58:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:58:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:58:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:58:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:58:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:58:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:58:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:58:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:58:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:58:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:58:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:58:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:58:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:58:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:58:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:58:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:58:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:58:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:58:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:59:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:59:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:59:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:59:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:59:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:59:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:59:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:59:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:59:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:59:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:59:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:59:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:59:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:59:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:59:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:59:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:59:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:59:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:59:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:59:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:59:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:59:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:59:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:59:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:59:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:59:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:59:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:59:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:59:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:59:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:59:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:59:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:59:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:59:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:59:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:59:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:59:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:59:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:59:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:59:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:59:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:59:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:59:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:59:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:59:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:59:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:59:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:59:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:59:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:59:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:59:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:59:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:59:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:59:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:59:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:59:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:59:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:59:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:59:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:59:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:59:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:59:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:59:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:59:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:59:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:59:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:59:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:59:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:59:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:59:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:59:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:59:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:59:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:59:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:59:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:59:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:59:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:59:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:59:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:59:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:59:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:59:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:59:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:59:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:59:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:59:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:59:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:59:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:59:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:59:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:59:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:59:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:59:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:59:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:59:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:59:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:59:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:59:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:59:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:59:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:59:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:59:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:59:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:59:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:59:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:59:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:59:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:59:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 01:59:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 01:59:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:59:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:59:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:59:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 01:59:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 01:59:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 01:59:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 01:59:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 01:59:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 01:59:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 01:59:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 01:59:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 01:59:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 01:59:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 01:59:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 01:59:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 01:59:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:01:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:01:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:01:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:01:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:01:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:01:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:01:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:01:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:01:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:01:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:01:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:01:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:01:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:01:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:01:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:01:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:01:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:01:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:01:38 rdma06 rsyslogd[1041]: imjournal: 5088 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 02:01:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:01:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:01:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:01:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:01:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:01:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:01:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:01:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:01:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:01:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:01:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:01:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:01:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:01:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:01:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:01:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:01:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:01:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:01:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:01:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:01:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:01:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:01:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:01:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:01:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:01:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:01:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:01:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:01:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:01:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:01:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:01:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:01:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:01:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:01:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:01:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:01:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:01:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:01:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:01:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:01:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:01:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:01:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:01:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:01:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:01:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:01:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:01:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:01:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:01:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:01:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:01:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:01:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:01:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:01:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:01:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:01:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:01:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:01:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:01:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:01:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:01:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:01:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:01:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:01:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:01:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:01:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:01:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:01:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:01:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:01:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:01:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:02:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:02:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:02:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:02:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:02:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:02:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:02:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:02:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:02:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:02:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:02:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:02:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:02:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:02:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:02:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:02:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:02:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:02:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:02:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:02:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:02:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:02:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:02:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:02:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:02:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:02:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:02:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:02:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:02:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:02:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:02:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:02:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:02:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:02:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:02:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:02:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:02:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:02:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:02:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:02:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:02:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:02:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:02:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:02:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:02:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:02:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:02:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:02:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:02:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:02:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:02:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:02:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:02:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:02:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:02:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:02:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:02:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:02:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:02:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:02:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:02:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:02:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:02:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:02:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:02:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:02:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:02:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:02:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:02:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:02:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:02:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:02:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:02:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:02:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:02:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:02:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:02:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:02:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:02:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:02:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:02:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:02:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:02:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:02:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:02:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:02:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:02:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:02:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:02:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:02:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:02:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:02:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:02:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:02:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:02:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:02:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:02:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:02:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:02:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:02:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:02:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:02:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:02:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:02:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:02:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:02:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:02:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:02:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:02:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:02:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:02:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:02:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:02:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:02:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:02:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:02:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:02:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:02:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:02:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:02:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:02:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:02:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:02:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:02:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:02:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:02:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:02:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:02:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:02:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:02:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:02:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:02:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:02:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:02:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:02:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:02:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:02:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:02:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:02:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:02:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:02:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:02:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:02:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:02:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:02:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:02:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:02:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:02:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:02:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:02:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:02:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:02:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:02:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:02:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:02:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:02:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:02:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:02:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:02:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:02:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:02:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:02:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:02:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:02:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:02:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:02:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:02:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:02:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:02:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:02:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:02:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:02:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:02:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:02:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:02:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:02:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:02:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:02:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:02:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:02:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:02:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:02:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:02:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:02:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:02:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:02:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:02:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:02:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:02:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:02:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:02:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:02:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:02:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:02:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:02:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:02:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:02:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:02:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:02:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:02:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:02:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:02:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:02:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:02:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:02:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:02:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:02:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:02:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:02:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:02:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:02:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:02:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:02:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:02:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:02:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:02:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:03:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:03:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:03:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:03:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:03:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:03:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:03:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:03:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:03:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:03:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:03:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:03:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:03:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:03:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:03:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:03:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:03:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:03:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:03:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:03:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:03:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:03:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:03:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:03:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:03:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:03:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:03:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:03:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:03:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:03:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:03:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:03:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:03:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:03:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:03:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:03:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:03:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:03:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:03:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:03:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:03:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:03:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:03:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:03:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:03:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:03:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:03:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:03:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:03:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:03:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:03:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:03:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:03:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:03:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:03:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:03:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:03:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:03:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:03:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:03:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:03:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:03:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:03:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:03:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:03:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:03:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:03:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:03:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:03:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:03:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:03:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:03:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:03:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:03:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:03:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:03:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:03:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:03:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:03:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:03:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:03:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:03:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:03:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:03:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:03:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:03:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:03:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:03:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:03:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:03:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:03:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:03:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:03:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:03:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:03:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:03:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:03:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:03:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:03:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:03:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:03:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:03:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:03:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:03:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:03:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:03:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:03:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:03:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:03:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:03:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:03:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:03:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:03:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:03:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:03:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:03:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:03:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:03:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:03:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:03:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:03:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:03:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:03:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:03:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:03:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:03:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:03:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:03:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:03:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:03:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:03:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:03:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:03:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:03:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:03:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:03:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:03:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:03:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:03:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:03:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:03:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:03:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:03:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:03:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:03:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:03:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:03:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:03:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:03:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:03:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:03:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:03:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:03:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:03:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:03:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:03:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:03:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:03:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:03:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:03:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:03:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:03:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:03:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:03:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:03:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:03:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:03:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:03:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:03:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:03:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:03:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:03:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:03:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:03:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:03:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:03:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:03:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:03:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:03:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:03:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:03:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:03:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:03:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:03:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:03:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:03:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:03:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:03:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:03:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:03:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:03:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:03:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:03:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:03:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:03:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:03:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:03:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:03:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:03:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:03:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:03:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:03:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:03:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:03:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:03:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:03:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:03:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:03:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:03:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:03:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:03:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:03:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:03:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:03:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:03:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:03:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:04:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:04:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:04:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:04:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:04:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:04:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:04:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:04:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:04:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:04:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:04:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:04:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:04:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:04:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:04:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:04:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:04:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:04:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:04:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:04:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:04:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:04:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:04:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:04:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:04:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:04:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:04:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:04:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:04:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:04:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:04:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:04:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:04:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:04:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:04:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:04:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:04:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:04:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:04:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:04:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:04:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:04:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:04:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:04:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:04:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:04:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:04:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:04:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:04:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:04:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:04:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:04:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:04:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:04:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:04:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:04:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:04:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:04:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:04:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:04:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:04:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:04:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:04:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:04:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:04:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:04:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:04:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:04:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:04:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:04:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:04:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:04:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:04:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:04:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:04:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:04:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:04:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:04:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:04:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:04:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:04:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:04:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:04:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:04:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:04:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:04:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:04:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:04:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:04:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:04:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:04:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:04:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:04:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:04:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:04:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:04:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:04:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:04:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:04:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:04:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:04:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:04:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:04:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:04:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:04:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:04:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:04:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:04:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:04:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:04:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:04:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:04:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:04:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:04:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:04:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:04:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:04:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:04:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:04:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:04:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:04:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:04:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:04:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:04:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:04:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:04:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:04:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:04:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:04:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:04:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:04:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:04:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:04:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:04:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:04:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:04:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:04:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:04:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:04:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:04:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:04:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:04:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:04:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:04:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:04:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:04:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:04:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:04:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:04:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:04:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:04:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:04:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:04:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:04:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:04:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:04:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:04:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:04:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:04:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:04:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:04:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:04:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:04:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:04:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:04:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:04:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:04:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:04:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:04:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:04:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:04:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:04:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:04:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:04:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:04:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:04:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:04:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:04:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:04:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:04:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:04:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:04:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:04:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:04:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:04:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:04:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:04:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:04:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:04:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:04:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:04:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:04:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:04:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:04:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:04:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:04:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:04:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:04:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:04:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:04:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:04:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:04:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:04:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:04:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:04:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:04:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:04:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:04:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:04:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:04:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:04:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:04:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:04:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:04:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:04:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:04:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:05:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:05:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:05:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:05:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:05:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:05:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:05:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:05:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:05:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:05:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:05:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:05:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:05:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:05:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:05:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:05:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:05:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:05:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:05:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:05:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:05:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:05:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:05:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:05:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:05:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:05:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:05:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:05:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:05:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:05:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:05:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:05:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:05:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:05:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:05:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:05:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:05:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:05:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:05:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:05:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:05:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:05:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:05:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:05:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:05:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:05:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:05:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:05:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:05:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:05:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:05:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:05:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:05:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:05:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:05:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:05:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:05:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:05:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:05:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:05:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:05:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:05:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:05:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:05:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:05:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:05:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:05:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:05:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:05:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:05:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:05:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:05:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:05:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:05:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:05:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:05:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:05:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:05:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:05:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:05:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:05:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:05:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:05:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:05:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:05:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:05:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:05:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:05:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:05:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:05:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:05:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:05:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:05:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:05:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:05:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:05:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:05:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:05:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:05:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:05:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:05:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:05:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:05:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:05:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:05:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:05:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:05:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:05:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:05:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:05:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:05:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:05:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:05:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:05:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:05:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:05:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:05:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:05:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:05:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:05:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:05:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:05:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:05:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:05:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:05:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:05:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:05:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:05:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:05:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:05:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:05:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:05:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:05:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:05:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:05:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:05:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:05:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:05:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:05:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:05:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:05:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:05:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:05:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:05:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:05:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:05:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:05:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:05:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:05:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:05:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:05:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:05:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:05:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:05:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:05:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:05:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:05:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:05:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:05:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:05:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:05:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:05:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:05:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:05:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:05:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:05:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:05:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:05:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:05:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:05:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:05:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:05:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:05:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:05:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:05:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:05:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:05:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:05:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:05:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:05:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:05:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:05:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:05:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:05:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:05:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:05:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:05:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:05:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:05:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:05:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:05:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:05:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:05:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:05:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:05:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:05:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:05:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:05:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:05:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:05:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:05:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:05:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:05:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:05:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:05:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:05:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:05:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:05:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:05:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:05:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:05:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:05:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:05:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:05:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:05:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:05:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:06:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:06:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:06:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:06:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:06:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:06:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:06:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:06:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:06:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:06:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:06:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:06:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:06:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:06:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:06:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:06:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:06:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:06:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:06:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:06:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:06:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:06:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:06:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:06:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:06:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:06:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:06:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:06:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:06:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:06:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:06:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:06:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:06:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:06:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:06:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:06:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:06:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:06:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:06:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:06:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:06:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:06:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:06:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:06:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:06:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:06:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:06:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:06:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:06:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:06:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:06:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:06:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:06:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:06:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:06:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:06:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:06:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:06:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:06:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:06:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:06:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:06:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:06:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:06:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:06:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:06:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:06:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:06:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:06:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:06:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:06:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:06:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:06:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:06:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:06:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:06:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:06:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:06:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:06:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:06:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:06:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:06:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:06:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:06:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:06:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:06:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:06:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:06:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:06:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:06:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:06:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:06:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:06:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:06:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:06:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:06:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:06:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:06:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:06:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:06:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:06:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:06:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:06:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:06:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:06:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:06:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:06:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:06:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:06:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:06:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:06:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:06:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:06:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:06:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:06:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:06:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:06:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:06:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:06:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:06:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:06:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:06:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:06:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:06:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:06:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:06:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:06:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:06:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:06:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:06:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:06:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:06:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:06:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:06:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:06:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:06:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:06:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:06:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:06:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:06:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:06:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:06:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:06:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:06:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:06:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:06:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:06:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:06:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:06:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:06:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:06:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:06:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:06:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:06:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:06:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:06:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:06:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:06:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:06:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:06:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:06:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:06:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:06:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:06:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:06:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:06:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:06:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:06:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:06:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:06:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:06:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:06:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:06:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:06:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:06:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:06:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:06:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:06:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:06:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:06:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:06:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:06:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:06:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:06:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:06:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:06:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:06:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:06:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:06:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:06:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:06:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:06:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:06:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:06:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:06:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:06:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:06:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:06:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:06:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:06:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:06:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:06:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:06:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:06:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:06:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:06:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:06:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:06:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:06:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:06:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:06:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:06:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:06:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:06:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:06:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:06:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:07:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:07:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:07:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:07:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:07:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:07:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:07:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:07:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:07:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:07:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:07:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:07:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:07:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:07:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:07:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:07:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:07:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:07:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:07:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:07:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:07:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:07:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:07:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:07:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:07:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:07:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:07:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:07:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:07:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:07:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:07:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:07:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:07:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:07:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:07:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:07:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:07:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:07:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:07:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:07:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:07:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:07:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:07:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:07:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:07:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:07:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:07:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:07:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:07:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:07:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:07:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:07:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:07:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:07:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:07:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:07:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:07:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:07:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:07:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:07:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:07:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:07:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:07:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:07:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:07:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:07:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:07:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:07:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:07:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:07:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:07:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:07:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:07:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:07:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:07:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:07:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:07:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:07:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:07:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:07:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:07:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:07:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:07:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:07:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:07:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:07:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:07:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:07:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:07:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:07:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:07:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:07:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:07:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:07:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:07:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:07:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:07:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:07:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:07:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:07:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:07:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:07:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:07:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:07:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:07:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:07:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:07:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:07:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:07:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:07:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:07:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:07:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:07:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:07:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:07:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:07:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:07:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:07:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:07:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:07:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:07:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:07:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:07:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:07:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:07:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:07:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:07:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:07:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:07:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:07:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:07:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:07:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:07:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:07:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:07:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:07:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:07:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:07:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:07:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:07:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:07:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:07:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:07:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:07:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:07:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:07:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:07:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:07:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:07:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:07:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:07:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:07:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:07:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:07:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:07:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:07:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:07:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:07:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:07:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:07:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:07:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:07:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:07:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:07:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:07:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:07:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:07:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:07:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:07:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:07:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:07:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:07:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:07:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:07:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:07:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:07:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:07:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:07:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:07:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:07:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:07:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:07:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:07:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:07:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:07:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:07:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:07:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:07:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:07:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:07:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:07:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:07:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:07:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:07:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:07:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:07:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:07:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:07:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:07:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:07:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:07:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:07:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:07:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:07:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:07:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:07:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:07:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:07:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:07:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:07:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:07:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:07:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:07:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:07:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:07:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:07:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:08:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:08:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:08:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:08:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:08:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:08:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:08:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:08:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:08:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:08:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:08:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:08:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:08:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:08:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:08:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:08:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:08:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:08:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:08:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:08:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:08:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:08:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:08:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:08:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:08:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:08:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:08:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:08:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:08:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:08:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:08:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:08:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:08:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:08:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:08:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:08:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:08:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:08:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:08:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:08:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:08:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:08:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:08:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:08:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:08:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:08:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:08:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:08:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:08:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:08:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:08:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:08:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:08:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:08:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:08:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:08:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:08:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:08:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:08:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:08:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:08:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:08:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:08:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:08:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:08:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:08:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:08:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:08:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:08:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:08:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:08:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:08:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:08:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:08:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:08:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:08:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:08:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:08:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:08:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:08:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:08:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:08:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:08:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:08:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:08:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:08:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:08:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:08:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:08:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:08:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:08:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:08:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:08:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:08:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:08:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:08:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:08:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:08:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:08:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:08:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:08:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:08:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:08:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:08:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:08:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:08:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:08:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:08:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:08:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:08:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:08:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:08:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:08:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:08:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:08:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:08:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:08:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:08:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:08:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:08:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:08:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:08:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:08:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:08:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:08:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:08:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:08:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:08:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:08:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:08:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:08:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:08:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:08:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:08:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:08:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:08:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:08:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:08:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:08:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:08:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:08:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:08:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:08:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:08:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:08:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:08:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:08:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:08:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:08:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:08:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:08:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:08:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:08:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:08:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:08:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:08:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:08:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:08:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:08:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:08:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:08:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:08:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:08:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:08:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:08:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:08:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:08:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:08:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:08:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:08:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:08:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:08:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:08:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:08:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:08:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:08:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:08:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:08:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:08:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:08:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:08:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:08:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:08:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:08:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:08:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:08:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:08:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:08:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:08:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:08:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:08:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:08:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:08:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:08:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:08:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:08:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:08:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:08:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:08:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:08:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:08:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:08:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:08:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:08:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:08:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:08:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:08:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:08:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:08:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:08:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:08:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:08:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:08:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:08:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:08:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:08:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:09:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:09:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:09:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:09:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:09:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:09:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:09:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:09:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:09:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:09:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:09:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:09:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:09:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:09:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:09:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:09:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:09:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:09:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:09:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:09:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:09:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:09:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:09:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:09:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:09:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:09:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:09:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:09:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:09:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:09:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:09:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:09:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:09:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:09:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:09:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:09:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:09:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:09:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:09:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:09:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:09:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:09:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:09:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:09:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:09:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:09:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:09:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:09:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:09:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:09:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:09:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:09:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:09:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:09:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:09:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:09:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:09:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:09:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:09:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:09:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:09:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:09:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:09:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:09:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:09:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:09:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:09:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:09:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:09:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:09:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:09:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:09:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:09:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:09:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:09:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:09:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:09:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:09:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:09:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:09:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:09:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:09:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:09:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:09:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:09:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:09:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:09:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:09:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:09:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:09:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:09:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:09:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:09:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:09:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:09:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:09:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:09:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:09:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:09:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:09:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:09:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:09:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:09:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:09:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:09:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:09:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:09:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:09:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:09:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:09:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:09:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:09:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:09:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:09:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:09:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:09:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:09:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:09:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:09:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:09:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:09:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:09:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:09:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:09:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:09:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:09:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:11:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:11:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:11:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:11:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:11:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:11:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:11:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:11:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:11:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:11:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:11:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:11:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:11:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:11:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:11:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:11:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:11:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:11:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:11:43 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 02:11:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:11:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:11:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:11:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:11:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:11:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:11:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:11:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:11:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:11:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:11:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:11:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:11:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:11:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:11:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:11:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:11:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:11:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:11:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:11:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:11:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:11:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:11:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:11:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:11:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:11:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:11:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:11:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:11:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:11:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:11:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:11:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:11:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:11:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:11:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:11:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:11:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:11:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:11:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:11:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:11:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:11:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:11:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:11:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:11:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:11:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:11:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:11:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:11:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:11:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:11:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:11:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:11:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:11:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:12:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:12:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:12:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:12:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:12:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:12:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:12:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:12:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:12:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:12:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:12:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:12:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:12:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:12:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:12:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:12:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:12:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:12:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:12:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:12:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:12:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:12:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:12:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:12:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:12:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:12:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:12:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:12:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:12:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:12:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:12:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:12:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:12:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:12:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:12:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:12:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:12:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:12:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:12:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:12:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:12:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:12:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:12:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:12:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:12:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:12:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:12:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:12:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:12:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:12:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:12:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:12:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:12:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:12:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:12:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:12:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:12:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:12:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:12:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:12:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:12:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:12:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:12:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:12:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:12:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:12:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:12:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:12:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:12:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:12:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:12:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:12:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:12:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:12:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:12:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:12:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:12:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:12:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:12:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:12:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:12:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:12:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:12:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:12:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:12:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:12:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:12:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:12:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:12:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:12:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:12:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:12:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:12:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:12:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:12:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:12:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:12:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:12:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:12:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:12:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:12:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:12:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:12:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:12:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:12:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:12:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:12:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:12:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:12:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:12:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:12:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:12:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:12:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:12:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:12:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:12:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:12:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:12:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:12:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:12:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:12:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:12:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:12:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:12:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:12:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:12:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:12:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:12:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:12:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:12:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:12:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:12:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:12:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:12:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:12:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:12:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:12:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:12:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:12:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:12:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:12:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:12:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:12:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:12:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:12:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:12:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:12:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:12:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:12:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:12:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:12:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:12:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:12:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:12:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:12:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:12:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:12:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:12:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:12:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:12:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:12:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:12:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:12:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:12:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:12:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:12:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:12:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:12:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:12:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:12:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:12:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:12:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:12:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:12:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:12:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:12:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:12:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:12:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:12:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:12:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:12:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:12:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:12:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:12:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:12:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:12:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:12:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:12:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:12:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:12:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:12:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:12:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:12:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:12:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:12:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:12:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:12:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:12:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:12:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:12:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:12:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:12:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:12:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:12:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:12:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:12:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:12:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:12:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:12:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:12:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:12:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:12:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:12:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:12:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:12:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:12:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:13:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:13:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:13:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:13:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:13:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:13:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:13:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:13:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:13:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:13:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:13:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:13:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:13:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:13:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:13:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:13:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:13:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:13:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:13:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:13:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:13:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:13:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:13:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:13:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:13:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:13:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:13:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:13:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:13:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:13:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:13:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:13:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:13:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:13:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:13:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:13:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:13:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:13:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:13:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:13:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:13:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:13:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:13:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:13:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:13:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:13:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:13:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:13:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:13:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:13:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:13:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:13:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:13:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:13:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:13:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:13:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:13:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:13:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:13:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:13:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:13:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:13:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:13:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:13:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:13:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:13:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:13:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:13:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:13:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:13:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:13:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:13:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:13:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:13:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:13:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:13:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:13:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:13:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:13:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:13:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:13:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:13:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:13:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:13:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:13:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:13:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:13:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:13:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:13:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:13:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:13:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:13:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:13:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:13:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:13:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:13:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:13:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:13:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:13:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:13:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:13:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:13:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:13:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:13:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:13:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:13:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:13:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:13:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:13:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:13:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:13:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:13:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:13:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:13:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:13:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:13:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:13:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:13:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:13:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:13:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:13:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:13:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:13:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:13:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:13:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:13:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:13:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:13:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:13:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:13:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:13:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:13:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:13:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:13:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:13:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:13:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:13:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:13:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:13:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:13:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:13:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:13:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:13:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:13:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:13:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:13:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:13:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:13:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:13:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:13:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:13:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:13:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:13:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:13:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:13:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:13:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:13:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:13:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:13:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:13:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:13:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:13:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:13:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:13:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:13:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:13:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:13:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:13:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:13:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:13:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:13:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:13:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:13:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:13:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:13:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:13:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:13:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:13:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:13:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:13:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:13:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:13:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:13:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:13:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:13:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:13:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:13:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:13:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:13:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:13:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:13:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:13:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:13:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:13:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:13:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:13:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:13:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:13:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:13:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:13:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:13:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:13:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:13:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:13:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:13:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:13:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:13:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:13:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:13:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:13:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:13:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:13:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:13:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:13:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:13:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:13:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:14:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:14:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:14:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:14:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:14:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:14:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:14:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:14:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:14:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:14:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:14:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:14:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:14:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:14:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:14:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:14:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:14:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:14:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:14:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:14:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:14:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:14:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:14:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:14:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:14:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:14:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:14:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:14:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:14:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:14:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:14:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:14:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:14:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:14:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:14:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:14:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:14:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:14:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:14:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:14:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:14:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:14:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:14:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:14:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:14:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:14:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:14:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:14:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:14:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:14:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:14:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:14:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:14:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:14:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:14:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:14:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:14:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:14:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:14:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:14:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:14:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:14:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:14:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:14:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:14:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:14:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:14:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:14:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:14:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:14:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:14:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:14:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:14:18 rdma06 systemd[1]: Starting dnf makecache...
Apr 18 02:14:18 rdma06 dnf[1175482]: Updating Subscription Management repositories.
Apr 18 02:14:18 rdma06 dnf[1175482]: Unable to read consumer identity
Apr 18 02:14:18 rdma06 dnf[1175482]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 18 02:14:18 rdma06 dnf[1175482]: Metadata cache refreshed recently.
Apr 18 02:14:18 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 18 02:14:18 rdma06 systemd[1]: Finished dnf makecache.
Apr 18 02:14:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:14:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:14:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:14:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:14:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:14:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:14:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:14:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:14:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:14:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:14:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:14:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:14:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:14:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:14:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:14:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:14:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:14:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:14:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:14:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:14:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:14:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:14:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:14:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:14:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:14:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:14:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:14:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:14:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:14:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:14:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:14:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:14:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:14:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:14:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:14:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:14:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:14:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:14:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:14:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:14:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:14:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:14:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:14:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:14:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:14:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:14:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:14:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:14:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:14:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:14:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:14:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:14:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:14:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:14:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:14:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:14:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:14:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:14:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:14:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:14:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:14:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:14:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:14:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:14:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:14:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:14:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:14:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:14:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:14:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:14:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:14:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:14:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:14:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:14:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:14:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:14:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:14:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:14:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:14:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:14:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:14:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:14:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:14:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:14:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:14:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:14:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:14:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:14:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:14:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:14:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:14:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:14:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:14:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:14:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:14:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:14:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:14:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:14:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:14:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:14:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:14:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:14:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:14:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:14:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:14:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:14:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:14:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:14:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:14:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:14:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:14:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:14:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:14:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:14:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:14:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:14:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:14:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:14:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:14:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:14:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:14:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:14:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:14:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:14:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:14:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:14:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:14:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:14:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:14:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:14:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:14:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:14:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:14:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:14:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:14:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:14:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:14:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:14:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:14:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:14:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:14:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:14:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:14:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:15:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:15:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:15:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:15:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:15:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:15:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:15:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:15:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:15:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:15:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:15:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:15:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:15:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:15:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:15:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:15:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:15:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:15:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:15:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:15:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:15:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:15:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:15:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:15:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:15:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:15:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:15:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:15:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:15:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:15:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:15:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:15:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:15:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:15:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:15:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:15:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:15:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:15:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:15:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:15:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:15:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:15:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:15:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:15:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:15:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:15:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:15:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:15:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:15:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:15:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:15:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:15:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:15:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:15:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:15:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:15:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:15:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:15:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:15:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:15:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:15:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:15:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:15:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:15:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:15:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:15:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:15:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:15:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:15:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:15:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:15:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:15:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:15:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:15:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:15:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:15:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:15:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:15:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:15:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:15:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:15:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:15:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:15:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:15:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:15:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:15:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:15:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:15:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:15:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:15:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:15:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:15:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:15:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:15:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:15:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:15:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:15:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:15:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:15:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:15:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:15:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:15:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:15:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:15:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:15:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:15:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:15:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:15:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:15:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:15:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:15:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:15:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:15:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:15:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:15:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:15:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:15:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:15:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:15:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:15:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:15:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:15:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:15:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:15:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:15:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:15:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:15:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:15:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:15:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:15:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:15:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:15:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:15:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:15:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:15:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:15:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:15:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:15:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:15:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:15:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:15:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:15:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:15:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:15:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:15:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:15:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:15:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:15:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:15:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:15:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:15:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:15:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:15:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:15:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:15:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:15:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:15:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:15:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:15:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:15:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:15:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:15:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:15:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:15:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:15:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:15:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:15:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:15:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:15:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:15:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:15:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:15:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:15:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:15:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:15:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:15:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:15:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:15:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:15:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:15:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:15:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:15:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:15:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:15:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:15:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:15:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:15:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:15:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:15:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:15:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:15:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:15:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:15:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:15:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:15:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:15:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:15:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:15:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:15:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:15:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:15:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:15:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:15:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:15:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:15:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:15:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:15:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:15:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:15:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:15:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:15:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:15:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:15:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:15:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:15:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:15:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:16:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:16:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:16:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:16:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:16:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:16:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:16:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:16:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:16:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:16:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:16:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:16:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:16:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:16:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:16:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:16:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:16:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:16:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:16:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:16:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:16:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:16:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:16:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:16:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:16:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:16:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:16:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:16:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:16:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:16:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:16:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:16:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:16:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:16:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:16:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:16:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:16:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:16:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:16:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:16:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:16:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:16:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:16:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:16:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:16:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:16:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:16:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:16:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:16:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:16:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:16:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:16:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:16:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:16:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:16:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:16:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:16:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:16:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:16:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:16:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:16:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:16:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:16:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:16:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:16:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:16:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:16:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:16:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:16:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:16:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:16:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:16:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:16:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:16:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:16:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:16:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:16:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:16:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:16:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:16:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:16:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:16:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:16:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:16:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:16:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:16:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:16:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:16:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:16:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:16:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:16:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:16:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:16:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:16:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:16:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:16:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:16:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:16:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:16:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:16:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:16:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:16:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:16:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:16:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:16:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:16:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:16:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:16:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:16:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:16:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:16:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:16:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:16:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:16:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:16:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:16:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:16:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:16:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:16:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:16:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:16:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:16:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:16:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:16:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:16:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:16:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:16:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:16:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:16:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:16:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:16:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:16:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:16:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:16:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:16:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:16:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:16:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:16:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:16:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:16:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:16:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:16:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:16:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:16:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:16:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:16:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:16:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:16:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:16:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:16:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:16:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:16:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:16:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:16:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:16:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:16:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:16:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:16:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:16:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:16:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:16:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:16:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:16:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:16:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:16:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:16:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:16:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:16:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:16:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:16:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:16:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:16:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:16:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:16:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:16:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:16:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:16:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:16:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:16:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:16:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:16:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:16:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:16:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:16:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:16:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:16:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:16:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:16:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:16:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:16:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:16:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:16:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:16:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:16:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:16:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:16:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:16:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:16:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:16:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:16:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:16:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:16:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:16:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:16:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:16:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:16:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:16:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:16:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:16:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:16:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:16:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:16:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:16:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:16:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:16:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:16:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:17:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:17:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:17:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:17:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:17:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:17:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:17:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:17:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:17:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:17:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:17:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:17:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:17:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:17:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:17:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:17:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:17:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:17:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:17:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:17:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:17:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:17:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:17:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:17:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:17:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:17:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:17:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:17:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:17:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:17:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:17:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:17:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:17:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:17:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:17:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:17:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:17:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:17:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:17:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:17:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:17:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:17:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:17:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:17:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:17:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:17:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:17:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:17:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:17:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:17:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:17:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:17:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:17:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:17:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:17:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:17:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:17:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:17:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:17:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:17:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:17:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:17:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:17:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:17:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:17:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:17:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:17:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:17:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:17:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:17:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:17:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:17:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:17:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:17:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:17:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:17:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:17:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:17:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:17:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:17:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:17:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:17:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:17:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:17:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:17:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:17:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:17:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:17:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:17:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:17:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:17:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:17:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:17:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:17:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:17:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:17:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:17:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:17:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:17:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:17:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:17:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:17:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:17:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:17:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:17:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:17:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:17:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:17:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:17:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:17:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:17:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:17:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:17:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:17:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:17:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:17:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:17:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:17:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:17:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:17:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:17:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:17:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:17:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:17:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:17:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:17:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:17:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:17:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:17:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:17:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:17:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:17:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:17:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:17:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:17:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:17:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:17:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:17:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:17:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:17:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:17:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:17:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:17:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:17:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:17:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:17:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:17:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:17:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:17:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:17:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:17:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:17:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:17:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:17:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:17:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:17:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:17:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:17:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:17:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:17:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:17:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:17:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:17:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:17:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:17:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:17:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:17:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:17:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:17:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:17:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:17:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:17:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:17:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:17:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:17:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:17:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:17:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:17:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:17:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:17:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:17:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:17:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:17:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:17:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:17:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:17:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:17:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:17:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:17:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:17:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:17:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:17:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:17:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:17:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:17:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:17:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:17:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:17:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:17:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:17:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:17:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:17:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:17:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:17:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:17:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:17:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:17:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:17:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:17:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:17:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:17:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:17:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:17:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:17:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:17:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:17:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:18:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:18:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:18:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:18:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:18:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:18:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:18:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:18:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:18:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:18:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:18:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:18:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:18:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:18:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:18:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:18:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:18:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:18:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:18:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:18:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:18:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:18:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:18:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:18:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:18:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:18:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:18:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:18:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:18:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:18:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:18:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:18:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:18:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:18:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:18:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:18:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:18:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:18:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:18:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:18:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:18:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:18:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:18:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:18:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:18:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:18:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:18:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:18:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:18:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:18:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:18:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:18:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:18:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:18:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:18:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:18:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:18:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:18:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:18:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:18:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:18:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:18:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:18:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:18:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:18:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:18:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:18:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:18:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:18:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:18:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:18:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:18:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:18:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:18:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:18:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:18:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:18:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:18:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:18:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:18:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:18:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:18:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:18:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:18:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:18:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:18:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:18:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:18:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:18:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:18:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:18:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:18:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:18:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:18:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:18:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:18:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:18:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:18:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:18:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:18:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:18:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:18:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:18:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:18:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:18:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:18:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:18:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:18:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:18:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:18:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:18:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:18:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:18:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:18:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:18:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:18:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:18:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:18:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:18:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:18:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:18:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:18:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:18:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:18:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:18:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:18:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:18:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:18:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:18:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:18:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:18:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:18:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:18:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:18:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:18:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:18:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:18:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:18:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:18:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:18:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:18:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:18:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:18:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:18:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:18:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:18:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:18:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:18:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:18:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:18:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:18:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:18:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:18:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:18:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:18:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:18:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:18:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:18:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:18:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:18:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:18:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:18:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:18:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:18:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:18:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:18:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:18:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:18:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:18:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:18:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:18:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:18:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:18:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:18:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:18:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:18:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:18:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:18:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:18:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:18:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:18:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:18:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:18:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:18:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:18:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:18:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:18:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:18:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:18:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:18:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:18:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:18:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:18:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:18:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:18:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:18:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:18:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:18:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:18:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:18:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:18:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:18:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:18:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:18:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:18:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:18:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:18:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:18:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:18:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:18:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:18:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:18:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:18:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:18:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:18:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:18:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:19:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:19:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:19:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:19:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:19:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:19:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:19:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:19:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:19:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:19:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:19:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:19:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:19:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:19:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:19:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:19:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:19:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:19:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:19:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:19:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:19:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:19:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:19:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:19:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:19:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:19:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:19:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:19:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:19:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:19:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:19:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:19:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:19:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:19:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:19:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:19:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:19:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:19:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:19:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:19:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:19:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:19:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:19:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:19:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:19:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:19:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:19:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:19:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:19:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:19:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:19:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:19:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:19:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:19:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:19:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:19:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:19:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:19:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:19:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:19:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:19:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:19:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:19:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:19:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:19:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:19:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:19:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:19:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:19:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:19:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:19:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:19:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:19:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:19:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:19:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:19:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:19:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:19:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:19:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:19:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:19:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:19:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:19:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:19:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:19:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:19:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:19:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:19:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:19:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:19:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:19:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:19:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:19:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:19:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:19:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:19:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:19:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:19:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:19:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:19:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:19:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:19:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:19:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:19:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:19:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:19:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:19:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:19:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:19:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:19:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:19:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:19:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:19:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:19:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:19:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:19:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:19:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:19:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:19:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:19:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:19:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:19:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:19:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:19:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:19:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:19:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:19:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:19:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:19:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:19:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:19:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:19:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:19:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:19:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:19:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:19:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:19:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:19:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:19:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:19:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:19:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:19:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:19:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:19:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:21:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:21:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:21:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:21:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:21:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:21:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:21:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:21:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:21:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:21:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:21:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:21:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:21:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:21:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:21:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:21:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:21:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:21:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:21:48 rdma06 rsyslogd[1041]: imjournal: 5298 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 02:21:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:21:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:21:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:21:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:21:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:21:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:21:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:21:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:21:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:21:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:21:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:21:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:21:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:21:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:21:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:21:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:21:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:21:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:21:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:21:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:21:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:21:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:21:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:21:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:21:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:21:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:21:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:21:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:21:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:21:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:21:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:21:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:21:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:21:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:21:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:21:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:22:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:22:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:22:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:22:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:22:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:22:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:22:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:22:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:22:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:22:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:22:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:22:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:22:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:22:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:22:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:22:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:22:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:22:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:22:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:22:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:22:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:22:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:22:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:22:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:22:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:22:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:22:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:22:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:22:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:22:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:22:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:22:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:22:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:22:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:22:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:22:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:22:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:22:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:22:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:22:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:22:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:22:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:22:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:22:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:22:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:22:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:22:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:22:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:22:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:22:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:22:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:22:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:22:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:22:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:22:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:22:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:22:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:22:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:22:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:22:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:22:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:22:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:22:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:22:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:22:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:22:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:22:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:22:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:22:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:22:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:22:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:22:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:22:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:22:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:22:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:22:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:22:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:22:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:22:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:22:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:22:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:22:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:22:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:22:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:22:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:22:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:22:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:22:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:22:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:22:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:22:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:22:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:22:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:22:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:22:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:22:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:22:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:22:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:22:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:22:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:22:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:22:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:22:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:22:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:22:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:22:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:22:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:22:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:22:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:22:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:22:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:22:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:22:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:22:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:22:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:22:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:22:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:22:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:22:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:22:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:22:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:22:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:22:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:22:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:22:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:22:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:22:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:22:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:22:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:22:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:22:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:22:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:22:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:22:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:22:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:22:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:22:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:22:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:22:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:22:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:22:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:22:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:22:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:22:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:22:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:22:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:22:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:22:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:22:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:22:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:22:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:22:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:22:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:22:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:22:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:22:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:22:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:22:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:22:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:22:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:22:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:22:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:22:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:22:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:22:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:22:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:22:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:22:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:22:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:22:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:22:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:22:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:22:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:22:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:22:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:22:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:22:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:22:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:22:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:22:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:22:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:22:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:22:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:22:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:22:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:22:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:22:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:22:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:22:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:22:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:22:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:22:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:22:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:22:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:22:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:22:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:22:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:22:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:22:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:22:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:22:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:22:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:22:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:22:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:22:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:22:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:22:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:22:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:22:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:22:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:22:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:22:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:22:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:22:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:22:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:22:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:23:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:23:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:23:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:23:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:23:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:23:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:23:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:23:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:23:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:23:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:23:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:23:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:23:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:23:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:23:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:23:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:23:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:23:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:23:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:23:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:23:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:23:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:23:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:23:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:23:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:23:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:23:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:23:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:23:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:23:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:23:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:23:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:23:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:23:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:23:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:23:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:23:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:23:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:23:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:23:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:23:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:23:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:23:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:23:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:23:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:23:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:23:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:23:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:23:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:23:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:23:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:23:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:23:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:23:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:23:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:23:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:23:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:23:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:23:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:23:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:23:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:23:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:23:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:23:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:23:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:23:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:23:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:23:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:23:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:23:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:23:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:23:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:23:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:23:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:23:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:23:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:23:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:23:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:23:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:23:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:23:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:23:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:23:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:23:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:23:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:23:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:23:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:23:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:23:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:23:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:23:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:23:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:23:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:23:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:23:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:23:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:23:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:23:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:23:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:23:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:23:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:23:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:23:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:23:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:23:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:23:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:23:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:23:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:23:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:23:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:23:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:23:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:23:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:23:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:23:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:23:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:23:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:23:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:23:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:23:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:23:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:23:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:23:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:23:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:23:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:23:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:23:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:23:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:23:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:23:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:23:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:23:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:23:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:23:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:23:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:23:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:23:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:23:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:23:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:23:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:23:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:23:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:23:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:23:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:23:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:23:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:23:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:23:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:23:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:23:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:23:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:23:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:23:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:23:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:23:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:23:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:23:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:23:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:23:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:23:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:23:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:23:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:23:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:23:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:23:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:23:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:24:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:24:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:24:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:24:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:24:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:24:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:24:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:24:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:24:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:24:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:24:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:24:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:24:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:24:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:24:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:24:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:24:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:24:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:24:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:24:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:24:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:24:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:24:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:24:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:24:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:24:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:24:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:24:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:24:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:24:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:24:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:24:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:24:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:24:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:24:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:24:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:24:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:24:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:24:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:24:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:24:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:24:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:24:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:24:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:24:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:24:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:24:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:24:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:24:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:24:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:24:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:24:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:24:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:24:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:24:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:24:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:24:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:24:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:24:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:24:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:24:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:24:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:24:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:24:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:24:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:24:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:24:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:24:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:24:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:24:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:24:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:24:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:24:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:24:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:24:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:24:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:24:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:24:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:24:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:24:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:24:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:24:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:24:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:24:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:24:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:24:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:24:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:24:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:24:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:24:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:24:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:24:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:24:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:24:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:24:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:24:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:24:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:24:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:24:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:24:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:24:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:24:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:24:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:24:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:24:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:24:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:24:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:24:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:24:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:24:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:24:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:24:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:24:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:24:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:24:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:24:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:24:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:24:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:24:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:24:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:24:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:24:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:24:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:24:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:24:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:24:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:24:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:24:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:24:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:24:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:24:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:24:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:24:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:24:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:24:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:24:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:24:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:24:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:24:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:24:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:24:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:24:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:24:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:24:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:24:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:24:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:24:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:24:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:24:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:24:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:24:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:24:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:24:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:24:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:24:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:24:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:24:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:24:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:24:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:24:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:24:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:24:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:24:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:24:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:24:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:24:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:24:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:24:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:24:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:24:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:24:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:24:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:24:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:24:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:24:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:24:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:24:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:24:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:24:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:24:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:24:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:24:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:24:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:24:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:24:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:24:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:24:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:24:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:24:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:24:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:24:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:24:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:24:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:24:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:24:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:24:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:25:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:25:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:25:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:25:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:25:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:25:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:25:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:25:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:25:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:25:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:25:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:25:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:25:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:25:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:25:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:25:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:25:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:25:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:25:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:25:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:25:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:25:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:25:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:25:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:25:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:25:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:25:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:25:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:25:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:25:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:25:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:25:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:25:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:25:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:25:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:25:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:25:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:25:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:25:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:25:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:25:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:25:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:25:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:25:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:25:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:25:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:25:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:25:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:25:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:25:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:25:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:25:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:25:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:25:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:25:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:25:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:25:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:25:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:25:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:25:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:25:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:25:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:25:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:25:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:25:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:25:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:25:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:25:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:25:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:25:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:25:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:25:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:25:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:25:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:25:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:25:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:25:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:25:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:25:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:25:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:25:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:25:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:25:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:25:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:25:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:25:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:25:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:25:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:25:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:25:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:25:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:25:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:25:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:25:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:25:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:25:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:25:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:25:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:25:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:25:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:25:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:25:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:25:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:25:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:25:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:25:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:25:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:25:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:25:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:25:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:25:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:25:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:25:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:25:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:25:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:25:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:25:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:25:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:25:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:25:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:25:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:25:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:25:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:25:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:25:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:25:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:25:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:25:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:25:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:25:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:25:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:25:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:25:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:25:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:25:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:25:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:25:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:25:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:25:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:25:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:25:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:25:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:25:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:25:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:25:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:25:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:25:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:25:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:25:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:25:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:25:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:25:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:25:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:25:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:25:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:25:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:25:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:25:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:25:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:25:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:25:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:25:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:25:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:25:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:25:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:25:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:26:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:26:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:26:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:26:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:26:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:26:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:26:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:26:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:26:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:26:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:26:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:26:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:26:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:26:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:26:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:26:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:26:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:26:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:26:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:26:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:26:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:26:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:26:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:26:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:26:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:26:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:26:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:26:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:26:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:26:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:26:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:26:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:26:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:26:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:26:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:26:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:26:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:26:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:26:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:26:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:26:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:26:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:26:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:26:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:26:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:26:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:26:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:26:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:26:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:26:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:26:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:26:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:26:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:26:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:26:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:26:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:26:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:26:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:26:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:26:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:26:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:26:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:26:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:26:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:26:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:26:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:26:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:26:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:26:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:26:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:26:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:26:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:26:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:26:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:26:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:26:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:26:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:26:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:26:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:26:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:26:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:26:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:26:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:26:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:26:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:26:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:26:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:26:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:26:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:26:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:26:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:26:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:26:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:26:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:26:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:26:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:26:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:26:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:26:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:26:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:26:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:26:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:26:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:26:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:26:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:26:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:26:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:26:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:26:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:26:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:26:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:26:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:26:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:26:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:26:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:26:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:26:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:26:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:26:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:26:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:26:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:26:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:26:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:26:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:26:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:26:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:26:33 rdma06 systemd-journald[790]: Data hash table of /run/log/journal/95379d2f723847b68d9e976dee8967a9/system.journal has a fill level at 75.0 (163750 of 218332 items, 75497472 file size, 461 bytes per hash table item), suggesting rotation.
Apr 18 02:26:33 rdma06 systemd-journald[790]: /run/log/journal/95379d2f723847b68d9e976dee8967a9/system.journal: Journal header limits reached or header out-of-date, rotating.
Apr 18 02:26:33 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 18 02:26:33 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 18 02:26:34 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 18 02:26:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:26:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:26:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:26:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:26:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:26:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:26:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:26:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:26:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:26:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:26:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:26:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:26:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:26:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:26:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:26:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:26:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:26:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:26:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:26:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:26:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:26:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:26:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:26:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:26:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:26:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:26:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:26:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:26:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:26:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:26:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:26:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:26:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:26:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:26:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:26:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:26:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:26:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:26:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:26:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:26:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:26:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:26:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:26:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:26:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:26:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:26:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:26:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:26:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:26:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:26:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:26:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:26:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:26:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:26:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:26:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:26:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:26:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:26:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:26:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:26:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:26:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:26:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:26:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:26:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:26:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:26:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:26:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:26:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:26:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:26:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:26:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:26:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:26:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:26:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:26:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:26:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:26:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:26:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:26:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:26:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:26:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:26:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:26:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:26:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:26:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:26:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:26:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:26:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:26:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:27:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:27:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:27:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:27:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:27:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:27:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:27:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:27:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:27:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:27:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:27:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:27:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:27:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:27:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:27:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:27:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:27:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:27:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:27:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:27:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:27:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:27:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:27:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:27:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:27:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:27:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:27:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:27:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:27:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:27:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:27:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:27:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:27:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:27:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:27:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:27:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:27:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:27:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:27:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:27:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:27:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:27:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:27:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:27:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:27:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:27:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:27:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:27:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:27:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:27:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:27:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:27:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:27:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:27:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:27:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:27:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:27:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:27:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:27:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:27:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:27:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:27:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:27:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:27:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:27:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:27:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:27:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:27:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:27:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:27:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:27:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:27:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:27:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:27:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:27:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:27:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:27:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:27:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:27:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:27:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:27:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:27:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:27:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:27:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:27:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:27:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:27:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:27:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:27:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:27:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:27:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:27:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:27:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:27:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:27:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:27:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:27:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:27:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:27:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:27:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:27:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:27:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:27:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:27:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:27:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:27:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:27:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:27:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:27:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:27:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:27:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:27:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:27:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:27:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:27:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:27:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:27:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:27:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:27:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:27:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:27:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:27:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:27:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:27:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:27:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:27:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:27:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:27:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:27:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:27:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:27:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:27:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:27:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:27:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:27:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:27:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:27:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:27:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:27:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:27:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:27:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:27:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:27:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:27:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:27:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:27:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:27:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:27:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:27:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:27:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:27:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:27:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:27:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:27:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:27:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:27:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:27:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:27:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:27:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:27:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:27:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:27:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:27:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:27:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:27:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:27:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:27:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:27:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:27:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:27:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:27:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:27:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:27:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:27:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:27:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:27:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:27:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:27:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:27:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:27:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:27:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:27:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:27:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:27:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:27:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:27:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:27:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:27:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:27:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:27:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:27:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:27:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:27:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:27:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:27:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:27:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:27:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:27:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:27:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:27:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:27:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:27:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:27:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:27:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:27:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:27:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:27:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:27:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:27:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:27:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:27:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:27:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:27:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:27:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:27:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:27:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:28:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:28:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:28:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:28:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:28:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:28:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:28:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:28:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:28:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:28:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:28:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:28:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:28:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:28:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:28:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:28:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:28:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:28:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:28:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:28:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:28:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:28:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:28:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:28:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:28:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:28:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:28:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:28:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:28:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:28:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:28:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:28:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:28:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:28:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:28:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:28:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:28:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:28:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:28:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:28:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:28:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:28:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:28:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:28:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:28:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:28:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:28:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:28:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:28:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:28:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:28:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:28:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:28:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:28:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:28:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:28:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:28:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:28:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:28:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:28:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:28:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:28:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:28:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:28:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:28:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:28:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:28:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:28:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:28:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:28:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:28:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:28:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:28:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:28:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:28:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:28:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:28:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:28:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:28:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:28:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:28:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:28:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:28:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:28:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:28:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:28:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:28:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:28:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:28:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:28:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:28:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:28:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:28:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:28:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:28:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:28:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:28:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:28:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:28:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:28:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:28:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:28:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:28:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:28:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:28:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:28:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:28:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:28:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:28:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:28:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:28:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:28:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:28:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:28:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:28:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:28:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:28:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:28:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:28:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:28:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:28:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:28:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:28:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:28:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:28:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:28:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:28:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:28:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:28:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:28:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:28:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:28:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:28:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:28:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:28:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:28:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:28:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:28:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:28:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:28:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:28:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:28:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:28:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:28:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:28:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:28:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:28:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:28:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:28:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:28:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:28:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:28:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:28:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:28:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:28:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:28:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:28:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:28:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:28:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:28:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:28:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:28:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:28:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:28:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:28:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:28:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:28:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:28:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:28:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:28:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:28:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:28:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:28:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:28:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:28:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:28:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:28:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:28:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:28:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:28:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:28:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:28:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:28:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:28:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:28:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:28:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:28:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:28:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:28:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:28:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:28:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:28:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:28:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:28:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:28:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:28:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:28:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:28:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:28:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:28:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:28:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:28:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:28:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:28:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:28:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:28:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:28:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:28:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:28:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:28:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:28:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:28:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:28:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:28:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:28:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:28:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:29:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:29:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:29:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:29:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:29:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:29:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:29:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:29:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:29:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:29:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:29:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:29:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:29:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:29:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:29:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:29:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:29:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:29:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:29:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:29:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:29:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:29:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:29:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:29:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:29:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:29:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:29:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:29:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:29:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:29:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:29:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:29:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:29:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:29:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:29:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:29:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:29:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:29:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:29:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:29:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:29:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:29:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:29:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:29:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:29:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:29:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:29:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:29:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:29:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:29:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:29:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:29:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:29:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:29:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:29:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:29:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:29:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:29:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:29:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:29:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:29:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:29:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:29:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:29:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:29:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:29:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:29:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:29:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:29:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:29:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:29:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:29:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:29:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:29:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:29:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:29:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:29:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:29:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:29:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:29:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:29:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:29:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:29:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:29:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:29:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:29:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:29:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:29:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:29:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:29:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:29:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:29:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:29:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:29:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:29:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:29:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:29:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:29:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:29:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:29:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:29:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:29:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:29:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:29:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:29:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:29:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:29:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:29:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:29:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:29:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:29:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:29:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:29:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:29:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:29:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:29:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:29:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:29:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:29:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:29:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:29:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:29:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:29:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:29:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:29:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:29:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:29:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:29:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:29:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:29:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:29:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:29:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:29:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:29:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:29:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:29:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:29:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:29:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:29:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:29:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:29:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:29:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:29:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:29:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:29:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:29:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:29:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:29:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:29:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:29:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:29:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:29:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:29:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:29:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:29:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:29:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:29:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:29:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:29:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:29:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:29:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:29:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:31:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:31:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:31:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:31:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:31:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:31:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:31:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:31:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:31:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:31:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:31:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:31:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:31:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:31:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:31:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:31:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:31:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:31:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:31:49 rdma06 rsyslogd[1041]: imjournal: 5087 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 02:31:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:31:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:31:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:31:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:31:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:31:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:31:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:31:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:31:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:31:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:31:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:31:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:31:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:31:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:31:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:31:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:31:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:31:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:31:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:31:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:31:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:31:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:31:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:31:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:31:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:31:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:31:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:31:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:31:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:31:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:31:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:31:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:31:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:31:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:31:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:31:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:32:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:32:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:32:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:32:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:32:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:32:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:32:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:32:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:32:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:32:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:32:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:32:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:32:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:32:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:32:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:32:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:32:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:32:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:32:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:32:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:32:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:32:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:32:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:32:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:32:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:32:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:32:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:32:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:32:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:32:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:32:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:32:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:32:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:32:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:32:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:32:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:32:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:32:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:32:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:32:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:32:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:32:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:32:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:32:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:32:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:32:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:32:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:32:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:32:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:32:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:32:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:32:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:32:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:32:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:32:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:32:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:32:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:32:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:32:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:32:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:32:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:32:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:32:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:32:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:32:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:32:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:32:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:32:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:32:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:32:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:32:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:32:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:32:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:32:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:32:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:32:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:32:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:32:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:32:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:32:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:32:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:32:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:32:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:32:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:32:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:32:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:32:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:32:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:32:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:32:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:32:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:32:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:32:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:32:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:32:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:32:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:32:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:32:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:32:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:32:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:32:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:32:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:32:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:32:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:32:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:32:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:32:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:32:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:32:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:32:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:32:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:32:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:32:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:32:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:32:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:32:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:32:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:32:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:32:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:32:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:32:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:32:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:32:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:32:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:32:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:32:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:32:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:32:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:32:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:32:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:32:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:32:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:32:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:32:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:32:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:32:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:32:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:32:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:32:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:32:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:32:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:32:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:32:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:32:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:32:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:32:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:32:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:32:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:32:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:32:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:32:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:32:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:32:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:32:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:32:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:32:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:32:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:32:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:32:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:32:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:32:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:32:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:32:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:32:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:32:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:32:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:32:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:32:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:32:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:32:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:32:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:32:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:32:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:32:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:32:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:32:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:32:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:32:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:32:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:32:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:32:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:32:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:32:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:32:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:32:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:32:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:32:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:32:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:32:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:32:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:32:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:32:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:32:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:32:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:32:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:32:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:32:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:32:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:32:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:32:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:32:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:32:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:32:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:32:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:32:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:32:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:32:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:32:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:32:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:32:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:32:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:32:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:32:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:32:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:32:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:32:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:33:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:33:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:33:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:33:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:33:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:33:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:33:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:33:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:33:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:33:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:33:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:33:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:33:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:33:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:33:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:33:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:33:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:33:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:33:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:33:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:33:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:33:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:33:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:33:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:33:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:33:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:33:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:33:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:33:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:33:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:33:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:33:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:33:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:33:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:33:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:33:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:33:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:33:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:33:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:33:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:33:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:33:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:33:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:33:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:33:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:33:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:33:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:33:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:33:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:33:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:33:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:33:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:33:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:33:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:33:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:33:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:33:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:33:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:33:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:33:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:33:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:33:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:33:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:33:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:33:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:33:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:33:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:33:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:33:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:33:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:33:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:33:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:33:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:33:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:33:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:33:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:33:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:33:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:33:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:33:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:33:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:33:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:33:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:33:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:33:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:33:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:33:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:33:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:33:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:33:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:33:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:33:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:33:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:33:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:33:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:33:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:33:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:33:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:33:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:33:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:33:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:33:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:33:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:33:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:33:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:33:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:33:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:33:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:33:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:33:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:33:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:33:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:33:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:33:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:33:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:33:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:33:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:33:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:33:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:33:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:33:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:33:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:33:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:33:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:33:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:33:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:33:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:33:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:33:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:33:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:33:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:33:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:33:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:33:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:33:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:33:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:33:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:33:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:33:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:33:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:33:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:33:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:33:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:33:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:33:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:33:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:33:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:33:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:33:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:33:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:33:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:33:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:33:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:33:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:33:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:33:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:33:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:33:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:33:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:33:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:33:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:33:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:33:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:33:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:33:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:33:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:33:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:33:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:33:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:33:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:33:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:33:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:33:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:33:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:33:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:33:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:33:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:33:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:33:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:33:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:33:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:33:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:33:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:33:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:33:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:33:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:33:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:33:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:33:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:33:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:33:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:33:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:33:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:33:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:33:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:33:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:33:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:33:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:33:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:33:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:33:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:33:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:33:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:33:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:33:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:33:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:33:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:33:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:33:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:33:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:33:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:33:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:33:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:33:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:33:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:33:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:34:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:34:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:34:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:34:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:34:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:34:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:34:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:34:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:34:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:34:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:34:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:34:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:34:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:34:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:34:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:34:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:34:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:34:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:34:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:34:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:34:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:34:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:34:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:34:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:34:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:34:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:34:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:34:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:34:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:34:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:34:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:34:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:34:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:34:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:34:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:34:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:34:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:34:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:34:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:34:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:34:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:34:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:34:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:34:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:34:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:34:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:34:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:34:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:34:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:34:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:34:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:34:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:34:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:34:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:34:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:34:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:34:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:34:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:34:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:34:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:34:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:34:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:34:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:34:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:34:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:34:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:34:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:34:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:34:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:34:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:34:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:34:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:34:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:34:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:34:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:34:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:34:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:34:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:34:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:34:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:34:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:34:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:34:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:34:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:34:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:34:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:34:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:34:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:34:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:34:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:34:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:34:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:34:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:34:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:34:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:34:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:34:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:34:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:34:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:34:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:34:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:34:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:34:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:34:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:34:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:34:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:34:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:34:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:34:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:34:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:34:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:34:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:34:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:34:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:34:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:34:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:34:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:34:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:34:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:34:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:34:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:34:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:34:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:34:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:34:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:34:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:34:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:34:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:34:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:34:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:34:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:34:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:34:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:34:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:34:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:34:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:34:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:34:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:34:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:34:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:34:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:34:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:34:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:34:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:34:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:34:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:34:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:34:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:34:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:34:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:34:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:34:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:34:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:34:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:34:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:34:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:34:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:34:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:34:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:34:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:34:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:34:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:34:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:34:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:34:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:34:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:34:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:34:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:34:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:34:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:34:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:34:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:34:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:34:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:34:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:34:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:34:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:34:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:34:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:34:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:34:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:34:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:34:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:34:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:34:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:34:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:34:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:34:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:34:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:34:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:34:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:34:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:34:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:34:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:34:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:34:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:34:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:34:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:34:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:34:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:34:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:34:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:34:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:34:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:34:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:34:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:34:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:34:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:34:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:34:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:34:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:34:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:34:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:34:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:34:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:34:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:35:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:35:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:35:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:35:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:35:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:35:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:35:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:35:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:35:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:35:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:35:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:35:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:35:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:35:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:35:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:35:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:35:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:35:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:35:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:35:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:35:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:35:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:35:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:35:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:35:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:35:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:35:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:35:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:35:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:35:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:35:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:35:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:35:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:35:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:35:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:35:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:35:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:35:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:35:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:35:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:35:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:35:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:35:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:35:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:35:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:35:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:35:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:35:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:35:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:35:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:35:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:35:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:35:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:35:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:35:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:35:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:35:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:35:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:35:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:35:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:35:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:35:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:35:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:35:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:35:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:35:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:35:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:35:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:35:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:35:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:35:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:35:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:35:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:35:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:35:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:35:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:35:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:35:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:35:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:35:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:35:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:35:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:35:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:35:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:35:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:35:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:35:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:35:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:35:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:35:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:35:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:35:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:35:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:35:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:35:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:35:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:35:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:35:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:35:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:35:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:35:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:35:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:35:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:35:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:35:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:35:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:35:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:35:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:35:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:35:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:35:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:35:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:35:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:35:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:35:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:35:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:35:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:35:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:35:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:35:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:35:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:35:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:35:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:35:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:35:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:35:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:35:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:35:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:35:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:35:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:35:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:35:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:35:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:35:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:35:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:35:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:35:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:35:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:35:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:35:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:35:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:35:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:35:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:35:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:35:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:35:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:35:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:35:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:35:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:35:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:35:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:35:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:35:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:35:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:35:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:35:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:35:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:35:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:35:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:35:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:35:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:35:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:35:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:35:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:35:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:35:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:35:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:35:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:35:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:35:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:35:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:35:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:35:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:35:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:35:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:35:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:35:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:35:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:35:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:35:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:35:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:35:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:35:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:35:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:35:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:35:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:35:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:35:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:35:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:35:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:35:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:35:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:35:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:35:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:35:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:35:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:35:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:35:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:35:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:35:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:35:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:35:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:35:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:35:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:35:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:35:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:35:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:35:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:35:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:35:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:35:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:35:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:35:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:35:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:35:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:35:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:36:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:36:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:36:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:36:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:36:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:36:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:36:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:36:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:36:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:36:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:36:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:36:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:36:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:36:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:36:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:36:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:36:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:36:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:36:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:36:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:36:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:36:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:36:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:36:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:36:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:36:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:36:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:36:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:36:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:36:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:36:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:36:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:36:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:36:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:36:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:36:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:36:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:36:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:36:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:36:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:36:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:36:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:36:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:36:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:36:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:36:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:36:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:36:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:36:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:36:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:36:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:36:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:36:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:36:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:36:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:36:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:36:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:36:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:36:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:36:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:36:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:36:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:36:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:36:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:36:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:36:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:36:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:36:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:36:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:36:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:36:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:36:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:36:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:36:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:36:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:36:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:36:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:36:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:36:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:36:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:36:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:36:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:36:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:36:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:36:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:36:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:36:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:36:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:36:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:36:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:36:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:36:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:36:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:36:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:36:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:36:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:36:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:36:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:36:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:36:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:36:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:36:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:36:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:36:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:36:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:36:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:36:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:36:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:36:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:36:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:36:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:36:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:36:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:36:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:36:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:36:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:36:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:36:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:36:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:36:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:36:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:36:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:36:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:36:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:36:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:36:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:36:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:36:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:36:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:36:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:36:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:36:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:36:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:36:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:36:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:36:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:36:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:36:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:36:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:36:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:36:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:36:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:36:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:36:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:36:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:36:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:36:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:36:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:36:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:36:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:36:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:36:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:36:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:36:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:36:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:36:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:36:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:36:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:36:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:36:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:36:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:36:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:36:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:36:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:36:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:36:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:36:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:36:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:36:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:36:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:36:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:36:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:36:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:36:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:36:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:36:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:36:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:36:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:36:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:36:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:36:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:36:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:36:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:36:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:36:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:36:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:36:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:36:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:36:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:36:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:36:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:36:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:36:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:36:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:36:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:36:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:36:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:36:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:36:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:36:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:36:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:36:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:36:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:36:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:36:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:36:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:36:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:36:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:36:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:36:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:36:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:36:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:36:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:36:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:36:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:36:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:37:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:37:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:37:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:37:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:37:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:37:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:37:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:37:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:37:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:37:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:37:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:37:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:37:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:37:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:37:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:37:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:37:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:37:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:37:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:37:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:37:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:37:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:37:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:37:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:37:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:37:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:37:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:37:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:37:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:37:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:37:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:37:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:37:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:37:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:37:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:37:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:37:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:37:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:37:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:37:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:37:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:37:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:37:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:37:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:37:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:37:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:37:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:37:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:37:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:37:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:37:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:37:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:37:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:37:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:37:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:37:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:37:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:37:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:37:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:37:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:37:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:37:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:37:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:37:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:37:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:37:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:37:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:37:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:37:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:37:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:37:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:37:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:37:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:37:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:37:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:37:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:37:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:37:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:37:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:37:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:37:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:37:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:37:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:37:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:37:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:37:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:37:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:37:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:37:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:37:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:37:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:37:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:37:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:37:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:37:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:37:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:37:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:37:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:37:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:37:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:37:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:37:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:37:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:37:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:37:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:37:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:37:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:37:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:37:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:37:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:37:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:37:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:37:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:37:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:37:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:37:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:37:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:37:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:37:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:37:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:37:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:37:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:37:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:37:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:37:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:37:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:37:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:37:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:37:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:37:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:37:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:37:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:37:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:37:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:37:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:37:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:37:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:37:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:37:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:37:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:37:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:37:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:37:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:37:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:37:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:37:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:37:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:37:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:37:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:37:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:37:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:37:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:37:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:37:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:37:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:37:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:37:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:37:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:37:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:37:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:37:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:37:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:37:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:37:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:37:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:37:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:37:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:37:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:37:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:37:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:37:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:37:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:37:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:37:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:37:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:37:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:37:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:37:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:37:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:37:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:37:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:37:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:37:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:37:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:37:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:37:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:37:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:37:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:37:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:37:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:37:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:37:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:37:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:37:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:37:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:37:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:37:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:37:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:37:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:37:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:37:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:37:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:37:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:37:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:37:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:37:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:37:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:37:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:37:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:37:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:37:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:37:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:37:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:37:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:37:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:37:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:38:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:38:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:38:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:38:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:38:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:38:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:38:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:38:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:38:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:38:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:38:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:38:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:38:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:38:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:38:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:38:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:38:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:38:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:38:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:38:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:38:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:38:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:38:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:38:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:38:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:38:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:38:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:38:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:38:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:38:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:38:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:38:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:38:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:38:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:38:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:38:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:38:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:38:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:38:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:38:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:38:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:38:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:38:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:38:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:38:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:38:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:38:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:38:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:38:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:38:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:38:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:38:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:38:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:38:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:38:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:38:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:38:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:38:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:38:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:38:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:38:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:38:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:38:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:38:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:38:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:38:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:38:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:38:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:38:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:38:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:38:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:38:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:38:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:38:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:38:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:38:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:38:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:38:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:38:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:38:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:38:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:38:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:38:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:38:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:38:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:38:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:38:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:38:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:38:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:38:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:38:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:38:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:38:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:38:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:38:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:38:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:38:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:38:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:38:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:38:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:38:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:38:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:38:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:38:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:38:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:38:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:38:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:38:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:38:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:38:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:38:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:38:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:38:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:38:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:38:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:38:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:38:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:38:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:38:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:38:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:38:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:38:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:38:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:38:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:38:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:38:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:38:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:38:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:38:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:38:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:38:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:38:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:38:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:38:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:38:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:38:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:38:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:38:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:38:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:38:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:38:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:38:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:38:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:38:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:38:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:38:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:38:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:38:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:38:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:38:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:38:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:38:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:38:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:38:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:38:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:38:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:38:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:38:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:38:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:38:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:38:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:38:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:38:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:38:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:38:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:38:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:38:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:38:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:38:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:38:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:38:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:38:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:38:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:38:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:38:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:38:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:38:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:38:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:38:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:38:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:38:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:38:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:38:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:38:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:38:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:38:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:38:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:38:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:38:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:38:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:38:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:38:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:38:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:38:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:38:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:38:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:38:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:38:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:38:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:38:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:38:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:38:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:38:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:38:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:38:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:38:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:38:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:38:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:38:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:38:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:38:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:38:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:38:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:38:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:38:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:38:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:39:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:39:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:39:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:39:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:39:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:39:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:39:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:39:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:39:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:39:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:39:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:39:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:39:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:39:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:39:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:39:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:39:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:39:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:39:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:39:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:39:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:39:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:39:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:39:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:39:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:39:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:39:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:39:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:39:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:39:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:39:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:39:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:39:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:39:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:39:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:39:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:39:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:39:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:39:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:39:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:39:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:39:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:39:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:39:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:39:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:39:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:39:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:39:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:39:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:39:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:39:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:39:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:39:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:39:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:39:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:39:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:39:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:39:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:39:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:39:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:39:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:39:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:39:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:39:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:39:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:39:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:39:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:39:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:39:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:39:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:39:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:39:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:39:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:39:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:39:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:39:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:39:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:39:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:39:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:39:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:39:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:39:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:39:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:39:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:39:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:39:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:39:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:39:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:39:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:39:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:39:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:39:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:39:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:39:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:39:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:39:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:39:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:39:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:39:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:39:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:39:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:39:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:41:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:41:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:41:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:41:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:41:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:41:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:41:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:41:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:41:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:41:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:41:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:41:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:41:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:41:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:41:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:41:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:41:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:41:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:41:54 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 02:41:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:41:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:41:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:41:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:41:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:41:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:41:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:41:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:41:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:41:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:41:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:41:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:41:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:41:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:41:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:41:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:41:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:41:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:42:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:42:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:42:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:42:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:42:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:42:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:42:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:42:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:42:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:42:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:42:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:42:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:42:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:42:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:42:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:42:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:42:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:42:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:42:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:42:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:42:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:42:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:42:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:42:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:42:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:42:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:42:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:42:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:42:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:42:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:42:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:42:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:42:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:42:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:42:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:42:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:42:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:42:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:42:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:42:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:42:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:42:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:42:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:42:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:42:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:42:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:42:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:42:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:42:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:42:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:42:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:42:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:42:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:42:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:42:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:42:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:42:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:42:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:42:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:42:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:42:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:42:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:42:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:42:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:42:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:42:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:42:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:42:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:42:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:42:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:42:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:42:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:42:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:42:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:42:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:42:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:42:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:42:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:42:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:42:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:42:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:42:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:42:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:42:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:42:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:42:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:42:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:42:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:42:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:42:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:42:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:42:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:42:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:42:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:42:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:42:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:42:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:42:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:42:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:42:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:42:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:42:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:42:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:42:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:42:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:42:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:42:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:42:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:42:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:42:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:42:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:42:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:42:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:42:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:42:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:42:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:42:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:42:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:42:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:42:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:42:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:42:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:42:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:42:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:42:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:42:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:42:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:42:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:42:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:42:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:42:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:42:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:42:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:42:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:42:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:42:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:42:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:42:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:42:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:42:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:42:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:42:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:42:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:42:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:42:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:42:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:42:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:42:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:42:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:42:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:42:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:42:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:42:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:42:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:42:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:42:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:42:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:42:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:42:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:42:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:42:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:42:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:42:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:42:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:42:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:42:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:42:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:42:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:42:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:42:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:42:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:42:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:42:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:42:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:42:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:42:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:42:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:42:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:42:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:42:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:42:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:42:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:42:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:42:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:42:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:42:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:42:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:42:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:42:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:42:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:42:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:42:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:42:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:42:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:42:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:42:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:42:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:42:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:42:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:42:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:42:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:42:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:42:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:42:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:42:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:42:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:42:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:42:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:42:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:42:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:42:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:42:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:42:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:42:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:42:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:42:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:43:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:43:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:43:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:43:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:43:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:43:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:43:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:43:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:43:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:43:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:43:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:43:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:43:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:43:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:43:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:43:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:43:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:43:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:43:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:43:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:43:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:43:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:43:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:43:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:43:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:43:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:43:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:43:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:43:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:43:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:43:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:43:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:43:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:43:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:43:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:43:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:43:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:43:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:43:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:43:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:43:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:43:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:43:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:43:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:43:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:43:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:43:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:43:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:43:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:43:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:43:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:43:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:43:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:43:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:43:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:43:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:43:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:43:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:43:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:43:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:43:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:43:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:43:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:43:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:43:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:43:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:43:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:43:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:43:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:43:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:43:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:43:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:43:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:43:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:43:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:43:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:43:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:43:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:43:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:43:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:43:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:43:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:43:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:43:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:43:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:43:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:43:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:43:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:43:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:43:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:43:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:43:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:43:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:43:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:43:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:43:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:43:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:43:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:43:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:43:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:43:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:43:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:43:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:43:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:43:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:43:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:43:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:43:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:43:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:43:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:43:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:43:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:43:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:43:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:43:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:43:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:43:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:43:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:43:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:43:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:43:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:43:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:43:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:43:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:43:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:43:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:43:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:43:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:43:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:43:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:43:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:43:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:43:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:43:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:43:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:43:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:43:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:43:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:43:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:43:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:43:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:43:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:43:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:43:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:43:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:43:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:43:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:43:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:43:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:43:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:43:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:43:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:43:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:43:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:43:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:43:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:43:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:43:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:43:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:43:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:43:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:43:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:43:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:43:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:43:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:43:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:43:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:43:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:43:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:43:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:43:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:43:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:43:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:43:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:43:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:43:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:43:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:43:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:43:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:43:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:43:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:43:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:43:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:43:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:43:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:43:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:43:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:43:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:43:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:43:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:43:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:43:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:43:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:43:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:43:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:43:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:43:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:43:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:43:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:43:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:43:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:43:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:43:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:43:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:43:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:43:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:43:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:43:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:43:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:43:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:43:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:43:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:43:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:43:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:43:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:43:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:44:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:44:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:44:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:44:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:44:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:44:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:44:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:44:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:44:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:44:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:44:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:44:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:44:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:44:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:44:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:44:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:44:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:44:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:44:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:44:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:44:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:44:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:44:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:44:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:44:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:44:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:44:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:44:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:44:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:44:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:44:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:44:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:44:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:44:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:44:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:44:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:44:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:44:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:44:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:44:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:44:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:44:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:44:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:44:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:44:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:44:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:44:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:44:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:44:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:44:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:44:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:44:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:44:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:44:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:44:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:44:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:44:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:44:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:44:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:44:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:44:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:44:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:44:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:44:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:44:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:44:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:44:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:44:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:44:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:44:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:44:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:44:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:44:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:44:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:44:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:44:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:44:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:44:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:44:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:44:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:44:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:44:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:44:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:44:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:44:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:44:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:44:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:44:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:44:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:44:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:44:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:44:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:44:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:44:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:44:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:44:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:44:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:44:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:44:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:44:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:44:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:44:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:44:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:44:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:44:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:44:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:44:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:44:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:44:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:44:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:44:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:44:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:44:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:44:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:44:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:44:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:44:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:44:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:44:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:44:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:44:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:44:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:44:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:44:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:44:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:44:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:44:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:44:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:44:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:44:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:44:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:44:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:44:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:44:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:44:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:44:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:44:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:44:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:44:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:44:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:44:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:44:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:44:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:44:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:44:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:44:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:44:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:44:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:44:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:44:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:44:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:44:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:44:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:44:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:44:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:44:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:44:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:44:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:44:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:44:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:44:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:44:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:44:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:44:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:44:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:44:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:44:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:44:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:44:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:44:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:44:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:44:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:44:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:44:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:44:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:44:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:44:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:44:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:44:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:44:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:44:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:44:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:44:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:44:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:44:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:44:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:44:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:44:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:44:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:44:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:44:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:44:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:44:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:44:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:44:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:44:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:44:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:44:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:44:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:44:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:44:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:44:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:44:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:44:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:44:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:44:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:44:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:44:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:44:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:44:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:44:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:44:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:44:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:44:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:44:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:44:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:45:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:45:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:45:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:45:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:45:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:45:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:45:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:45:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:45:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:45:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:45:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:45:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:45:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:45:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:45:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:45:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:45:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:45:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:45:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:45:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:45:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:45:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:45:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:45:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:45:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:45:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:45:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:45:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:45:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:45:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:45:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:45:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:45:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:45:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:45:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:45:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:45:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:45:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:45:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:45:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:45:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:45:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:45:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:45:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:45:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:45:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:45:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:45:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:45:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:45:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:45:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:45:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:45:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:45:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:45:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:45:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:45:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:45:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:45:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:45:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:45:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:45:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:45:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:45:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:45:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:45:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:45:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:45:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:45:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:45:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:45:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:45:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:45:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:45:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:45:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:45:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:45:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:45:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:45:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:45:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:45:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:45:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:45:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:45:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:45:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:45:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:45:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:45:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:45:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:45:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:45:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:45:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:45:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:45:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:45:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:45:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:45:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:45:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:45:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:45:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:45:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:45:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:45:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:45:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:45:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:45:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:45:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:45:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:45:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:45:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:45:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:45:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:45:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:45:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:45:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:45:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:45:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:45:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:45:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:45:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:45:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:45:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:45:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:45:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:45:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:45:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:45:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:45:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:45:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:45:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:45:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:45:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:45:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:45:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:45:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:45:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:45:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:45:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:45:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:45:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:45:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:45:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:45:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:45:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:45:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:45:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:45:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:45:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:45:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:45:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:45:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:45:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:45:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:45:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:45:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:45:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:45:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:45:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:45:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:45:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:45:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:45:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:45:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:45:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:45:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:45:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:45:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:45:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:45:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:45:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:45:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:45:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:45:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:45:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:45:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:45:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:45:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:45:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:45:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:45:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:45:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:45:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:45:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:45:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:45:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:45:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:45:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:45:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:45:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:45:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:45:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:45:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:45:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:45:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:45:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:45:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:45:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:45:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:45:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:45:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:45:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:45:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:45:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:45:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:45:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:45:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:45:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:45:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:45:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:45:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:45:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:45:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:45:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:45:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:45:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:45:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:46:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:46:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:46:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:46:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:46:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:46:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:46:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:46:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:46:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:46:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:46:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:46:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:46:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:46:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:46:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:46:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:46:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:46:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:46:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:46:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:46:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:46:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:46:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:46:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:46:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:46:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:46:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:46:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:46:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:46:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:46:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:46:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:46:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:46:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:46:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:46:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:46:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:46:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:46:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:46:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:46:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:46:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:46:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:46:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:46:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:46:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:46:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:46:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:46:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:46:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:46:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:46:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:46:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:46:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:46:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:46:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:46:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:46:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:46:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:46:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:46:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:46:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:46:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:46:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:46:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:46:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:46:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:46:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:46:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:46:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:46:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:46:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:46:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:46:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:46:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:46:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:46:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:46:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:46:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:46:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:46:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:46:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:46:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:46:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:46:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:46:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:46:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:46:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:46:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:46:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:46:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:46:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:46:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:46:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:46:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:46:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:46:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:46:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:46:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:46:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:46:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:46:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:46:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:46:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:46:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:46:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:46:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:46:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:46:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:46:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:46:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:46:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:46:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:46:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:46:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:46:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:46:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:46:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:46:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:46:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:46:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:46:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:46:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:46:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:46:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:46:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:46:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:46:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:46:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:46:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:46:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:46:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:46:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:46:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:46:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:46:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:46:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:46:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:46:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:46:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:46:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:46:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:46:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:46:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:46:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:46:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:46:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:46:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:46:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:46:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:46:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:46:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:46:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:46:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:46:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:46:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:46:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:46:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:46:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:46:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:46:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:46:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:46:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:46:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:46:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:46:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:46:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:46:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:46:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:46:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:46:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:46:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:46:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:46:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:46:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:46:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:46:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:46:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:46:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:46:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:46:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:46:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:46:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:46:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:46:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:46:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:46:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:46:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:46:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:46:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:46:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:46:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:46:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:46:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:46:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:46:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:46:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:46:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:46:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:46:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:46:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:46:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:46:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:46:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:46:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:46:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:46:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:46:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:46:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:46:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:46:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:46:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:46:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:46:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:46:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:46:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:47:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:47:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:47:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:47:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:47:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:47:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:47:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:47:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:47:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:47:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:47:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:47:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:47:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:47:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:47:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:47:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:47:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:47:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:47:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:47:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:47:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:47:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:47:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:47:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:47:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:47:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:47:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:47:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:47:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:47:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:47:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:47:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:47:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:47:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:47:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:47:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:47:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:47:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:47:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:47:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:47:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:47:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:47:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:47:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:47:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:47:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:47:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:47:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:47:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:47:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:47:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:47:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:47:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:47:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:47:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:47:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:47:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:47:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:47:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:47:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:47:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:47:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:47:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:47:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:47:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:47:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:47:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:47:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:47:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:47:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:47:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:47:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:47:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:47:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:47:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:47:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:47:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:47:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:47:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:47:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:47:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:47:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:47:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:47:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:47:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:47:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:47:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:47:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:47:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:47:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:47:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:47:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:47:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:47:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:47:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:47:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:47:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:47:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:47:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:47:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:47:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:47:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:47:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:47:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:47:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:47:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:47:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:47:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:47:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:47:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:47:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:47:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:47:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:47:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:47:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:47:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:47:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:47:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:47:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:47:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:47:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:47:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:47:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:47:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:47:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:47:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:47:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:47:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:47:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:47:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:47:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:47:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:47:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:47:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:47:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:47:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:47:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:47:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:47:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:47:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:47:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:47:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:47:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:47:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:47:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:47:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:47:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:47:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:47:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:47:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:47:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:47:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:47:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:47:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:47:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:47:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:47:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:47:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:47:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:47:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:47:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:47:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:47:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:47:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:47:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:47:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:47:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:47:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:47:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:47:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:47:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:47:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:47:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:47:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:47:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:47:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:48:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:48:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:48:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:48:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:48:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:48:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:48:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:48:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:48:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:48:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:48:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:48:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:48:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:48:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:48:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:48:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:48:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:48:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:48:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:48:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:48:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:48:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:48:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:48:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:48:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:48:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:48:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:48:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:48:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:48:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:48:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:48:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:48:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:48:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:48:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:48:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:48:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:48:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:48:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:48:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:48:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:48:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:48:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:48:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:48:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:48:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:48:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:48:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:48:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:48:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:48:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:48:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:48:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:48:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:48:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:48:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:48:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:48:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:48:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:48:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:48:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:48:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:48:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:48:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:48:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:48:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:48:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:48:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:48:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:48:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:48:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:48:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:48:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:48:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:48:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:48:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:48:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:48:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:48:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:48:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:48:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:48:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:48:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:48:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:48:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:48:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:48:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:48:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:48:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:48:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:48:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:48:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:48:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:48:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:48:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:48:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:48:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:48:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:48:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:48:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:48:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:48:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:48:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:48:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:48:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:48:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:48:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:48:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:48:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:48:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:48:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:48:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:48:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:48:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:48:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:48:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:48:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:48:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:48:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:48:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:48:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:48:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:48:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:48:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:48:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:48:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:48:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:48:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:48:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:48:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:48:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:48:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:48:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:48:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:48:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:48:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:48:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:48:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:48:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:48:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:48:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:48:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:48:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:48:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:48:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:48:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:48:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:48:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:48:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:48:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:48:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:48:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:48:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:48:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:48:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:48:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:48:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:48:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:48:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:48:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:48:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:48:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:48:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:48:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:48:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:48:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:48:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:48:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:48:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:48:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:48:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:48:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:48:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:48:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:48:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:48:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:48:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:48:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:48:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:48:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:48:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:48:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:48:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:48:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:48:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:48:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:48:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:48:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:48:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:48:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:48:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:48:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:48:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:48:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:48:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:48:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:48:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:48:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:48:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:48:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:48:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:48:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:48:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:48:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:48:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:48:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:48:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:48:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:48:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:48:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:48:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:48:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:48:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:48:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:48:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:48:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:49:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:49:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:49:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:49:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:49:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:49:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:49:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:49:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:49:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:49:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:49:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:49:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:49:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:49:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:49:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:49:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:49:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:49:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:49:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:49:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:49:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:49:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:49:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:49:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:49:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:49:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:49:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:49:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:49:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:49:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:49:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:49:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:49:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:49:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:49:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:49:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:49:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:49:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:49:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:49:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:49:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:49:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:49:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:49:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:49:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:49:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:49:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:49:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:49:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:49:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:49:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:49:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:49:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:49:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:49:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:49:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:49:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:49:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:49:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:49:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:49:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:49:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:49:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:49:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:49:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:49:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:49:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:49:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:49:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:49:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:49:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:49:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:49:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:49:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:49:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:49:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:49:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:49:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:49:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:49:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:49:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:49:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:49:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:49:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:49:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:49:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:49:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:49:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:49:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:49:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:49:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:49:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:49:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:49:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:49:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:49:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:49:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:49:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:49:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:49:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:49:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:49:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:49:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:49:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:49:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:49:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:49:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:49:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:49:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:49:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:49:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:49:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:49:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:49:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:49:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:49:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:49:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:49:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:49:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:49:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:49:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:49:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:49:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:49:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:49:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:49:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:49:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:49:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:49:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:49:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:49:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:49:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:49:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:49:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:49:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:49:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:49:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:49:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:49:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:49:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:49:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:49:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:49:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:49:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:49:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:49:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:49:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:49:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:49:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:49:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:49:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:49:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:49:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:49:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:49:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:49:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:49:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:49:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:49:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:49:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:49:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:49:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:49:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:49:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:49:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:49:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:49:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:49:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:49:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:49:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:51:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:51:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:51:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:51:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:51:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:51:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:51:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:51:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:51:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:51:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:51:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:51:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:51:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:51:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:51:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:51:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:51:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:51:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:51:59 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 02:52:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:52:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:52:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:52:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:52:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:52:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:52:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:52:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:52:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:52:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:52:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:52:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:52:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:52:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:52:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:52:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:52:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:52:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:52:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:52:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:52:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:52:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:52:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:52:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:52:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:52:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:52:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:52:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:52:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:52:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:52:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:52:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:52:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:52:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:52:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:52:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:52:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:52:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:52:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:52:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:52:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:52:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:52:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:52:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:52:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:52:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:52:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:52:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:52:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:52:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:52:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:52:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:52:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:52:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:52:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:52:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:52:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:52:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:52:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:52:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:52:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:52:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:52:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:52:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:52:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:52:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:52:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:52:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:52:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:52:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:52:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:52:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:52:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:52:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:52:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:52:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:52:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:52:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:52:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:52:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:52:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:52:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:52:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:52:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:52:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:52:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:52:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:52:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:52:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:52:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:52:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:52:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:52:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:52:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:52:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:52:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:52:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:52:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:52:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:52:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:52:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:52:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:52:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:52:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:52:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:52:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:52:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:52:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:52:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:52:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:52:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:52:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:52:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:52:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:52:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:52:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:52:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:52:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:52:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:52:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:52:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:52:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:52:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:52:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:52:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:52:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:52:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:52:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:52:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:52:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:52:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:52:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:52:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:52:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:52:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:52:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:52:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:52:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:52:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:52:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:52:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:52:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:52:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:52:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:52:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:52:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:52:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:52:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:52:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:52:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:52:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:52:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:52:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:52:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:52:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:52:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:52:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:52:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:52:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:52:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:52:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:52:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:52:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:52:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:52:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:52:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:52:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:52:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:52:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:52:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:52:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:52:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:52:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:52:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:52:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:52:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:52:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:52:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:52:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:52:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:52:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:52:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:52:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:52:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:52:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:52:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:52:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:52:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:52:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:52:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:52:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:52:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:52:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:52:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:52:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:52:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:52:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:52:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:52:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:52:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:52:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:52:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:52:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:52:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:52:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:52:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:52:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:52:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:52:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:52:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:52:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:52:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:52:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:52:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:52:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:52:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:53:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:53:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:53:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:53:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:53:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:53:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:53:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:53:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:53:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:53:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:53:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:53:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:53:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:53:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:53:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:53:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:53:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:53:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:53:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:53:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:53:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:53:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:53:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:53:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:53:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:53:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:53:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:53:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:53:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:53:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:53:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:53:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:53:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:53:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:53:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:53:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:53:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:53:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:53:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:53:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:53:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:53:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:53:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:53:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:53:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:53:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:53:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:53:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:53:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:53:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:53:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:53:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:53:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:53:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:53:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:53:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:53:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:53:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:53:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:53:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:53:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:53:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:53:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:53:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:53:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:53:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:53:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:53:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:53:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:53:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:53:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:53:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:53:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:53:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:53:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:53:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:53:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:53:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:53:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:53:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:53:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:53:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:53:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:53:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:53:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:53:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:53:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:53:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:53:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:53:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:53:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:53:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:53:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:53:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:53:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:53:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:53:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:53:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:53:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:53:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:53:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:53:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:53:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:53:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:53:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:53:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:53:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:53:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:53:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:53:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:53:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:53:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:53:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:53:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:53:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:53:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:53:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:53:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:53:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:53:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:53:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:53:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:53:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:53:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:53:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:53:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:53:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:53:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:53:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:53:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:53:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:53:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:53:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:53:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:53:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:53:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:54:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:54:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:54:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:54:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:54:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:54:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:54:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:54:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:54:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:54:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:54:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:54:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:54:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:54:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:54:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:54:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:54:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:54:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:54:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:54:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:54:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:54:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:54:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:54:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:54:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:54:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:54:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:54:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:54:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:54:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:54:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:54:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:54:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:54:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:54:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:54:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:54:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:54:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:54:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:54:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:54:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:54:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:54:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:54:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:54:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:54:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:54:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:54:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:54:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:54:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:54:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:54:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:54:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:54:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:54:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:54:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:54:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:54:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:54:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:54:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:54:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:54:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:54:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:54:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:54:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:54:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:54:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:54:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:54:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:54:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:54:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:54:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:54:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:54:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:54:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:54:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:54:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:54:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:54:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:54:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:54:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:54:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:54:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:54:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:54:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:54:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:54:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:54:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:54:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:54:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:54:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:54:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:54:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:54:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:54:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:54:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:54:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:54:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:54:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:54:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:54:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:54:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:54:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:54:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:54:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:54:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:54:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:54:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:54:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:54:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:54:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:54:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:54:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:54:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:54:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:54:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:54:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:54:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:54:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:54:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:54:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:54:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:54:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:54:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:54:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:54:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:54:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:54:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:54:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:54:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:54:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:54:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:54:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:54:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:54:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:54:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:54:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:54:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:54:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:54:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:54:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:54:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:54:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:54:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:54:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:54:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:54:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:54:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:54:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:54:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:54:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:54:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:54:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:54:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:54:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:54:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:54:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:54:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:54:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:54:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:54:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:54:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:54:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:54:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:54:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:54:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:54:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:54:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:54:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:54:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:54:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:54:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:54:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:54:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:54:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:54:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:55:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:55:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:55:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:55:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:55:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:55:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:55:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:55:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:55:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:55:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:55:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:55:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:55:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:55:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:55:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:55:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:55:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:55:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:55:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:55:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:55:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:55:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:55:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:55:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:55:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:55:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:55:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:55:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:55:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:55:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:55:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:55:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:55:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:55:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:55:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:55:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:55:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:55:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:55:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:55:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:55:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:55:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:55:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:55:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:55:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:55:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:55:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:55:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:55:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:55:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:55:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:55:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:55:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:55:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:55:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:55:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:55:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:55:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:55:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:55:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:55:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:55:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:55:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:55:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:55:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:55:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:55:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:55:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:55:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:55:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:55:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:55:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:55:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:55:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:55:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:55:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:55:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:55:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:55:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:55:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:55:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:55:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:55:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:55:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:55:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:55:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:55:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:55:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:55:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:55:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:55:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:55:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:55:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:55:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:55:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:55:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:55:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:55:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:55:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:55:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:55:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:55:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:55:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:55:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:55:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:55:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:55:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:55:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:55:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:55:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:55:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:55:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:55:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:55:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:55:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:55:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:55:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:55:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:55:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:55:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:55:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:55:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:55:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:55:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:55:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:55:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:55:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:55:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:55:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:55:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:55:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:55:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:55:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:55:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:55:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:55:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:55:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:55:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:55:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:55:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:55:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:55:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:55:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:55:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:55:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:55:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:55:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:55:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:55:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:55:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:55:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:55:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:55:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:55:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:55:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:55:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:55:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:55:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:55:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:55:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:55:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:55:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:55:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:55:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:55:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:55:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:55:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:55:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:55:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:55:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:55:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:55:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:55:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:55:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:55:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:55:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:55:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:55:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:55:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:55:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:55:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:55:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:55:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:55:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:55:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:55:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:55:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:55:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:55:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:55:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:55:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:55:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:55:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:55:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:55:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:55:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:55:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:55:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:55:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:55:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:55:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:55:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:55:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:55:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:55:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:55:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:55:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:55:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:55:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:55:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:55:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:55:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:55:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:55:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:55:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:55:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:56:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:56:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:56:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:56:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:56:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:56:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:56:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:56:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:56:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:56:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:56:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:56:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:56:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:56:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:56:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:56:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:56:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:56:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:56:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:56:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:56:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:56:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:56:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:56:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:56:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:56:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:56:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:56:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:56:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:56:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:56:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:56:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:56:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:56:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:56:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:56:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:56:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:56:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:56:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:56:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:56:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:56:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:56:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:56:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:56:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:56:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:56:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:56:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:56:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:56:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:56:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:56:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:56:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:56:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:56:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:56:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:56:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:56:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:56:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:56:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:56:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:56:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:56:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:56:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:56:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:56:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:56:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:56:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:56:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:56:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:56:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:56:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:56:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:56:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:56:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:56:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:56:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:56:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:56:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:56:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:56:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:56:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:56:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:56:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:56:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:56:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:56:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:56:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:56:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:56:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:56:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:56:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:56:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:56:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:56:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:56:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:56:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:56:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:56:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:56:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:56:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:56:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:56:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:56:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:56:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:56:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:56:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:56:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:56:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:56:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:56:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:56:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:56:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:56:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:56:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:56:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:56:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:56:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:56:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:56:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:56:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:56:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:56:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:56:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:56:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:56:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:56:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:56:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:56:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:56:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:56:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:56:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:56:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:56:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:56:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:56:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:56:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:56:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:56:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:56:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:56:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:56:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:56:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:56:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:56:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:56:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:56:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:56:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:56:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:56:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:56:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:56:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:56:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:56:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:56:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:56:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:56:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:56:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:56:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:56:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:56:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:56:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:56:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:56:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:56:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:56:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:56:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:56:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:56:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:56:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:56:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:56:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:56:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:56:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:56:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:56:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:56:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:56:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:56:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:56:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:56:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:56:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:56:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:56:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:56:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:56:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:56:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:56:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:56:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:56:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:56:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:56:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:56:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:56:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:56:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:56:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:56:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:56:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:56:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:56:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:56:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:56:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:56:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:56:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:56:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:56:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:56:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:56:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:56:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:56:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:56:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:56:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:56:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:56:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:56:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:56:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:57:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:57:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:57:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:57:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:57:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:57:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:57:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:57:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:57:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:57:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:57:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:57:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:57:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:57:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:57:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:57:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:57:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:57:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:57:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:57:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:57:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:57:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:57:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:57:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:57:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:57:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:57:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:57:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:57:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:57:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:57:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:57:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:57:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:57:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:57:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:57:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:57:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:57:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:57:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:57:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:57:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:57:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:57:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:57:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:57:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:57:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:57:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:57:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:57:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:57:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:57:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:57:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:57:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:57:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:57:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:57:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:57:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:57:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:57:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:57:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:57:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:57:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:57:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:57:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:57:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:57:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:57:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:57:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:57:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:57:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:57:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:57:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:57:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:57:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:57:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:57:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:57:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:57:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:57:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:57:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:57:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:57:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:57:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:57:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:57:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:57:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:57:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:57:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:57:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:57:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:57:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:57:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:57:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:57:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:57:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:57:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:57:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:57:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:57:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:57:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:57:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:57:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:57:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:57:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:57:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:57:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:57:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:57:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:57:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:57:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:57:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:57:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:57:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:57:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:57:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:57:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:57:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:57:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:57:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:57:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:57:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:57:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:57:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:57:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:57:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:57:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:57:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:57:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:57:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:57:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:57:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:57:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:57:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:57:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:57:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:57:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:57:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:57:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:57:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:57:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:57:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:57:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:57:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:57:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:57:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:57:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:57:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:57:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:57:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:57:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:57:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:57:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:57:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:57:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:57:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:57:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:57:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:57:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:57:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:57:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:57:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:57:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:57:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:57:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:57:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:57:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:57:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:57:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:57:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:57:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:57:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:57:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:57:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:57:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:57:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:57:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:57:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:57:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:57:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:57:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:57:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:57:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:57:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:57:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:57:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:57:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:57:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:57:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:57:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:57:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:57:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:57:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:57:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:57:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:57:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:57:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:57:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:57:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:57:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:57:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:57:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:57:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:57:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:57:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:57:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:57:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:57:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:57:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:57:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:57:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:57:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:57:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:57:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:57:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:57:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:57:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:58:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:58:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:58:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:58:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:58:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:58:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:58:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:58:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:58:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:58:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:58:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:58:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:58:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:58:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:58:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:58:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:58:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:58:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:58:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:58:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:58:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:58:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:58:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:58:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:58:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:58:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:58:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:58:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:58:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:58:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:58:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:58:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:58:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:58:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:58:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:58:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:58:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:58:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:58:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:58:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:58:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:58:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:58:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:58:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:58:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:58:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:58:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:58:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:58:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:58:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:58:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:58:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:58:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:58:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:58:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:58:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:58:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:58:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:58:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:58:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:58:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:58:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:58:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:58:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:58:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:58:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:58:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:58:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:58:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:58:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:58:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:58:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:58:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:58:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:58:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:58:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:58:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:58:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:58:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:58:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:58:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:58:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:58:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:58:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:58:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:58:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:58:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:58:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:58:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:58:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:58:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:58:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:58:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:58:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:58:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:58:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:58:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:58:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:58:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:58:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:58:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:58:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:58:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:58:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:58:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:58:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:58:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:58:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:58:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:58:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:58:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:58:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:58:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:58:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:58:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:58:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:58:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:58:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:58:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:58:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:58:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:58:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:58:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:58:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:58:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:58:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:58:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:58:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:58:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:58:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:58:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:58:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:58:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:58:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:58:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:58:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:58:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:58:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:58:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:58:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:58:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:58:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:58:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:58:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:58:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:58:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:58:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:58:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:58:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:58:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:58:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:58:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:58:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:58:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:58:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:58:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:58:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:58:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:58:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:58:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:58:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:58:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:58:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:58:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:58:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:58:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:58:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:58:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:58:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:58:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:58:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:58:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:58:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:58:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:58:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:58:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:58:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:58:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:58:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:58:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:58:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:58:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:58:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:58:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:58:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:58:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:58:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:58:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:58:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:58:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:58:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:58:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:58:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:58:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:58:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:58:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:58:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:58:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:58:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:58:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:58:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:58:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:58:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:58:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:58:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:58:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:58:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:58:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:58:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:58:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:58:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:58:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:58:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:58:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:58:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:58:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:59:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:59:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:59:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:59:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:59:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:59:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:59:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:59:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:59:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:59:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:59:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:59:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:59:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:59:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:59:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:59:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:59:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:59:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:59:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:59:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:59:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:59:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:59:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:59:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:59:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:59:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:59:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:59:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:59:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:59:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:59:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:59:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:59:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:59:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:59:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:59:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:59:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:59:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:59:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:59:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:59:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:59:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:59:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:59:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:59:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:59:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:59:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:59:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:59:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:59:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:59:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:59:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:59:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:59:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:59:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:59:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:59:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:59:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:59:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:59:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:59:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:59:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:59:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:59:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:59:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:59:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:59:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:59:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:59:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:59:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:59:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:59:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:59:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:59:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:59:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:59:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:59:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:59:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:59:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:59:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:59:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:59:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:59:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:59:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:59:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:59:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:59:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:59:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:59:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:59:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:59:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:59:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:59:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:59:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:59:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:59:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:59:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:59:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:59:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:59:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:59:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:59:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:59:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:59:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:59:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:59:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:59:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:59:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:59:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:59:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:59:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:59:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:59:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:59:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:59:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:59:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:59:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:59:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:59:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:59:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:59:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:59:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:59:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:59:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:59:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:59:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:59:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:59:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:59:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:59:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 02:59:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 02:59:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:59:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:59:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:59:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 02:59:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 02:59:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 02:59:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 02:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 02:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 02:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 02:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 02:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 02:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 02:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 02:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 02:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 02:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:01:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:01:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:01:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:01:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:01:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:01:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:01:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:01:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:01:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:01:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:01:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:01:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:01:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:01:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:01:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:01:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:01:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:01:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:02:00 rdma06 rsyslogd[1041]: imjournal: 5090 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 03:02:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:02:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:02:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:02:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:02:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:02:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:02:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:02:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:02:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:02:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:02:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:02:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:02:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:02:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:02:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:02:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:02:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:02:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:02:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:02:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:02:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:02:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:02:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:02:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:02:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:02:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:02:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:02:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:02:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:02:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:02:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:02:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:02:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:02:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:02:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:02:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:02:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:02:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:02:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:02:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:02:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:02:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:02:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:02:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:02:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:02:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:02:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:02:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:02:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:02:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:02:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:02:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:02:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:02:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:02:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:02:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:02:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:02:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:02:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:02:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:02:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:02:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:02:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:02:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:02:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:02:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:02:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:02:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:02:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:02:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:02:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:02:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:02:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:02:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:02:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:02:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:02:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:02:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:02:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:02:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:02:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:02:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:02:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:02:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:02:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:02:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:02:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:02:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:02:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:02:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:02:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:02:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:02:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:02:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:02:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:02:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:02:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:02:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:02:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:02:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:02:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:02:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:02:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:02:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:02:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:02:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:02:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:02:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:02:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:02:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:02:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:02:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:02:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:02:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:02:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:02:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:02:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:02:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:02:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:02:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:02:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:02:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:02:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:02:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:02:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:02:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:02:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:02:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:02:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:02:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:02:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:02:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:02:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:02:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:02:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:02:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:02:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:02:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:02:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:02:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:02:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:02:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:02:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:02:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:02:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:02:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:02:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:02:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:02:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:02:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:02:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:02:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:02:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:02:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:02:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:02:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:02:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:02:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:02:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:02:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:02:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:02:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:02:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:02:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:02:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:02:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:02:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:02:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:02:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:02:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:02:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:02:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:02:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:02:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:02:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:02:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:02:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:02:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:02:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:02:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:02:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:02:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:02:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:02:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:02:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:02:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:02:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:02:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:02:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:02:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:02:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:02:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:02:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:02:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:02:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:02:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:02:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:02:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:02:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:02:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:02:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:02:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:02:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:02:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:02:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:02:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:02:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:02:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:02:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:02:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:02:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:02:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:02:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:02:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:02:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:02:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:03:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:03:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:03:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:03:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:03:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:03:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:03:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:03:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:03:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:03:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:03:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:03:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:03:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:03:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:03:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:03:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:03:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:03:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:03:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:03:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:03:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:03:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:03:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:03:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:03:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:03:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:03:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:03:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:03:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:03:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:03:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:03:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:03:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:03:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:03:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:03:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:03:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:03:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:03:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:03:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:03:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:03:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:03:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:03:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:03:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:03:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:03:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:03:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:03:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:03:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:03:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:03:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:03:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:03:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:03:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:03:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:03:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:03:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:03:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:03:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:03:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:03:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:03:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:03:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:03:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:03:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:03:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:03:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:03:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:03:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:03:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:03:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:03:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:03:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:03:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:03:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:03:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:03:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:03:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:03:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:03:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:03:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:03:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:03:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:03:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:03:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:03:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:03:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:03:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:03:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:03:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:03:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:03:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:03:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:03:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:03:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:03:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:03:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:03:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:03:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:03:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:03:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:03:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:03:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:03:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:03:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:03:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:03:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:03:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:03:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:03:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:03:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:03:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:03:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:03:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:03:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:03:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:03:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:03:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:03:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:03:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:03:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:03:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:03:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:03:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:03:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:03:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:03:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:03:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:03:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:03:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:03:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:03:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:03:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:03:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:03:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:03:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:03:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:03:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:03:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:03:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:03:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:03:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:03:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:03:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:03:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:03:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:03:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:03:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:03:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:03:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:03:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:03:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:03:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:03:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:03:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:03:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:03:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:03:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:03:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:03:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:03:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:03:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:03:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:03:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:03:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:03:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:03:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:03:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:03:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:03:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:03:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:03:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:03:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:03:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:03:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:03:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:03:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:03:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:03:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:03:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:03:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:03:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:03:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:03:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:03:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:03:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:03:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:03:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:03:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:03:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:03:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:03:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:03:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:03:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:03:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:03:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:03:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:03:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:03:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:03:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:03:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:03:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:03:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:03:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:03:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:03:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:03:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:03:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:03:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:03:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:03:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:03:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:03:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:03:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:03:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:04:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:04:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:04:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:04:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:04:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:04:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:04:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:04:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:04:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:04:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:04:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:04:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:04:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:04:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:04:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:04:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:04:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:04:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:04:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:04:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:04:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:04:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:04:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:04:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:04:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:04:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:04:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:04:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:04:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:04:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:04:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:04:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:04:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:04:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:04:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:04:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:04:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:04:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:04:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:04:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:04:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:04:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:04:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:04:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:04:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:04:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:04:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:04:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:04:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:04:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:04:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:04:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:04:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:04:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:04:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:04:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:04:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:04:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:04:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:04:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:04:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:04:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:04:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:04:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:04:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:04:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:04:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:04:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:04:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:04:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:04:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:04:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:04:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:04:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:04:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:04:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:04:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:04:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:04:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:04:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:04:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:04:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:04:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:04:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:04:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:04:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:04:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:04:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:04:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:04:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:04:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:04:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:04:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:04:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:04:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:04:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:04:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:04:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:04:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:04:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:04:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:04:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:04:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:04:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:04:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:04:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:04:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:04:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:04:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:04:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:04:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:04:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:04:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:04:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:04:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:04:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:04:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:04:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:04:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:04:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:04:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:04:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:04:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:04:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:04:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:04:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:04:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:04:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:04:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:04:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:04:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:04:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:04:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:04:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:04:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:04:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:04:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:04:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:04:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:04:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:04:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:04:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:04:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:04:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:04:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:04:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:04:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:04:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:04:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:04:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:04:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:04:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:04:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:04:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:04:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:04:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:04:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:04:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:04:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:04:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:04:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:04:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:04:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:04:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:04:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:04:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:04:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:04:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:04:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:04:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:04:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:04:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:04:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:04:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:04:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:04:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:04:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:04:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:04:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:04:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:04:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:04:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:04:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:04:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:04:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:04:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:04:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:04:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:04:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:04:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:04:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:04:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:04:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:04:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:04:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:04:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:04:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:04:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:04:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:04:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:04:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:04:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:04:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:04:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:04:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:04:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:04:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:04:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:04:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:04:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:04:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:04:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:04:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:04:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:04:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:04:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:05:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:05:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:05:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:05:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:05:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:05:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:05:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:05:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:05:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:05:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:05:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:05:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:05:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:05:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:05:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:05:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:05:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:05:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:05:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:05:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:05:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:05:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:05:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:05:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:05:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:05:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:05:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:05:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:05:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:05:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:05:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:05:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:05:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:05:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:05:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:05:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:05:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:05:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:05:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:05:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:05:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:05:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:05:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:05:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:05:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:05:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:05:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:05:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:05:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:05:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:05:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:05:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:05:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:05:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:05:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:05:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:05:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:05:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:05:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:05:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:05:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:05:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:05:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:05:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:05:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:05:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:05:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:05:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:05:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:05:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:05:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:05:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:05:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:05:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:05:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:05:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:05:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:05:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:05:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:05:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:05:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:05:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:05:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:05:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:05:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:05:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:05:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:05:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:05:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:05:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:05:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:05:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:05:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:05:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:05:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:05:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:05:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:05:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:05:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:05:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:05:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:05:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:05:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:05:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:05:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:05:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:05:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:05:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:05:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:05:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:05:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:05:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:05:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:05:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:05:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:05:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:05:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:05:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:05:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:05:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:05:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:05:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:05:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:05:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:05:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:05:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:05:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:05:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:05:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:05:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:05:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:05:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:05:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:05:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:05:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:05:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:05:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:05:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:05:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:05:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:05:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:05:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:05:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:05:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:05:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:05:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:05:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:05:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:05:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:05:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:05:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:05:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:05:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:05:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:05:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:05:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:05:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:05:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:05:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:05:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:05:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:05:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:05:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:05:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:05:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:05:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:05:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:05:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:05:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:05:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:05:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:05:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:05:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:05:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:05:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:05:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:05:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:05:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:05:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:05:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:05:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:05:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:05:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:05:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:05:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:05:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:05:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:05:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:05:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:05:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:05:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:05:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:05:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:05:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:05:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:05:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:05:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:05:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:05:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:05:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:05:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:05:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:05:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:05:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:05:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:05:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:05:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:05:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:05:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:05:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:05:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:05:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:05:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:05:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:05:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:05:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:06:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:06:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:06:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:06:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:06:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:06:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:06:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:06:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:06:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:06:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:06:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:06:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:06:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:06:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:06:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:06:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:06:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:06:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:06:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:06:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:06:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:06:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:06:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:06:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:06:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:06:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:06:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:06:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:06:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:06:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:06:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:06:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:06:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:06:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:06:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:06:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:06:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:06:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:06:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:06:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:06:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:06:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:06:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:06:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:06:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:06:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:06:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:06:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:06:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:06:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:06:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:06:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:06:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:06:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:06:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:06:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:06:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:06:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:06:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:06:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:06:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:06:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:06:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:06:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:06:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:06:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:06:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:06:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:06:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:06:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:06:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:06:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:06:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:06:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:06:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:06:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:06:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:06:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:06:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:06:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:06:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:06:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:06:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:06:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:06:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:06:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:06:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:06:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:06:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:06:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:06:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:06:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:06:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:06:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:06:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:06:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:06:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:06:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:06:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:06:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:06:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:06:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:06:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:06:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:06:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:06:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:06:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:06:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:06:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:06:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:06:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:06:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:06:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:06:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:06:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:06:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:06:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:06:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:06:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:06:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:06:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:06:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:06:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:06:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:06:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:06:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:06:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:06:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:06:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:06:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:06:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:06:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:06:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:06:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:06:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:06:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:06:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:06:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:06:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:06:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:06:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:06:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:06:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:06:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:06:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:06:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:06:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:06:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:06:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:06:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:06:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:06:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:06:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:06:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:06:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:06:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:06:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:06:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:06:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:06:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:06:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:06:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:06:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:06:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:06:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:06:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:06:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:06:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:06:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:06:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:06:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:06:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:06:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:06:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:06:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:06:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:06:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:06:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:06:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:06:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:06:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:06:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:06:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:06:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:06:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:06:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:06:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:06:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:06:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:06:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:06:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:06:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:06:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:06:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:06:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:06:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:06:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:06:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:06:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:06:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:06:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:06:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:06:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:06:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:06:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:06:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:06:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:06:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:06:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:06:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:06:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:06:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:06:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:06:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:06:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:06:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:07:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:07:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:07:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:07:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:07:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:07:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:07:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:07:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:07:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:07:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:07:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:07:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:07:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:07:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:07:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:07:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:07:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:07:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:07:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:07:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:07:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:07:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:07:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:07:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:07:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:07:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:07:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:07:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:07:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:07:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:07:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:07:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:07:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:07:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:07:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:07:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:07:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:07:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:07:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:07:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:07:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:07:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:07:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:07:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:07:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:07:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:07:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:07:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:07:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:07:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:07:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:07:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:07:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:07:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:07:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:07:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:07:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:07:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:07:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:07:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:07:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:07:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:07:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:07:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:07:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:07:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:07:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:07:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:07:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:07:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:07:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:07:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:07:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:07:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:07:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:07:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:07:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:07:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:07:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:07:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:07:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:07:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:07:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:07:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:07:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:07:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:07:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:07:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:07:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:07:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:07:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:07:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:07:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:07:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:07:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:07:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:07:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:07:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:07:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:07:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:07:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:07:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:07:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:07:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:07:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:07:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:07:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:07:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:07:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:07:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:07:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:07:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:07:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:07:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:07:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:07:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:07:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:07:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:07:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:07:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:07:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:07:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:07:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:07:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:07:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:07:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:07:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:07:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:07:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:07:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:07:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:07:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:07:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:07:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:07:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:07:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:07:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:07:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:07:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:07:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:07:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:07:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:07:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:07:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:07:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:07:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:07:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:07:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:07:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:07:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:07:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:07:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:07:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:07:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:07:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:07:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:07:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:07:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:07:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:07:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:07:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:07:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:07:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:07:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:07:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:07:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:07:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:07:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:07:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:07:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:07:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:07:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:07:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:07:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:07:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:07:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:07:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:07:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:07:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:07:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:07:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:07:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:07:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:07:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:07:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:07:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:07:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:07:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:07:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:07:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:07:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:07:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:07:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:07:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:07:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:07:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:07:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:07:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:07:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:07:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:07:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:07:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:07:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:07:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:07:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:07:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:07:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:07:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:07:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:07:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:07:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:07:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:07:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:07:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:07:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:07:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:08:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:08:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:08:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:08:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:08:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:08:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:08:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:08:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:08:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:08:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:08:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:08:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:08:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:08:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:08:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:08:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:08:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:08:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:08:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:08:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:08:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:08:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:08:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:08:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:08:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:08:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:08:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:08:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:08:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:08:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:08:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:08:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:08:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:08:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:08:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:08:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:08:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:08:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:08:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:08:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:08:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:08:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:08:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:08:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:08:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:08:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:08:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:08:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:08:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:08:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:08:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:08:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:08:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:08:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:08:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:08:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:08:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:08:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:08:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:08:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:08:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:08:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:08:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:08:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:08:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:08:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:08:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:08:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:08:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:08:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:08:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:08:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:08:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:08:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:08:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:08:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:08:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:08:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:08:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:08:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:08:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:08:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:08:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:08:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:08:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:08:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:08:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:08:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:08:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:08:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:08:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:08:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:08:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:08:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:08:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:08:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:08:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:08:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:08:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:08:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:08:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:08:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:08:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:08:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:08:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:08:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:08:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:08:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:08:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:08:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:08:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:08:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:08:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:08:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:08:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:08:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:08:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:08:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:08:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:08:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:08:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:08:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:08:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:08:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:08:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:08:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:08:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:08:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:08:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:08:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:08:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:08:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:08:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:08:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:08:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:08:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:08:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:08:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:08:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:08:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:08:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:08:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:08:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:08:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:08:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:08:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:08:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:08:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:08:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:08:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:08:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:08:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:08:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:08:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:08:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:08:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:08:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:08:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:08:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:08:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:08:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:08:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:08:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:08:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:08:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:08:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:08:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:08:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:08:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:08:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:08:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:08:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:08:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:08:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:08:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:08:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:08:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:08:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:08:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:08:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:08:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:08:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:08:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:08:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:08:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:08:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:08:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:08:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:08:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:08:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:08:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:08:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:08:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:08:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:08:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:08:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:08:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:08:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:09:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:09:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:09:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:09:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:09:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:09:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:09:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:09:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:09:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:09:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:09:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:09:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:09:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:09:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:09:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:09:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:09:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:09:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:09:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:09:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:09:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:09:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:09:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:09:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:09:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:09:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:09:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:09:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:09:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:09:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:09:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:09:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:09:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:09:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:09:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:09:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:09:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:09:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:09:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:09:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:09:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:09:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:09:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:09:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:09:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:09:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:09:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:09:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:09:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:09:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:09:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:09:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:09:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:09:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:09:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:09:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:09:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:09:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:09:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:09:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:09:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:09:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:09:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:09:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:09:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:09:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:09:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:09:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:09:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:09:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:09:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:09:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:09:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:09:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:09:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:09:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:09:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:09:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:09:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:09:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:09:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:09:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:09:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:09:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:09:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:09:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:09:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:09:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:09:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:09:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:09:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:09:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:09:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:09:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:09:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:09:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:09:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:09:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:09:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:09:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:09:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:09:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:09:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:09:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:09:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:09:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:09:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:09:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:09:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:09:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:09:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:09:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:09:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:09:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:09:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:09:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:09:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:09:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:09:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:09:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:09:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:09:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:09:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:09:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:09:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:09:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:09:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:09:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:09:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:09:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:09:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:09:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:09:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:09:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:09:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:09:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:09:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:09:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:09:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:09:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:09:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:09:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:09:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:09:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:09:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:09:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:09:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:09:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:09:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:09:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:09:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:09:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:09:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:09:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:09:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:09:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:09:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:09:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:09:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:09:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:09:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:09:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:09:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:09:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:09:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:09:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:09:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:09:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:09:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:09:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:09:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:09:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:09:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:09:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:09:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:09:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:09:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:09:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:09:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:09:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:09:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:09:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:09:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:09:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:09:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:09:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:09:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:09:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:09:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:09:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:09:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:09:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:09:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:09:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:09:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:09:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:09:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:09:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:09:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:09:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:09:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:09:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:09:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:09:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:09:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:09:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:09:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:09:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:09:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:09:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:09:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:09:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:09:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:09:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:09:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:09:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:12:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:12:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:12:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:12:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:12:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:12:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:12:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:12:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:12:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:12:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:12:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:12:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:12:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:12:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:12:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:12:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:12:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:12:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:12:05 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 03:12:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:12:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:12:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:12:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:12:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:12:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:12:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:12:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:12:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:12:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:12:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:12:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:12:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:12:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:12:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:12:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:12:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:12:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:12:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:12:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:12:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:12:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:12:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:12:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:12:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:12:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:12:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:12:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:12:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:12:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:12:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:12:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:12:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:12:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:12:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:12:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:12:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:12:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:12:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:12:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:12:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:12:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:12:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:12:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:12:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:12:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:12:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:12:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:12:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:12:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:12:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:12:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:12:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:12:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:12:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:12:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:12:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:12:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:12:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:12:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:12:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:12:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:12:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:12:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:12:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:12:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:12:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:12:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:12:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:12:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:12:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:12:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:12:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:12:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:12:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:12:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:12:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:12:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:12:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:12:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:12:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:12:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:12:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:12:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:12:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:12:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:12:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:12:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:12:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:12:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:12:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:12:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:12:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:12:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:12:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:12:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:12:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:12:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:12:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:12:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:12:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:12:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:12:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:12:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:12:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:12:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:12:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:12:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:12:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:12:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:12:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:12:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:12:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:12:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:12:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:12:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:12:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:12:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:12:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:12:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:12:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:12:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:12:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:12:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:12:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:12:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:12:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:12:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:12:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:12:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:12:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:12:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:12:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:12:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:12:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:12:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:12:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:12:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:12:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:12:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:12:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:12:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:12:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:12:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:12:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:12:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:12:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:12:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:12:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:12:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:12:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:12:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:12:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:12:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:12:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:12:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:12:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:12:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:12:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:12:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:12:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:12:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:12:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:12:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:12:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:12:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:12:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:12:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:12:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:12:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:12:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:12:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:12:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:12:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:12:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:12:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:12:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:12:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:12:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:12:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:13:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:13:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:13:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:13:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:13:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:13:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:13:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:13:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:13:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:13:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:13:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:13:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:13:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:13:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:13:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:13:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:13:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:13:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:13:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:13:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:13:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:13:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:13:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:13:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:13:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:13:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:13:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:13:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:13:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:13:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:13:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:13:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:13:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:13:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:13:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:13:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:13:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:13:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:13:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:13:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:13:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:13:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:13:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:13:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:13:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:13:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:13:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:13:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:13:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:13:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:13:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:13:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:13:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:13:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:13:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:13:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:13:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:13:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:13:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:13:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:13:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:13:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:13:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:13:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:13:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:13:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:13:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:13:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:13:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:13:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:13:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:13:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:13:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:13:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:13:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:13:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:13:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:13:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:13:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:13:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:13:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:13:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:13:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:13:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:13:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:13:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:13:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:13:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:13:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:13:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:13:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:13:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:13:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:13:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:13:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:13:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:13:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:13:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:13:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:13:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:13:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:13:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:13:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:13:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:13:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:13:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:13:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:13:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:13:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:13:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:13:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:13:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:13:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:13:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:13:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:13:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:13:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:13:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:13:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:13:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:13:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:13:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:13:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:13:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:13:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:13:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:13:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:13:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:13:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:13:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:13:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:13:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:13:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:13:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:13:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:13:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:13:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:13:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:13:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:13:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:13:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:13:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:13:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:13:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:13:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:13:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:13:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:13:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:13:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:13:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:13:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:13:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:13:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:13:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:13:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:13:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:13:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:13:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:13:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:13:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:13:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:13:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:13:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:13:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:13:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:13:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:13:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:13:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:13:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:13:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:13:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:13:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:13:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:13:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:13:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:13:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:13:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:13:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:13:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:13:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:13:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:13:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:13:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:13:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:13:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:13:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:13:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:13:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:13:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:13:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:13:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:13:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:13:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:13:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:13:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:13:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:13:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:13:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:13:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:13:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:13:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:13:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:13:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:13:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:13:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:13:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:13:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:13:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:13:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:13:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:13:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:13:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:13:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:13:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:13:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:13:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:14:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:14:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:14:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:14:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:14:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:14:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:14:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:14:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:14:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:14:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:14:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:14:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:14:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:14:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:14:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:14:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:14:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:14:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:14:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:14:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:14:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:14:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:14:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:14:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:14:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:14:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:14:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:14:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:14:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:14:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:14:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:14:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:14:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:14:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:14:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:14:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:14:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:14:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:14:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:14:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:14:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:14:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:14:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:14:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:14:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:14:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:14:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:14:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:14:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:14:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:14:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:14:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:14:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:14:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:14:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:14:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:14:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:14:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:14:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:14:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:14:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:14:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:14:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:14:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:14:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:14:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:14:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:14:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:14:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:14:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:14:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:14:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:14:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:14:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:14:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:14:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:14:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:14:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:14:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:14:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:14:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:14:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:14:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:14:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:14:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:14:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:14:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:14:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:14:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:14:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:14:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:14:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:14:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:14:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:14:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:14:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:14:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:14:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:14:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:14:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:14:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:14:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:14:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:14:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:14:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:14:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:14:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:14:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:14:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:14:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:14:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:14:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:14:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:14:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:14:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:14:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:14:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:14:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:14:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:14:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:14:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:14:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:14:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:14:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:14:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:14:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:14:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:14:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:14:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:14:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:14:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:14:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:14:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:14:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:14:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:14:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:14:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:14:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:14:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:14:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:14:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:14:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:14:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:14:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:14:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:14:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:14:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:14:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:14:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:14:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:14:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:14:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:14:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:14:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:14:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:14:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:14:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:14:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:14:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:14:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:14:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:14:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:14:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:14:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:14:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:14:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:14:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:14:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:14:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:14:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:14:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:14:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:14:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:14:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:14:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:14:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:14:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:14:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:14:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:14:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:14:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:14:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:14:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:14:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:14:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:14:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:14:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:14:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:14:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:14:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:14:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:14:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:14:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:14:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:14:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:14:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:14:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:14:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:14:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:14:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:14:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:14:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:14:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:14:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:14:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:14:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:14:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:14:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:14:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:14:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:14:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:14:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:14:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:14:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:14:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:14:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:15:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:15:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:15:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:15:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:15:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:15:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:15:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:15:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:15:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:15:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:15:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:15:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:15:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:15:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:15:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:15:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:15:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:15:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:15:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:15:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:15:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:15:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:15:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:15:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:15:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:15:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:15:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:15:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:15:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:15:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:15:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:15:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:15:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:15:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:15:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:15:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:15:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:15:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:15:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:15:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:15:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:15:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:15:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:15:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:15:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:15:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:15:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:15:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:15:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:15:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:15:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:15:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:15:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:15:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:15:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:15:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:15:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:15:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:15:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:15:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:15:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:15:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:15:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:15:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:15:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:15:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:15:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:15:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:15:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:15:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:15:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:15:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:15:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:15:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:15:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:15:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:15:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:15:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:15:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:15:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:15:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:15:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:15:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:15:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:15:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:15:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:15:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:15:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:15:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:15:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:15:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:15:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:15:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:15:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:15:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:15:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:15:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:15:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:15:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:15:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:15:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:15:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:15:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:15:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:15:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:15:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:15:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:15:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:15:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:15:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:15:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:15:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:15:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:15:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:15:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:15:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:15:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:15:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:15:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:15:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:15:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:15:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:15:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:15:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:15:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:15:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:15:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:15:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:15:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:15:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:15:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:15:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:15:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:15:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:15:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:15:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:15:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:15:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:15:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:15:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:15:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:15:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:15:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:15:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:15:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:15:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:15:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:15:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:15:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:15:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:15:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:15:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:15:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:15:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:15:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:15:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:15:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:15:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:15:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:15:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:15:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:15:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:15:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:15:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:15:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:15:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:15:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:15:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:15:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:15:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:15:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:15:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:15:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:15:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:15:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:15:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:15:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:15:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:15:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:15:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:15:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:15:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:15:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:15:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:15:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:15:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:15:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:15:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:15:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:15:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:15:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:15:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:15:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:15:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:15:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:15:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:15:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:15:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:15:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:15:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:15:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:15:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:15:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:15:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:15:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:15:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:15:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:15:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:15:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:15:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:15:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:15:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:15:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:15:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:15:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:15:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:16:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:16:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:16:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:16:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:16:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:16:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:16:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:16:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:16:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:16:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:16:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:16:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:16:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:16:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:16:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:16:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:16:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:16:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:16:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:16:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:16:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:16:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:16:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:16:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:16:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:16:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:16:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:16:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:16:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:16:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:16:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:16:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:16:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:16:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:16:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:16:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:16:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:16:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:16:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:16:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:16:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:16:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:16:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:16:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:16:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:16:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:16:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:16:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:16:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:16:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:16:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:16:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:16:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:16:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:16:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:16:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:16:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:16:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:16:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:16:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:16:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:16:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:16:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:16:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:16:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:16:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:16:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:16:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:16:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:16:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:16:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:16:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:16:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:16:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:16:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:16:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:16:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:16:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:16:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:16:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:16:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:16:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:16:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:16:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:16:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:16:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:16:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:16:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:16:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:16:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:16:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:16:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:16:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:16:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:16:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:16:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:16:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:16:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:16:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:16:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:16:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:16:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:16:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:16:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:16:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:16:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:16:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:16:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:16:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:16:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:16:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:16:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:16:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:16:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:16:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:16:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:16:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:16:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:16:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:16:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:16:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:16:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:16:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:16:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:16:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:16:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:16:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:16:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:16:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:16:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:16:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:16:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:16:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:16:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:16:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:16:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:16:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:16:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:16:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:16:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:16:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:16:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:16:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:16:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:16:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:16:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:16:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:16:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:16:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:16:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:16:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:16:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:16:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:16:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:16:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:16:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:16:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:16:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:16:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:16:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:16:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:16:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:16:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:16:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:16:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:16:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:16:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:16:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:16:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:16:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:16:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:16:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:16:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:16:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:16:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:16:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:16:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:16:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:16:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:16:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:16:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:16:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:16:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:16:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:16:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:16:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:16:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:16:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:16:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:16:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:16:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:16:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:16:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:16:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:16:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:16:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:16:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:16:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:16:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:16:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:16:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:16:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:16:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:16:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:16:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:16:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:16:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:16:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:16:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:16:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:16:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:16:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:16:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:16:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:16:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:16:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:17:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:17:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:17:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:17:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:17:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:17:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:17:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:17:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:17:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:17:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:17:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:17:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:17:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:17:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:17:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:17:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:17:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:17:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:17:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:17:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:17:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:17:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:17:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:17:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:17:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:17:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:17:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:17:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:17:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:17:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:17:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:17:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:17:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:17:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:17:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:17:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:17:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:17:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:17:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:17:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:17:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:17:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:17:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:17:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:17:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:17:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:17:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:17:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:17:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:17:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:17:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:17:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:17:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:17:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:17:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:17:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:17:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:17:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:17:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:17:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:17:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:17:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:17:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:17:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:17:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:17:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:17:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:17:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:17:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:17:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:17:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:17:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:17:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:17:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:17:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:17:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:17:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:17:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:17:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:17:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:17:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:17:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:17:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:17:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:17:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:17:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:17:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:17:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:17:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:17:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:17:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:17:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:17:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:17:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:17:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:17:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:17:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:17:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:17:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:17:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:17:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:17:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:17:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:17:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:17:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:17:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:17:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:17:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:17:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:17:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:17:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:17:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:17:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:17:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:17:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:17:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:17:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:17:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:17:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:17:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:17:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:17:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:17:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:17:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:17:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:17:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:17:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:17:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:17:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:17:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:17:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:17:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:17:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:17:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:17:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:17:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:17:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:17:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:17:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:17:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:17:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:17:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:17:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:17:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:17:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:17:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:17:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:17:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:17:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:17:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:17:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:17:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:17:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:17:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:17:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:17:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:17:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:17:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:17:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:17:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:17:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:17:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:17:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:17:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:17:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:17:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:17:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:17:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:17:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:17:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:17:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:17:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:17:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:17:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:17:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:17:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:17:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:17:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:17:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:17:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:17:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:17:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:17:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:17:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:17:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:17:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:17:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:17:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:17:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:17:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:17:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:17:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:17:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:17:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:17:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:17:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:17:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:17:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:17:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:17:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:17:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:17:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:17:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:17:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:17:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:17:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:17:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:17:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:17:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:17:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:17:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:17:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:17:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:17:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:17:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:17:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:18:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:18:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:18:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:18:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:18:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:18:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:18:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:18:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:18:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:18:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:18:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:18:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:18:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:18:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:18:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:18:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:18:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:18:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:18:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:18:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:18:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:18:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:18:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:18:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:18:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:18:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:18:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:18:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:18:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:18:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:18:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:18:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:18:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:18:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:18:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:18:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:18:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:18:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:18:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:18:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:18:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:18:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:18:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:18:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:18:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:18:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:18:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:18:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:18:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:18:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:18:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:18:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:18:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:18:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:18:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:18:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:18:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:18:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:18:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:18:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:18:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:18:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:18:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:18:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:18:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:18:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:18:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:18:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:18:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:18:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:18:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:18:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:18:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:18:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:18:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:18:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:18:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:18:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:18:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:18:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:18:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:18:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:18:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:18:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:18:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:18:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:18:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:18:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:18:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:18:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:18:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:18:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:18:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:18:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:18:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:18:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:18:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:18:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:18:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:18:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:18:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:18:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:18:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:18:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:18:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:18:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:18:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:18:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:18:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:18:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:18:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:18:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:18:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:18:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:18:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:18:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:18:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:18:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:18:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:18:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:18:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:18:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:18:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:18:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:18:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:18:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:18:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:18:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:18:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:18:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:18:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:18:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:18:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:18:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:18:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:18:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:18:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:18:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:18:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:18:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:18:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:18:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:18:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:18:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:18:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:18:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:18:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:18:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:18:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:18:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:18:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:18:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:18:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:18:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:18:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:18:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:18:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:18:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:18:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:18:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:18:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:18:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:18:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:18:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:18:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:18:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:18:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:18:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:18:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:18:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:18:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:18:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:18:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:18:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:18:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:18:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:18:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:18:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:18:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:18:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:18:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:18:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:18:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:18:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:18:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:18:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:18:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:18:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:18:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:18:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:18:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:18:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:18:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:18:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:18:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:18:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:18:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:18:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:18:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:18:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:18:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:18:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:18:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:18:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:18:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:18:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:18:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:18:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:18:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:18:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:18:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:18:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:18:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:18:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:18:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:18:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:19:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:19:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:19:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:19:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:19:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:19:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:19:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:19:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:19:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:19:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:19:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:19:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:19:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:19:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:19:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:19:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:19:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:19:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:19:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:19:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:19:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:19:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:19:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:19:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:19:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:19:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:19:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:19:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:19:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:19:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:19:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:19:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:19:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:19:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:19:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:19:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:19:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:19:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:19:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:19:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:19:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:19:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:19:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:19:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:19:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:19:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:19:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:19:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:19:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:19:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:19:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:19:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:19:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:19:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:19:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:19:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:19:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:19:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:19:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:19:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:19:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:19:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:19:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:19:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:19:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:19:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:19:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:19:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:19:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:19:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:19:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:19:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:19:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:19:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:19:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:19:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:19:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:19:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:19:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:19:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:19:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:19:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:19:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:19:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:19:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:19:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:19:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:19:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:19:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:19:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:19:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:19:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:19:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:19:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:19:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:19:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:19:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:19:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:19:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:19:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:19:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:19:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:19:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:19:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:19:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:19:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:19:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:19:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:19:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:19:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:19:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:19:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:19:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:19:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:19:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:19:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:19:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:19:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:19:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:19:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:19:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:19:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:19:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:19:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:19:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:19:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:19:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:19:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:19:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:19:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:19:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:19:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:19:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:19:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:19:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:19:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:19:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:19:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:19:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:19:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:19:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:19:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:19:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:19:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:19:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:19:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:19:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:19:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:19:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:19:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:19:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:19:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:19:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:19:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:19:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:19:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:19:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:19:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:19:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:19:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:19:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:19:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:19:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:19:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:19:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:19:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:19:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:19:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:19:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:19:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:19:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:19:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:19:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:19:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:19:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:19:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:19:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:19:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:19:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:19:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:19:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:19:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:19:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:19:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:19:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:19:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:19:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:19:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:19:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:19:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:19:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:19:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:19:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:19:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:19:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:19:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:19:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:19:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:19:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:19:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:19:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:19:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:19:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:19:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:19:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:19:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:19:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:19:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:19:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:19:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:19:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:19:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:19:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:19:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:19:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:19:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:20:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:20:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:20:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:20:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:20:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:20:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:20:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:20:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:20:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:20:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:20:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:20:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:20:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:20:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:20:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:20:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:20:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:20:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:22:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:22:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:22:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:22:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:22:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:22:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:22:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:22:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:22:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:22:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:22:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:22:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:22:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:22:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:22:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:22:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:22:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:22:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:22:10 rdma06 rsyslogd[1041]: imjournal: 5293 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 03:22:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:22:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:22:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:22:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:22:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:22:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:22:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:22:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:22:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:22:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:22:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:22:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:22:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:22:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:22:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:22:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:22:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:22:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:22:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:22:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:22:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:22:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:22:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:22:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:22:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:22:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:22:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:22:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:22:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:22:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:22:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:22:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:22:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:22:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:22:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:22:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:22:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:22:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:22:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:22:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:22:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:22:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:22:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:22:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:22:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:22:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:22:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:22:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:22:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:22:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:22:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:22:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:22:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:22:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:22:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:22:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:22:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:22:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:22:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:22:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:22:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:22:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:22:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:22:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:22:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:22:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:22:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:22:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:22:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:22:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:22:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:22:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:22:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:22:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:22:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:22:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:22:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:22:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:22:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:22:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:22:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:22:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:22:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:22:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:22:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:22:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:22:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:22:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:22:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:22:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:22:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:22:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:22:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:22:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:22:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:22:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:22:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:22:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:22:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:22:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:22:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:22:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:22:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:22:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:22:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:22:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:22:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:22:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:22:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:22:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:22:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:22:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:22:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:22:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:22:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:22:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:22:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:22:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:22:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:22:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:22:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:22:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:22:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:22:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:22:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:22:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:22:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:22:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:22:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:22:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:22:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:22:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:22:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:22:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:22:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:22:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:22:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:22:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:22:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:22:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:22:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:22:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:22:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:22:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:22:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:22:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:22:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:22:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:22:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:22:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:22:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:22:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:22:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:22:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:22:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:22:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:22:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:22:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:22:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:22:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:22:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:22:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:23:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:23:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:23:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:23:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:23:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:23:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:23:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:23:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:23:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:23:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:23:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:23:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:23:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:23:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:23:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:23:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:23:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:23:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:23:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:23:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:23:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:23:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:23:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:23:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:23:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:23:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:23:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:23:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:23:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:23:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:23:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:23:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:23:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:23:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:23:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:23:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:23:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:23:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:23:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:23:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:23:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:23:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:23:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:23:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:23:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:23:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:23:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:23:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:23:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:23:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:23:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:23:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:23:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:23:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:23:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:23:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:23:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:23:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:23:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:23:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:23:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:23:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:23:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:23:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:23:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:23:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:23:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:23:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:23:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:23:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:23:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:23:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:23:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:23:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:23:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:23:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:23:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:23:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:23:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:23:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:23:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:23:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:23:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:23:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:23:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:23:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:23:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:23:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:23:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:23:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:23:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:23:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:23:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:23:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:23:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:23:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:23:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:23:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:23:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:23:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:23:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:23:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:23:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:23:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:23:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:23:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:23:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:23:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:23:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:23:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:23:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:23:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:23:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:23:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:23:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:23:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:23:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:23:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:23:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:23:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:23:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:23:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:23:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:23:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:23:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:23:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:23:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:23:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:23:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:23:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:23:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:23:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:23:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:23:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:23:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:23:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:23:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:23:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:23:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:23:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:23:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:23:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:23:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:23:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:23:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:23:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:23:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:23:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:23:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:23:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:23:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:23:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:23:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:23:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:23:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:23:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:23:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:23:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:23:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:23:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:23:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:23:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:23:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:23:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:23:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:23:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:23:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:23:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:23:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:23:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:23:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:23:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:23:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:23:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:23:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:23:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:23:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:23:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:23:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:23:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:23:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:23:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:23:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:23:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:23:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:23:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:23:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:23:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:23:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:23:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:23:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:23:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:23:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:23:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:23:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:23:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:23:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:23:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:23:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:23:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:23:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:23:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:23:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:23:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:23:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:23:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:23:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:23:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:23:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:23:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:23:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:23:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:23:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:23:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:23:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:23:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:24:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:24:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:24:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:24:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:24:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:24:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:24:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:24:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:24:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:24:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:24:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:24:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:24:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:24:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:24:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:24:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:24:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:24:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:24:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:24:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:24:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:24:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:24:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:24:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:24:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:24:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:24:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:24:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:24:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:24:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:24:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:24:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:24:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:24:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:24:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:24:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:24:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:24:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:24:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:24:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:24:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:24:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:24:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:24:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:24:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:24:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:24:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:24:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:24:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:24:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:24:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:24:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:24:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:24:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:24:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:24:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:24:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:24:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:24:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:24:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:24:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:24:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:24:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:24:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:24:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:24:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:24:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:24:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:24:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:24:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:24:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:24:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:24:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:24:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:24:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:24:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:24:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:24:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:24:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:24:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:24:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:24:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:24:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:24:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:24:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:24:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:24:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:24:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:24:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:24:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:24:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:24:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:24:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:24:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:24:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:24:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:24:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:24:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:24:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:24:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:24:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:24:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:24:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:24:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:24:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:24:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:24:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:24:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:24:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:24:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:24:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:24:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:24:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:24:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:24:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:24:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:24:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:24:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:24:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:24:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:24:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:24:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:24:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:24:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:24:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:24:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:24:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:24:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:24:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:24:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:24:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:24:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:24:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:24:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:24:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:24:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:24:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:24:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:24:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:24:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:24:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:24:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:24:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:24:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:24:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:24:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:24:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:24:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:24:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:24:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:24:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:24:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:24:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:24:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:24:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:24:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:24:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:24:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:24:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:24:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:24:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:24:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:24:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:24:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:24:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:24:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:24:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:24:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:24:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:24:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:24:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:24:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:24:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:24:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:24:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:24:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:24:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:24:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:24:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:24:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:24:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:24:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:24:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:24:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:24:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:24:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:24:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:24:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:24:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:24:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:24:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:24:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:24:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:24:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:24:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:24:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:24:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:24:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:24:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:24:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:24:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:24:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:24:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:24:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:24:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:24:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:24:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:24:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:24:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:24:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:24:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:24:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:24:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:24:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:24:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:24:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:25:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:25:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:25:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:25:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:25:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:25:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:25:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:25:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:25:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:25:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:25:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:25:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:25:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:25:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:25:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:25:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:25:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:25:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:25:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:25:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:25:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:25:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:25:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:25:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:25:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:25:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:25:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:25:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:25:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:25:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:25:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:25:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:25:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:25:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:25:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:25:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:25:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:25:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:25:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:25:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:25:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:25:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:25:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:25:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:25:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:25:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:25:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:25:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:25:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:25:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:25:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:25:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:25:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:25:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:25:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:25:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:25:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:25:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:25:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:25:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:25:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:25:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:25:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:25:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:25:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:25:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:25:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:25:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:25:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:25:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:25:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:25:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:25:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:25:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:25:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:25:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:25:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:25:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:25:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:25:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:25:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:25:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:25:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:25:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:25:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:25:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:25:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:25:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:25:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:25:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:25:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:25:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:25:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:25:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:25:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:25:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:25:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:25:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:25:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:25:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:25:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:25:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:25:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:25:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:25:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:25:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:25:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:25:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:25:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:25:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:25:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:25:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:25:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:25:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:25:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:25:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:25:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:25:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:25:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:25:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:25:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:25:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:25:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:25:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:25:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:25:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:25:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:25:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:25:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:25:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:25:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:25:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:25:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:25:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:25:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:25:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:25:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:25:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:25:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:25:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:25:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:25:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:25:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:25:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:25:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:25:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:25:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:25:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:25:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:25:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:25:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:25:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:25:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:25:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:25:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:25:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:25:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:25:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:25:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:25:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:25:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:25:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:25:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:25:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:25:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:25:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:25:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:25:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:25:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:25:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:25:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:25:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:25:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:25:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:25:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:25:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:25:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:25:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:25:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:25:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:25:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:25:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:25:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:25:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:25:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:25:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:25:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:25:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:25:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:25:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:25:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:25:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:25:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:25:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:25:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:25:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:25:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:25:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:25:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:25:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:25:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:25:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:25:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:25:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:25:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:25:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:25:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:25:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:25:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:25:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:25:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:25:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:25:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:25:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:25:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:25:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:26:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:26:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:26:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:26:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:26:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:26:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:26:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:26:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:26:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:26:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:26:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:26:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:26:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:26:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:26:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:26:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:26:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:26:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:26:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:26:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:26:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:26:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:26:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:26:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:26:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:26:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:26:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:26:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:26:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:26:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:26:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:26:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:26:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:26:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:26:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:26:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:26:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:26:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:26:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:26:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:26:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:26:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:26:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:26:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:26:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:26:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:26:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:26:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:26:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:26:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:26:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:26:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:26:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:26:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:26:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:26:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:26:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:26:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:26:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:26:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:26:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:26:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:26:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:26:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:26:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:26:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:26:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:26:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:26:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:26:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:26:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:26:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:26:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:26:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:26:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:26:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:26:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:26:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:26:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:26:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:26:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:26:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:26:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:26:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:26:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:26:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:26:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:26:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:26:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:26:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:26:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:26:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:26:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:26:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:26:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:26:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:26:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:26:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:26:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:26:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:26:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:26:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:26:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:26:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:26:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:26:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:26:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:26:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:26:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:26:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:26:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:26:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:26:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:26:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:26:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:26:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:26:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:26:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:26:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:26:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:26:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:26:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:26:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:26:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:26:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:26:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:26:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:26:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:26:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:26:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:26:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:26:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:26:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:26:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:26:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:26:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:26:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:26:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:26:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:26:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:26:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:26:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:26:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:26:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:26:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:26:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:26:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:26:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:26:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:26:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:26:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:26:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:26:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:26:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:26:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:26:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:26:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:26:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:26:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:26:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:26:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:26:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:26:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:26:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:26:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:26:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:26:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:26:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:26:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:26:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:26:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:26:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:26:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:26:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:26:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:26:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:26:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:26:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:26:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:26:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:26:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:26:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:26:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:26:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:26:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:26:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:26:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:26:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:26:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:26:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:26:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:26:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:26:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:26:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:26:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:26:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:26:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:26:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:26:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:26:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:26:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:26:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:26:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:26:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:26:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:26:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:26:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:26:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:26:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:26:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:26:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:26:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:26:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:26:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:26:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:26:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:27:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:27:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:27:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:27:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:27:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:27:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:27:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:27:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:27:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:27:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:27:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:27:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:27:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:27:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:27:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:27:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:27:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:27:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:27:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:27:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:27:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:27:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:27:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:27:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:27:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:27:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:27:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:27:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:27:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:27:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:27:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:27:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:27:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:27:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:27:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:27:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:27:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:27:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:27:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:27:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:27:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:27:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:27:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:27:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:27:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:27:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:27:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:27:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:27:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:27:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:27:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:27:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:27:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:27:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:27:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:27:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:27:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:27:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:27:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:27:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:27:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:27:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:27:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:27:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:27:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:27:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:27:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:27:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:27:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:27:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:27:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:27:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:27:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:27:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:27:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:27:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:27:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:27:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:27:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:27:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:27:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:27:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:27:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:27:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:27:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:27:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:27:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:27:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:27:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:27:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:27:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:27:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:27:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:27:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:27:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:27:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:27:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:27:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:27:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:27:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:27:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:27:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:27:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:27:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:27:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:27:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:27:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:27:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:27:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:27:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:27:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:27:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:27:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:27:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:27:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:27:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:27:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:27:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:27:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:27:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:27:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:27:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:27:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:27:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:27:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:27:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:27:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:27:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:27:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:27:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:27:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:27:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:27:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:27:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:27:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:27:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:27:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:27:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:27:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:27:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:27:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:27:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:27:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:27:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:27:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:27:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:27:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:27:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:27:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:27:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:27:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:27:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:27:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:27:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:27:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:27:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:27:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:27:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:27:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:27:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:27:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:27:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:27:40 rdma06 systemd-journald[790]: Data hash table of /run/log/journal/95379d2f723847b68d9e976dee8967a9/system.journal has a fill level at 75.0 (163750 of 218332 items, 75497472 file size, 461 bytes per hash table item), suggesting rotation.
Apr 18 03:27:40 rdma06 systemd-journald[790]: /run/log/journal/95379d2f723847b68d9e976dee8967a9/system.journal: Journal header limits reached or header out-of-date, rotating.
Apr 18 03:27:40 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 18 03:27:40 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 18 03:27:41 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 18 03:27:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:27:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:27:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:27:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:27:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:27:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:27:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:27:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:27:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:27:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:27:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:27:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:27:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:27:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:27:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:27:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:27:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:27:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:27:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:27:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:27:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:27:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:27:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:27:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:27:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:27:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:27:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:27:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:27:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:27:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:27:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:27:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:27:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:27:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:27:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:27:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:27:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:27:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:27:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:27:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:27:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:27:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:27:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:27:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:27:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:27:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:27:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:27:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:27:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:27:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:27:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:27:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:27:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:27:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:28:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:28:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:28:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:28:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:28:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:28:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:28:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:28:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:28:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:28:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:28:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:28:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:28:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:28:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:28:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:28:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:28:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:28:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:28:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:28:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:28:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:28:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:28:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:28:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:28:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:28:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:28:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:28:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:28:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:28:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:28:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:28:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:28:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:28:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:28:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:28:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:28:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:28:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:28:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:28:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:28:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:28:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:28:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:28:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:28:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:28:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:28:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:28:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:28:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:28:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:28:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:28:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:28:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:28:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:28:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:28:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:28:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:28:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:28:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:28:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:28:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:28:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:28:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:28:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:28:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:28:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:28:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:28:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:28:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:28:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:28:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:28:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:28:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:28:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:28:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:28:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:28:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:28:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:28:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:28:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:28:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:28:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:28:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:28:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:28:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:28:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:28:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:28:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:28:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:28:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:28:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:28:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:28:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:28:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:28:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:28:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:28:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:28:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:28:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:28:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:28:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:28:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:28:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:28:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:28:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:28:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:28:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:28:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:28:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:28:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:28:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:28:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:28:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:28:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:28:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:28:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:28:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:28:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:28:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:28:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:28:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:28:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:28:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:28:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:28:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:28:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:28:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:28:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:28:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:28:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:28:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:28:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:28:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:28:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:28:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:28:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:28:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:28:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:28:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:28:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:28:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:28:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:28:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:28:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:28:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:28:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:28:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:28:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:28:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:28:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:28:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:28:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:28:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:28:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:28:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:28:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:28:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:28:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:28:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:28:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:28:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:28:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:28:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:28:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:28:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:28:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:28:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:28:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:28:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:28:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:28:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:28:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:28:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:28:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:28:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:28:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:28:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:28:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:28:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:28:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:28:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:28:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:28:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:28:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:28:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:28:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:28:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:28:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:28:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:28:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:28:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:28:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:28:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:28:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:28:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:28:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:28:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:28:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:28:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:28:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:28:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:28:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:28:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:28:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:28:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:28:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:28:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:28:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:28:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:28:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:28:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:28:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:28:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:28:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:28:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:28:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:29:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:29:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:29:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:29:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:29:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:29:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:29:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:29:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:29:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:29:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:29:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:29:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:29:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:29:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:29:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:29:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:29:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:29:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:29:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:29:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:29:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:29:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:29:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:29:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:29:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:29:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:29:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:29:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:29:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:29:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:29:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:29:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:29:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:29:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:29:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:29:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:29:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:29:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:29:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:29:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:29:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:29:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:29:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:29:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:29:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:29:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:29:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:29:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:29:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:29:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:29:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:29:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:29:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:29:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:29:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:29:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:29:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:29:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:29:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:29:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:29:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:29:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:29:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:29:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:29:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:29:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:29:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:29:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:29:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:29:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:29:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:29:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:29:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:29:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:29:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:29:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:29:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:29:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:29:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:29:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:29:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:29:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:29:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:29:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:29:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:29:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:29:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:29:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:29:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:29:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:29:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:29:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:29:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:29:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:29:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:29:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:29:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:29:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:29:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:29:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:29:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:29:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:29:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:29:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:29:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:29:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:29:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:29:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:29:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:29:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:29:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:29:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:29:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:29:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:29:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:29:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:29:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:29:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:29:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:29:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:29:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:29:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:29:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:29:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:29:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:29:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:29:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:29:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:29:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:29:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:29:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:29:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:29:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:29:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:29:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:29:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:29:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:29:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:29:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:29:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:29:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:29:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:29:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:29:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:29:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:29:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:29:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:29:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:29:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:29:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:29:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:29:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:29:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:29:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:29:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:29:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:29:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:29:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:29:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:29:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:29:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:29:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:29:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:29:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:29:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:29:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:29:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:29:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:29:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:29:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:29:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:29:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:29:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:29:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:29:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:29:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:29:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:29:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:29:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:29:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:29:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:29:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:29:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:29:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:29:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:29:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:29:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:29:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:29:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:29:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:29:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:29:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:29:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:29:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:29:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:29:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:29:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:29:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:29:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:29:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:29:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:29:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:29:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:29:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:29:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:29:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:29:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:29:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:29:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:29:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:29:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:29:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:29:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:29:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:29:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:29:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:30:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:30:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:30:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:30:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:30:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:30:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:30:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:30:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:30:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:30:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:30:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:30:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:30:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:30:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:30:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:30:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:30:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:30:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:30:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:30:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:30:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:30:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:30:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:30:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:30:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:30:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:30:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:30:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:30:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:30:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:30:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:30:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:30:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:30:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:30:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:30:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:32:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:32:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:32:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:32:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:32:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:32:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:32:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:32:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:32:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:32:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:32:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:32:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:32:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:32:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:32:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:32:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:32:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:32:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:32:11 rdma06 rsyslogd[1041]: imjournal: 5087 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 03:32:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:32:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:32:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:32:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:32:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:32:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:32:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:32:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:32:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:32:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:32:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:32:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:32:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:32:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:32:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:32:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:32:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:32:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:32:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:32:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:32:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:32:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:32:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:32:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:32:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:32:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:32:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:32:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:32:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:32:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:32:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:32:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:32:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:32:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:32:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:32:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:32:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:32:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:32:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:32:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:32:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:32:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:32:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:32:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:32:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:32:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:32:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:32:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:32:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:32:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:32:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:32:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:32:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:32:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:32:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:32:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:32:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:32:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:32:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:32:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:32:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:32:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:32:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:32:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:32:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:32:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:32:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:32:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:32:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:32:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:32:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:32:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:32:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:32:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:32:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:32:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:32:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:32:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:32:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:32:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:32:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:32:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:32:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:32:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:32:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:32:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:32:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:32:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:32:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:32:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:32:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:32:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:32:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:32:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:32:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:32:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:32:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:32:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:32:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:32:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:32:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:32:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:32:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:32:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:32:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:32:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:32:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:32:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:32:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:32:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:32:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:32:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:32:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:32:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:32:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:32:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:32:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:32:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:32:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:32:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:32:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:32:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:32:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:32:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:32:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:32:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:32:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:32:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:32:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:32:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:32:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:32:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:32:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:32:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:32:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:32:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:32:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:32:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:32:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:32:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:32:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:32:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:32:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:32:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:32:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:32:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:32:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:32:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:32:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:32:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:32:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:32:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:32:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:32:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:32:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:32:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:32:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:32:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:32:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:32:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:32:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:32:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:33:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:33:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:33:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:33:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:33:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:33:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:33:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:33:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:33:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:33:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:33:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:33:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:33:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:33:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:33:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:33:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:33:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:33:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:33:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:33:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:33:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:33:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:33:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:33:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:33:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:33:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:33:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:33:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:33:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:33:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:33:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:33:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:33:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:33:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:33:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:33:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:33:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:33:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:33:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:33:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:33:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:33:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:33:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:33:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:33:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:33:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:33:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:33:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:33:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:33:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:33:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:33:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:33:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:33:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:33:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:33:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:33:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:33:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:33:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:33:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:33:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:33:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:33:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:33:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:33:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:33:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:33:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:33:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:33:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:33:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:33:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:33:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:33:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:33:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:33:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:33:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:33:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:33:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:33:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:33:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:33:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:33:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:33:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:33:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:33:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:33:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:33:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:33:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:33:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:33:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:33:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:33:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:33:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:33:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:33:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:33:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:33:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:33:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:33:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:33:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:33:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:33:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:33:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:33:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:33:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:33:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:33:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:33:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:33:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:33:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:33:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:33:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:33:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:33:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:33:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:33:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:33:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:33:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:33:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:33:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:33:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:33:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:33:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:33:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:33:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:33:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:33:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:33:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:33:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:33:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:33:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:33:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:33:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:33:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:33:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:33:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:33:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:33:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:33:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:33:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:33:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:33:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:33:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:33:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:33:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:33:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:33:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:33:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:33:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:33:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:33:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:33:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:33:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:33:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:33:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:33:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:33:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:33:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:33:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:33:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:33:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:33:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:33:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:33:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:33:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:33:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:33:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:33:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:33:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:33:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:33:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:33:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:33:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:33:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:33:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:33:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:33:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:33:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:33:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:33:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:33:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:33:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:33:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:33:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:33:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:33:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:33:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:33:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:33:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:33:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:33:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:33:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:33:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:33:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:33:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:33:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:33:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:33:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:33:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:33:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:33:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:33:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:33:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:33:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:33:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:33:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:33:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:33:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:33:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:33:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:33:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:33:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:33:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:33:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:33:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:33:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:34:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:34:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:34:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:34:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:34:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:34:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:34:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:34:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:34:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:34:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:34:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:34:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:34:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:34:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:34:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:34:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:34:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:34:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:34:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:34:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:34:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:34:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:34:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:34:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:34:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:34:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:34:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:34:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:34:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:34:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:34:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:34:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:34:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:34:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:34:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:34:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:34:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:34:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:34:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:34:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:34:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:34:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:34:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:34:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:34:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:34:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:34:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:34:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:34:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:34:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:34:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:34:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:34:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:34:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:34:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:34:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:34:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:34:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:34:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:34:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:34:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:34:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:34:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:34:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:34:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:34:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:34:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:34:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:34:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:34:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:34:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:34:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:34:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:34:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:34:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:34:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:34:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:34:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:34:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:34:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:34:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:34:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:34:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:34:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:34:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:34:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:34:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:34:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:34:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:34:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:34:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:34:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:34:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:34:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:34:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:34:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:34:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:34:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:34:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:34:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:34:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:34:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:34:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:34:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:34:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:34:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:34:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:34:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:34:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:34:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:34:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:34:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:34:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:34:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:34:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:34:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:34:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:34:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:34:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:34:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:34:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:34:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:34:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:34:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:34:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:34:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:34:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:34:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:34:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:34:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:34:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:34:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:34:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:34:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:34:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:34:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:34:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:34:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:34:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:34:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:34:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:34:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:34:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:34:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:34:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:34:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:34:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:34:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:34:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:34:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:34:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:34:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:34:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:34:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:34:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:34:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:34:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:34:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:34:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:34:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:34:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:34:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:34:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:34:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:34:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:34:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:34:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:34:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:34:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:34:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:34:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:34:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:34:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:34:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:34:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:34:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:34:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:34:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:34:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:34:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:34:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:34:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:34:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:34:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:34:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:34:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:34:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:34:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:34:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:34:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:34:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:34:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:34:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:34:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:34:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:34:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:34:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:34:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:34:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:34:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:34:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:34:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:34:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:34:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:34:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:34:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:34:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:34:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:34:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:34:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:34:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:34:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:34:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:34:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:34:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:34:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:35:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:35:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:35:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:35:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:35:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:35:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:35:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:35:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:35:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:35:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:35:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:35:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:35:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:35:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:35:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:35:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:35:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:35:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:35:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:35:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:35:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:35:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:35:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:35:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:35:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:35:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:35:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:35:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:35:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:35:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:35:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:35:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:35:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:35:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:35:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:35:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:35:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:35:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:35:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:35:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:35:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:35:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:35:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:35:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:35:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:35:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:35:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:35:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:35:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:35:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:35:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:35:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:35:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:35:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:35:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:35:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:35:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:35:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:35:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:35:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:35:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:35:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:35:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:35:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:35:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:35:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:35:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:35:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:35:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:35:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:35:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:35:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:35:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:35:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:35:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:35:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:35:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:35:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:35:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:35:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:35:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:35:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:35:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:35:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:35:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:35:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:35:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:35:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:35:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:35:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:35:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:35:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:35:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:35:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:35:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:35:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:35:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:35:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:35:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:35:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:35:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:35:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:35:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:35:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:35:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:35:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:35:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:35:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:35:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:35:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:35:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:35:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:35:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:35:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:35:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:35:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:35:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:35:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:35:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:35:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:35:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:35:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:35:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:35:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:35:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:35:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:35:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:35:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:35:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:35:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:35:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:35:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:35:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:35:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:35:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:35:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:35:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:35:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:35:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:35:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:35:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:35:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:35:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:35:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:35:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:35:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:35:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:35:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:35:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:35:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:35:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:35:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:35:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:35:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:35:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:35:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:35:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:35:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:35:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:35:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:35:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:35:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:35:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:35:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:35:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:35:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:35:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:35:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:35:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:35:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:35:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:35:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:35:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:35:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:35:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:35:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:35:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:35:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:35:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:35:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:35:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:35:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:35:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:35:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:35:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:35:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:35:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:35:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:35:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:35:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:35:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:35:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:35:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:35:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:35:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:35:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:35:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:35:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:35:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:35:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:35:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:35:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:35:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:35:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:35:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:35:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:35:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:35:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:35:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:35:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:35:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:35:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:35:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:35:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:35:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:35:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:36:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:36:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:36:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:36:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:36:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:36:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:36:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:36:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:36:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:36:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:36:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:36:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:36:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:36:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:36:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:36:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:36:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:36:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:36:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:36:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:36:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:36:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:36:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:36:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:36:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:36:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:36:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:36:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:36:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:36:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:36:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:36:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:36:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:36:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:36:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:36:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:36:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:36:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:36:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:36:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:36:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:36:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:36:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:36:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:36:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:36:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:36:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:36:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:36:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:36:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:36:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:36:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:36:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:36:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:36:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:36:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:36:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:36:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:36:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:36:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:36:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:36:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:36:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:36:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:36:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:36:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:36:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:36:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:36:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:36:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:36:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:36:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:36:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:36:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:36:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:36:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:36:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:36:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:36:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:36:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:36:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:36:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:36:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:36:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:36:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:36:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:36:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:36:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:36:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:36:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:36:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:36:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:36:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:36:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:36:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:36:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:36:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:36:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:36:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:36:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:36:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:36:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:36:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:36:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:36:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:36:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:36:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:36:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:36:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:36:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:36:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:36:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:36:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:36:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:36:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:36:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:36:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:36:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:36:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:36:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:36:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:36:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:36:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:36:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:36:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:36:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:36:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:36:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:36:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:36:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:36:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:36:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:36:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:36:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:36:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:36:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:36:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:36:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:36:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:36:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:36:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:36:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:36:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:36:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:36:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:36:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:36:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:36:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:36:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:36:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:36:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:36:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:36:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:36:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:36:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:36:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:36:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:36:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:36:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:36:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:36:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:36:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:36:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:36:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:36:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:36:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:36:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:36:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:36:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:36:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:36:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:36:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:36:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:36:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:36:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:36:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:36:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:36:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:36:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:36:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:36:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:36:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:36:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:36:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:36:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:36:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:36:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:36:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:36:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:36:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:36:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:36:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:36:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:36:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:36:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:36:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:36:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:36:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:36:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:36:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:36:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:36:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:36:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:36:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:36:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:36:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:36:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:36:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:36:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:36:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:36:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:36:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:36:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:36:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:36:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:36:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:37:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:37:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:37:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:37:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:37:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:37:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:37:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:37:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:37:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:37:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:37:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:37:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:37:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:37:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:37:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:37:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:37:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:37:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:37:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:37:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:37:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:37:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:37:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:37:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:37:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:37:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:37:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:37:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:37:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:37:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:37:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:37:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:37:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:37:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:37:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:37:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:37:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:37:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:37:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:37:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:37:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:37:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:37:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:37:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:37:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:37:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:37:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:37:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:37:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:37:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:37:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:37:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:37:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:37:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:37:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:37:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:37:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:37:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:37:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:37:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:37:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:37:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:37:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:37:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:37:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:37:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:37:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:37:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:37:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:37:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:37:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:37:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:37:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:37:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:37:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:37:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:37:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:37:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:37:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:37:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:37:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:37:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:37:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:37:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:37:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:37:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:37:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:37:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:37:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:37:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:37:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:37:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:37:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:37:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:37:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:37:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:37:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:37:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:37:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:37:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:37:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:37:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:37:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:37:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:37:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:37:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:37:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:37:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:37:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:37:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:37:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:37:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:37:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:37:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:37:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:37:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:37:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:37:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:37:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:37:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:37:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:37:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:37:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:37:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:37:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:37:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:37:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:37:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:37:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:37:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:37:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:37:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:37:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:37:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:37:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:37:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:37:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:37:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:37:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:37:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:37:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:37:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:37:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:37:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:37:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:37:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:37:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:37:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:37:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:37:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:37:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:37:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:37:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:37:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:37:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:37:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:37:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:37:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:37:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:37:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:37:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:37:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:37:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:37:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:37:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:37:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:37:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:37:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:37:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:37:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:37:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:37:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:37:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:37:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:37:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:37:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:37:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:37:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:37:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:37:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:37:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:37:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:37:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:37:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:37:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:37:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:37:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:37:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:37:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:37:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:37:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:37:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:37:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:37:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:37:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:37:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:37:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:37:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:37:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:37:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:37:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:37:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:37:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:37:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:37:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:37:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:37:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:37:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:37:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:37:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:37:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:37:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:37:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:37:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:37:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:37:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:38:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:38:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:38:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:38:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:38:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:38:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:38:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:38:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:38:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:38:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:38:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:38:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:38:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:38:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:38:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:38:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:38:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:38:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:38:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:38:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:38:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:38:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:38:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:38:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:38:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:38:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:38:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:38:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:38:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:38:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:38:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:38:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:38:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:38:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:38:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:38:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:38:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:38:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:38:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:38:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:38:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:38:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:38:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:38:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:38:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:38:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:38:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:38:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:38:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:38:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:38:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:38:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:38:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:38:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:38:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:38:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:38:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:38:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:38:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:38:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:38:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:38:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:38:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:38:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:38:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:38:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:38:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:38:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:38:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:38:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:38:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:38:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:38:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:38:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:38:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:38:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:38:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:38:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:38:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:38:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:38:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:38:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:38:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:38:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:38:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:38:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:38:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:38:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:38:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:38:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:38:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:38:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:38:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:38:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:38:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:38:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:38:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:38:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:38:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:38:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:38:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:38:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:38:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:38:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:38:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:38:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:38:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:38:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:38:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:38:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:38:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:38:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:38:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:38:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:38:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:38:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:38:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:38:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:38:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:38:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:38:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:38:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:38:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:38:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:38:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:38:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:38:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:38:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:38:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:38:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:38:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:38:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:38:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:38:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:38:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:38:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:38:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:38:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:38:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:38:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:38:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:38:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:38:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:38:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:38:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:38:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:38:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:38:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:38:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:38:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:38:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:38:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:38:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:38:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:38:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:38:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:38:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:38:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:38:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:38:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:38:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:38:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:38:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:38:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:38:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:38:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:38:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:38:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:38:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:38:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:38:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:38:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:38:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:38:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:38:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:38:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:38:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:38:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:38:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:38:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:38:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:38:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:38:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:38:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:38:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:38:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:38:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:38:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:38:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:38:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:38:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:38:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:38:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:38:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:38:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:38:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:38:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:38:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:38:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:38:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:38:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:38:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:38:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:38:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:38:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:38:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:38:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:38:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:38:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:38:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:38:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:38:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:38:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:38:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:38:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:38:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:39:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:39:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:39:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:39:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:39:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:39:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:39:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:39:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:39:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:39:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:39:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:39:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:39:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:39:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:39:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:39:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:39:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:39:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:39:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:39:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:39:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:39:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:39:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:39:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:39:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:39:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:39:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:39:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:39:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:39:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:39:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:39:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:39:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:39:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:39:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:39:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:39:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:39:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:39:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:39:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:39:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:39:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:39:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:39:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:39:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:39:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:39:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:39:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:39:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:39:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:39:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:39:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:39:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:39:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:39:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:39:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:39:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:39:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:39:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:39:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:39:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:39:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:39:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:39:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:39:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:39:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:39:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:39:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:39:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:39:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:39:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:39:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:39:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:39:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:39:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:39:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:39:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:39:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:39:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:39:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:39:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:39:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:39:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:39:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:39:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:39:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:39:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:39:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:39:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:39:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:39:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:39:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:39:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:39:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:39:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:39:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:39:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:39:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:39:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:39:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:39:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:39:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:39:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:39:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:39:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:39:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:39:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:39:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:39:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:39:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:39:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:39:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:39:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:39:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:39:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:39:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:39:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:39:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:39:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:39:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:39:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:39:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:39:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:39:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:39:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:39:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:39:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:39:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:39:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:39:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:39:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:39:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:39:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:39:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:39:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:39:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:39:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:39:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:39:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:39:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:39:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:39:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:39:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:39:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:39:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:39:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:39:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:39:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:39:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:39:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:39:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:39:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:39:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:39:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:39:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:39:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:39:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:39:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:39:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:39:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:39:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:39:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:39:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:39:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:39:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:39:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:39:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:39:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:39:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:39:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:39:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:39:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:39:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:39:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:39:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:39:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:39:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:39:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:39:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:39:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:39:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:39:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:39:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:39:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:39:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:39:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:39:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:39:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:39:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:39:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:39:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:39:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:39:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:39:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:39:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:39:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:39:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:39:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:39:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:39:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:39:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:39:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:39:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:39:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:39:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:39:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:39:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:39:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:39:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:39:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:39:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:39:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:39:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:39:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:39:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:39:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:40:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:40:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:40:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:40:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:40:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:40:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:40:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:40:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:40:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:40:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:40:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:40:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:40:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:40:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:40:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:40:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:40:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:40:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:40:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:40:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:40:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:40:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:40:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:40:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:40:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:40:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:40:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:40:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:40:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:40:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:40:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:40:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:40:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:40:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:40:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:40:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:42:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:42:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:42:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:42:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:42:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:42:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:42:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:42:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:42:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:42:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:42:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:42:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:42:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:42:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:42:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:42:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:42:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:42:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:42:16 rdma06 rsyslogd[1041]: imjournal: 5294 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 03:42:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:42:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:42:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:42:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:42:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:42:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:42:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:42:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:42:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:42:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:42:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:42:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:42:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:42:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:42:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:42:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:42:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:42:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:42:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:42:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:42:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:42:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:42:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:42:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:42:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:42:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:42:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:42:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:42:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:42:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:42:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:42:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:42:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:42:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:42:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:42:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:42:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:42:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:42:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:42:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:42:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:42:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:42:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:42:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:42:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:42:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:42:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:42:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:42:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:42:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:42:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:42:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:42:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:42:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:42:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:42:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:42:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:42:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:42:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:42:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:42:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:42:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:42:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:42:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:42:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:42:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:42:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:42:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:42:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:42:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:42:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:42:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:42:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:42:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:42:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:42:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:42:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:42:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:42:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:42:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:42:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:42:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:42:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:42:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:42:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:42:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:42:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:42:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:42:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:42:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:42:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:42:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:42:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:42:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:42:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:42:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:42:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:42:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:42:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:42:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:42:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:42:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:42:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:42:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:42:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:42:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:42:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:42:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:42:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:42:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:42:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:42:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:42:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:42:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:42:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:42:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:42:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:42:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:42:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:42:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:42:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:42:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:42:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:42:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:42:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:42:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:42:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:42:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:42:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:42:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:42:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:42:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:42:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:42:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:42:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:42:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:42:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:42:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:42:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:42:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:42:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:42:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:42:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:42:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:43:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:43:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:43:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:43:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:43:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:43:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:43:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:43:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:43:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:43:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:43:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:43:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:43:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:43:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:43:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:43:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:43:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:43:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:43:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:43:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:43:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:43:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:43:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:43:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:43:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:43:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:43:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:43:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:43:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:43:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:43:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:43:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:43:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:43:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:43:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:43:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:43:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:43:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:43:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:43:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:43:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:43:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:43:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:43:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:43:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:43:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:43:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:43:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:43:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:43:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:43:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:43:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:43:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:43:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:43:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:43:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:43:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:43:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:43:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:43:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:43:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:43:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:43:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:43:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:43:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:43:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:43:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:43:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:43:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:43:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:43:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:43:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:43:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:43:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:43:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:43:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:43:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:43:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:43:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:43:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:43:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:43:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:43:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:43:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:43:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:43:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:43:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:43:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:43:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:43:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:43:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:43:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:43:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:43:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:43:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:43:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:43:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:43:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:43:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:43:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:43:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:43:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:43:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:43:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:43:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:43:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:43:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:43:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:43:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:43:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:43:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:43:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:43:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:43:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:43:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:43:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:43:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:43:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:43:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:43:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:43:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:43:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:43:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:43:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:43:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:43:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:43:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:43:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:43:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:43:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:43:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:43:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:43:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:43:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:43:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:43:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:43:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:43:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:43:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:43:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:43:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:43:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:43:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:43:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:43:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:43:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:43:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:43:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:43:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:43:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:43:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:43:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:43:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:43:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:43:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:43:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:43:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:43:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:43:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:43:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:43:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:43:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:43:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:43:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:43:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:43:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:43:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:43:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:43:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:43:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:43:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:43:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:43:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:43:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:43:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:43:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:43:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:43:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:43:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:43:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:43:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:43:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:43:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:43:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:43:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:43:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:43:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:43:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:43:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:43:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:43:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:43:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:43:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:43:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:43:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:43:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:43:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:43:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:43:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:43:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:43:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:43:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:43:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:43:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:43:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:43:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:43:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:43:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:43:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:43:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:43:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:43:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:43:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:43:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:43:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:43:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:44:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:44:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:44:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:44:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:44:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:44:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:44:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:44:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:44:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:44:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:44:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:44:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:44:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:44:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:44:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:44:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:44:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:44:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:44:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:44:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:44:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:44:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:44:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:44:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:44:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:44:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:44:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:44:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:44:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:44:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:44:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:44:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:44:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:44:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:44:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:44:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:44:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:44:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:44:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:44:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:44:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:44:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:44:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:44:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:44:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:44:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:44:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:44:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:44:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:44:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:44:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:44:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:44:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:44:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:44:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:44:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:44:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:44:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:44:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:44:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:44:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:44:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:44:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:44:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:44:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:44:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:44:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:44:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:44:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:44:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:44:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:44:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:44:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:44:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:44:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:44:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:44:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:44:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:44:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:44:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:44:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:44:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:44:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:44:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:44:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:44:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:44:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:44:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:44:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:44:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:44:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:44:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:44:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:44:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:44:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:44:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:44:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:44:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:44:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:44:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:44:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:44:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:44:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:44:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:44:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:44:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:44:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:44:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:44:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:44:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:44:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:44:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:44:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:44:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:44:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:44:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:44:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:44:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:44:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:44:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:44:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:44:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:44:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:44:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:44:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:44:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:44:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:44:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:44:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:44:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:44:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:44:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:44:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:44:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:44:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:44:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:44:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:44:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:44:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:44:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:44:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:44:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:44:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:44:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:44:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:44:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:44:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:44:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:44:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:44:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:44:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:44:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:44:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:44:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:44:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:44:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:44:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:44:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:44:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:44:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:44:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:44:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:44:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:44:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:44:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:44:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:44:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:44:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:44:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:44:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:44:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:44:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:44:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:44:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:44:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:44:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:44:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:44:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:44:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:44:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:44:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:44:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:44:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:44:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:44:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:44:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:44:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:44:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:44:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:44:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:44:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:44:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:44:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:44:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:44:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:44:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:44:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:44:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:44:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:44:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:44:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:44:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:44:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:44:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:44:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:44:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:44:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:44:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:44:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:44:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:44:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:44:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:44:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:44:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:44:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:44:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:45:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:45:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:45:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:45:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:45:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:45:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:45:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:45:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:45:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:45:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:45:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:45:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:45:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:45:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:45:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:45:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:45:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:45:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:45:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:45:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:45:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:45:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:45:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:45:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:45:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:45:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:45:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:45:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:45:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:45:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:45:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:45:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:45:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:45:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:45:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:45:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:45:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:45:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:45:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:45:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:45:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:45:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:45:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:45:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:45:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:45:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:45:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:45:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:45:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:45:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:45:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:45:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:45:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:45:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:45:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:45:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:45:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:45:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:45:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:45:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:45:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:45:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:45:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:45:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:45:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:45:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:45:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:45:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:45:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:45:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:45:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:45:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:45:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:45:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:45:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:45:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:45:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:45:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:45:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:45:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:45:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:45:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:45:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:45:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:45:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:45:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:45:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:45:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:45:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:45:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:45:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:45:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:45:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:45:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:45:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:45:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:45:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:45:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:45:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:45:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:45:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:45:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:45:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:45:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:45:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:45:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:45:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:45:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:45:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:45:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:45:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:45:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:45:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:45:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:45:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:45:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:45:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:45:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:45:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:45:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:45:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:45:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:45:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:45:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:45:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:45:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:45:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:45:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:45:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:45:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:45:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:45:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:45:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:45:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:45:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:45:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:45:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:45:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:45:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:45:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:45:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:45:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:45:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:45:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:45:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:45:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:45:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:45:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:45:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:45:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:45:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:45:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:45:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:45:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:45:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:45:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:45:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:45:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:45:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:45:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:45:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:45:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:45:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:45:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:45:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:45:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:45:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:45:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:45:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:45:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:45:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:45:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:45:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:45:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:45:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:45:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:45:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:45:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:45:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:45:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:45:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:45:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:45:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:45:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:45:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:45:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:45:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:45:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:45:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:45:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:45:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:45:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:45:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:45:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:45:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:45:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:45:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:45:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:45:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:45:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:45:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:45:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:45:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:45:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:45:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:45:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:45:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:45:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:45:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:45:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:45:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:45:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:45:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:45:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:45:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:45:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:46:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:46:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:46:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:46:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:46:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:46:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:46:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:46:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:46:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:46:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:46:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:46:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:46:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:46:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:46:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:46:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:46:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:46:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:46:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:46:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:46:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:46:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:46:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:46:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:46:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:46:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:46:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:46:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:46:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:46:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:46:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:46:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:46:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:46:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:46:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:46:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:46:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:46:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:46:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:46:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:46:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:46:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:46:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:46:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:46:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:46:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:46:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:46:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:46:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:46:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:46:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:46:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:46:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:46:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:46:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:46:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:46:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:46:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:46:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:46:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:46:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:46:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:46:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:46:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:46:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:46:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:46:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:46:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:46:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:46:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:46:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:46:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:46:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:46:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:46:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:46:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:46:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:46:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:46:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:46:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:46:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:46:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:46:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:46:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:46:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:46:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:46:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:46:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:46:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:46:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:46:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:46:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:46:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:46:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:46:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:46:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:46:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:46:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:46:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:46:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:46:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:46:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:46:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:46:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:46:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:46:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:46:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:46:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:46:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:46:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:46:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:46:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:46:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:46:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:46:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:46:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:46:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:46:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:46:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:46:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:46:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:46:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:46:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:46:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:46:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:46:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:46:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:46:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:46:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:46:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:46:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:46:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:46:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:46:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:46:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:46:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:46:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:46:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:46:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:46:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:46:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:46:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:46:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:46:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:46:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:46:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:46:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:46:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:46:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:46:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:46:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:46:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:46:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:46:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:46:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:46:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:46:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:46:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:46:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:46:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:46:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:46:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:46:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:46:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:46:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:46:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:46:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:46:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:46:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:46:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:46:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:46:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:46:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:46:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:46:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:46:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:46:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:46:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:46:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:46:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:46:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:46:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:46:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:46:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:46:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:46:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:46:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:46:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:46:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:46:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:46:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:46:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:46:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:46:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:46:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:46:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:46:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:46:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:46:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:46:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:46:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:46:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:46:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:46:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:46:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:46:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:46:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:46:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:46:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:46:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:46:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:46:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:46:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:46:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:46:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:46:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:47:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:47:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:47:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:47:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:47:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:47:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:47:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:47:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:47:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:47:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:47:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:47:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:47:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:47:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:47:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:47:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:47:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:47:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:47:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:47:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:47:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:47:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:47:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:47:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:47:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:47:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:47:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:47:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:47:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:47:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:47:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:47:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:47:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:47:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:47:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:47:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:47:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:47:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:47:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:47:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:47:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:47:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:47:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:47:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:47:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:47:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:47:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:47:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:47:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:47:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:47:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:47:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:47:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:47:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:47:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:47:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:47:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:47:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:47:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:47:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:47:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:47:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:47:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:47:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:47:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:47:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:47:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:47:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:47:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:47:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:47:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:47:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:47:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:47:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:47:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:47:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:47:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:47:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:47:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:47:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:47:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:47:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:47:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:47:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:47:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:47:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:47:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:47:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:47:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:47:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:47:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:47:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:47:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:47:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:47:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:47:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:47:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:47:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:47:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:47:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:47:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:47:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:47:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:47:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:47:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:47:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:47:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:47:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:47:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:47:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:47:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:47:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:47:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:47:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:47:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:47:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:47:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:47:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:47:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:47:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:47:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:47:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:47:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:47:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:47:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:47:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:47:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:47:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:47:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:47:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:47:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:47:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:47:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:47:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:47:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:47:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:47:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:47:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:47:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:47:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:47:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:47:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:47:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:47:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:47:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:47:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:47:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:47:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:47:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:47:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:47:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:47:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:47:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:47:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:47:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:47:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:47:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:47:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:47:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:47:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:47:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:47:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:47:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:47:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:47:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:47:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:47:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:47:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:47:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:47:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:47:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:47:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:47:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:47:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:47:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:47:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:47:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:47:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:47:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:47:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:47:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:47:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:47:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:47:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:47:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:47:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:47:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:47:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:47:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:47:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:47:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:47:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:47:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:47:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:47:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:47:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:47:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:47:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:47:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:47:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:47:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:47:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:47:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:47:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:47:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:47:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:47:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:47:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:47:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:47:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:47:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:47:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:47:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:47:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:47:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:47:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:48:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:48:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:48:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:48:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:48:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:48:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:48:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:48:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:48:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:48:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:48:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:48:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:48:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:48:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:48:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:48:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:48:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:48:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:48:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:48:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:48:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:48:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:48:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:48:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:48:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:48:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:48:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:48:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:48:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:48:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:48:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:48:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:48:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:48:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:48:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:48:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:48:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:48:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:48:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:48:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:48:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:48:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:48:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:48:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:48:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:48:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:48:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:48:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:48:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:48:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:48:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:48:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:48:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:48:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:48:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:48:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:48:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:48:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:48:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:48:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:48:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:48:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:48:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:48:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:48:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:48:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:48:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:48:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:48:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:48:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:48:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:48:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:48:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:48:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:48:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:48:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:48:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:48:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:48:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:48:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:48:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:48:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:48:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:48:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:48:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:48:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:48:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:48:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:48:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:48:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:48:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:48:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:48:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:48:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:48:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:48:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:48:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:48:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:48:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:48:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:48:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:48:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:48:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:48:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:48:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:48:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:48:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:48:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:48:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:48:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:48:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:48:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:48:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:48:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:48:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:48:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:48:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:48:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:48:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:48:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:48:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:48:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:48:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:48:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:48:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:48:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:48:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:48:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:48:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:48:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:48:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:48:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:48:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:48:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:48:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:48:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:48:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:48:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:48:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:48:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:48:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:48:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:48:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:48:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:48:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:48:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:48:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:48:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:48:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:48:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:48:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:48:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:48:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:48:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:48:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:48:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:48:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:48:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:48:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:48:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:48:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:48:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:48:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:48:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:48:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:48:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:48:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:48:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:48:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:48:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:48:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:48:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:48:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:48:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:48:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:48:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:48:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:48:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:48:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:48:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:48:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:48:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:48:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:48:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:48:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:48:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:48:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:48:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:48:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:48:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:48:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:48:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:48:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:48:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:48:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:48:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:48:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:48:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:48:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:48:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:48:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:48:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:48:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:48:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:48:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:48:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:48:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:48:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:48:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:48:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:48:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:48:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:48:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:48:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:48:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:48:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:49:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:49:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:49:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:49:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:49:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:49:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:49:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:49:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:49:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:49:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:49:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:49:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:49:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:49:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:49:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:49:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:49:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:49:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:49:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:49:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:49:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:49:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:49:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:49:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:49:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:49:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:49:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:49:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:49:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:49:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:49:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:49:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:49:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:49:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:49:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:49:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:49:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:49:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:49:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:49:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:49:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:49:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:49:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:49:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:49:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:49:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:49:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:49:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:49:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:49:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:49:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:49:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:49:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:49:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:49:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:49:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:49:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:49:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:49:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:49:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:49:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:49:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:49:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:49:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:49:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:49:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:49:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:49:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:49:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:49:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:49:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:49:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:49:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:49:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:49:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:49:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:49:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:49:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:49:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:49:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:49:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:49:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:49:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:49:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:49:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:49:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:49:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:49:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:49:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:49:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:49:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:49:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:49:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:49:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:49:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:49:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:49:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:49:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:49:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:49:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:49:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:49:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:49:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:49:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:49:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:49:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:49:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:49:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:49:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:49:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:49:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:49:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:49:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:49:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:49:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:49:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:49:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:49:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:49:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:49:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:49:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:49:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:49:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:49:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:49:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:49:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:49:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:49:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:49:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:49:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:49:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:49:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:49:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:49:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:49:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:49:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:49:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:49:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:49:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:49:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:49:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:49:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:49:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:49:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:49:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:49:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:49:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:49:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:49:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:49:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:49:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:49:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:49:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:49:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:49:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:49:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:49:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:49:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:49:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:49:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:49:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:49:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:49:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:49:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:49:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:49:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:49:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:49:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:49:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:49:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:49:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:49:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:49:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:49:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:49:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:49:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:49:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:49:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:49:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:49:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:49:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:49:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:49:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:49:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:49:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:49:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:49:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:49:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:49:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:49:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:49:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:49:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:49:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:49:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:49:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:49:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:49:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:49:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:49:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:49:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:49:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:49:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:49:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:49:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:49:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:49:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:49:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:49:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:49:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:49:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:49:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:49:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:49:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:49:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:49:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:49:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:50:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:50:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:50:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:50:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:50:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:50:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:50:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:50:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:50:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:50:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:50:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:50:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:50:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:50:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:50:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:50:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:50:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:50:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:50:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:50:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:50:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:50:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:50:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:50:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:50:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:50:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:50:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:50:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:50:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:50:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:50:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:50:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:50:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:50:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:50:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:50:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:50:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:50:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:50:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:50:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:50:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:50:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:50:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:50:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:50:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:50:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:50:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:50:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:50:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:50:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:50:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:50:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:50:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:50:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:52:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:52:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:52:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:52:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:52:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:52:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:52:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:52:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:52:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:52:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:52:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:52:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:52:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:52:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:52:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:52:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:52:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:52:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:52:21 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 03:52:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:52:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:52:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:52:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:52:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:52:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:52:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:52:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:52:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:52:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:52:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:52:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:52:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:52:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:52:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:52:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:52:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:52:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:52:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:52:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:52:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:52:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:52:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:52:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:52:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:52:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:52:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:52:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:52:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:52:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:52:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:52:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:52:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:52:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:52:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:52:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:52:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:52:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:52:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:52:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:52:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:52:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:52:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:52:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:52:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:52:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:52:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:52:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:52:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:52:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:52:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:52:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:52:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:52:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:52:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:52:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:52:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:52:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:52:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:52:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:52:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:52:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:52:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:52:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:52:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:52:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:52:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:52:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:52:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:52:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:52:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:52:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:52:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:52:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:52:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:52:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:52:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:52:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:52:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:52:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:52:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:52:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:52:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:52:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:52:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:52:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:52:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:52:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:52:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:52:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:52:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:52:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:52:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:52:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:52:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:52:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:52:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:52:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:52:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:52:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:52:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:52:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:52:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:52:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:52:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:52:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:52:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:52:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:52:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:52:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:52:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:52:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:52:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:52:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:52:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:52:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:52:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:52:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:52:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:52:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:52:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:52:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:52:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:52:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:52:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:52:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:53:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:53:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:53:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:53:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:53:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:53:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:53:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:53:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:53:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:53:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:53:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:53:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:53:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:53:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:53:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:53:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:53:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:53:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:53:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:53:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:53:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:53:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:53:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:53:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:53:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:53:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:53:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:53:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:53:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:53:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:53:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:53:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:53:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:53:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:53:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:53:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:53:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:53:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:53:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:53:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:53:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:53:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:53:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:53:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:53:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:53:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:53:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:53:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:53:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:53:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:53:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:53:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:53:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:53:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:53:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:53:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:53:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:53:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:53:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:53:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:53:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:53:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:53:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:53:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:53:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:53:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:53:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:53:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:53:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:53:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:53:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:53:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:53:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:53:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:53:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:53:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:53:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:53:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:53:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:53:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:53:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:53:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:53:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:53:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:53:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:53:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:53:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:53:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:53:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:53:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:53:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:53:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:53:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:53:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:53:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:53:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:53:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:53:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:53:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:53:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:53:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:53:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:53:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:53:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:53:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:53:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:53:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:53:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:53:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:53:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:53:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:53:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:53:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:53:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:53:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:53:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:53:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:53:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:53:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:53:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:53:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:53:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:53:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:53:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:53:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:53:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:53:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:53:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:53:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:53:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:53:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:53:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:53:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:53:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:53:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:53:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:53:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:53:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:53:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:53:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:53:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:53:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:53:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:53:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:53:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:53:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:53:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:53:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:53:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:53:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:53:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:53:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:53:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:53:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:53:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:53:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:53:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:53:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:53:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:53:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:53:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:53:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:53:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:53:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:53:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:53:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:53:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:53:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:53:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:53:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:53:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:53:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:53:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:53:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:53:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:53:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:53:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:53:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:53:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:53:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:53:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:53:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:53:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:53:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:53:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:53:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:53:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:53:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:53:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:53:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:53:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:53:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:53:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:53:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:53:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:53:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:53:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:53:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:53:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:53:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:53:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:53:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:53:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:53:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:53:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:53:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:53:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:53:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:53:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:53:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:53:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:53:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:53:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:53:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:53:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:53:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:54:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:54:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:54:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:54:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:54:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:54:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:54:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:54:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:54:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:54:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:54:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:54:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:54:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:54:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:54:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:54:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:54:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:54:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:54:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:54:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:54:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:54:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:54:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:54:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:54:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:54:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:54:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:54:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:54:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:54:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:54:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:54:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:54:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:54:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:54:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:54:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:54:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:54:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:54:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:54:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:54:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:54:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:54:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:54:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:54:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:54:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:54:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:54:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:54:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:54:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:54:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:54:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:54:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:54:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:54:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:54:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:54:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:54:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:54:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:54:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:54:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:54:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:54:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:54:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:54:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:54:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:54:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:54:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:54:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:54:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:54:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:54:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:54:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:54:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:54:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:54:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:54:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:54:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:54:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:54:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:54:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:54:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:54:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:54:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:54:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:54:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:54:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:54:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:54:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:54:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:54:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:54:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:54:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:54:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:54:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:54:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:54:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:54:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:54:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:54:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:54:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:54:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:54:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:54:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:54:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:54:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:54:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:54:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:54:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:54:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:54:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:54:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:54:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:54:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:54:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:54:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:54:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:54:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:54:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:54:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:54:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:54:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:54:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:54:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:54:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:54:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:54:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:54:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:54:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:54:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:54:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:54:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:54:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:54:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:54:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:54:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:54:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:54:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:54:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:54:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:54:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:54:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:54:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:54:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:54:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:54:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:54:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:54:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:54:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:54:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:54:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:54:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:54:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:54:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:54:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:54:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:54:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:54:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:54:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:54:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:54:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:54:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:54:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:54:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:54:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:54:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:54:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:54:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:54:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:54:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:54:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:54:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:54:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:54:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:54:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:54:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:54:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:54:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:54:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:54:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:54:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:54:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:54:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:54:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:54:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:54:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:54:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:54:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:54:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:54:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:54:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:54:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:54:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:54:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:54:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:54:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:54:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:54:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:54:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:54:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:54:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:54:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:54:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:54:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:54:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:54:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:54:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:54:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:54:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:54:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:54:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:54:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:54:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:54:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:54:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:54:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:55:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:55:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:55:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:55:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:55:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:55:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:55:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:55:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:55:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:55:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:55:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:55:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:55:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:55:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:55:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:55:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:55:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:55:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:55:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:55:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:55:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:55:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:55:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:55:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:55:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:55:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:55:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:55:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:55:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:55:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:55:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:55:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:55:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:55:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:55:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:55:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:55:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:55:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:55:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:55:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:55:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:55:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:55:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:55:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:55:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:55:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:55:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:55:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:55:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:55:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:55:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:55:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:55:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:55:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:55:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:55:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:55:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:55:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:55:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:55:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:55:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:55:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:55:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:55:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:55:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:55:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:55:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:55:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:55:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:55:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:55:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:55:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:55:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:55:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:55:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:55:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:55:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:55:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:55:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:55:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:55:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:55:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:55:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:55:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:55:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:55:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:55:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:55:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:55:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:55:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:55:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:55:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:55:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:55:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:55:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:55:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:55:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:55:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:55:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:55:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:55:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:55:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:55:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:55:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:55:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:55:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:55:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:55:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:55:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:55:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:55:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:55:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:55:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:55:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:55:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:55:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:55:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:55:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:55:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:55:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:55:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:55:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:55:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:55:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:55:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:55:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:55:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:55:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:55:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:55:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:55:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:55:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:55:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:55:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:55:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:55:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:55:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:55:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:55:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:55:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:55:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:55:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:55:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:55:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:55:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:55:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:55:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:55:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:55:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:55:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:55:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:55:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:55:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:55:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:55:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:55:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:55:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:55:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:55:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:55:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:55:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:55:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:55:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:55:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:55:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:55:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:55:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:55:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:55:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:55:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:55:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:55:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:55:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:55:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:55:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:55:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:55:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:55:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:55:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:55:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:55:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:55:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:55:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:55:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:55:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:55:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:55:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:55:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:55:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:55:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:55:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:55:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:55:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:55:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:55:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:55:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:55:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:55:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:55:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:55:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:55:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:55:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:55:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:55:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:55:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:55:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:55:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:55:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:55:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:55:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:55:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:55:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:55:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:55:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:55:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:55:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:56:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:56:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:56:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:56:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:56:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:56:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:56:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:56:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:56:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:56:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:56:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:56:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:56:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:56:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:56:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:56:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:56:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:56:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:56:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:56:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:56:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:56:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:56:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:56:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:56:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:56:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:56:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:56:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:56:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:56:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:56:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:56:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:56:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:56:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:56:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:56:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:56:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:56:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:56:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:56:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:56:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:56:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:56:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:56:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:56:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:56:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:56:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:56:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:56:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:56:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:56:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:56:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:56:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:56:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:56:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:56:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:56:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:56:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:56:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:56:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:56:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:56:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:56:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:56:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:56:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:56:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:56:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:56:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:56:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:56:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:56:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:56:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:56:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:56:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:56:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:56:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:56:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:56:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:56:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:56:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:56:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:56:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:56:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:56:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:56:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:56:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:56:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:56:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:56:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:56:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:56:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:56:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:56:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:56:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:56:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:56:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:56:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:56:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:56:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:56:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:56:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:56:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:56:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:56:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:56:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:56:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:56:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:56:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:56:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:56:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:56:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:56:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:56:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:56:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:56:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:56:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:56:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:56:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:56:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:56:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:56:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:56:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:56:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:56:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:56:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:56:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:56:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:56:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:56:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:56:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:56:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:56:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:56:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:56:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:56:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:56:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:56:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:56:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:56:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:56:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:56:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:56:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:56:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:56:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:56:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:56:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:56:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:56:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:56:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:56:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:56:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:56:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:56:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:56:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:56:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:56:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:56:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:56:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:56:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:56:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:56:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:56:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:56:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:56:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:56:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:56:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:56:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:56:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:56:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:56:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:56:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:56:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:56:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:56:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:56:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:56:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:56:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:56:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:56:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:56:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:56:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:56:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:56:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:56:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:56:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:56:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:56:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:56:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:56:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:56:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:56:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:56:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:56:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:56:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:56:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:56:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:56:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:56:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:56:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:56:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:56:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:56:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:56:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:56:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:56:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:56:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:56:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:56:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:56:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:56:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:56:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:56:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:56:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:56:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:56:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:56:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:57:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:57:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:57:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:57:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:57:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:57:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:57:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:57:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:57:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:57:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:57:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:57:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:57:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:57:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:57:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:57:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:57:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:57:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:57:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:57:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:57:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:57:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:57:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:57:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:57:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:57:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:57:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:57:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:57:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:57:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:57:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:57:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:57:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:57:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:57:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:57:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:57:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:57:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:57:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:57:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:57:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:57:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:57:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:57:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:57:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:57:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:57:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:57:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:57:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:57:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:57:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:57:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:57:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:57:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:57:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:57:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:57:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:57:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:57:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:57:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:57:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:57:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:57:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:57:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:57:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:57:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:57:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:57:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:57:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:57:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:57:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:57:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:57:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:57:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:57:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:57:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:57:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:57:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:57:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:57:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:57:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:57:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:57:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:57:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:57:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:57:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:57:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:57:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:57:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:57:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:57:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:57:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:57:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:57:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:57:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:57:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:57:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:57:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:57:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:57:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:57:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:57:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:57:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:57:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:57:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:57:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:57:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:57:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:57:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:57:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:57:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:57:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:57:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:57:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:57:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:57:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:57:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:57:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:57:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:57:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:57:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:57:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:57:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:57:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:57:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:57:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:57:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:57:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:57:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:57:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:57:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:57:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:57:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:57:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:57:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:57:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:57:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:57:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:57:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:57:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:57:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:57:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:57:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:57:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:57:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:57:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:57:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:57:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:57:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:57:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:57:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:57:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:57:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:57:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:57:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:57:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:57:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:57:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:57:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:57:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:57:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:57:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:57:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:57:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:57:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:57:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:57:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:57:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:57:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:57:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:57:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:57:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:57:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:57:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:57:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:57:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:57:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:57:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:57:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:57:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:57:46 rdma06 systemd[1]: Starting dnf makecache...
Apr 18 03:57:47 rdma06 dnf[1178014]: Updating Subscription Management repositories.
Apr 18 03:57:47 rdma06 dnf[1178014]: Unable to read consumer identity
Apr 18 03:57:47 rdma06 dnf[1178014]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 18 03:57:50 rdma06 dnf[1178014]: Copr repo for fmf owned by psss                 1.1 kB/s | 3.3 kB     00:03
Apr 18 03:57:50 rdma06 dnf[1178014]: beaker-AppStream                                 48 kB/s | 168  B     00:00
Apr 18 03:57:50 rdma06 dnf[1178014]: Errors during downloading metadata for repository 'beaker-AppStream':
Apr 18 03:57:50 rdma06 dnf[1178014]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 03:57:50 rdma06 dnf[1178014]: Error: Failed to download metadata for repo 'beaker-AppStream': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 03:57:50 rdma06 dnf[1178014]: beaker-AppStream-debuginfo                       55 kB/s | 168  B     00:00
Apr 18 03:57:50 rdma06 dnf[1178014]: Errors during downloading metadata for repository 'beaker-AppStream-debuginfo':
Apr 18 03:57:50 rdma06 dnf[1178014]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 03:57:50 rdma06 dnf[1178014]: Error: Failed to download metadata for repo 'beaker-AppStream-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 03:57:50 rdma06 dnf[1178014]: beaker-BaseOS                                    57 kB/s | 168  B     00:00
Apr 18 03:57:50 rdma06 dnf[1178014]: Errors during downloading metadata for repository 'beaker-BaseOS':
Apr 18 03:57:50 rdma06 dnf[1178014]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 03:57:50 rdma06 dnf[1178014]: Error: Failed to download metadata for repo 'beaker-BaseOS': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 03:57:50 rdma06 dnf[1178014]: beaker-BaseOS-debuginfo                          54 kB/s | 168  B     00:00
Apr 18 03:57:50 rdma06 dnf[1178014]: Errors during downloading metadata for repository 'beaker-BaseOS-debuginfo':
Apr 18 03:57:50 rdma06 dnf[1178014]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 03:57:50 rdma06 dnf[1178014]: Error: Failed to download metadata for repo 'beaker-BaseOS-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 03:57:50 rdma06 dnf[1178014]: beaker-CRB                                       57 kB/s | 168  B     00:00
Apr 18 03:57:50 rdma06 dnf[1178014]: Errors during downloading metadata for repository 'beaker-CRB':
Apr 18 03:57:50 rdma06 dnf[1178014]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 03:57:50 rdma06 dnf[1178014]: Error: Failed to download metadata for repo 'beaker-CRB': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 03:57:50 rdma06 dnf[1178014]: beaker-CRB-debuginfo                             55 kB/s | 168  B     00:00
Apr 18 03:57:50 rdma06 dnf[1178014]: Errors during downloading metadata for repository 'beaker-CRB-debuginfo':
Apr 18 03:57:50 rdma06 dnf[1178014]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 03:57:50 rdma06 dnf[1178014]: Error: Failed to download metadata for repo 'beaker-CRB-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 03:57:50 rdma06 dnf[1178014]: beaker-HighAvailability                          57 kB/s | 168  B     00:00
Apr 18 03:57:50 rdma06 dnf[1178014]: Errors during downloading metadata for repository 'beaker-HighAvailability':
Apr 18 03:57:50 rdma06 dnf[1178014]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 03:57:50 rdma06 dnf[1178014]: Error: Failed to download metadata for repo 'beaker-HighAvailability': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 03:57:50 rdma06 dnf[1178014]: beaker-HighAvailability-debuginfo                67 kB/s | 168  B     00:00
Apr 18 03:57:50 rdma06 dnf[1178014]: Errors during downloading metadata for repository 'beaker-HighAvailability-debuginfo':
Apr 18 03:57:50 rdma06 dnf[1178014]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 03:57:50 rdma06 dnf[1178014]: Error: Failed to download metadata for repo 'beaker-HighAvailability-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 03:57:50 rdma06 dnf[1178014]: beaker-NFV                                       63 kB/s | 168  B     00:00
Apr 18 03:57:50 rdma06 dnf[1178014]: Errors during downloading metadata for repository 'beaker-NFV':
Apr 18 03:57:50 rdma06 dnf[1178014]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 03:57:50 rdma06 dnf[1178014]: Error: Failed to download metadata for repo 'beaker-NFV': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 03:57:50 rdma06 dnf[1178014]: beaker-NFV-debuginfo                             56 kB/s | 168  B     00:00
Apr 18 03:57:50 rdma06 dnf[1178014]: Errors during downloading metadata for repository 'beaker-NFV-debuginfo':
Apr 18 03:57:50 rdma06 dnf[1178014]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 03:57:50 rdma06 dnf[1178014]: Error: Failed to download metadata for repo 'beaker-NFV-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 03:57:50 rdma06 dnf[1178014]: beaker-RT                                        63 kB/s | 168  B     00:00
Apr 18 03:57:50 rdma06 dnf[1178014]: Errors during downloading metadata for repository 'beaker-RT':
Apr 18 03:57:50 rdma06 dnf[1178014]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 03:57:50 rdma06 dnf[1178014]: Error: Failed to download metadata for repo 'beaker-RT': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 03:57:50 rdma06 dnf[1178014]: beaker-RT-debuginfo                              58 kB/s | 168  B     00:00
Apr 18 03:57:50 rdma06 dnf[1178014]: Errors during downloading metadata for repository 'beaker-RT-debuginfo':
Apr 18 03:57:50 rdma06 dnf[1178014]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 03:57:50 rdma06 dnf[1178014]: Error: Failed to download metadata for repo 'beaker-RT-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 03:57:50 rdma06 dnf[1178014]: beaker-ResilientStorage                          65 kB/s | 168  B     00:00
Apr 18 03:57:50 rdma06 dnf[1178014]: Errors during downloading metadata for repository 'beaker-ResilientStorage':
Apr 18 03:57:50 rdma06 dnf[1178014]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 03:57:50 rdma06 dnf[1178014]: Error: Failed to download metadata for repo 'beaker-ResilientStorage': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 03:57:50 rdma06 dnf[1178014]: beaker-ResilientStorage-debuginfo                56 kB/s | 168  B     00:00
Apr 18 03:57:50 rdma06 dnf[1178014]: Errors during downloading metadata for repository 'beaker-ResilientStorage-debuginfo':
Apr 18 03:57:50 rdma06 dnf[1178014]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 03:57:50 rdma06 dnf[1178014]: Error: Failed to download metadata for repo 'beaker-ResilientStorage-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 03:57:50 rdma06 dnf[1178014]: beaker-SAP                                       62 kB/s | 168  B     00:00
Apr 18 03:57:50 rdma06 dnf[1178014]: Errors during downloading metadata for repository 'beaker-SAP':
Apr 18 03:57:50 rdma06 dnf[1178014]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 03:57:50 rdma06 dnf[1178014]: Error: Failed to download metadata for repo 'beaker-SAP': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 03:57:50 rdma06 dnf[1178014]: beaker-SAP-debuginfo                             61 kB/s | 168  B     00:00
Apr 18 03:57:50 rdma06 dnf[1178014]: Errors during downloading metadata for repository 'beaker-SAP-debuginfo':
Apr 18 03:57:50 rdma06 dnf[1178014]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 03:57:50 rdma06 dnf[1178014]: Error: Failed to download metadata for repo 'beaker-SAP-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 03:57:50 rdma06 dnf[1178014]: beaker-SAPHANA                                   57 kB/s | 168  B     00:00
Apr 18 03:57:50 rdma06 dnf[1178014]: Errors during downloading metadata for repository 'beaker-SAPHANA':
Apr 18 03:57:50 rdma06 dnf[1178014]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 03:57:50 rdma06 dnf[1178014]: Error: Failed to download metadata for repo 'beaker-SAPHANA': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 03:57:50 rdma06 dnf[1178014]: beaker-SAPHANA-debuginfo                         59 kB/s | 168  B     00:00
Apr 18 03:57:50 rdma06 dnf[1178014]: Errors during downloading metadata for repository 'beaker-SAPHANA-debuginfo':
Apr 18 03:57:50 rdma06 dnf[1178014]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 03:57:50 rdma06 dnf[1178014]: Error: Failed to download metadata for repo 'beaker-SAPHANA-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 03:57:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:57:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:57:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:57:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:57:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:57:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:57:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:57:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:57:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:57:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:57:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:57:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:57:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:57:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:57:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:57:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:57:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:57:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:57:52 rdma06 dnf[1178014]: beaker-harness                                  570  B/s | 1.3 kB     00:02
Apr 18 03:57:53 rdma06 dnf[1178014]: beaker-tasks                                    1.9 kB/s | 1.3 kB     00:00
Apr 18 03:57:53 rdma06 dnf[1178014]: Ignoring repositories: beaker-AppStream, beaker-AppStream-debuginfo, beaker-BaseOS, beaker-BaseOS-debuginfo, beaker-CRB, beaker-CRB-debuginfo, beaker-HighAvailability, beaker-HighAvailability-debuginfo, beaker-NFV, beaker-NFV-debuginfo, beaker-RT, beaker-RT-debuginfo, beaker-ResilientStorage, beaker-ResilientStorage-debuginfo, beaker-SAP, beaker-SAP-debuginfo, beaker-SAPHANA, beaker-SAPHANA-debuginfo
Apr 18 03:57:53 rdma06 dnf[1178014]: Metadata cache created.
Apr 18 03:57:53 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 18 03:57:53 rdma06 systemd[1]: Finished dnf makecache.
Apr 18 03:57:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:57:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:57:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:57:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:57:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:57:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:57:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:57:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:57:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:57:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:57:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:57:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:57:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:57:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:57:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:57:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:57:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:57:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:58:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:58:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:58:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:58:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:58:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:58:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:58:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:58:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:58:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:58:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:58:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:58:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:58:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:58:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:58:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:58:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:58:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:58:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:58:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:58:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:58:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:58:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:58:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:58:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:58:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:58:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:58:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:58:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:58:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:58:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:58:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:58:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:58:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:58:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:58:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:58:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:58:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:58:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:58:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:58:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:58:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:58:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:58:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:58:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:58:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:58:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:58:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:58:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:58:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:58:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:58:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:58:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:58:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:58:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:58:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:58:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:58:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:58:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:58:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:58:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:58:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:58:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:58:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:58:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:58:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:58:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:58:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:58:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:58:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:58:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:58:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:58:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:58:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:58:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:58:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:58:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:58:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:58:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:58:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:58:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:58:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:58:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:58:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:58:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:58:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:58:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:58:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:58:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:58:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:58:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:58:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:58:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:58:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:58:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:58:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:58:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:58:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:58:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:58:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:58:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:58:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:58:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:58:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:58:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:58:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:58:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:58:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:58:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:58:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:58:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:58:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:58:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:58:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:58:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:58:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:58:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:58:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:58:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:58:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:58:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:58:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:58:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:58:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:58:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:58:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:58:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:58:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:58:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:58:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:58:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:58:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:58:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:58:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:58:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:58:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:58:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:58:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:58:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:58:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:58:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:58:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:58:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:58:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:58:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:58:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:58:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:58:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:58:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:58:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:58:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:58:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:58:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:58:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:58:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:58:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:58:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:58:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:58:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:58:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:58:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:58:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:58:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:58:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:58:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:58:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:58:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:58:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:58:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:58:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:58:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:58:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:58:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:58:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:58:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:58:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:58:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:58:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:58:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:58:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:58:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:58:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:58:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:58:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:58:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:58:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:58:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:58:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:58:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:58:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:58:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:58:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:58:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:58:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:58:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:58:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:58:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:58:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:58:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:58:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:58:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:58:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:58:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:58:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:58:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:58:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:58:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:58:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:58:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:58:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:58:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:58:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:58:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:58:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:58:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:58:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:58:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:59:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:59:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:59:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:59:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:59:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:59:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:59:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:59:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:59:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:59:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:59:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:59:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:59:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:59:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:59:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:59:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:59:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:59:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:59:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:59:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:59:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:59:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:59:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:59:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:59:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:59:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:59:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:59:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:59:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:59:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:59:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:59:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:59:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:59:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:59:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:59:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:59:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:59:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:59:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:59:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:59:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:59:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:59:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:59:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:59:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:59:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:59:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:59:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:59:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:59:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:59:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:59:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:59:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:59:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:59:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:59:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:59:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:59:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:59:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:59:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:59:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:59:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:59:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:59:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:59:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:59:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:59:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:59:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:59:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:59:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:59:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:59:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:59:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:59:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:59:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:59:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:59:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:59:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:59:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:59:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:59:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:59:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:59:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:59:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:59:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:59:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:59:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:59:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:59:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:59:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:59:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:59:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:59:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:59:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:59:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:59:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:59:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:59:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:59:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:59:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:59:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:59:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:59:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:59:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:59:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:59:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:59:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:59:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:59:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:59:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:59:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:59:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:59:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:59:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:59:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:59:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:59:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:59:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:59:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:59:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:59:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:59:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:59:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:59:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:59:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:59:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:59:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:59:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:59:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:59:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:59:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:59:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:59:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:59:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:59:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:59:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:59:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:59:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:59:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:59:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:59:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:59:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:59:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:59:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:59:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:59:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:59:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:59:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:59:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:59:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:59:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:59:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:59:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:59:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:59:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:59:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:59:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:59:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:59:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:59:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:59:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:59:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:59:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:59:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:59:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:59:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:59:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:59:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:59:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:59:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:59:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:59:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:59:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:59:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:59:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:59:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:59:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:59:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:59:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:59:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:59:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:59:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:59:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:59:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:59:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:59:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:59:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:59:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:59:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:59:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:59:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:59:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:59:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:59:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:59:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:59:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:59:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:59:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 03:59:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 03:59:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:59:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:59:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:59:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 03:59:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 03:59:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 03:59:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 03:59:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 03:59:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 03:59:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 03:59:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 03:59:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 03:59:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 03:59:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 03:59:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 03:59:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 03:59:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:00:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:00:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:00:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:00:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:00:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:00:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:00:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:00:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:00:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:00:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:00:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:00:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:00:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:00:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:00:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:00:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:00:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:00:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:00:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:00:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:00:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:00:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:00:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:00:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:00:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:00:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:00:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:00:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:00:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:00:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:00:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:00:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:00:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:00:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:00:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:00:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:00:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:00:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:00:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:00:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:00:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:00:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:00:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:00:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:00:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:00:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:00:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:00:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:00:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:00:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:00:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:00:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:00:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:00:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:00:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:00:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:00:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:00:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:00:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:00:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:00:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:00:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:00:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:00:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:00:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:00:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:00:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:00:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:00:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:02:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:02:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:02:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:02:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:02:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:02:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:02:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:02:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:02:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:02:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:02:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:02:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:02:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:02:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:02:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:02:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:02:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:02:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:02:22 rdma06 rsyslogd[1041]: imjournal: 5169 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 04:02:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:02:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:02:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:02:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:02:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:02:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:02:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:02:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:02:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:02:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:02:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:02:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:02:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:02:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:02:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:02:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:02:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:02:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:02:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:02:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:02:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:02:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:02:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:02:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:02:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:02:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:02:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:02:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:02:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:02:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:02:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:02:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:02:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:02:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:02:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:02:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:02:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:02:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:02:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:02:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:02:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:02:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:02:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:02:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:02:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:02:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:02:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:02:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:02:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:02:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:02:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:02:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:02:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:02:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:02:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:02:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:02:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:02:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:02:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:02:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:02:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:02:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:02:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:02:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:02:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:02:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:02:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:02:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:02:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:02:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:02:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:02:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:02:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:02:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:02:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:02:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:02:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:02:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:02:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:02:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:02:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:02:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:02:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:02:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:02:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:02:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:02:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:02:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:02:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:02:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:02:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:02:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:02:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:02:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:02:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:02:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:02:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:02:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:02:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:02:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:02:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:02:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:02:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:02:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:02:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:02:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:02:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:02:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:02:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:02:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:02:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:02:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:02:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:02:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:02:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:02:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:02:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:02:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:02:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:02:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:02:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:02:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:02:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:02:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:02:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:02:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:03:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:03:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:03:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:03:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:03:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:03:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:03:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:03:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:03:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:03:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:03:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:03:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:03:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:03:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:03:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:03:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:03:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:03:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:03:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:03:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:03:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:03:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:03:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:03:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:03:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:03:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:03:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:03:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:03:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:03:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:03:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:03:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:03:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:03:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:03:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:03:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:03:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:03:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:03:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:03:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:03:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:03:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:03:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:03:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:03:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:03:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:03:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:03:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:03:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:03:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:03:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:03:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:03:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:03:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:03:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:03:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:03:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:03:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:03:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:03:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:03:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:03:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:03:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:03:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:03:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:03:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:03:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:03:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:03:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:03:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:03:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:03:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:03:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:03:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:03:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:03:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:03:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:03:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:03:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:03:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:03:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:03:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:03:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:03:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:03:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:03:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:03:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:03:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:03:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:03:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:03:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:03:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:03:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:03:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:03:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:03:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:03:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:03:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:03:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:03:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:03:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:03:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:03:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:03:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:03:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:03:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:03:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:03:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:03:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:03:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:03:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:03:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:03:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:03:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:03:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:03:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:03:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:03:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:03:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:03:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:03:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:03:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:03:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:03:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:03:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:03:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:03:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:03:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:03:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:03:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:03:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:03:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:03:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:03:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:03:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:03:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:03:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:03:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:03:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:03:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:03:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:03:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:03:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:03:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:03:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:03:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:03:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:03:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:03:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:03:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:03:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:03:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:03:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:03:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:03:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:03:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:03:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:03:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:03:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:03:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:03:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:03:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:03:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:03:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:03:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:03:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:03:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:03:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:03:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:03:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:03:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:03:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:03:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:03:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:03:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:03:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:03:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:03:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:03:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:03:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:03:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:03:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:03:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:03:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:03:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:03:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:03:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:03:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:03:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:03:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:03:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:03:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:03:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:03:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:03:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:03:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:03:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:03:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:03:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:03:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:03:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:03:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:03:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:03:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:03:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:03:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:03:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:03:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:03:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:03:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:03:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:03:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:03:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:03:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:03:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:03:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:04:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:04:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:04:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:04:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:04:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:04:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:04:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:04:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:04:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:04:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:04:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:04:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:04:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:04:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:04:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:04:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:04:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:04:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:04:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:04:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:04:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:04:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:04:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:04:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:04:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:04:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:04:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:04:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:04:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:04:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:04:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:04:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:04:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:04:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:04:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:04:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:04:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:04:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:04:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:04:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:04:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:04:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:04:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:04:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:04:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:04:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:04:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:04:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:04:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:04:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:04:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:04:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:04:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:04:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:04:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:04:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:04:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:04:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:04:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:04:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:04:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:04:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:04:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:04:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:04:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:04:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:04:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:04:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:04:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:04:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:04:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:04:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:04:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:04:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:04:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:04:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:04:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:04:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:04:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:04:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:04:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:04:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:04:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:04:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:04:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:04:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:04:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:04:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:04:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:04:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:04:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:04:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:04:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:04:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:04:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:04:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:04:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:04:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:04:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:04:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:04:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:04:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:04:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:04:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:04:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:04:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:04:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:04:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:04:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:04:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:04:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:04:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:04:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:04:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:04:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:04:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:04:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:04:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:04:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:04:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:04:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:04:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:04:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:04:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:04:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:04:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:04:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:04:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:04:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:04:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:04:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:04:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:04:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:04:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:04:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:04:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:04:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:04:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:04:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:04:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:04:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:04:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:04:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:04:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:04:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:04:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:04:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:04:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:04:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:04:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:04:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:04:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:04:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:04:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:04:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:04:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:04:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:04:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:04:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:04:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:04:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:04:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:04:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:04:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:04:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:04:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:04:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:04:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:04:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:04:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:04:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:04:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:04:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:04:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:04:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:04:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:04:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:04:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:04:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:04:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:04:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:04:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:04:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:04:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:04:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:04:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:04:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:04:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:04:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:04:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:04:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:04:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:04:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:04:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:04:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:04:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:04:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:04:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:04:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:04:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:04:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:04:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:04:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:04:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:04:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:04:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:04:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:04:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:04:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:04:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:04:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:04:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:04:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:04:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:04:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:04:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:05:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:05:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:05:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:05:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:05:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:05:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:05:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:05:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:05:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:05:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:05:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:05:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:05:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:05:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:05:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:05:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:05:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:05:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:05:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:05:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:05:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:05:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:05:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:05:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:05:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:05:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:05:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:05:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:05:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:05:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:05:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:05:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:05:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:05:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:05:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:05:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:05:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:05:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:05:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:05:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:05:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:05:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:05:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:05:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:05:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:05:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:05:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:05:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:05:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:05:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:05:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:05:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:05:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:05:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:05:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:05:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:05:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:05:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:05:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:05:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:05:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:05:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:05:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:05:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:05:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:05:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:05:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:05:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:05:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:05:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:05:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:05:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:05:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:05:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:05:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:05:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:05:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:05:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:05:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:05:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:05:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:05:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:05:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:05:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:05:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:05:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:05:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:05:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:05:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:05:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:05:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:05:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:05:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:05:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:05:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:05:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:05:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:05:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:05:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:05:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:05:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:05:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:05:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:05:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:05:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:05:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:05:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:05:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:05:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:05:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:05:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:05:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:05:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:05:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:05:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:05:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:05:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:05:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:05:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:05:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:05:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:05:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:05:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:05:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:05:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:05:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:05:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:05:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:05:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:05:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:05:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:05:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:05:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:05:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:05:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:05:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:05:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:05:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:05:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:05:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:05:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:05:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:05:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:05:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:05:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:05:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:05:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:05:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:05:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:05:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:05:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:05:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:05:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:05:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:05:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:05:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:05:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:05:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:05:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:05:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:05:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:05:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:05:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:05:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:05:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:05:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:05:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:05:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:05:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:05:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:05:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:05:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:05:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:05:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:05:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:05:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:05:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:05:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:05:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:05:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:05:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:05:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:05:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:05:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:05:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:05:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:05:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:05:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:05:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:05:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:05:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:05:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:05:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:05:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:05:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:05:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:05:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:05:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:05:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:05:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:05:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:05:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:05:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:05:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:05:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:05:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:05:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:05:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:05:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:05:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:05:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:05:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:05:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:05:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:05:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:05:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:06:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:06:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:06:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:06:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:06:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:06:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:06:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:06:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:06:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:06:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:06:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:06:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:06:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:06:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:06:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:06:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:06:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:06:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:06:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:06:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:06:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:06:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:06:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:06:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:06:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:06:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:06:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:06:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:06:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:06:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:06:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:06:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:06:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:06:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:06:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:06:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:06:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:06:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:06:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:06:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:06:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:06:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:06:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:06:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:06:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:06:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:06:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:06:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:06:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:06:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:06:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:06:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:06:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:06:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:06:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:06:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:06:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:06:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:06:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:06:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:06:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:06:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:06:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:06:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:06:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:06:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:06:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:06:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:06:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:06:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:06:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:06:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:06:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:06:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:06:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:06:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:06:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:06:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:06:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:06:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:06:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:06:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:06:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:06:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:06:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:06:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:06:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:06:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:06:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:06:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:06:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:06:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:06:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:06:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:06:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:06:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:06:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:06:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:06:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:06:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:06:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:06:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:06:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:06:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:06:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:06:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:06:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:06:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:06:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:06:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:06:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:06:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:06:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:06:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:06:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:06:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:06:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:06:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:06:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:06:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:06:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:06:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:06:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:06:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:06:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:06:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:06:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:06:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:06:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:06:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:06:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:06:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:06:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:06:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:06:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:06:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:06:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:06:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:06:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:06:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:06:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:06:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:06:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:06:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:06:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:06:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:06:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:06:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:06:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:06:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:06:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:06:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:06:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:06:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:06:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:06:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:06:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:06:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:06:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:06:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:06:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:06:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:06:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:06:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:06:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:06:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:06:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:06:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:06:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:06:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:06:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:06:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:06:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:06:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:06:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:06:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:06:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:06:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:06:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:06:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:06:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:06:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:06:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:06:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:06:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:06:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:06:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:06:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:06:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:06:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:06:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:06:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:06:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:06:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:06:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:06:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:06:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:06:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:06:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:06:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:06:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:06:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:06:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:06:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:06:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:06:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:06:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:06:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:06:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:06:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:06:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:06:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:06:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:06:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:06:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:06:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:07:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:07:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:07:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:07:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:07:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:07:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:07:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:07:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:07:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:07:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:07:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:07:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:07:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:07:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:07:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:07:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:07:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:07:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:07:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:07:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:07:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:07:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:07:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:07:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:07:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:07:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:07:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:07:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:07:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:07:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:07:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:07:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:07:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:07:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:07:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:07:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:07:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:07:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:07:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:07:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:07:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:07:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:07:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:07:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:07:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:07:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:07:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:07:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:07:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:07:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:07:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:07:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:07:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:07:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:07:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:07:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:07:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:07:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:07:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:07:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:07:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:07:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:07:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:07:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:07:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:07:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:07:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:07:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:07:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:07:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:07:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:07:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:07:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:07:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:07:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:07:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:07:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:07:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:07:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:07:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:07:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:07:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:07:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:07:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:07:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:07:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:07:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:07:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:07:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:07:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:07:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:07:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:07:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:07:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:07:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:07:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:07:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:07:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:07:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:07:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:07:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:07:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:07:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:07:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:07:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:07:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:07:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:07:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:07:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:07:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:07:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:07:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:07:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:07:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:07:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:07:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:07:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:07:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:07:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:07:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:07:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:07:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:07:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:07:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:07:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:07:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:07:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:07:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:07:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:07:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:07:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:07:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:07:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:07:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:07:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:07:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:07:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:07:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:07:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:07:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:07:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:07:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:07:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:07:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:07:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:07:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:07:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:07:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:07:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:07:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:07:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:07:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:07:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:07:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:07:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:07:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:07:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:07:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:07:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:07:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:07:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:07:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:07:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:07:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:07:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:07:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:07:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:07:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:07:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:07:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:07:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:07:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:07:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:07:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:07:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:07:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:07:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:07:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:07:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:07:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:07:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:07:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:07:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:07:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:07:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:07:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:07:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:07:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:07:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:07:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:07:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:07:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:07:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:07:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:07:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:07:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:07:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:07:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:07:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:07:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:07:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:07:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:07:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:07:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:07:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:07:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:07:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:07:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:07:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:07:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:07:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:07:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:07:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:07:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:07:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:07:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:08:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:08:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:08:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:08:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:08:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:08:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:08:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:08:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:08:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:08:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:08:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:08:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:08:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:08:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:08:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:08:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:08:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:08:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:08:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:08:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:08:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:08:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:08:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:08:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:08:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:08:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:08:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:08:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:08:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:08:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:08:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:08:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:08:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:08:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:08:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:08:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:08:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:08:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:08:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:08:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:08:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:08:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:08:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:08:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:08:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:08:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:08:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:08:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:08:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:08:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:08:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:08:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:08:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:08:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:08:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:08:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:08:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:08:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:08:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:08:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:08:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:08:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:08:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:08:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:08:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:08:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:08:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:08:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:08:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:08:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:08:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:08:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:08:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:08:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:08:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:08:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:08:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:08:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:08:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:08:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:08:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:08:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:08:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:08:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:08:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:08:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:08:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:08:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:08:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:08:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:08:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:08:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:08:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:08:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:08:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:08:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:08:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:08:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:08:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:08:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:08:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:08:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:08:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:08:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:08:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:08:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:08:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:08:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:08:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:08:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:08:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:08:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:08:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:08:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:08:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:08:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:08:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:08:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:08:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:08:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:08:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:08:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:08:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:08:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:08:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:08:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:08:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:08:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:08:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:08:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:08:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:08:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:08:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:08:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:08:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:08:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:08:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:08:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:08:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:08:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:08:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:08:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:08:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:08:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:08:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:08:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:08:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:08:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:08:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:08:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:08:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:08:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:08:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:08:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:08:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:08:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:08:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:08:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:08:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:08:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:08:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:08:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:08:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:08:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:08:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:08:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:08:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:08:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:08:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:08:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:08:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:08:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:08:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:08:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:08:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:08:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:08:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:08:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:08:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:08:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:08:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:08:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:08:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:08:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:08:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:08:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:08:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:08:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:08:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:08:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:08:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:08:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:08:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:08:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:08:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:08:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:08:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:08:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:08:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:08:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:08:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:08:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:08:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:08:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:08:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:08:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:08:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:08:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:08:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:08:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:08:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:08:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:08:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:08:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:08:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:08:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:09:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:09:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:09:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:09:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:09:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:09:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:09:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:09:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:09:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:09:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:09:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:09:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:09:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:09:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:09:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:09:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:09:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:09:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:09:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:09:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:09:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:09:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:09:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:09:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:09:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:09:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:09:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:09:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:09:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:09:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:09:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:09:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:09:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:09:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:09:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:09:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:09:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:09:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:09:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:09:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:09:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:09:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:09:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:09:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:09:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:09:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:09:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:09:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:09:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:09:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:09:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:09:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:09:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:09:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:09:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:09:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:09:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:09:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:09:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:09:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:09:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:09:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:09:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:09:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:09:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:09:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:09:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:09:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:09:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:09:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:09:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:09:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:09:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:09:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:09:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:09:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:09:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:09:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:09:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:09:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:09:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:09:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:09:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:09:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:09:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:09:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:09:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:09:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:09:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:09:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:09:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:09:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:09:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:09:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:09:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:09:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:09:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:09:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:09:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:09:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:09:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:09:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:09:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:09:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:09:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:09:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:09:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:09:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:09:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:09:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:09:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:09:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:09:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:09:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:09:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:09:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:09:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:09:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:09:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:09:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:09:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:09:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:09:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:09:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:09:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:09:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:09:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:09:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:09:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:09:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:09:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:09:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:09:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:09:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:09:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:09:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:09:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:09:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:09:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:09:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:09:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:09:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:09:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:09:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:09:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:09:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:09:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:09:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:09:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:09:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:09:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:09:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:09:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:09:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:09:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:09:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:09:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:09:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:09:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:09:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:09:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:09:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:09:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:09:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:09:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:09:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:09:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:09:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:09:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:09:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:09:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:09:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:09:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:09:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:09:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:09:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:09:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:09:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:09:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:09:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:09:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:09:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:09:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:09:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:09:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:09:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:09:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:09:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:09:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:09:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:09:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:09:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:09:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:09:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:09:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:09:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:09:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:09:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:09:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:09:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:09:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:09:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:09:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:09:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:09:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:09:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:09:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:09:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:09:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:09:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:09:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:09:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:09:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:09:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:09:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:09:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:10:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:10:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:10:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:10:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:10:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:10:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:10:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:10:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:10:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:10:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:10:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:10:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:10:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:10:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:10:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:10:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:10:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:10:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:10:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:10:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:10:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:10:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:10:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:10:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:10:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:10:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:10:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:10:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:10:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:10:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:10:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:10:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:10:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:10:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:10:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:10:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:10:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:10:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:10:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:10:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:10:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:10:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:10:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:10:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:10:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:10:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:10:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:10:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:10:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:10:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:10:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:10:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:10:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:10:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:10:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:10:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:10:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:10:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:10:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:10:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:10:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:10:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:10:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:10:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:10:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:10:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:10:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:10:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:10:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:10:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:10:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:10:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:12:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:12:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:12:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:12:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:12:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:12:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:12:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:12:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:12:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:12:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:12:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:12:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:12:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:12:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:12:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:12:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:12:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:12:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:12:27 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 04:12:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:12:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:12:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:12:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:12:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:12:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:12:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:12:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:12:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:12:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:12:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:12:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:12:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:12:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:12:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:12:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:12:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:12:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:12:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:12:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:12:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:12:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:12:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:12:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:12:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:12:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:12:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:12:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:12:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:12:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:12:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:12:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:12:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:12:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:12:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:12:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:12:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:12:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:12:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:12:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:12:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:12:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:12:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:12:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:12:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:12:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:12:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:12:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:12:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:12:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:12:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:12:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:12:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:12:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:12:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:12:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:12:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:12:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:12:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:12:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:12:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:12:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:12:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:12:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:12:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:12:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:12:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:12:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:12:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:12:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:12:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:12:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:12:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:12:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:12:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:12:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:12:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:12:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:12:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:12:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:12:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:12:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:12:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:12:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:12:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:12:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:12:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:12:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:12:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:12:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:12:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:12:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:12:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:12:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:12:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:12:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:12:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:12:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:12:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:12:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:12:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:12:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:12:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:12:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:12:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:12:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:12:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:12:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:13:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:13:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:13:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:13:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:13:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:13:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:13:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:13:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:13:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:13:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:13:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:13:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:13:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:13:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:13:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:13:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:13:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:13:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:13:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:13:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:13:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:13:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:13:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:13:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:13:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:13:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:13:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:13:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:13:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:13:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:13:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:13:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:13:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:13:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:13:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:13:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:13:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:13:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:13:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:13:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:13:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:13:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:13:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:13:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:13:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:13:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:13:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:13:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:13:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:13:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:13:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:13:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:13:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:13:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:13:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:13:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:13:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:13:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:13:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:13:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:13:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:13:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:13:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:13:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:13:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:13:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:13:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:13:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:13:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:13:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:13:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:13:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:13:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:13:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:13:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:13:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:13:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:13:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:13:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:13:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:13:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:13:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:13:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:13:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:13:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:13:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:13:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:13:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:13:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:13:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:13:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:13:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:13:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:13:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:13:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:13:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:13:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:13:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:13:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:13:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:13:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:13:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:13:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:13:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:13:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:13:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:13:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:13:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:13:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:13:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:13:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:13:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:13:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:13:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:13:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:13:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:13:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:13:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:13:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:13:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:13:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:13:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:13:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:13:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:13:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:13:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:13:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:13:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:13:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:13:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:13:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:13:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:13:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:13:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:13:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:13:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:13:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:13:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:13:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:13:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:13:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:13:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:13:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:13:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:13:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:13:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:13:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:13:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:13:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:13:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:13:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:13:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:13:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:13:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:13:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:13:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:13:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:13:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:13:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:13:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:13:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:13:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:13:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:13:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:13:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:13:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:13:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:13:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:13:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:13:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:13:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:13:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:13:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:13:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:13:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:13:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:13:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:13:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:13:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:13:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:13:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:13:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:13:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:13:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:13:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:13:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:13:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:13:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:13:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:13:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:13:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:13:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:13:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:13:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:13:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:13:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:13:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:13:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:13:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:13:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:13:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:13:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:13:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:13:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:13:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:13:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:13:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:13:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:13:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:13:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:13:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:13:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:13:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:13:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:13:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:13:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:14:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:14:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:14:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:14:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:14:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:14:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:14:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:14:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:14:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:14:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:14:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:14:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:14:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:14:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:14:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:14:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:14:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:14:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:14:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:14:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:14:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:14:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:14:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:14:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:14:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:14:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:14:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:14:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:14:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:14:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:14:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:14:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:14:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:14:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:14:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:14:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:14:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:14:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:14:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:14:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:14:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:14:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:14:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:14:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:14:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:14:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:14:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:14:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:14:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:14:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:14:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:14:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:14:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:14:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:14:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:14:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:14:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:14:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:14:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:14:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:14:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:14:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:14:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:14:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:14:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:14:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:14:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:14:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:14:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:14:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:14:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:14:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:14:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:14:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:14:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:14:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:14:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:14:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:14:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:14:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:14:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:14:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:14:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:14:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:14:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:14:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:14:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:14:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:14:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:14:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:14:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:14:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:14:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:14:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:14:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:14:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:14:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:14:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:14:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:14:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:14:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:14:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:14:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:14:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:14:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:14:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:14:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:14:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:14:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:14:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:14:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:14:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:14:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:14:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:14:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:14:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:14:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:14:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:14:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:14:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:14:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:14:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:14:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:14:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:14:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:14:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:14:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:14:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:14:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:14:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:14:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:14:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:14:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:14:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:14:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:14:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:14:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:14:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:14:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:14:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:14:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:14:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:14:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:14:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:14:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:14:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:14:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:14:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:14:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:14:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:14:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:14:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:14:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:14:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:14:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:14:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:14:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:14:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:14:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:14:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:14:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:14:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:14:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:14:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:14:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:14:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:14:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:14:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:14:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:14:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:14:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:14:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:14:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:14:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:14:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:14:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:14:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:14:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:14:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:14:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:14:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:14:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:14:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:14:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:14:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:14:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:14:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:14:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:14:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:14:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:14:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:14:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:14:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:14:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:14:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:14:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:14:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:14:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:14:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:14:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:14:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:14:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:14:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:14:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:14:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:14:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:14:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:14:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:14:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:14:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:14:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:14:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:14:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:14:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:14:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:14:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:15:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:15:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:15:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:15:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:15:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:15:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:15:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:15:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:15:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:15:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:15:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:15:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:15:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:15:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:15:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:15:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:15:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:15:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:15:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:15:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:15:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:15:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:15:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:15:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:15:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:15:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:15:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:15:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:15:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:15:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:15:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:15:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:15:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:15:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:15:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:15:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:15:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:15:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:15:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:15:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:15:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:15:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:15:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:15:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:15:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:15:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:15:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:15:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:15:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:15:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:15:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:15:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:15:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:15:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:15:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:15:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:15:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:15:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:15:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:15:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:15:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:15:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:15:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:15:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:15:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:15:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:15:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:15:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:15:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:15:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:15:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:15:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:15:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:15:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:15:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:15:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:15:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:15:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:15:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:15:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:15:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:15:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:15:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:15:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:15:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:15:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:15:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:15:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:15:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:15:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:15:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:15:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:15:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:15:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:15:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:15:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:15:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:15:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:15:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:15:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:15:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:15:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:15:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:15:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:15:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:15:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:15:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:15:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:15:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:15:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:15:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:15:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:15:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:15:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:15:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:15:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:15:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:15:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:15:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:15:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:15:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:15:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:15:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:15:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:15:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:15:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:15:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:15:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:15:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:15:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:15:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:15:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:15:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:15:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:15:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:15:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:15:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:15:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:15:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:15:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:15:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:15:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:15:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:15:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:15:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:15:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:15:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:15:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:15:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:15:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:15:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:15:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:15:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:15:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:15:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:15:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:15:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:15:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:15:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:15:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:15:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:15:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:15:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:15:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:15:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:15:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:15:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:15:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:15:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:15:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:15:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:15:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:15:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:15:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:15:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:15:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:15:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:15:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:15:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:15:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:15:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:15:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:15:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:15:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:15:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:15:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:15:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:15:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:15:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:15:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:15:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:15:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:15:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:15:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:15:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:15:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:15:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:15:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:15:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:15:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:15:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:15:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:15:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:15:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:15:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:15:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:15:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:15:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:15:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:15:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:15:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:15:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:15:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:15:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:15:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:15:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:16:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:16:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:16:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:16:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:16:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:16:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:16:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:16:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:16:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:16:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:16:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:16:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:16:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:16:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:16:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:16:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:16:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:16:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:16:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:16:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:16:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:16:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:16:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:16:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:16:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:16:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:16:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:16:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:16:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:16:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:16:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:16:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:16:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:16:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:16:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:16:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:16:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:16:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:16:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:16:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:16:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:16:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:16:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:16:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:16:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:16:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:16:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:16:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:16:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:16:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:16:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:16:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:16:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:16:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:16:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:16:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:16:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:16:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:16:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:16:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:16:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:16:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:16:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:16:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:16:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:16:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:16:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:16:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:16:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:16:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:16:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:16:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:16:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:16:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:16:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:16:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:16:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:16:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:16:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:16:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:16:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:16:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:16:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:16:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:16:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:16:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:16:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:16:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:16:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:16:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:16:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:16:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:16:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:16:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:16:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:16:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:16:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:16:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:16:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:16:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:16:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:16:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:16:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:16:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:16:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:16:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:16:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:16:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:16:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:16:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:16:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:16:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:16:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:16:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:16:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:16:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:16:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:16:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:16:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:16:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:16:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:16:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:16:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:16:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:16:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:16:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:16:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:16:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:16:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:16:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:16:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:16:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:16:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:16:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:16:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:16:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:16:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:16:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:16:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:16:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:16:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:16:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:16:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:16:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:16:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:16:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:16:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:16:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:16:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:16:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:16:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:16:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:16:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:16:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:16:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:16:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:16:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:16:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:16:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:16:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:16:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:16:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:16:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:16:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:16:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:16:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:16:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:16:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:16:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:16:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:16:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:16:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:16:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:16:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:16:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:16:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:16:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:16:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:16:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:16:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:16:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:16:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:16:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:16:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:16:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:16:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:16:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:16:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:16:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:16:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:16:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:16:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:16:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:16:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:16:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:16:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:16:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:16:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:16:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:16:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:16:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:16:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:16:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:16:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:16:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:16:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:16:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:16:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:16:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:16:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:16:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:16:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:16:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:16:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:16:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:16:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:17:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:17:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:17:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:17:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:17:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:17:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:17:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:17:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:17:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:17:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:17:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:17:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:17:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:17:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:17:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:17:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:17:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:17:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:17:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:17:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:17:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:17:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:17:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:17:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:17:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:17:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:17:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:17:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:17:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:17:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:17:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:17:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:17:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:17:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:17:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:17:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:17:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:17:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:17:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:17:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:17:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:17:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:17:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:17:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:17:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:17:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:17:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:17:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:17:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:17:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:17:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:17:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:17:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:17:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:17:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:17:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:17:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:17:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:17:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:17:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:17:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:17:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:17:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:17:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:17:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:17:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:17:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:17:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:17:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:17:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:17:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:17:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:17:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:17:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:17:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:17:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:17:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:17:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:17:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:17:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:17:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:17:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:17:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:17:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:17:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:17:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:17:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:17:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:17:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:17:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:17:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:17:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:17:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:17:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:17:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:17:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:17:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:17:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:17:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:17:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:17:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:17:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:17:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:17:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:17:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:17:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:17:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:17:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:17:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:17:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:17:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:17:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:17:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:17:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:17:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:17:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:17:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:17:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:17:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:17:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:17:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:17:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:17:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:17:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:17:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:17:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:17:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:17:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:17:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:17:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:17:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:17:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:17:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:17:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:17:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:17:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:17:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:17:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:17:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:17:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:17:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:17:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:17:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:17:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:17:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:17:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:17:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:17:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:17:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:17:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:17:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:17:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:17:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:17:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:17:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:17:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:17:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:17:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:17:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:17:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:17:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:17:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:17:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:17:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:17:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:17:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:17:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:17:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:17:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:17:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:17:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:17:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:17:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:17:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:17:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:17:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:17:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:17:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:17:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:17:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:17:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:17:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:17:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:17:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:17:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:17:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:17:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:17:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:17:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:17:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:17:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:17:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:17:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:17:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:17:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:17:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:17:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:17:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:17:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:17:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:17:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:17:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:17:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:17:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:17:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:17:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:17:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:17:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:17:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:17:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:17:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:17:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:17:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:17:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:17:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:17:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:18:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:18:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:18:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:18:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:18:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:18:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:18:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:18:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:18:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:18:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:18:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:18:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:18:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:18:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:18:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:18:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:18:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:18:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:18:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:18:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:18:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:18:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:18:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:18:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:18:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:18:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:18:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:18:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:18:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:18:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:18:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:18:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:18:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:18:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:18:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:18:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:18:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:18:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:18:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:18:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:18:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:18:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:18:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:18:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:18:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:18:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:18:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:18:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:18:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:18:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:18:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:18:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:18:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:18:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:18:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:18:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:18:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:18:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:18:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:18:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:18:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:18:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:18:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:18:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:18:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:18:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:18:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:18:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:18:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:18:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:18:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:18:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:18:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:18:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:18:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:18:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:18:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:18:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:18:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:18:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:18:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:18:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:18:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:18:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:18:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:18:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:18:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:18:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:18:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:18:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:18:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:18:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:18:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:18:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:18:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:18:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:18:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:18:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:18:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:18:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:18:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:18:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:18:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:18:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:18:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:18:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:18:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:18:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:18:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:18:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:18:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:18:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:18:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:18:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:18:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:18:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:18:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:18:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:18:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:18:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:18:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:18:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:18:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:18:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:18:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:18:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:18:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:18:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:18:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:18:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:18:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:18:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:18:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:18:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:18:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:18:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:18:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:18:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:18:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:18:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:18:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:18:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:18:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:18:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:18:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:18:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:18:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:18:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:18:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:18:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:18:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:18:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:18:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:18:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:18:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:18:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:18:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:18:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:18:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:18:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:18:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:18:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:18:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:18:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:18:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:18:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:18:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:18:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:18:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:18:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:18:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:18:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:18:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:18:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:18:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:18:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:18:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:18:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:18:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:18:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:18:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:18:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:18:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:18:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:18:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:18:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:18:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:18:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:18:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:18:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:18:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:18:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:18:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:18:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:18:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:18:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:18:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:18:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:18:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:18:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:18:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:18:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:18:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:18:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:18:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:18:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:18:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:18:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:18:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:18:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:18:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:18:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:18:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:18:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:18:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:18:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:19:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:19:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:19:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:19:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:19:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:19:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:19:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:19:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:19:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:19:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:19:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:19:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:19:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:19:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:19:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:19:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:19:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:19:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:19:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:19:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:19:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:19:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:19:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:19:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:19:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:19:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:19:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:19:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:19:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:19:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:19:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:19:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:19:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:19:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:19:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:19:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:19:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:19:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:19:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:19:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:19:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:19:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:19:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:19:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:19:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:19:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:19:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:19:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:19:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:19:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:19:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:19:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:19:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:19:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:19:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:19:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:19:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:19:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:19:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:19:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:19:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:19:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:19:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:19:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:19:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:19:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:19:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:19:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:19:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:19:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:19:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:19:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:19:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:19:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:19:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:19:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:19:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:19:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:19:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:19:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:19:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:19:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:19:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:19:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:19:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:19:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:19:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:19:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:19:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:19:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:19:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:19:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:19:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:19:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:19:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:19:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:19:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:19:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:19:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:19:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:19:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:19:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:19:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:19:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:19:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:19:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:19:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:19:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:19:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:19:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:19:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:19:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:19:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:19:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:19:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:19:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:19:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:19:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:19:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:19:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:19:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:19:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:19:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:19:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:19:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:19:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:19:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:19:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:19:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:19:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:19:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:19:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:19:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:19:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:19:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:19:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:19:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:19:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:19:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:19:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:19:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:19:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:19:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:19:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:19:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:19:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:19:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:19:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:19:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:19:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:19:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:19:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:19:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:19:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:19:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:19:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:19:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:19:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:19:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:19:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:19:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:19:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:19:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:19:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:19:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:19:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:19:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:19:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:19:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:19:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:19:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:19:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:19:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:19:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:19:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:19:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:19:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:19:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:19:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:19:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:19:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:19:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:19:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:19:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:19:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:19:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:19:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:19:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:19:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:19:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:19:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:19:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:19:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:19:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:19:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:19:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:19:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:19:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:19:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:19:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:19:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:19:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:19:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:19:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:19:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:19:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:19:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:19:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:19:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:19:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:19:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:19:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:19:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:19:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:19:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:19:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:20:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:20:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:20:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:20:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:20:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:20:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:20:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:20:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:20:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:20:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:20:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:20:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:20:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:20:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:20:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:20:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:20:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:20:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:20:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:20:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:20:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:20:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:20:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:20:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:20:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:20:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:20:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:20:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:20:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:20:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:20:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:20:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:20:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:20:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:20:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:20:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:20:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:20:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:20:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:20:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:20:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:20:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:20:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:20:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:20:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:20:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:20:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:20:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:20:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:20:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:20:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:20:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:20:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:20:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:20:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:20:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:20:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:20:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:20:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:20:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:20:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:20:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:20:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:20:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:20:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:20:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:20:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:20:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:20:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:20:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:20:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:20:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:20:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:20:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:20:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:20:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:20:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:20:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:20:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:20:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:20:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:20:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:20:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:20:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:20:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:20:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:20:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:20:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:20:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:20:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:22:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:22:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:22:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:22:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:22:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:22:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:22:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:22:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:22:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:22:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:22:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:22:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:22:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:22:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:22:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:22:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:22:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:22:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:22:32 rdma06 rsyslogd[1041]: imjournal: 5292 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 04:22:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:22:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:22:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:22:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:22:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:22:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:22:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:22:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:22:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:22:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:22:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:22:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:22:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:22:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:22:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:22:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:22:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:22:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:22:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:22:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:22:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:22:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:22:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:22:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:22:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:22:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:22:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:22:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:22:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:22:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:22:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:22:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:22:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:22:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:22:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:22:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:22:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:22:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:22:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:22:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:22:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:22:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:22:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:22:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:22:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:22:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:22:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:22:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:22:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:22:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:22:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:22:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:22:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:22:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:22:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:22:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:22:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:22:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:22:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:22:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:22:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:22:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:22:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:22:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:22:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:22:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:22:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:22:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:22:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:22:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:22:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:22:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:22:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:22:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:22:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:22:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:22:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:22:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:22:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:22:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:22:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:22:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:22:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:22:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:22:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:22:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:22:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:22:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:22:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:22:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:23:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:23:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:23:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:23:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:23:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:23:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:23:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:23:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:23:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:23:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:23:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:23:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:23:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:23:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:23:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:23:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:23:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:23:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:23:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:23:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:23:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:23:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:23:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:23:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:23:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:23:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:23:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:23:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:23:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:23:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:23:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:23:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:23:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:23:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:23:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:23:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:23:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:23:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:23:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:23:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:23:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:23:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:23:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:23:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:23:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:23:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:23:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:23:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:23:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:23:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:23:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:23:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:23:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:23:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:23:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:23:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:23:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:23:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:23:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:23:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:23:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:23:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:23:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:23:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:23:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:23:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:23:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:23:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:23:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:23:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:23:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:23:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:23:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:23:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:23:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:23:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:23:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:23:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:23:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:23:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:23:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:23:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:23:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:23:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:23:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:23:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:23:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:23:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:23:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:23:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:23:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:23:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:23:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:23:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:23:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:23:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:23:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:23:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:23:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:23:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:23:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:23:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:23:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:23:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:23:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:23:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:23:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:23:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:23:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:23:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:23:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:23:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:23:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:23:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:23:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:23:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:23:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:23:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:23:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:23:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:23:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:23:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:23:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:23:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:23:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:23:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:23:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:23:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:23:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:23:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:23:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:23:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:23:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:23:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:23:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:23:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:23:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:23:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:23:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:23:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:23:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:23:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:23:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:23:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:23:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:23:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:23:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:23:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:23:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:23:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:23:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:23:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:23:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:23:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:23:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:23:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:23:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:23:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:23:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:23:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:23:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:23:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:23:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:23:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:23:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:23:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:23:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:23:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:23:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:23:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:23:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:23:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:23:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:23:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:23:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:23:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:23:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:23:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:23:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:23:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:23:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:23:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:23:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:23:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:23:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:23:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:23:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:23:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:23:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:23:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:23:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:23:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:23:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:23:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:23:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:23:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:23:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:23:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:23:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:23:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:23:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:23:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:23:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:23:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:23:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:23:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:23:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:23:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:23:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:23:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:23:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:23:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:23:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:23:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:23:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:23:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:24:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:24:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:24:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:24:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:24:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:24:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:24:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:24:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:24:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:24:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:24:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:24:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:24:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:24:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:24:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:24:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:24:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:24:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:24:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:24:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:24:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:24:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:24:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:24:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:24:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:24:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:24:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:24:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:24:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:24:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:24:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:24:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:24:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:24:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:24:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:24:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:24:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:24:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:24:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:24:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:24:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:24:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:24:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:24:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:24:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:24:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:24:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:24:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:24:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:24:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:24:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:24:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:24:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:24:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:24:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:24:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:24:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:24:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:24:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:24:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:24:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:24:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:24:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:24:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:24:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:24:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:24:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:24:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:24:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:24:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:24:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:24:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:24:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:24:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:24:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:24:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:24:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:24:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:24:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:24:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:24:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:24:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:24:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:24:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:24:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:24:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:24:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:24:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:24:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:24:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:24:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:24:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:24:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:24:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:24:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:24:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:24:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:24:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:24:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:24:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:24:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:24:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:24:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:24:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:24:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:24:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:24:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:24:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:24:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:24:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:24:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:24:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:24:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:24:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:24:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:24:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:24:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:24:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:24:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:24:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:24:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:24:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:24:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:24:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:24:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:24:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:24:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:24:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:24:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:24:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:24:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:24:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:24:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:24:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:24:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:24:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:24:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:24:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:24:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:24:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:24:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:24:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:24:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:24:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:24:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:24:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:24:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:24:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:24:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:24:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:24:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:24:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:24:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:24:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:24:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:24:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:24:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:24:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:24:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:24:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:24:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:24:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:24:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:24:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:24:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:24:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:24:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:24:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:24:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:24:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:24:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:24:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:24:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:24:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:24:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:24:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:24:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:24:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:24:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:24:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:24:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:24:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:24:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:24:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:24:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:24:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:24:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:24:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:24:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:24:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:24:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:24:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:24:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:24:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:24:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:24:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:24:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:24:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:24:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:24:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:24:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:24:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:24:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:24:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:24:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:24:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:24:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:24:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:24:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:24:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:24:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:24:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:24:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:24:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:24:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:24:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:25:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:25:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:25:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:25:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:25:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:25:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:25:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:25:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:25:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:25:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:25:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:25:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:25:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:25:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:25:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:25:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:25:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:25:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:25:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:25:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:25:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:25:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:25:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:25:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:25:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:25:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:25:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:25:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:25:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:25:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:25:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:25:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:25:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:25:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:25:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:25:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:25:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:25:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:25:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:25:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:25:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:25:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:25:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:25:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:25:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:25:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:25:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:25:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:25:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:25:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:25:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:25:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:25:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:25:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:25:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:25:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:25:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:25:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:25:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:25:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:25:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:25:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:25:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:25:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:25:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:25:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:25:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:25:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:25:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:25:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:25:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:25:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:25:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:25:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:25:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:25:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:25:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:25:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:25:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:25:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:25:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:25:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:25:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:25:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:25:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:25:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:25:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:25:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:25:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:25:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:25:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:25:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:25:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:25:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:25:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:25:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:25:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:25:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:25:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:25:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:25:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:25:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:25:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:25:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:25:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:25:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:25:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:25:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:25:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:25:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:25:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:25:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:25:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:25:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:25:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:25:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:25:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:25:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:25:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:25:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:25:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:25:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:25:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:25:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:25:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:25:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:25:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:25:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:25:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:25:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:25:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:25:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:25:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:25:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:25:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:25:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:25:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:25:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:25:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:25:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:25:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:25:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:25:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:25:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:25:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:25:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:25:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:25:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:25:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:25:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:25:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:25:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:25:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:25:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:25:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:25:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:25:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:25:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:25:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:25:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:25:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:25:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:25:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:25:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:25:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:25:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:25:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:25:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:25:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:25:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:25:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:25:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:25:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:25:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:25:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:25:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:25:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:25:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:25:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:25:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:25:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:25:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:25:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:25:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:25:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:25:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:25:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:25:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:25:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:25:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:25:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:25:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:25:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:25:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:25:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:25:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:25:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:25:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:25:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:25:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:25:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:25:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:25:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:25:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:25:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:25:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:25:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:25:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:25:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:25:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:25:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:25:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:25:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:25:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:25:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:25:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:26:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:26:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:26:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:26:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:26:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:26:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:26:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:26:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:26:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:26:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:26:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:26:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:26:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:26:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:26:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:26:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:26:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:26:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:26:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:26:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:26:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:26:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:26:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:26:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:26:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:26:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:26:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:26:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:26:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:26:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:26:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:26:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:26:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:26:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:26:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:26:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:26:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:26:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:26:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:26:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:26:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:26:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:26:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:26:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:26:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:26:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:26:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:26:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:26:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:26:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:26:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:26:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:26:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:26:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:26:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:26:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:26:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:26:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:26:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:26:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:26:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:26:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:26:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:26:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:26:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:26:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:26:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:26:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:26:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:26:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:26:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:26:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:26:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:26:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:26:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:26:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:26:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:26:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:26:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:26:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:26:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:26:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:26:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:26:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:26:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:26:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:26:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:26:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:26:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:26:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:26:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:26:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:26:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:26:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:26:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:26:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:26:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:26:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:26:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:26:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:26:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:26:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:26:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:26:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:26:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:26:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:26:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:26:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:26:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:26:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:26:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:26:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:26:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:26:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:26:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:26:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:26:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:26:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:26:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:26:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:26:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:26:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:26:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:26:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:26:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:26:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:26:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:26:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:26:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:26:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:26:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:26:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:26:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:26:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:26:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:26:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:26:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:26:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:26:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:26:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:26:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:26:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:26:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:26:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:26:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:26:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:26:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:26:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:26:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:26:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:26:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:26:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:26:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:26:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:26:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:26:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:26:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:26:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:26:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:26:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:26:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:26:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:26:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:26:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:26:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:26:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:26:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:26:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:26:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:26:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:26:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:26:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:26:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:26:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:26:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:26:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:26:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:26:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:26:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:26:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:26:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:26:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:26:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:26:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:26:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:26:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:26:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:26:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:26:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:26:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:26:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:26:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:26:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:26:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:26:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:26:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:26:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:26:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:26:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:26:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:26:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:26:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:26:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:26:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:26:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:26:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:26:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:26:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:26:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:26:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:26:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:26:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:26:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:26:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:26:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:26:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:27:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:27:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:27:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:27:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:27:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:27:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:27:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:27:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:27:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:27:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:27:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:27:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:27:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:27:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:27:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:27:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:27:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:27:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:27:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:27:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:27:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:27:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:27:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:27:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:27:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:27:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:27:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:27:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:27:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:27:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:27:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:27:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:27:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:27:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:27:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:27:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:27:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:27:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:27:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:27:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:27:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:27:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:27:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:27:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:27:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:27:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:27:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:27:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:27:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:27:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:27:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:27:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:27:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:27:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:27:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:27:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:27:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:27:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:27:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:27:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:27:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:27:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:27:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:27:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:27:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:27:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:27:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:27:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:27:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:27:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:27:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:27:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:27:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:27:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:27:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:27:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:27:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:27:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:27:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:27:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:27:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:27:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:27:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:27:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:27:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:27:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:27:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:27:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:27:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:27:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:27:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:27:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:27:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:27:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:27:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:27:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:27:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:27:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:27:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:27:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:27:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:27:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:27:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:27:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:27:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:27:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:27:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:27:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:27:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:27:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:27:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:27:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:27:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:27:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:27:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:27:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:27:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:27:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:27:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:27:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:27:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:27:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:27:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:27:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:27:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:27:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:27:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:27:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:27:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:27:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:27:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:27:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:27:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:27:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:27:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:27:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:27:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:27:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:27:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:27:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:27:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:27:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:27:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:27:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:27:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:27:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:27:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:27:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:27:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:27:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:27:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:27:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:27:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:27:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:27:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:27:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:27:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:27:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:27:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:27:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:27:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:27:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:27:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:27:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:27:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:27:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:27:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:27:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:27:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:27:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:27:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:27:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:27:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:27:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:27:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:27:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:27:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:27:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:27:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:27:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:27:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:27:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:27:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:27:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:27:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:27:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:27:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:27:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:27:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:27:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:27:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:27:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:27:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:27:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:27:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:27:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:27:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:27:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:27:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:27:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:27:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:27:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:27:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:27:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:27:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:27:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:27:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:27:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:27:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:27:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:27:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:27:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:27:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:27:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:27:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:27:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:28:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:28:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:28:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:28:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:28:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:28:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:28:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:28:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:28:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:28:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:28:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:28:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:28:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:28:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:28:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:28:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:28:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:28:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:28:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:28:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:28:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:28:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:28:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:28:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:28:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:28:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:28:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:28:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:28:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:28:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:28:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:28:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:28:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:28:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:28:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:28:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:28:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:28:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:28:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:28:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:28:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:28:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:28:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:28:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:28:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:28:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:28:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:28:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:28:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:28:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:28:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:28:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:28:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:28:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:28:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:28:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:28:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:28:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:28:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:28:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:28:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:28:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:28:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:28:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:28:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:28:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:28:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:28:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:28:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:28:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:28:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:28:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:28:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:28:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:28:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:28:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:28:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:28:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:28:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:28:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:28:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:28:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:28:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:28:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:28:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:28:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:28:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:28:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:28:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:28:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:28:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:28:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:28:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:28:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:28:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:28:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:28:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:28:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:28:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:28:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:28:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:28:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:28:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:28:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:28:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:28:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:28:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:28:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:28:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:28:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:28:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:28:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:28:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:28:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:28:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:28:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:28:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:28:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:28:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:28:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:28:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:28:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:28:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:28:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:28:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:28:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:28:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:28:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:28:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:28:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:28:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:28:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:28:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:28:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:28:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:28:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:28:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:28:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:28:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:28:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:28:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:28:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:28:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:28:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:28:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:28:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:28:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:28:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:28:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:28:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:28:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:28:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:28:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:28:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:28:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:28:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:28:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:28:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:28:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:28:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:28:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:28:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:28:47 rdma06 systemd-journald[790]: Data hash table of /run/log/journal/95379d2f723847b68d9e976dee8967a9/system.journal has a fill level at 75.0 (163751 of 218332 items, 75497472 file size, 461 bytes per hash table item), suggesting rotation.
Apr 18 04:28:47 rdma06 systemd-journald[790]: /run/log/journal/95379d2f723847b68d9e976dee8967a9/system.journal: Journal header limits reached or header out-of-date, rotating.
Apr 18 04:28:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:28:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:28:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:28:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:28:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:28:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:28:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:28:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:28:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:28:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:28:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:28:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:28:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:28:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:28:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:28:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:28:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:28:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:28:47 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 18 04:28:47 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 18 04:28:48 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 18 04:28:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:28:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:28:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:28:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:28:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:28:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:28:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:28:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:28:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:28:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:28:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:28:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:28:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:28:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:28:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:28:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:28:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:28:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:28:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:28:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:28:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:28:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:28:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:28:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:28:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:28:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:28:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:28:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:28:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:28:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:28:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:28:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:28:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:28:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:28:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:28:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:29:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:29:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:29:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:29:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:29:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:29:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:29:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:29:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:29:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:29:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:29:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:29:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:29:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:29:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:29:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:29:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:29:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:29:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:29:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:29:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:29:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:29:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:29:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:29:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:29:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:29:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:29:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:29:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:29:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:29:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:29:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:29:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:29:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:29:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:29:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:29:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:29:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:29:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:29:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:29:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:29:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:29:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:29:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:29:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:29:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:29:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:29:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:29:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:29:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:29:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:29:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:29:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:29:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:29:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:29:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:29:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:29:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:29:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:29:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:29:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:29:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:29:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:29:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:29:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:29:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:29:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:29:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:29:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:29:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:29:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:29:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:29:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:29:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:29:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:29:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:29:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:29:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:29:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:29:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:29:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:29:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:29:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:29:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:29:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:29:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:29:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:29:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:29:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:29:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:29:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:29:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:29:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:29:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:29:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:29:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:29:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:29:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:29:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:29:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:29:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:29:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:29:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:29:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:29:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:29:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:29:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:29:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:29:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:29:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:29:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:29:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:29:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:29:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:29:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:29:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:29:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:29:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:29:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:29:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:29:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:29:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:29:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:29:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:29:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:29:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:29:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:29:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:29:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:29:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:29:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:29:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:29:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:29:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:29:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:29:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:29:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:29:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:29:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:29:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:29:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:29:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:29:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:29:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:29:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:29:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:29:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:29:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:29:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:29:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:29:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:29:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:29:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:29:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:29:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:29:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:29:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:29:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:29:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:29:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:29:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:29:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:29:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:29:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:29:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:29:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:29:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:29:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:29:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:29:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:29:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:29:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:29:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:29:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:29:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:29:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:29:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:29:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:29:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:29:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:29:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:29:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:29:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:29:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:29:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:29:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:29:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:29:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:29:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:29:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:29:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:29:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:29:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:29:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:29:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:29:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:29:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:29:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:29:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:29:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:29:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:29:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:29:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:29:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:29:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:29:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:29:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:29:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:29:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:29:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:29:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:29:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:29:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:29:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:29:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:29:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:29:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:30:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:30:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:30:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:30:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:30:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:30:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:30:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:30:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:30:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:30:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:30:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:30:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:30:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:30:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:30:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:30:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:30:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:30:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:30:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:30:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:30:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:30:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:30:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:30:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:30:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:30:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:30:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:30:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:30:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:30:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:30:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:30:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:30:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:30:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:30:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:30:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:30:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:30:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:30:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:30:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:30:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:30:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:30:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:30:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:30:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:30:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:30:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:30:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:30:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:30:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:30:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:30:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:30:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:30:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:30:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:30:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:30:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:30:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:30:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:30:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:30:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:30:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:30:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:30:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:30:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:30:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:30:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:30:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:30:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:30:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:30:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:30:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:30:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:30:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:30:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:30:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:30:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:30:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:30:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:30:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:30:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:30:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:30:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:30:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:30:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:30:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:30:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:30:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:30:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:30:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:30:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:30:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:30:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:30:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:30:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:30:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:30:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:30:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:30:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:30:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:30:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:30:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:30:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:30:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:30:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:30:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:30:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:30:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:32:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:32:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:32:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:32:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:32:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:32:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:32:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:32:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:32:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:32:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:32:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:32:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:32:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:32:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:32:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:32:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:32:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:32:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:32:38 rdma06 rsyslogd[1041]: imjournal: 5296 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 04:32:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:32:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:32:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:32:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:32:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:32:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:32:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:32:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:32:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:32:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:32:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:32:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:32:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:32:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:32:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:32:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:32:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:32:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:32:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:32:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:32:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:32:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:32:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:32:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:32:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:32:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:32:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:32:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:32:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:32:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:32:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:32:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:32:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:32:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:32:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:32:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:32:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:32:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:32:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:32:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:32:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:32:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:32:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:32:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:32:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:32:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:32:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:32:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:32:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:32:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:32:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:32:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:32:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:32:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:32:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:32:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:32:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:32:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:32:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:32:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:32:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:32:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:32:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:32:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:32:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:32:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:32:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:32:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:32:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:32:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:32:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:32:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:33:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:33:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:33:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:33:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:33:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:33:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:33:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:33:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:33:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:33:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:33:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:33:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:33:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:33:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:33:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:33:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:33:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:33:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:33:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:33:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:33:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:33:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:33:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:33:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:33:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:33:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:33:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:33:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:33:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:33:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:33:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:33:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:33:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:33:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:33:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:33:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:33:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:33:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:33:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:33:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:33:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:33:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:33:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:33:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:33:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:33:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:33:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:33:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:33:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:33:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:33:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:33:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:33:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:33:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:33:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:33:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:33:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:33:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:33:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:33:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:33:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:33:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:33:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:33:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:33:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:33:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:33:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:33:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:33:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:33:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:33:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:33:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:33:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:33:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:33:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:33:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:33:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:33:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:33:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:33:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:33:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:33:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:33:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:33:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:33:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:33:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:33:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:33:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:33:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:33:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:33:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:33:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:33:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:33:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:33:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:33:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:33:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:33:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:33:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:33:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:33:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:33:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:33:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:33:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:33:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:33:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:33:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:33:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:33:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:33:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:33:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:33:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:33:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:33:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:33:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:33:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:33:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:33:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:33:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:33:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:33:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:33:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:33:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:33:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:33:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:33:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:33:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:33:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:33:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:33:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:33:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:33:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:33:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:33:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:33:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:33:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:33:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:33:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:33:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:33:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:33:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:33:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:33:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:33:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:33:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:33:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:33:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:33:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:33:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:33:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:33:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:33:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:33:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:33:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:33:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:33:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:33:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:33:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:33:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:33:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:33:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:33:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:33:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:33:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:33:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:33:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:33:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:33:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:33:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:33:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:33:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:33:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:33:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:33:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:33:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:33:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:33:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:33:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:33:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:33:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:33:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:33:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:33:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:33:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:33:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:33:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:33:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:33:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:33:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:33:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:33:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:33:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:33:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:33:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:33:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:33:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:33:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:33:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:33:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:33:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:33:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:33:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:33:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:33:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:33:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:33:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:33:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:33:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:33:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:33:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:33:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:33:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:33:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:33:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:33:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:33:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:34:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:34:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:34:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:34:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:34:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:34:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:34:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:34:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:34:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:34:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:34:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:34:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:34:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:34:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:34:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:34:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:34:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:34:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:34:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:34:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:34:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:34:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:34:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:34:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:34:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:34:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:34:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:34:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:34:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:34:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:34:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:34:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:34:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:34:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:34:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:34:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:34:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:34:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:34:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:34:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:34:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:34:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:34:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:34:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:34:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:34:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:34:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:34:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:34:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:34:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:34:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:34:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:34:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:34:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:34:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:34:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:34:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:34:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:34:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:34:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:34:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:34:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:34:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:34:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:34:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:34:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:34:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:34:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:34:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:34:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:34:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:34:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:34:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:34:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:34:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:34:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:34:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:34:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:34:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:34:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:34:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:34:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:34:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:34:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:34:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:34:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:34:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:34:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:34:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:34:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:34:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:34:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:34:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:34:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:34:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:34:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:34:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:34:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:34:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:34:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:34:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:34:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:34:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:34:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:34:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:34:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:34:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:34:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:34:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:34:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:34:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:34:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:34:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:34:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:34:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:34:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:34:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:34:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:34:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:34:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:34:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:34:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:34:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:34:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:34:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:34:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:34:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:34:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:34:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:34:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:34:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:34:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:34:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:34:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:34:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:34:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:34:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:34:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:34:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:34:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:34:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:34:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:34:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:34:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:34:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:34:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:34:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:34:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:34:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:34:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:34:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:34:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:34:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:34:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:34:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:34:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:34:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:34:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:34:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:34:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:34:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:34:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:34:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:34:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:34:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:34:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:34:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:34:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:34:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:34:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:34:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:34:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:34:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:34:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:34:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:34:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:34:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:34:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:34:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:34:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:34:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:34:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:34:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:34:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:34:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:34:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:34:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:34:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:34:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:34:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:34:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:34:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:34:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:34:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:34:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:34:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:34:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:34:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:34:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:34:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:34:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:34:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:34:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:34:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:34:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:34:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:34:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:34:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:34:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:34:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:34:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:34:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:34:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:34:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:34:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:34:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:35:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:35:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:35:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:35:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:35:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:35:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:35:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:35:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:35:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:35:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:35:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:35:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:35:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:35:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:35:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:35:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:35:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:35:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:35:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:35:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:35:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:35:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:35:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:35:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:35:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:35:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:35:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:35:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:35:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:35:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:35:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:35:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:35:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:35:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:35:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:35:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:35:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:35:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:35:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:35:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:35:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:35:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:35:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:35:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:35:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:35:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:35:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:35:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:35:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:35:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:35:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:35:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:35:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:35:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:35:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:35:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:35:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:35:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:35:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:35:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:35:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:35:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:35:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:35:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:35:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:35:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:35:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:35:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:35:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:35:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:35:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:35:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:35:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:35:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:35:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:35:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:35:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:35:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:35:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:35:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:35:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:35:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:35:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:35:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:35:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:35:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:35:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:35:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:35:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:35:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:35:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:35:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:35:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:35:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:35:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:35:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:35:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:35:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:35:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:35:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:35:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:35:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:35:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:35:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:35:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:35:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:35:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:35:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:35:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:35:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:35:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:35:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:35:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:35:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:35:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:35:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:35:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:35:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:35:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:35:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:35:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:35:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:35:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:35:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:35:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:35:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:35:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:35:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:35:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:35:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:35:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:35:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:35:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:35:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:35:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:35:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:35:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:35:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:35:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:35:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:35:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:35:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:35:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:35:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:35:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:35:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:35:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:35:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:35:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:35:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:35:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:35:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:35:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:35:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:35:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:35:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:35:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:35:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:35:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:35:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:35:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:35:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:35:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:35:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:35:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:35:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:35:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:35:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:35:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:35:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:35:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:35:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:35:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:35:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:35:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:35:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:35:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:35:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:35:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:35:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:35:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:35:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:35:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:35:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:35:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:35:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:35:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:35:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:35:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:35:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:35:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:35:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:35:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:35:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:35:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:35:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:35:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:35:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:35:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:35:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:35:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:35:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:35:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:35:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:35:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:35:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:35:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:35:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:35:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:35:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:35:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:35:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:35:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:35:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:35:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:35:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:36:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:36:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:36:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:36:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:36:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:36:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:36:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:36:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:36:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:36:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:36:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:36:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:36:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:36:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:36:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:36:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:36:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:36:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:36:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:36:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:36:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:36:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:36:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:36:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:36:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:36:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:36:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:36:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:36:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:36:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:36:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:36:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:36:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:36:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:36:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:36:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:36:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:36:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:36:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:36:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:36:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:36:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:36:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:36:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:36:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:36:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:36:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:36:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:36:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:36:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:36:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:36:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:36:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:36:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:36:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:36:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:36:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:36:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:36:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:36:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:36:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:36:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:36:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:36:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:36:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:36:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:36:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:36:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:36:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:36:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:36:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:36:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:36:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:36:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:36:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:36:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:36:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:36:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:36:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:36:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:36:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:36:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:36:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:36:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:36:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:36:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:36:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:36:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:36:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:36:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:36:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:36:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:36:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:36:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:36:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:36:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:36:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:36:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:36:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:36:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:36:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:36:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:36:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:36:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:36:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:36:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:36:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:36:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:36:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:36:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:36:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:36:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:36:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:36:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:36:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:36:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:36:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:36:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:36:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:36:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:36:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:36:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:36:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:36:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:36:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:36:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:36:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:36:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:36:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:36:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:36:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:36:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:36:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:36:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:36:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:36:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:36:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:36:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:36:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:36:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:36:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:36:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:36:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:36:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:36:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:36:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:36:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:36:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:36:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:36:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:36:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:36:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:36:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:36:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:36:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:36:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:36:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:36:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:36:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:36:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:36:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:36:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:36:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:36:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:36:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:36:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:36:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:36:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:36:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:36:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:36:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:36:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:36:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:36:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:36:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:36:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:36:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:36:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:36:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:36:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:36:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:36:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:36:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:36:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:36:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:36:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:36:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:36:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:36:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:36:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:36:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:36:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:36:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:36:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:36:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:36:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:36:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:36:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:36:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:36:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:36:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:36:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:36:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:36:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:36:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:36:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:36:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:36:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:36:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:36:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:36:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:36:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:36:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:36:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:36:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:36:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:37:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:37:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:37:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:37:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:37:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:37:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:37:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:37:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:37:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:37:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:37:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:37:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:37:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:37:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:37:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:37:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:37:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:37:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:37:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:37:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:37:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:37:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:37:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:37:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:37:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:37:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:37:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:37:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:37:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:37:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:37:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:37:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:37:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:37:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:37:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:37:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:37:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:37:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:37:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:37:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:37:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:37:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:37:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:37:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:37:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:37:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:37:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:37:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:37:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:37:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:37:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:37:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:37:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:37:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:37:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:37:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:37:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:37:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:37:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:37:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:37:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:37:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:37:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:37:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:37:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:37:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:37:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:37:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:37:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:37:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:37:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:37:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:37:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:37:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:37:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:37:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:37:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:37:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:37:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:37:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:37:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:37:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:37:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:37:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:37:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:37:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:37:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:37:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:37:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:37:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:37:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:37:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:37:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:37:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:37:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:37:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:37:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:37:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:37:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:37:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:37:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:37:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:37:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:37:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:37:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:37:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:37:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:37:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:37:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:37:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:37:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:37:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:37:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:37:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:37:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:37:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:37:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:37:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:37:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:37:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:37:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:37:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:37:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:37:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:37:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:37:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:37:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:37:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:37:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:37:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:37:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:37:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:37:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:37:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:37:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:37:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:37:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:37:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:37:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:37:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:37:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:37:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:37:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:37:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:37:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:37:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:37:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:37:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:37:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:37:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:37:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:37:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:37:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:37:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:37:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:37:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:37:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:37:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:37:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:37:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:37:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:37:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:37:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:37:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:37:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:37:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:37:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:37:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:37:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:37:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:37:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:37:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:37:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:37:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:37:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:37:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:37:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:37:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:37:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:37:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:37:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:37:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:37:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:37:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:37:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:37:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:37:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:37:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:37:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:37:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:37:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:37:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:37:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:37:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:37:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:37:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:37:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:37:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:37:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:37:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:37:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:37:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:37:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:37:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:37:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:37:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:37:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:37:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:37:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:37:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:37:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:37:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:37:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:37:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:37:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:37:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:38:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:38:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:38:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:38:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:38:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:38:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:38:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:38:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:38:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:38:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:38:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:38:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:38:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:38:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:38:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:38:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:38:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:38:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:38:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:38:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:38:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:38:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:38:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:38:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:38:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:38:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:38:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:38:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:38:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:38:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:38:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:38:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:38:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:38:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:38:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:38:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:38:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:38:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:38:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:38:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:38:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:38:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:38:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:38:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:38:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:38:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:38:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:38:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:38:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:38:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:38:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:38:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:38:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:38:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:38:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:38:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:38:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:38:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:38:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:38:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:38:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:38:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:38:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:38:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:38:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:38:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:38:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:38:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:38:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:38:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:38:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:38:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:38:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:38:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:38:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:38:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:38:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:38:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:38:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:38:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:38:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:38:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:38:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:38:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:38:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:38:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:38:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:38:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:38:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:38:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:38:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:38:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:38:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:38:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:38:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:38:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:38:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:38:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:38:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:38:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:38:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:38:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:38:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:38:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:38:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:38:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:38:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:38:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:38:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:38:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:38:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:38:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:38:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:38:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:38:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:38:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:38:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:38:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:38:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:38:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:38:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:38:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:38:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:38:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:38:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:38:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:38:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:38:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:38:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:38:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:38:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:38:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:38:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:38:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:38:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:38:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:38:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:38:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:38:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:38:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:38:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:38:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:38:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:38:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:38:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:38:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:38:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:38:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:38:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:38:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:38:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:38:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:38:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:38:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:38:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:38:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:38:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:38:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:38:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:38:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:38:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:38:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:38:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:38:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:38:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:38:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:38:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:38:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:38:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:38:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:38:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:38:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:38:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:38:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:38:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:38:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:38:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:38:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:38:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:38:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:38:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:38:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:38:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:38:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:38:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:38:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:38:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:38:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:38:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:38:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:38:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:38:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:38:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:38:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:38:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:38:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:38:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:38:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:38:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:38:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:38:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:38:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:38:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:38:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:38:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:38:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:38:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:38:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:38:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:38:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:38:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:38:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:38:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:38:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:38:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:38:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:39:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:39:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:39:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:39:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:39:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:39:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:39:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:39:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:39:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:39:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:39:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:39:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:39:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:39:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:39:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:39:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:39:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:39:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:39:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:39:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:39:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:39:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:39:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:39:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:39:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:39:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:39:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:39:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:39:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:39:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:39:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:39:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:39:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:39:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:39:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:39:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:39:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:39:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:39:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:39:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:39:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:39:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:39:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:39:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:39:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:39:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:39:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:39:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:39:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:39:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:39:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:39:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:39:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:39:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:39:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:39:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:39:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:39:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:39:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:39:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:39:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:39:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:39:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:39:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:39:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:39:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:39:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:39:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:39:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:39:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:39:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:39:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:39:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:39:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:39:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:39:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:39:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:39:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:39:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:39:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:39:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:39:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:39:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:39:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:39:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:39:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:39:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:39:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:39:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:39:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:39:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:39:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:39:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:39:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:39:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:39:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:39:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:39:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:39:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:39:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:39:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:39:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:39:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:39:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:39:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:39:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:39:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:39:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:39:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:39:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:39:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:39:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:39:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:39:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:39:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:39:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:39:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:39:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:39:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:39:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:39:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:39:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:39:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:39:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:39:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:39:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:39:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:39:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:39:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:39:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:39:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:39:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:39:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:39:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:39:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:39:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:39:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:39:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:39:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:39:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:39:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:39:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:39:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:39:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:39:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:39:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:39:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:39:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:39:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:39:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:39:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:39:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:39:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:39:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:39:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:39:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:39:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:39:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:39:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:39:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:39:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:39:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:39:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:39:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:39:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:39:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:39:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:39:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:39:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:39:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:39:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:39:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:39:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:39:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:39:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:39:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:39:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:39:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:39:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:39:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:39:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:39:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:39:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:39:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:39:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:39:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:39:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:39:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:39:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:39:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:39:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:39:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:39:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:39:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:39:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:39:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:39:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:39:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:39:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:39:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:39:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:39:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:39:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:39:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:39:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:39:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:39:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:39:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:39:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:39:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:39:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:39:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:39:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:39:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:39:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:39:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:40:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:40:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:40:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:40:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:40:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:40:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:40:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:40:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:40:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:40:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:40:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:40:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:40:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:40:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:40:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:40:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:40:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:40:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:40:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:40:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:40:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:40:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:40:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:40:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:40:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:40:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:40:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:40:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:40:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:40:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:40:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:40:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:40:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:40:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:40:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:40:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:40:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:40:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:40:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:40:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:40:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:40:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:40:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:40:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:40:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:40:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:40:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:40:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:40:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:40:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:40:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:40:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:40:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:40:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:40:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:40:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:40:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:40:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:40:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:40:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:40:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:40:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:40:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:40:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:40:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:40:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:40:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:40:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:40:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:40:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:40:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:40:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:40:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:40:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:40:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:40:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:40:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:40:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:40:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:40:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:40:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:40:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:40:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:40:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:40:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:40:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:40:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:40:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:40:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:40:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:40:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:40:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:40:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:40:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:40:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:40:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:40:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:40:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:40:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:40:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:40:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:40:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:40:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:40:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:40:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:40:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:40:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:40:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:40:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:40:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:40:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:40:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:40:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:40:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:40:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:40:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:40:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:40:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:40:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:40:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:40:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:40:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:40:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:40:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:40:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:40:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:42:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:42:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:42:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:42:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:42:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:42:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:42:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:42:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:42:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:42:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:42:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:42:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:42:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:42:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:42:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:42:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:42:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:42:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:42:43 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 04:42:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:42:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:42:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:42:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:42:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:42:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:42:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:42:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:42:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:42:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:42:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:42:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:42:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:42:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:42:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:42:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:42:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:42:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:42:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:42:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:42:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:42:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:42:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:42:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:42:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:42:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:42:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:42:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:42:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:42:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:42:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:42:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:42:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:42:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:42:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:42:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:42:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:42:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:42:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:42:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:42:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:42:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:42:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:42:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:42:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:42:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:42:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:42:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:42:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:42:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:42:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:42:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:42:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:42:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:43:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:43:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:43:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:43:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:43:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:43:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:43:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:43:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:43:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:43:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:43:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:43:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:43:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:43:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:43:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:43:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:43:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:43:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:43:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:43:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:43:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:43:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:43:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:43:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:43:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:43:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:43:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:43:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:43:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:43:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:43:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:43:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:43:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:43:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:43:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:43:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:43:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:43:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:43:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:43:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:43:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:43:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:43:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:43:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:43:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:43:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:43:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:43:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:43:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:43:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:43:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:43:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:43:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:43:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:43:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:43:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:43:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:43:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:43:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:43:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:43:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:43:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:43:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:43:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:43:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:43:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:43:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:43:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:43:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:43:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:43:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:43:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:43:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:43:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:43:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:43:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:43:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:43:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:43:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:43:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:43:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:43:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:43:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:43:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:43:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:43:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:43:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:43:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:43:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:43:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:43:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:43:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:43:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:43:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:43:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:43:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:43:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:43:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:43:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:43:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:43:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:43:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:43:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:43:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:43:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:43:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:43:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:43:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:43:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:43:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:43:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:43:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:43:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:43:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:43:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:43:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:43:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:43:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:43:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:43:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:43:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:43:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:43:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:43:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:43:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:43:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:43:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:43:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:43:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:43:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:43:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:43:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:43:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:43:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:43:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:43:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:43:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:43:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:43:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:43:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:43:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:43:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:43:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:43:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:43:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:43:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:43:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:43:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:43:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:43:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:43:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:43:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:43:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:43:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:43:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:43:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:43:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:43:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:43:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:43:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:43:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:43:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:43:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:43:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:43:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:43:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:43:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:43:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:43:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:43:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:43:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:43:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:43:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:43:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:43:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:43:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:43:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:43:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:43:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:43:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:43:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:43:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:43:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:43:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:43:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:43:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:43:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:43:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:43:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:43:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:43:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:43:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:43:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:43:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:43:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:43:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:43:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:43:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:43:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:43:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:43:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:43:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:43:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:43:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:43:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:43:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:43:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:43:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:43:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:43:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:43:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:43:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:43:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:43:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:43:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:43:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:44:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:44:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:44:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:44:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:44:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:44:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:44:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:44:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:44:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:44:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:44:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:44:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:44:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:44:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:44:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:44:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:44:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:44:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:44:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:44:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:44:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:44:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:44:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:44:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:44:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:44:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:44:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:44:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:44:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:44:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:44:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:44:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:44:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:44:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:44:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:44:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:44:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:44:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:44:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:44:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:44:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:44:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:44:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:44:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:44:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:44:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:44:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:44:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:44:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:44:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:44:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:44:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:44:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:44:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:44:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:44:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:44:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:44:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:44:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:44:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:44:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:44:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:44:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:44:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:44:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:44:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:44:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:44:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:44:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:44:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:44:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:44:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:44:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:44:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:44:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:44:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:44:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:44:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:44:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:44:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:44:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:44:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:44:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:44:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:44:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:44:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:44:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:44:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:44:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:44:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:44:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:44:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:44:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:44:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:44:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:44:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:44:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:44:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:44:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:44:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:44:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:44:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:44:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:44:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:44:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:44:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:44:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:44:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:44:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:44:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:44:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:44:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:44:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:44:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:44:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:44:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:44:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:44:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:44:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:44:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:44:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:44:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:44:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:44:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:44:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:44:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:44:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:44:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:44:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:44:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:44:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:44:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:44:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:44:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:44:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:44:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:44:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:44:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:44:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:44:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:44:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:44:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:44:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:44:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:44:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:44:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:44:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:44:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:44:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:44:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:44:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:44:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:44:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:44:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:44:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:44:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:44:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:44:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:44:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:44:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:44:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:44:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:44:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:44:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:44:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:44:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:44:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:44:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:44:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:44:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:44:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:44:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:44:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:44:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:44:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:44:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:44:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:44:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:44:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:44:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:44:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:44:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:44:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:44:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:44:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:44:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:44:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:44:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:44:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:44:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:44:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:44:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:44:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:44:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:44:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:44:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:44:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:44:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:44:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:44:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:44:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:44:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:44:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:44:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:44:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:44:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:44:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:44:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:44:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:44:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:44:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:44:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:44:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:44:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:44:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:44:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:45:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:45:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:45:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:45:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:45:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:45:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:45:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:45:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:45:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:45:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:45:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:45:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:45:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:45:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:45:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:45:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:45:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:45:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:45:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:45:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:45:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:45:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:45:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:45:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:45:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:45:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:45:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:45:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:45:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:45:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:45:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:45:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:45:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:45:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:45:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:45:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:45:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:45:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:45:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:45:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:45:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:45:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:45:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:45:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:45:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:45:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:45:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:45:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:45:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:45:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:45:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:45:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:45:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:45:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:45:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:45:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:45:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:45:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:45:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:45:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:45:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:45:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:45:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:45:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:45:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:45:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:45:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:45:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:45:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:45:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:45:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:45:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:45:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:45:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:45:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:45:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:45:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:45:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:45:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:45:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:45:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:45:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:45:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:45:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:45:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:45:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:45:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:45:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:45:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:45:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:45:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:45:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:45:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:45:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:45:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:45:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:45:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:45:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:45:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:45:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:45:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:45:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:45:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:45:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:45:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:45:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:45:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:45:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:45:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:45:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:45:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:45:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:45:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:45:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:45:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:45:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:45:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:45:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:45:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:45:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:45:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:45:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:45:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:45:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:45:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:45:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:45:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:45:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:45:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:45:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:45:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:45:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:45:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:45:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:45:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:45:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:45:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:45:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:45:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:45:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:45:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:45:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:45:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:45:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:45:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:45:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:45:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:45:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:45:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:45:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:45:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:45:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:45:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:45:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:45:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:45:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:45:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:45:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:45:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:45:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:45:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:45:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:45:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:45:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:45:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:45:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:45:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:45:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:45:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:45:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:45:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:45:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:45:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:45:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:45:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:45:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:45:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:45:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:45:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:45:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:45:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:45:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:45:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:45:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:45:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:45:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:45:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:45:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:45:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:45:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:45:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:45:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:45:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:45:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:45:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:45:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:45:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:45:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:45:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:45:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:45:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:45:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:45:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:45:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:45:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:45:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:45:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:45:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:45:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:45:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:45:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:45:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:45:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:45:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:45:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:45:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:46:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:46:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:46:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:46:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:46:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:46:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:46:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:46:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:46:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:46:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:46:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:46:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:46:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:46:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:46:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:46:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:46:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:46:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:46:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:46:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:46:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:46:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:46:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:46:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:46:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:46:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:46:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:46:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:46:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:46:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:46:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:46:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:46:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:46:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:46:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:46:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:46:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:46:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:46:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:46:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:46:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:46:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:46:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:46:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:46:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:46:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:46:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:46:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:46:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:46:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:46:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:46:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:46:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:46:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:46:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:46:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:46:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:46:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:46:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:46:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:46:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:46:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:46:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:46:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:46:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:46:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:46:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:46:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:46:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:46:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:46:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:46:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:46:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:46:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:46:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:46:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:46:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:46:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:46:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:46:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:46:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:46:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:46:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:46:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:46:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:46:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:46:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:46:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:46:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:46:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:46:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:46:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:46:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:46:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:46:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:46:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:46:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:46:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:46:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:46:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:46:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:46:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:46:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:46:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:46:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:46:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:46:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:46:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:46:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:46:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:46:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:46:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:46:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:46:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:46:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:46:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:46:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:46:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:46:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:46:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:46:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:46:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:46:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:46:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:46:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:46:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:46:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:46:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:46:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:46:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:46:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:46:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:46:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:46:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:46:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:46:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:46:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:46:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:46:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:46:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:46:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:46:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:46:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:46:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:46:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:46:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:46:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:46:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:46:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:46:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:46:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:46:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:46:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:46:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:46:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:46:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:46:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:46:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:46:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:46:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:46:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:46:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:46:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:46:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:46:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:46:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:46:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:46:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:46:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:46:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:46:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:46:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:46:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:46:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:46:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:46:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:46:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:46:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:46:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:46:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:46:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:46:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:46:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:46:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:46:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:46:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:46:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:46:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:46:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:46:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:46:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:46:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:46:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:46:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:46:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:46:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:46:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:46:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:46:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:46:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:46:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:46:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:46:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:46:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:46:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:46:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:46:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:46:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:46:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:46:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:46:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:46:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:46:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:46:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:46:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:46:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:47:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:47:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:47:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:47:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:47:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:47:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:47:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:47:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:47:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:47:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:47:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:47:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:47:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:47:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:47:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:47:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:47:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:47:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:47:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:47:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:47:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:47:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:47:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:47:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:47:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:47:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:47:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:47:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:47:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:47:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:47:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:47:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:47:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:47:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:47:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:47:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:47:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:47:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:47:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:47:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:47:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:47:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:47:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:47:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:47:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:47:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:47:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:47:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:47:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:47:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:47:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:47:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:47:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:47:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:47:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:47:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:47:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:47:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:47:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:47:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:47:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:47:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:47:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:47:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:47:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:47:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:47:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:47:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:47:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:47:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:47:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:47:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:47:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:47:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:47:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:47:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:47:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:47:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:47:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:47:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:47:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:47:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:47:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:47:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:47:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:47:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:47:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:47:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:47:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:47:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:47:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:47:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:47:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:47:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:47:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:47:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:47:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:47:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:47:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:47:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:47:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:47:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:47:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:47:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:47:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:47:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:47:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:47:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:47:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:47:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:47:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:47:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:47:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:47:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:47:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:47:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:47:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:47:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:47:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:47:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:47:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:47:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:47:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:47:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:47:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:47:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:47:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:47:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:47:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:47:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:47:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:47:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:47:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:47:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:47:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:47:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:47:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:47:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:47:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:47:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:47:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:47:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:47:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:47:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:47:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:47:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:47:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:47:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:47:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:47:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:47:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:47:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:47:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:47:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:47:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:47:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:47:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:47:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:47:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:47:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:47:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:47:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:47:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:47:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:47:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:47:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:47:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:47:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:47:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:47:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:47:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:47:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:47:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:47:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:47:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:47:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:47:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:47:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:47:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:47:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:47:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:47:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:47:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:47:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:47:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:47:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:47:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:47:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:47:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:47:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:47:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:47:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:47:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:47:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:47:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:47:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:47:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:47:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:47:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:47:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:47:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:47:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:47:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:47:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:47:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:47:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:47:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:47:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:47:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:47:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:47:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:47:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:47:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:47:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:47:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:47:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:48:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:48:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:48:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:48:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:48:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:48:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:48:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:48:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:48:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:48:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:48:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:48:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:48:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:48:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:48:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:48:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:48:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:48:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:48:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:48:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:48:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:48:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:48:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:48:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:48:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:48:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:48:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:48:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:48:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:48:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:48:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:48:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:48:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:48:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:48:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:48:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:48:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:48:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:48:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:48:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:48:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:48:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:48:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:48:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:48:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:48:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:48:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:48:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:48:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:48:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:48:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:48:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:48:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:48:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:48:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:48:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:48:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:48:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:48:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:48:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:48:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:48:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:48:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:48:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:48:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:48:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:48:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:48:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:48:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:48:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:48:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:48:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:48:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:48:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:48:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:48:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:48:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:48:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:48:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:48:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:48:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:48:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:48:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:48:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:48:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:48:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:48:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:48:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:48:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:48:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:48:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:48:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:48:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:48:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:48:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:48:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:48:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:48:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:48:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:48:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:48:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:48:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:48:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:48:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:48:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:48:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:48:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:48:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:48:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:48:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:48:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:48:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:48:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:48:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:48:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:48:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:48:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:48:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:48:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:48:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:48:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:48:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:48:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:48:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:48:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:48:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:48:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:48:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:48:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:48:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:48:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:48:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:48:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:48:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:48:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:48:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:48:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:48:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:48:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:48:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:48:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:48:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:48:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:48:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:48:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:48:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:48:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:48:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:48:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:48:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:48:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:48:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:48:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:48:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:48:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:48:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:48:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:48:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:48:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:48:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:48:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:48:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:48:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:48:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:48:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:48:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:48:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:48:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:48:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:48:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:48:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:48:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:48:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:48:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:48:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:48:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:48:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:48:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:48:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:48:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:48:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:48:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:48:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:48:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:48:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:48:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:48:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:48:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:48:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:48:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:48:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:48:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:48:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:48:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:48:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:48:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:48:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:48:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:48:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:48:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:48:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:48:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:48:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:48:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:48:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:48:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:48:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:48:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:48:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:48:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:48:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:48:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:48:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:48:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:48:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:48:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:49:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:49:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:49:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:49:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:49:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:49:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:49:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:49:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:49:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:49:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:49:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:49:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:49:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:49:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:49:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:49:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:49:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:49:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:49:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:49:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:49:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:49:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:49:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:49:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:49:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:49:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:49:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:49:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:49:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:49:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:49:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:49:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:49:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:49:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:49:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:49:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:49:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:49:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:49:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:49:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:49:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:49:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:49:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:49:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:49:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:49:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:49:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:49:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:49:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:49:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:49:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:49:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:49:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:49:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:49:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:49:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:49:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:49:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:49:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:49:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:49:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:49:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:49:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:49:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:49:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:49:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:49:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:49:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:49:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:49:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:49:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:49:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:49:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:49:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:49:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:49:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:49:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:49:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:49:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:49:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:49:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:49:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:49:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:49:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:49:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:49:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:49:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:49:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:49:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:49:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:49:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:49:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:49:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:49:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:49:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:49:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:49:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:49:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:49:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:49:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:49:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:49:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:49:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:49:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:49:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:49:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:49:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:49:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:49:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:49:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:49:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:49:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:49:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:49:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:49:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:49:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:49:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:49:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:49:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:49:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:49:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:49:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:49:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:49:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:49:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:49:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:49:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:49:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:49:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:49:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:49:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:49:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:49:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:49:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:49:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:49:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:49:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:49:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:49:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:49:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:49:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:49:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:49:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:49:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:49:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:49:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:49:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:49:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:49:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:49:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:49:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:49:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:49:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:49:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:49:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:49:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:49:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:49:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:49:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:49:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:49:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:49:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:49:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:49:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:49:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:49:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:49:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:49:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:49:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:49:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:49:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:49:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:49:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:49:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:49:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:49:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:49:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:49:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:49:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:49:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:49:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:49:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:49:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:49:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:49:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:49:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:49:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:49:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:49:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:49:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:49:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:49:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:49:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:49:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:49:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:49:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:49:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:49:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:49:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:49:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:49:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:49:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:49:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:49:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:49:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:49:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:49:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:49:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:49:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:49:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:49:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:49:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:49:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:49:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:49:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:49:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:50:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:50:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:50:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:50:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:50:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:50:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:50:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:50:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:50:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:50:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:50:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:50:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:50:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:50:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:50:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:50:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:50:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:50:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:50:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:50:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:50:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:50:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:50:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:50:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:50:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:50:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:50:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:50:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:50:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:50:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:50:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:50:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:50:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:50:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:50:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:50:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:50:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:50:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:50:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:50:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:50:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:50:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:50:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:50:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:50:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:50:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:50:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:50:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:50:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:50:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:50:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:50:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:50:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:50:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:50:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:50:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:50:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:50:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:50:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:50:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:50:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:50:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:50:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:50:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:50:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:50:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:50:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:50:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:50:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:50:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:50:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:50:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:50:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:50:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:50:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:50:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:50:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:50:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:50:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:50:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:50:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:50:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:50:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:50:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:50:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:50:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:50:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:50:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:50:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:50:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:50:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:50:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:50:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:50:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:50:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:50:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:50:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:50:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:50:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:50:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:50:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:50:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:50:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:50:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:50:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:50:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:50:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:50:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:50:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:50:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:50:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:50:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:50:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:50:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:50:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:50:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:50:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:50:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:50:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:50:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:50:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:50:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:50:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:50:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:50:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:50:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:50:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:50:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:50:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:50:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:50:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:50:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:50:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:50:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:50:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:50:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:50:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:50:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:50:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:50:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:50:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:50:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:50:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:50:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:52:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:52:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:52:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:52:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:52:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:52:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:52:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:52:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:52:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:52:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:52:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:52:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:52:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:52:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:52:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:52:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:52:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:52:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:52:44 rdma06 rsyslogd[1041]: imjournal: 5082 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 04:52:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:52:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:52:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:52:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:52:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:52:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:52:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:52:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:52:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:52:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:52:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:52:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:52:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:52:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:52:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:52:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:52:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:52:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:52:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:52:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:52:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:52:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:52:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:52:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:52:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:52:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:52:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:52:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:52:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:52:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:52:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:52:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:52:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:52:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:52:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:52:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:52:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:52:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:52:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:52:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:52:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:52:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:52:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:52:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:52:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:52:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:52:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:52:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:52:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:52:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:52:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:52:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:52:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:52:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:53:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:53:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:53:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:53:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:53:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:53:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:53:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:53:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:53:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:53:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:53:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:53:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:53:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:53:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:53:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:53:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:53:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:53:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:53:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:53:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:53:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:53:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:53:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:53:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:53:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:53:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:53:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:53:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:53:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:53:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:53:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:53:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:53:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:53:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:53:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:53:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:53:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:53:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:53:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:53:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:53:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:53:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:53:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:53:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:53:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:53:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:53:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:53:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:53:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:53:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:53:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:53:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:53:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:53:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:53:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:53:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:53:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:53:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:53:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:53:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:53:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:53:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:53:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:53:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:53:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:53:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:53:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:53:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:53:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:53:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:53:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:53:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:53:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:53:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:53:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:53:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:53:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:53:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:53:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:53:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:53:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:53:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:53:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:53:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:53:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:53:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:53:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:53:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:53:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:53:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:53:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:53:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:53:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:53:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:53:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:53:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:53:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:53:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:53:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:53:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:53:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:53:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:53:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:53:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:53:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:53:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:53:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:53:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:53:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:53:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:53:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:53:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:53:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:53:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:53:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:53:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:53:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:53:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:53:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:53:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:53:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:53:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:53:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:53:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:53:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:53:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:53:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:53:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:53:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:53:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:53:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:53:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:53:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:53:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:53:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:53:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:53:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:53:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:53:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:53:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:53:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:53:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:53:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:53:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:53:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:53:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:53:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:53:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:53:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:53:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:53:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:53:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:53:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:53:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:53:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:53:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:53:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:53:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:53:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:53:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:53:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:53:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:53:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:53:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:53:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:53:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:53:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:53:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:53:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:53:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:53:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:53:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:53:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:53:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:53:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:53:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:53:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:53:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:53:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:53:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:53:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:53:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:53:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:53:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:53:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:53:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:53:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:53:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:53:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:53:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:53:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:53:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:53:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:53:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:53:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:53:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:53:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:53:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:53:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:53:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:53:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:53:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:53:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:53:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:53:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:53:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:53:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:53:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:53:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:53:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:53:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:53:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:53:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:53:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:53:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:53:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:54:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:54:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:54:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:54:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:54:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:54:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:54:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:54:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:54:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:54:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:54:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:54:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:54:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:54:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:54:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:54:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:54:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:54:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:54:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:54:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:54:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:54:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:54:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:54:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:54:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:54:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:54:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:54:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:54:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:54:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:54:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:54:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:54:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:54:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:54:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:54:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:54:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:54:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:54:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:54:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:54:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:54:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:54:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:54:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:54:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:54:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:54:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:54:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:54:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:54:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:54:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:54:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:54:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:54:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:54:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:54:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:54:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:54:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:54:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:54:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:54:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:54:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:54:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:54:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:54:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:54:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:54:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:54:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:54:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:54:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:54:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:54:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:54:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:54:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:54:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:54:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:54:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:54:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:54:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:54:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:54:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:54:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:54:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:54:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:54:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:54:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:54:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:54:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:54:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:54:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:54:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:54:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:54:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:54:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:54:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:54:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:54:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:54:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:54:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:54:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:54:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:54:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:54:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:54:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:54:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:54:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:54:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:54:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:54:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:54:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:54:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:54:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:54:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:54:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:54:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:54:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:54:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:54:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:54:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:54:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:54:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:54:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:54:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:54:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:54:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:54:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:54:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:54:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:54:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:54:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:54:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:54:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:54:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:54:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:54:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:54:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:54:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:54:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:54:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:54:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:54:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:54:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:54:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:54:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:54:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:54:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:54:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:54:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:54:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:54:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:54:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:54:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:54:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:54:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:54:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:54:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:54:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:54:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:54:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:54:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:54:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:54:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:54:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:54:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:54:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:54:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:54:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:54:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:54:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:54:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:54:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:54:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:54:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:54:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:54:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:54:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:54:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:54:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:54:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:54:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:54:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:54:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:54:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:54:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:54:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:54:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:54:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:54:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:54:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:54:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:54:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:54:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:54:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:54:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:54:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:54:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:54:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:54:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:54:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:54:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:54:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:54:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:54:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:54:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:54:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:54:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:54:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:54:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:54:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:54:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:54:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:54:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:54:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:54:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:54:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:54:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:55:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:55:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:55:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:55:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:55:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:55:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:55:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:55:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:55:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:55:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:55:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:55:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:55:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:55:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:55:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:55:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:55:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:55:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:55:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:55:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:55:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:55:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:55:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:55:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:55:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:55:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:55:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:55:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:55:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:55:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:55:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:55:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:55:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:55:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:55:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:55:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:55:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:55:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:55:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:55:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:55:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:55:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:55:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:55:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:55:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:55:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:55:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:55:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:55:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:55:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:55:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:55:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:55:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:55:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:55:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:55:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:55:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:55:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:55:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:55:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:55:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:55:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:55:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:55:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:55:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:55:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:55:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:55:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:55:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:55:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:55:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:55:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:55:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:55:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:55:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:55:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:55:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:55:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:55:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:55:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:55:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:55:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:55:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:55:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:55:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:55:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:55:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:55:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:55:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:55:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:55:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:55:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:55:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:55:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:55:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:55:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:55:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:55:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:55:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:55:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:55:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:55:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:55:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:55:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:55:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:55:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:55:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:55:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:55:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:55:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:55:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:55:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:55:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:55:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:55:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:55:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:55:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:55:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:55:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:55:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:55:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:55:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:55:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:55:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:55:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:55:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:55:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:55:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:55:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:55:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:55:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:55:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:55:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:55:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:55:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:55:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:55:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:55:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:55:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:55:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:55:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:55:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:55:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:55:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:55:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:55:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:55:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:55:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:55:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:55:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:55:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:55:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:55:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:55:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:55:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:55:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:55:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:55:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:55:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:55:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:55:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:55:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:55:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:55:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:55:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:55:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:55:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:55:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:55:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:55:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:55:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:55:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:55:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:55:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:55:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:55:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:55:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:55:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:55:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:55:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:55:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:55:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:55:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:55:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:55:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:55:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:55:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:55:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:55:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:55:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:55:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:55:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:55:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:55:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:55:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:55:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:55:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:55:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:55:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:55:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:55:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:55:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:55:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:55:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:55:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:55:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:55:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:55:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:55:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:55:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:55:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:55:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:55:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:55:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:55:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:55:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:56:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:56:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:56:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:56:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:56:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:56:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:56:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:56:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:56:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:56:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:56:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:56:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:56:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:56:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:56:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:56:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:56:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:56:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:56:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:56:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:56:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:56:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:56:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:56:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:56:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:56:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:56:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:56:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:56:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:56:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:56:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:56:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:56:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:56:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:56:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:56:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:56:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:56:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:56:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:56:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:56:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:56:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:56:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:56:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:56:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:56:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:56:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:56:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:56:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:56:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:56:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:56:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:56:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:56:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:56:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:56:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:56:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:56:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:56:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:56:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:56:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:56:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:56:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:56:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:56:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:56:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:56:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:56:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:56:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:56:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:56:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:56:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:56:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:56:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:56:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:56:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:56:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:56:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:56:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:56:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:56:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:56:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:56:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:56:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:56:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:56:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:56:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:56:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:56:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:56:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:56:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:56:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:56:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:56:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:56:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:56:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:56:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:56:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:56:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:56:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:56:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:56:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:56:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:56:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:56:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:56:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:56:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:56:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:56:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:56:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:56:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:56:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:56:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:56:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:56:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:56:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:56:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:56:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:56:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:56:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:56:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:56:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:56:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:56:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:56:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:56:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:56:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:56:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:56:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:56:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:56:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:56:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:56:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:56:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:56:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:56:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:56:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:56:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:56:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:56:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:56:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:56:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:56:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:56:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:56:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:56:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:56:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:56:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:56:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:56:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:56:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:56:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:56:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:56:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:56:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:56:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:56:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:56:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:56:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:56:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:56:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:56:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:56:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:56:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:56:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:56:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:56:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:56:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:56:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:56:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:56:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:56:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:56:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:56:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:56:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:56:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:56:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:56:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:56:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:56:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:56:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:56:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:56:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:56:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:56:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:56:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:56:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:56:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:56:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:56:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:56:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:56:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:56:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:56:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:56:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:56:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:56:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:56:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:56:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:56:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:56:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:56:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:56:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:56:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:56:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:56:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:56:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:56:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:56:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:56:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:56:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:56:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:56:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:56:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:56:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:56:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:57:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:57:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:57:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:57:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:57:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:57:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:57:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:57:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:57:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:57:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:57:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:57:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:57:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:57:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:57:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:57:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:57:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:57:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:57:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:57:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:57:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:57:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:57:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:57:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:57:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:57:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:57:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:57:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:57:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:57:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:57:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:57:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:57:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:57:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:57:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:57:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:57:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:57:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:57:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:57:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:57:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:57:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:57:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:57:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:57:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:57:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:57:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:57:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:57:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:57:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:57:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:57:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:57:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:57:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:57:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:57:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:57:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:57:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:57:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:57:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:57:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:57:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:57:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:57:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:57:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:57:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:57:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:57:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:57:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:57:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:57:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:57:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:57:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:57:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:57:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:57:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:57:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:57:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:57:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:57:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:57:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:57:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:57:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:57:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:57:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:57:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:57:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:57:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:57:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:57:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:57:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:57:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:57:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:57:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:57:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:57:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:57:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:57:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:57:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:57:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:57:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:57:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:57:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:57:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:57:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:57:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:57:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:57:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:57:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:57:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:57:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:57:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:57:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:57:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:57:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:57:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:57:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:57:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:57:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:57:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:57:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:57:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:57:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:57:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:57:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:57:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:57:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:57:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:57:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:57:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:57:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:57:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:57:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:57:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:57:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:57:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:57:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:57:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:57:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:57:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:57:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:57:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:57:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:57:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:57:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:57:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:57:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:57:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:57:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:57:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:57:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:57:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:57:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:57:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:57:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:57:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:57:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:57:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:57:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:57:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:57:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:57:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:57:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:57:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:57:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:57:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:57:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:57:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:57:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:57:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:57:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:57:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:57:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:57:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:57:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:57:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:57:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:57:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:57:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:57:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:57:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:57:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:57:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:57:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:57:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:57:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:57:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:57:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:57:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:57:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:57:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:57:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:57:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:57:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:57:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:57:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:57:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:57:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:57:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:57:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:57:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:57:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:57:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:57:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:57:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:57:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:57:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:57:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:57:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:57:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:57:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:57:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:57:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:57:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:57:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:57:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:58:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:58:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:58:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:58:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:58:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:58:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:58:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:58:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:58:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:58:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:58:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:58:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:58:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:58:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:58:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:58:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:58:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:58:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:58:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:58:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:58:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:58:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:58:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:58:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:58:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:58:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:58:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:58:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:58:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:58:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:58:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:58:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:58:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:58:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:58:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:58:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:58:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:58:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:58:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:58:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:58:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:58:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:58:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:58:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:58:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:58:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:58:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:58:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:58:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:58:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:58:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:58:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:58:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:58:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:58:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:58:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:58:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:58:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:58:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:58:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:58:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:58:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:58:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:58:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:58:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:58:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:58:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:58:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:58:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:58:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:58:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:58:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:58:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:58:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:58:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:58:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:58:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:58:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:58:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:58:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:58:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:58:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:58:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:58:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:58:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:58:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:58:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:58:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:58:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:58:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:58:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:58:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:58:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:58:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:58:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:58:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:58:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:58:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:58:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:58:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:58:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:58:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:58:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:58:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:58:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:58:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:58:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:58:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:58:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:58:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:58:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:58:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:58:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:58:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:58:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:58:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:58:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:58:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:58:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:58:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:58:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:58:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:58:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:58:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:58:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:58:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:58:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:58:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:58:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:58:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:58:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:58:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:58:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:58:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:58:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:58:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:58:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:58:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:58:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:58:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:58:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:58:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:58:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:58:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:58:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:58:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:58:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:58:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:58:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:58:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:58:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:58:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:58:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:58:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:58:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:58:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:58:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:58:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:58:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:58:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:58:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:58:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:58:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:58:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:58:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:58:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:58:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:58:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:58:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:58:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:58:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:58:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:58:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:58:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:58:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:58:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:58:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:58:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:58:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:58:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:58:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:58:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:58:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:58:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:58:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:58:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:58:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:58:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:58:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:58:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:58:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:58:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:58:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:58:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:58:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:58:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:58:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:58:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:58:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:58:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:58:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:58:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:58:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:58:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:58:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:58:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:58:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:58:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:58:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:58:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:58:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:58:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:58:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:58:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:58:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:58:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:59:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:59:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:59:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:59:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:59:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:59:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:59:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:59:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:59:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:59:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:59:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:59:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:59:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:59:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:59:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:59:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:59:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:59:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:59:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:59:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:59:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:59:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:59:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:59:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:59:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:59:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:59:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:59:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:59:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:59:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:59:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:59:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:59:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:59:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:59:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:59:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:59:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:59:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:59:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:59:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:59:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:59:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:59:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:59:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:59:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:59:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:59:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:59:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:59:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:59:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:59:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:59:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:59:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:59:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:59:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:59:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:59:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:59:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:59:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:59:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:59:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:59:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:59:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:59:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:59:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:59:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:59:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:59:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:59:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:59:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:59:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:59:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:59:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:59:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:59:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:59:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:59:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:59:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:59:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:59:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:59:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:59:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:59:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:59:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:59:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:59:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:59:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:59:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:59:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:59:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:59:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:59:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:59:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:59:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:59:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:59:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:59:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:59:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:59:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:59:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:59:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:59:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:59:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:59:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:59:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:59:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:59:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:59:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:59:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:59:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:59:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:59:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:59:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:59:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:59:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:59:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:59:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:59:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:59:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:59:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:59:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:59:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:59:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:59:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:59:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:59:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:59:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:59:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:59:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:59:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:59:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:59:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:59:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:59:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:59:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:59:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:59:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:59:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:59:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:59:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:59:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:59:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:59:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:59:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:59:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:59:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:59:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:59:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:59:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:59:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:59:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:59:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:59:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:59:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:59:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:59:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:59:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:59:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:59:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:59:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:59:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:59:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:59:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:59:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:59:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:59:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:59:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:59:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:59:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:59:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:59:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:59:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:59:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:59:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:59:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:59:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:59:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:59:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:59:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:59:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:59:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:59:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:59:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:59:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:59:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:59:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:59:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:59:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:59:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:59:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:59:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:59:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:59:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:59:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:59:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:59:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:59:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:59:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 04:59:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 04:59:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:59:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:59:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:59:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 04:59:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 04:59:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 04:59:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 04:59:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 04:59:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 04:59:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 04:59:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 04:59:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 04:59:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 04:59:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 04:59:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 04:59:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 04:59:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:00:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:00:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:00:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:00:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:00:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:00:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:00:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:00:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:00:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:00:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:00:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:00:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:00:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:00:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:00:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:00:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:00:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:00:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:00:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:00:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:00:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:00:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:00:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:00:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:00:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:00:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:00:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:00:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:00:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:00:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:00:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:00:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:00:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:00:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:00:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:00:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:00:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:00:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:00:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:00:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:00:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:00:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:00:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:00:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:00:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:00:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:00:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:00:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:00:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:00:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:00:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:00:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:00:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:00:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:00:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:00:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:00:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:00:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:00:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:00:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:00:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:00:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:00:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:00:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:00:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:00:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:00:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:00:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:00:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:00:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:00:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:00:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:00:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:00:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:00:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:00:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:00:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:00:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:00:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:00:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:00:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:00:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:00:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:00:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:00:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:00:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:00:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:00:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:00:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:00:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:00:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:00:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:00:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:00:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:00:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:00:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:00:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:00:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:00:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:00:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:00:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:00:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:00:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:00:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:00:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:00:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:00:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:00:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:00:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:00:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:00:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:00:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:00:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:00:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:00:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:00:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:00:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:00:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:00:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:00:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:00:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:00:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:00:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:00:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:00:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:00:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:00:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:00:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:00:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:00:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:00:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:00:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:00:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:00:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:00:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:00:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:00:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:00:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:00:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:00:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:00:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:00:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:00:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:00:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:02:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:02:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:02:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:02:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:02:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:02:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:02:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:02:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:02:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:02:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:02:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:02:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:02:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:02:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:02:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:02:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:02:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:02:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:02:48 rdma06 rsyslogd[1041]: imjournal: 5295 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 05:02:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:02:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:02:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:02:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:02:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:02:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:02:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:02:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:02:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:02:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:02:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:02:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:02:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:02:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:02:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:02:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:02:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:02:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:02:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:02:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:02:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:02:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:02:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:02:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:02:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:02:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:02:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:02:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:02:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:02:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:02:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:02:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:02:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:02:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:02:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:02:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:03:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:03:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:03:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:03:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:03:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:03:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:03:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:03:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:03:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:03:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:03:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:03:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:03:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:03:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:03:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:03:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:03:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:03:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:03:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:03:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:03:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:03:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:03:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:03:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:03:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:03:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:03:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:03:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:03:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:03:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:03:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:03:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:03:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:03:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:03:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:03:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:03:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:03:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:03:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:03:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:03:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:03:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:03:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:03:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:03:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:03:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:03:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:03:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:03:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:03:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:03:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:03:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:03:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:03:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:03:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:03:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:03:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:03:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:03:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:03:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:03:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:03:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:03:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:03:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:03:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:03:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:03:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:03:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:03:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:03:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:03:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:03:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:03:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:03:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:03:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:03:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:03:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:03:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:03:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:03:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:03:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:03:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:03:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:03:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:03:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:03:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:03:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:03:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:03:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:03:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:03:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:03:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:03:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:03:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:03:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:03:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:03:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:03:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:03:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:03:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:03:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:03:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:03:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:03:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:03:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:03:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:03:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:03:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:03:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:03:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:03:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:03:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:03:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:03:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:03:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:03:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:03:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:03:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:03:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:03:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:03:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:03:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:03:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:03:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:03:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:03:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:03:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:03:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:03:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:03:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:03:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:03:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:03:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:03:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:03:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:03:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:03:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:03:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:03:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:03:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:03:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:03:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:03:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:03:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:03:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:03:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:03:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:03:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:03:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:03:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:03:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:03:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:03:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:03:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:03:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:03:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:03:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:03:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:03:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:03:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:03:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:03:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:03:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:03:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:03:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:03:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:03:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:03:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:03:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:03:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:03:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:03:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:03:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:03:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:03:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:03:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:03:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:03:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:03:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:03:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:03:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:03:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:03:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:03:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:03:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:03:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:03:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:03:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:03:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:03:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:03:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:03:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:03:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:03:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:03:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:03:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:03:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:03:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:03:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:03:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:03:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:03:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:03:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:03:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:03:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:03:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:03:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:03:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:03:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:03:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:03:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:03:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:03:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:03:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:03:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:03:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:04:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:04:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:04:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:04:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:04:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:04:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:04:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:04:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:04:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:04:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:04:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:04:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:04:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:04:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:04:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:04:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:04:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:04:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:04:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:04:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:04:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:04:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:04:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:04:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:04:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:04:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:04:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:04:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:04:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:04:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:04:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:04:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:04:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:04:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:04:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:04:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:04:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:04:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:04:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:04:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:04:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:04:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:04:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:04:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:04:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:04:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:04:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:04:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:04:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:04:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:04:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:04:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:04:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:04:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:04:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:04:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:04:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:04:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:04:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:04:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:04:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:04:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:04:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:04:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:04:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:04:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:04:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:04:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:04:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:04:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:04:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:04:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:04:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:04:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:04:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:04:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:04:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:04:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:04:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:04:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:04:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:04:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:04:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:04:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:04:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:04:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:04:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:04:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:04:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:04:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:04:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:04:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:04:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:04:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:04:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:04:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:04:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:04:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:04:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:04:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:04:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:04:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:04:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:04:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:04:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:04:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:04:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:04:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:04:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:04:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:04:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:04:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:04:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:04:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:04:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:04:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:04:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:04:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:04:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:04:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:04:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:04:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:04:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:04:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:04:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:04:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:04:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:04:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:04:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:04:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:04:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:04:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:04:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:04:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:04:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:04:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:04:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:04:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:04:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:04:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:04:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:04:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:04:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:04:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:04:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:04:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:04:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:04:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:04:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:04:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:04:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:04:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:04:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:04:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:04:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:04:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:04:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:04:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:04:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:04:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:04:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:04:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:04:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:04:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:04:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:04:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:04:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:04:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:04:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:04:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:04:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:04:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:04:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:04:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:04:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:04:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:04:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:04:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:04:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:04:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:04:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:04:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:04:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:04:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:04:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:04:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:04:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:04:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:04:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:04:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:04:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:04:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:04:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:04:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:04:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:04:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:04:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:04:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:04:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:04:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:04:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:04:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:04:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:04:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:04:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:04:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:04:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:04:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:04:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:04:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:04:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:04:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:04:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:04:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:04:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:04:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:05:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:05:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:05:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:05:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:05:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:05:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:05:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:05:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:05:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:05:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:05:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:05:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:05:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:05:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:05:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:05:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:05:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:05:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:05:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:05:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:05:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:05:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:05:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:05:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:05:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:05:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:05:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:05:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:05:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:05:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:05:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:05:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:05:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:05:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:05:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:05:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:05:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:05:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:05:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:05:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:05:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:05:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:05:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:05:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:05:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:05:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:05:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:05:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:05:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:05:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:05:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:05:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:05:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:05:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:05:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:05:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:05:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:05:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:05:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:05:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:05:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:05:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:05:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:05:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:05:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:05:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:05:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:05:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:05:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:05:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:05:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:05:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:05:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:05:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:05:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:05:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:05:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:05:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:05:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:05:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:05:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:05:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:05:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:05:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:05:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:05:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:05:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:05:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:05:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:05:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:05:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:05:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:05:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:05:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:05:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:05:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:05:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:05:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:05:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:05:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:05:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:05:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:05:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:05:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:05:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:05:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:05:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:05:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:05:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:05:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:05:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:05:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:05:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:05:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:05:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:05:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:05:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:05:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:05:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:05:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:05:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:05:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:05:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:05:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:05:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:05:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:05:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:05:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:05:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:05:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:05:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:05:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:05:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:05:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:05:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:05:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:05:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:05:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:05:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:05:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:05:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:05:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:05:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:05:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:05:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:05:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:05:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:05:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:05:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:05:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:05:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:05:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:05:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:05:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:05:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:05:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:05:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:05:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:05:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:05:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:05:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:05:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:05:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:05:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:05:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:05:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:05:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:05:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:05:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:05:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:05:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:05:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:05:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:05:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:05:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:05:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:05:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:05:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:05:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:05:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:05:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:05:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:05:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:05:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:05:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:05:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:05:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:05:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:05:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:05:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:05:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:05:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:05:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:05:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:05:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:05:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:05:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:05:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:05:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:05:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:05:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:05:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:05:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:05:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:05:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:05:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:05:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:05:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:05:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:05:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:05:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:05:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:05:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:05:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:05:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:05:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:06:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:06:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:06:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:06:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:06:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:06:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:06:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:06:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:06:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:06:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:06:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:06:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:06:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:06:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:06:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:06:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:06:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:06:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:06:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:06:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:06:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:06:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:06:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:06:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:06:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:06:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:06:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:06:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:06:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:06:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:06:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:06:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:06:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:06:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:06:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:06:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:06:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:06:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:06:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:06:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:06:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:06:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:06:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:06:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:06:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:06:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:06:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:06:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:06:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:06:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:06:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:06:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:06:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:06:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:06:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:06:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:06:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:06:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:06:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:06:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:06:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:06:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:06:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:06:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:06:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:06:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:06:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:06:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:06:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:06:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:06:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:06:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:06:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:06:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:06:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:06:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:06:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:06:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:06:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:06:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:06:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:06:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:06:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:06:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:06:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:06:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:06:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:06:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:06:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:06:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:06:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:06:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:06:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:06:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:06:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:06:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:06:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:06:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:06:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:06:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:06:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:06:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:06:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:06:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:06:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:06:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:06:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:06:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:06:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:06:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:06:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:06:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:06:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:06:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:06:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:06:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:06:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:06:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:06:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:06:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:06:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:06:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:06:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:06:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:06:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:06:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:06:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:06:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:06:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:06:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:06:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:06:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:06:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:06:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:06:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:06:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:06:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:06:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:06:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:06:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:06:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:06:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:06:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:06:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:06:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:06:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:06:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:06:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:06:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:06:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:06:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:06:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:06:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:06:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:06:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:06:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:06:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:06:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:06:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:06:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:06:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:06:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:06:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:06:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:06:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:06:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:06:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:06:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:06:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:06:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:06:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:06:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:06:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:06:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:06:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:06:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:06:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:06:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:06:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:06:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:06:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:06:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:06:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:06:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:06:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:06:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:06:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:06:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:06:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:06:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:06:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:06:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:06:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:06:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:06:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:06:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:06:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:06:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:06:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:06:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:06:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:06:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:06:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:06:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:06:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:06:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:06:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:06:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:06:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:06:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:06:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:06:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:06:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:06:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:06:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:06:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:07:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:07:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:07:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:07:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:07:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:07:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:07:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:07:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:07:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:07:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:07:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:07:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:07:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:07:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:07:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:07:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:07:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:07:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:07:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:07:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:07:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:07:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:07:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:07:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:07:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:07:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:07:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:07:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:07:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:07:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:07:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:07:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:07:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:07:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:07:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:07:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:07:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:07:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:07:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:07:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:07:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:07:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:07:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:07:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:07:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:07:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:07:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:07:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:07:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:07:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:07:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:07:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:07:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:07:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:07:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:07:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:07:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:07:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:07:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:07:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:07:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:07:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:07:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:07:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:07:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:07:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:07:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:07:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:07:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:07:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:07:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:07:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:07:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:07:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:07:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:07:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:07:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:07:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:07:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:07:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:07:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:07:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:07:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:07:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:07:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:07:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:07:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:07:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:07:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:07:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:07:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:07:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:07:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:07:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:07:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:07:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:07:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:07:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:07:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:07:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:07:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:07:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:07:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:07:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:07:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:07:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:07:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:07:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:07:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:07:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:07:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:07:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:07:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:07:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:07:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:07:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:07:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:07:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:07:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:07:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:07:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:07:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:07:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:07:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:07:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:07:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:07:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:07:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:07:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:07:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:07:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:07:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:07:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:07:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:07:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:07:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:07:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:07:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:07:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:07:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:07:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:07:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:07:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:07:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:07:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:07:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:07:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:07:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:07:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:07:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:07:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:07:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:07:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:07:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:07:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:07:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:07:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:07:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:07:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:07:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:07:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:07:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:07:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:07:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:07:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:07:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:07:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:07:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:07:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:07:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:07:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:07:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:07:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:07:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:07:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:07:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:07:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:07:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:07:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:07:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:07:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:07:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:07:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:07:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:07:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:07:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:07:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:07:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:07:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:07:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:07:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:07:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:07:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:07:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:07:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:07:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:07:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:07:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:07:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:07:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:07:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:07:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:07:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:07:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:07:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:07:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:07:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:07:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:07:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:07:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:07:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:07:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:07:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:07:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:07:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:07:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:08:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:08:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:08:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:08:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:08:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:08:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:08:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:08:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:08:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:08:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:08:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:08:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:08:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:08:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:08:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:08:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:08:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:08:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:08:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:08:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:08:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:08:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:08:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:08:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:08:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:08:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:08:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:08:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:08:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:08:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:08:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:08:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:08:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:08:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:08:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:08:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:08:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:08:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:08:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:08:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:08:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:08:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:08:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:08:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:08:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:08:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:08:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:08:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:08:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:08:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:08:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:08:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:08:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:08:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:08:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:08:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:08:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:08:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:08:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:08:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:08:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:08:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:08:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:08:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:08:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:08:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:08:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:08:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:08:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:08:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:08:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:08:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:08:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:08:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:08:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:08:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:08:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:08:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:08:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:08:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:08:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:08:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:08:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:08:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:08:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:08:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:08:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:08:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:08:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:08:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:08:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:08:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:08:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:08:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:08:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:08:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:08:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:08:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:08:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:08:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:08:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:08:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:08:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:08:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:08:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:08:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:08:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:08:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:08:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:08:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:08:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:08:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:08:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:08:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:08:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:08:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:08:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:08:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:08:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:08:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:08:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:08:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:08:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:08:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:08:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:08:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:08:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:08:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:08:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:08:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:08:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:08:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:08:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:08:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:08:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:08:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:08:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:08:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:08:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:08:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:08:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:08:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:08:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:08:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:08:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:08:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:08:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:08:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:08:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:08:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:08:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:08:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:08:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:08:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:08:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:08:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:08:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:08:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:08:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:08:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:08:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:08:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:08:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:08:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:08:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:08:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:08:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:08:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:08:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:08:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:08:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:08:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:08:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:08:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:08:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:08:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:08:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:08:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:08:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:08:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:08:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:08:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:08:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:08:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:08:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:08:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:08:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:08:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:08:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:08:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:08:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:08:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:08:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:08:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:08:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:08:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:08:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:08:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:08:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:08:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:08:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:08:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:08:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:08:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:08:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:08:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:08:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:08:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:08:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:08:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:08:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:08:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:08:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:08:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:08:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:08:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:09:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:09:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:09:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:09:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:09:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:09:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:09:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:09:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:09:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:09:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:09:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:09:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:09:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:09:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:09:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:09:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:09:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:09:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:09:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:09:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:09:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:09:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:09:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:09:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:09:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:09:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:09:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:09:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:09:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:09:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:09:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:09:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:09:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:09:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:09:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:09:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:09:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:09:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:09:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:09:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:09:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:09:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:09:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:09:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:09:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:09:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:09:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:09:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:09:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:09:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:09:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:09:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:09:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:09:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:09:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:09:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:09:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:09:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:09:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:09:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:09:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:09:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:09:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:09:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:09:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:09:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:09:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:09:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:09:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:09:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:09:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:09:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:09:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:09:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:09:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:09:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:09:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:09:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:09:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:09:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:09:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:09:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:09:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:09:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:09:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:09:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:09:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:09:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:09:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:09:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:09:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:09:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:09:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:09:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:09:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:09:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:09:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:09:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:09:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:09:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:09:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:09:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:09:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:09:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:09:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:09:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:09:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:09:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:09:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:09:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:09:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:09:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:09:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:09:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:09:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:09:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:09:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:09:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:09:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:09:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:09:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:09:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:09:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:09:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:09:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:09:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:09:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:09:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:09:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:09:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:09:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:09:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:09:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:09:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:09:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:09:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:09:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:09:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:09:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:09:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:09:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:09:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:09:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:09:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:09:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:09:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:09:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:09:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:09:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:09:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:09:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:09:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:09:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:09:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:09:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:09:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:09:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:09:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:09:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:09:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:09:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:09:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:09:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:09:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:09:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:09:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:09:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:09:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:09:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:09:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:09:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:09:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:09:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:09:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:09:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:09:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:09:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:09:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:09:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:09:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:09:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:09:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:09:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:09:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:09:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:09:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:09:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:09:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:09:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:09:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:09:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:09:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:09:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:09:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:09:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:09:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:09:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:09:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:09:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:09:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:09:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:09:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:09:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:09:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:09:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:09:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:09:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:09:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:09:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:09:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:09:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:09:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:09:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:09:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:09:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:09:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:10:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:10:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:10:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:10:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:10:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:10:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:10:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:10:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:10:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:10:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:10:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:10:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:10:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:10:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:10:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:10:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:10:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:10:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:10:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:10:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:10:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:10:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:10:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:10:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:10:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:10:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:10:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:10:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:10:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:10:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:10:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:10:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:10:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:10:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:10:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:10:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:10:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:10:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:10:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:10:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:10:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:10:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:10:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:10:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:10:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:10:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:10:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:10:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:10:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:10:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:10:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:10:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:10:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:10:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:10:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:10:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:10:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:10:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:10:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:10:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:10:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:10:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:10:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:10:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:10:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:10:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:10:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:10:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:10:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:10:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:10:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:10:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:10:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:10:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:10:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:10:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:10:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:10:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:10:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:10:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:10:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:10:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:10:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:10:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:10:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:10:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:10:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:10:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:10:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:10:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:10:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:10:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:10:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:10:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:10:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:10:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:10:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:10:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:10:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:10:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:10:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:10:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:10:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:10:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:10:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:10:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:10:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:10:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:10:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:10:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:10:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:10:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:10:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:10:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:10:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:10:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:10:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:10:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:10:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:10:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:10:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:10:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:10:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:10:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:10:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:10:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:10:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:10:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:10:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:10:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:10:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:10:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:10:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:10:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:10:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:10:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:10:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:10:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:10:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:10:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:10:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:10:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:10:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:10:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:10:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:10:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:10:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:10:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:10:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:10:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:10:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:10:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:10:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:10:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:10:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:10:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:10:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:10:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:10:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:10:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:10:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:10:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:12:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:12:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:12:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:12:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:12:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:12:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:12:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:12:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:12:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:12:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:12:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:12:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:12:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:12:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:12:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:12:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:12:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:12:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:12:49 rdma06 rsyslogd[1041]: imjournal: 5082 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 05:12:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:12:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:12:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:12:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:12:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:12:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:12:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:12:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:12:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:12:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:12:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:12:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:12:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:12:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:12:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:12:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:12:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:12:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:12:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:12:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:12:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:12:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:12:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:12:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:12:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:12:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:12:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:12:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:12:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:12:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:12:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:12:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:12:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:12:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:12:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:12:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:13:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:13:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:13:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:13:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:13:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:13:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:13:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:13:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:13:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:13:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:13:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:13:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:13:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:13:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:13:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:13:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:13:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:13:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:13:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:13:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:13:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:13:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:13:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:13:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:13:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:13:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:13:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:13:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:13:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:13:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:13:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:13:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:13:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:13:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:13:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:13:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:13:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:13:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:13:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:13:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:13:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:13:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:13:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:13:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:13:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:13:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:13:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:13:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:13:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:13:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:13:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:13:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:13:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:13:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:13:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:13:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:13:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:13:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:13:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:13:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:13:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:13:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:13:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:13:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:13:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:13:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:13:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:13:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:13:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:13:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:13:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:13:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:13:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:13:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:13:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:13:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:13:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:13:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:13:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:13:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:13:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:13:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:13:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:13:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:13:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:13:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:13:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:13:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:13:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:13:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:13:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:13:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:13:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:13:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:13:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:13:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:13:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:13:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:13:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:13:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:13:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:13:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:13:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:13:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:13:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:13:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:13:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:13:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:13:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:13:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:13:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:13:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:13:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:13:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:13:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:13:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:13:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:13:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:13:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:13:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:13:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:13:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:13:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:13:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:13:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:13:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:13:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:13:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:13:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:13:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:13:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:13:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:13:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:13:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:13:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:13:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:13:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:13:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:13:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:13:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:13:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:13:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:13:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:13:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:13:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:13:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:13:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:13:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:13:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:13:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:13:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:13:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:13:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:13:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:13:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:13:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:13:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:13:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:13:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:13:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:13:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:13:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:13:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:13:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:13:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:13:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:13:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:13:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:13:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:13:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:13:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:13:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:13:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:13:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:13:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:13:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:13:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:13:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:13:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:13:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:13:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:13:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:13:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:13:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:13:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:13:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:13:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:13:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:13:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:13:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:13:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:13:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:13:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:13:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:13:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:13:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:13:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:13:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:13:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:13:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:13:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:13:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:13:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:13:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:13:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:13:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:13:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:13:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:13:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:13:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:13:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:13:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:13:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:13:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:13:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:13:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:14:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:14:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:14:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:14:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:14:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:14:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:14:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:14:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:14:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:14:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:14:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:14:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:14:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:14:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:14:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:14:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:14:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:14:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:14:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:14:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:14:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:14:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:14:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:14:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:14:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:14:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:14:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:14:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:14:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:14:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:14:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:14:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:14:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:14:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:14:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:14:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:14:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:14:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:14:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:14:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:14:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:14:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:14:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:14:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:14:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:14:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:14:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:14:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:14:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:14:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:14:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:14:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:14:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:14:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:14:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:14:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:14:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:14:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:14:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:14:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:14:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:14:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:14:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:14:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:14:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:14:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:14:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:14:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:14:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:14:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:14:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:14:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:14:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:14:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:14:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:14:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:14:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:14:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:14:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:14:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:14:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:14:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:14:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:14:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:14:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:14:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:14:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:14:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:14:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:14:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:14:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:14:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:14:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:14:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:14:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:14:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:14:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:14:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:14:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:14:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:14:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:14:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:14:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:14:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:14:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:14:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:14:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:14:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:14:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:14:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:14:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:14:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:14:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:14:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:14:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:14:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:14:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:14:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:14:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:14:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:14:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:14:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:14:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:14:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:14:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:14:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:14:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:14:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:14:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:14:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:14:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:14:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:14:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:14:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:14:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:14:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:14:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:14:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:14:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:14:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:14:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:14:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:14:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:14:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:14:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:14:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:14:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:14:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:14:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:14:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:14:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:14:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:14:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:14:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:14:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:14:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:14:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:14:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:14:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:14:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:14:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:14:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:14:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:14:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:14:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:14:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:14:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:14:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:14:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:14:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:14:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:14:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:14:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:14:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:14:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:14:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:14:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:14:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:14:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:14:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:14:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:14:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:14:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:14:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:14:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:14:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:14:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:14:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:14:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:14:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:14:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:14:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:14:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:14:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:14:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:14:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:14:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:14:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:14:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:14:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:14:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:14:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:14:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:14:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:14:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:14:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:14:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:14:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:14:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:14:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:14:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:14:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:14:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:14:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:14:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:14:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:15:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:15:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:15:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:15:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:15:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:15:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:15:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:15:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:15:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:15:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:15:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:15:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:15:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:15:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:15:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:15:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:15:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:15:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:15:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:15:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:15:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:15:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:15:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:15:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:15:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:15:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:15:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:15:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:15:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:15:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:15:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:15:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:15:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:15:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:15:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:15:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:15:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:15:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:15:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:15:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:15:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:15:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:15:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:15:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:15:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:15:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:15:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:15:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:15:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:15:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:15:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:15:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:15:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:15:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:15:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:15:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:15:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:15:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:15:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:15:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:15:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:15:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:15:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:15:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:15:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:15:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:15:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:15:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:15:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:15:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:15:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:15:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:15:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:15:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:15:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:15:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:15:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:15:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:15:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:15:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:15:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:15:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:15:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:15:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:15:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:15:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:15:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:15:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:15:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:15:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:15:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:15:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:15:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:15:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:15:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:15:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:15:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:15:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:15:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:15:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:15:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:15:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:15:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:15:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:15:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:15:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:15:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:15:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:15:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:15:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:15:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:15:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:15:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:15:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:15:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:15:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:15:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:15:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:15:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:15:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:15:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:15:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:15:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:15:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:15:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:15:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:15:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:15:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:15:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:15:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:15:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:15:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:15:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:15:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:15:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:15:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:15:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:15:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:15:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:15:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:15:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:15:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:15:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:15:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:15:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:15:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:15:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:15:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:15:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:15:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:15:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:15:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:15:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:15:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:15:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:15:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:15:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:15:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:15:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:15:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:15:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:15:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:15:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:15:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:15:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:15:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:15:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:15:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:15:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:15:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:15:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:15:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:15:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:15:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:15:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:15:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:15:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:15:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:15:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:15:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:15:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:15:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:15:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:15:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:15:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:15:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:15:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:15:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:15:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:15:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:15:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:15:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:15:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:15:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:15:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:15:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:15:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:15:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:15:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:15:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:15:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:15:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:15:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:15:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:15:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:15:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:15:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:15:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:15:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:15:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:15:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:15:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:15:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:15:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:15:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:15:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:16:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:16:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:16:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:16:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:16:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:16:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:16:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:16:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:16:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:16:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:16:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:16:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:16:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:16:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:16:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:16:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:16:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:16:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:16:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:16:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:16:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:16:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:16:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:16:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:16:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:16:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:16:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:16:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:16:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:16:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:16:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:16:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:16:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:16:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:16:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:16:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:16:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:16:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:16:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:16:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:16:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:16:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:16:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:16:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:16:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:16:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:16:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:16:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:16:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:16:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:16:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:16:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:16:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:16:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:16:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:16:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:16:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:16:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:16:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:16:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:16:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:16:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:16:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:16:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:16:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:16:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:16:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:16:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:16:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:16:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:16:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:16:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:16:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:16:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:16:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:16:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:16:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:16:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:16:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:16:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:16:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:16:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:16:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:16:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:16:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:16:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:16:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:16:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:16:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:16:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:16:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:16:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:16:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:16:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:16:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:16:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:16:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:16:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:16:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:16:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:16:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:16:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:16:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:16:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:16:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:16:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:16:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:16:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:16:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:16:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:16:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:16:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:16:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:16:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:16:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:16:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:16:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:16:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:16:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:16:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:16:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:16:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:16:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:16:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:16:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:16:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:16:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:16:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:16:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:16:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:16:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:16:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:16:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:16:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:16:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:16:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:16:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:16:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:16:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:16:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:16:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:16:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:16:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:16:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:16:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:16:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:16:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:16:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:16:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:16:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:16:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:16:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:16:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:16:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:16:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:16:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:16:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:16:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:16:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:16:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:16:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:16:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:16:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:16:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:16:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:16:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:16:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:16:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:16:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:16:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:16:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:16:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:16:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:16:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:16:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:16:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:16:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:16:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:16:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:16:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:16:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:16:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:16:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:16:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:16:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:16:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:16:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:16:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:16:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:16:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:16:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:16:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:16:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:16:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:16:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:16:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:16:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:16:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:16:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:16:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:16:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:16:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:16:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:16:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:16:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:16:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:16:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:16:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:16:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:16:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:16:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:16:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:16:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:16:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:16:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:16:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:17:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:17:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:17:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:17:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:17:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:17:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:17:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:17:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:17:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:17:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:17:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:17:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:17:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:17:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:17:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:17:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:17:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:17:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:17:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:17:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:17:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:17:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:17:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:17:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:17:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:17:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:17:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:17:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:17:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:17:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:17:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:17:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:17:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:17:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:17:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:17:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:17:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:17:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:17:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:17:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:17:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:17:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:17:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:17:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:17:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:17:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:17:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:17:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:17:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:17:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:17:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:17:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:17:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:17:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:17:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:17:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:17:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:17:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:17:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:17:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:17:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:17:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:17:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:17:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:17:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:17:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:17:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:17:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:17:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:17:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:17:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:17:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:17:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:17:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:17:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:17:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:17:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:17:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:17:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:17:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:17:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:17:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:17:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:17:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:17:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:17:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:17:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:17:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:17:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:17:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:17:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:17:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:17:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:17:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:17:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:17:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:17:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:17:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:17:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:17:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:17:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:17:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:17:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:17:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:17:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:17:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:17:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:17:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:17:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:17:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:17:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:17:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:17:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:17:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:17:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:17:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:17:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:17:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:17:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:17:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:17:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:17:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:17:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:17:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:17:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:17:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:17:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:17:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:17:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:17:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:17:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:17:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:17:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:17:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:17:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:17:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:17:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:17:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:17:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:17:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:17:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:17:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:17:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:17:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:17:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:17:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:17:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:17:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:17:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:17:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:17:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:17:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:17:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:17:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:17:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:17:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:17:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:17:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:17:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:17:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:17:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:17:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:17:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:17:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:17:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:17:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:17:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:17:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:17:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:17:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:17:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:17:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:17:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:17:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:17:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:17:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:17:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:17:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:17:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:17:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:17:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:17:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:17:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:17:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:17:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:17:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:17:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:17:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:17:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:17:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:17:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:17:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:17:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:17:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:17:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:17:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:17:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:17:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:17:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:17:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:17:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:17:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:17:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:17:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:17:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:17:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:17:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:17:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:17:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:17:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:17:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:17:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:17:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:17:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:17:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:17:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:18:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:18:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:18:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:18:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:18:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:18:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:18:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:18:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:18:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:18:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:18:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:18:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:18:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:18:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:18:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:18:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:18:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:18:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:18:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:18:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:18:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:18:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:18:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:18:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:18:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:18:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:18:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:18:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:18:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:18:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:18:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:18:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:18:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:18:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:18:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:18:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:18:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:18:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:18:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:18:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:18:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:18:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:18:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:18:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:18:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:18:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:18:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:18:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:18:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:18:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:18:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:18:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:18:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:18:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:18:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:18:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:18:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:18:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:18:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:18:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:18:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:18:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:18:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:18:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:18:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:18:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:18:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:18:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:18:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:18:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:18:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:18:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:18:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:18:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:18:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:18:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:18:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:18:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:18:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:18:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:18:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:18:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:18:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:18:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:18:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:18:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:18:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:18:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:18:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:18:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:18:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:18:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:18:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:18:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:18:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:18:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:18:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:18:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:18:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:18:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:18:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:18:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:18:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:18:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:18:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:18:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:18:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:18:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:18:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:18:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:18:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:18:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:18:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:18:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:18:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:18:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:18:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:18:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:18:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:18:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:18:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:18:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:18:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:18:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:18:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:18:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:18:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:18:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:18:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:18:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:18:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:18:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:18:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:18:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:18:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:18:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:18:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:18:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:18:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:18:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:18:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:18:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:18:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:18:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:18:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:18:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:18:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:18:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:18:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:18:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:18:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:18:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:18:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:18:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:18:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:18:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:18:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:18:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:18:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:18:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:18:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:18:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:18:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:18:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:18:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:18:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:18:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:18:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:18:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:18:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:18:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:18:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:18:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:18:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:18:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:18:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:18:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:18:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:18:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:18:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:18:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:18:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:18:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:18:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:18:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:18:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:18:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:18:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:18:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:18:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:18:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:18:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:18:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:18:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:18:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:18:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:18:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:18:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:18:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:18:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:18:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:18:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:18:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:18:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:18:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:18:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:18:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:18:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:18:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:18:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:18:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:18:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:18:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:18:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:18:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:18:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:19:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:19:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:19:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:19:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:19:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:19:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:19:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:19:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:19:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:19:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:19:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:19:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:19:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:19:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:19:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:19:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:19:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:19:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:19:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:19:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:19:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:19:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:19:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:19:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:19:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:19:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:19:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:19:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:19:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:19:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:19:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:19:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:19:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:19:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:19:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:19:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:19:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:19:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:19:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:19:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:19:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:19:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:19:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:19:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:19:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:19:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:19:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:19:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:19:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:19:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:19:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:19:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:19:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:19:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:19:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:19:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:19:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:19:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:19:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:19:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:19:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:19:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:19:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:19:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:19:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:19:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:19:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:19:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:19:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:19:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:19:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:19:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:19:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:19:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:19:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:19:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:19:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:19:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:19:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:19:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:19:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:19:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:19:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:19:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:19:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:19:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:19:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:19:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:19:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:19:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:19:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:19:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:19:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:19:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:19:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:19:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:19:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:19:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:19:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:19:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:19:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:19:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:19:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:19:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:19:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:19:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:19:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:19:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:19:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:19:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:19:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:19:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:19:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:19:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:19:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:19:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:19:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:19:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:19:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:19:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:19:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:19:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:19:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:19:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:19:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:19:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:19:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:19:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:19:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:19:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:19:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:19:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:19:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:19:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:19:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:19:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:19:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:19:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:19:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:19:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:19:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:19:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:19:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:19:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:19:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:19:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:19:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:19:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:19:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:19:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:19:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:19:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:19:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:19:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:19:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:19:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:19:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:19:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:19:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:19:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:19:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:19:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:19:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:19:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:19:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:19:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:19:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:19:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:19:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:19:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:19:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:19:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:19:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:19:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:19:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:19:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:19:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:19:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:19:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:19:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:19:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:19:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:19:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:19:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:19:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:19:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:19:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:19:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:19:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:19:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:19:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:19:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:19:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:19:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:19:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:19:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:19:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:19:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:19:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:19:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:19:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:19:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:19:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:19:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:19:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:19:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:19:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:19:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:19:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:19:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:19:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:19:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:19:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:19:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:19:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:19:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:20:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:20:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:20:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:20:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:20:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:20:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:20:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:20:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:20:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:20:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:20:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:20:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:20:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:20:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:20:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:20:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:20:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:20:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:20:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:20:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:20:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:20:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:20:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:20:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:20:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:20:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:20:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:20:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:20:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:20:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:20:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:20:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:20:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:20:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:20:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:20:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:20:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:20:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:20:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:20:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:20:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:20:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:20:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:20:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:20:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:20:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:20:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:20:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:20:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:20:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:20:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:20:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:20:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:20:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:20:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:20:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:20:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:20:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:20:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:20:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:20:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:20:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:20:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:20:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:20:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:20:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:20:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:20:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:20:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:20:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:20:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:20:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:20:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:20:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:20:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:20:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:20:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:20:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:20:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:20:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:20:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:20:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:20:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:20:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:20:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:20:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:20:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:20:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:20:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:20:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:20:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:20:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:20:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:20:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:20:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:20:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:20:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:20:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:20:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:20:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:20:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:20:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:20:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:20:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:20:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:20:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:20:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:20:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:20:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:20:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:20:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:20:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:20:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:20:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:20:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:20:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:20:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:20:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:20:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:20:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:20:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:20:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:20:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:20:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:20:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:20:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:20:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:20:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:20:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:20:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:20:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:20:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:20:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:20:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:20:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:20:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:20:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:20:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:20:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:20:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:20:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:20:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:20:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:20:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:20:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:20:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:20:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:20:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:20:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:20:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:20:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:20:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:20:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:20:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:20:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:20:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:20:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:20:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:20:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:20:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:20:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:20:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:22:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:22:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:22:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:22:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:22:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:22:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:22:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:22:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:22:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:22:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:22:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:22:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:22:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:22:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:22:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:22:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:22:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:22:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:22:50 rdma06 rsyslogd[1041]: imjournal: 5089 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 05:22:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:22:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:22:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:22:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:22:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:22:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:22:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:22:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:22:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:22:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:22:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:22:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:22:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:22:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:22:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:22:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:22:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:22:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:22:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:22:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:22:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:22:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:22:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:22:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:22:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:22:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:22:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:22:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:22:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:22:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:22:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:22:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:22:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:22:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:22:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:22:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:23:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:23:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:23:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:23:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:23:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:23:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:23:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:23:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:23:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:23:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:23:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:23:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:23:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:23:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:23:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:23:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:23:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:23:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:23:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:23:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:23:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:23:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:23:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:23:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:23:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:23:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:23:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:23:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:23:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:23:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:23:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:23:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:23:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:23:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:23:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:23:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:23:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:23:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:23:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:23:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:23:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:23:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:23:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:23:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:23:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:23:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:23:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:23:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:23:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:23:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:23:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:23:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:23:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:23:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:23:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:23:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:23:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:23:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:23:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:23:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:23:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:23:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:23:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:23:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:23:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:23:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:23:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:23:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:23:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:23:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:23:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:23:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:23:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:23:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:23:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:23:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:23:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:23:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:23:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:23:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:23:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:23:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:23:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:23:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:23:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:23:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:23:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:23:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:23:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:23:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:23:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:23:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:23:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:23:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:23:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:23:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:23:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:23:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:23:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:23:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:23:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:23:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:23:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:23:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:23:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:23:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:23:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:23:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:23:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:23:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:23:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:23:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:23:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:23:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:23:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:23:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:23:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:23:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:23:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:23:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:23:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:23:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:23:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:23:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:23:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:23:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:23:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:23:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:23:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:23:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:23:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:23:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:23:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:23:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:23:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:23:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:23:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:23:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:23:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:23:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:23:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:23:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:23:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:23:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:23:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:23:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:23:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:23:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:23:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:23:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:23:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:23:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:23:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:23:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:23:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:23:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:23:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:23:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:23:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:23:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:23:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:23:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:23:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:23:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:23:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:23:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:23:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:23:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:23:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:23:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:23:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:23:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:23:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:23:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:23:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:23:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:23:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:23:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:23:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:23:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:23:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:23:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:23:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:23:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:23:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:23:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:23:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:23:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:23:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:23:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:23:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:23:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:23:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:23:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:23:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:23:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:23:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:23:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:23:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:23:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:23:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:23:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:23:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:23:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:23:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:23:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:23:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:23:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:23:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:23:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:23:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:23:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:23:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:23:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:23:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:23:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:24:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:24:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:24:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:24:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:24:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:24:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:24:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:24:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:24:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:24:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:24:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:24:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:24:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:24:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:24:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:24:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:24:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:24:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:24:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:24:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:24:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:24:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:24:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:24:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:24:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:24:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:24:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:24:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:24:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:24:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:24:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:24:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:24:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:24:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:24:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:24:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:24:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:24:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:24:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:24:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:24:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:24:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:24:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:24:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:24:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:24:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:24:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:24:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:24:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:24:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:24:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:24:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:24:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:24:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:24:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:24:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:24:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:24:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:24:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:24:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:24:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:24:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:24:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:24:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:24:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:24:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:24:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:24:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:24:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:24:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:24:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:24:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:24:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:24:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:24:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:24:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:24:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:24:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:24:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:24:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:24:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:24:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:24:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:24:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:24:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:24:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:24:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:24:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:24:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:24:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:24:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:24:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:24:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:24:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:24:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:24:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:24:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:24:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:24:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:24:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:24:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:24:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:24:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:24:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:24:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:24:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:24:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:24:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:24:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:24:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:24:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:24:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:24:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:24:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:24:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:24:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:24:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:24:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:24:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:24:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:24:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:24:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:24:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:24:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:24:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:24:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:24:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:24:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:24:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:24:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:24:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:24:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:24:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:24:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:24:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:24:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:24:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:24:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:24:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:24:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:24:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:24:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:24:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:24:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:24:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:24:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:24:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:24:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:24:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:24:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:24:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:24:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:24:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:24:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:24:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:24:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:24:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:24:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:24:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:24:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:24:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:24:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:24:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:24:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:24:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:24:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:24:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:24:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:24:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:24:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:24:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:24:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:24:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:24:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:24:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:24:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:24:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:24:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:24:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:24:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:24:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:24:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:24:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:24:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:24:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:24:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:24:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:24:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:24:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:24:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:24:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:24:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:24:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:24:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:24:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:24:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:24:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:24:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:24:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:24:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:24:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:24:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:24:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:24:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:24:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:24:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:24:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:24:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:24:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:24:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:24:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:24:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:24:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:24:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:24:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:24:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:25:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:25:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:25:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:25:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:25:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:25:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:25:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:25:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:25:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:25:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:25:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:25:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:25:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:25:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:25:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:25:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:25:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:25:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:25:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:25:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:25:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:25:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:25:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:25:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:25:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:25:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:25:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:25:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:25:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:25:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:25:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:25:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:25:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:25:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:25:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:25:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:25:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:25:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:25:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:25:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:25:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:25:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:25:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:25:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:25:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:25:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:25:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:25:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:25:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:25:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:25:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:25:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:25:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:25:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:25:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:25:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:25:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:25:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:25:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:25:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:25:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:25:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:25:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:25:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:25:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:25:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:25:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:25:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:25:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:25:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:25:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:25:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:25:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:25:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:25:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:25:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:25:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:25:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:25:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:25:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:25:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:25:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:25:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:25:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:25:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:25:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:25:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:25:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:25:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:25:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:25:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:25:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:25:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:25:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:25:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:25:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:25:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:25:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:25:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:25:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:25:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:25:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:25:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:25:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:25:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:25:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:25:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:25:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:25:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:25:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:25:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:25:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:25:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:25:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:25:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:25:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:25:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:25:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:25:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:25:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:25:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:25:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:25:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:25:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:25:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:25:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:25:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:25:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:25:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:25:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:25:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:25:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:25:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:25:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:25:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:25:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:25:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:25:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:25:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:25:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:25:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:25:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:25:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:25:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:25:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:25:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:25:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:25:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:25:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:25:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:25:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:25:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:25:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:25:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:25:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:25:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:25:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:25:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:25:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:25:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:25:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:25:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:25:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:25:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:25:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:25:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:25:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:25:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:25:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:25:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:25:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:25:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:25:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:25:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:25:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:25:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:25:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:25:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:25:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:25:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:25:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:25:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:25:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:25:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:25:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:25:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:25:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:25:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:25:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:25:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:25:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:25:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:25:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:25:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:25:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:25:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:25:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:25:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:25:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:25:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:25:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:25:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:25:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:25:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:25:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:25:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:25:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:25:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:25:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:25:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:25:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:25:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:25:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:25:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:25:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:25:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:26:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:26:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:26:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:26:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:26:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:26:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:26:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:26:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:26:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:26:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:26:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:26:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:26:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:26:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:26:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:26:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:26:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:26:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:26:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:26:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:26:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:26:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:26:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:26:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:26:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:26:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:26:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:26:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:26:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:26:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:26:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:26:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:26:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:26:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:26:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:26:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:26:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:26:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:26:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:26:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:26:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:26:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:26:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:26:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:26:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:26:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:26:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:26:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:26:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:26:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:26:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:26:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:26:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:26:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:26:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:26:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:26:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:26:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:26:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:26:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:26:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:26:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:26:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:26:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:26:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:26:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:26:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:26:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:26:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:26:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:26:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:26:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:26:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:26:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:26:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:26:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:26:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:26:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:26:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:26:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:26:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:26:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:26:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:26:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:26:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:26:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:26:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:26:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:26:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:26:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:26:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:26:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:26:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:26:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:26:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:26:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:26:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:26:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:26:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:26:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:26:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:26:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:26:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:26:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:26:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:26:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:26:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:26:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:26:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:26:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:26:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:26:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:26:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:26:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:26:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:26:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:26:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:26:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:26:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:26:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:26:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:26:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:26:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:26:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:26:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:26:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:26:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:26:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:26:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:26:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:26:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:26:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:26:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:26:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:26:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:26:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:26:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:26:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:26:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:26:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:26:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:26:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:26:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:26:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:26:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:26:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:26:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:26:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:26:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:26:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:26:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:26:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:26:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:26:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:26:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:26:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:26:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:26:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:26:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:26:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:26:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:26:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:26:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:26:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:26:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:26:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:26:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:26:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:26:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:26:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:26:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:26:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:26:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:26:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:26:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:26:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:26:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:26:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:26:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:26:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:26:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:26:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:26:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:26:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:26:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:26:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:26:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:26:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:26:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:26:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:26:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:26:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:26:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:26:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:26:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:26:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:26:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:26:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:26:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:26:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:26:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:26:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:26:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:26:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:26:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:26:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:26:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:26:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:26:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:26:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:26:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:26:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:26:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:26:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:26:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:26:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:27:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:27:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:27:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:27:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:27:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:27:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:27:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:27:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:27:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:27:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:27:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:27:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:27:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:27:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:27:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:27:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:27:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:27:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:27:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:27:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:27:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:27:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:27:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:27:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:27:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:27:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:27:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:27:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:27:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:27:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:27:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:27:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:27:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:27:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:27:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:27:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:27:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:27:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:27:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:27:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:27:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:27:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:27:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:27:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:27:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:27:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:27:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:27:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:27:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:27:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:27:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:27:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:27:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:27:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:27:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:27:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:27:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:27:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:27:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:27:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:27:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:27:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:27:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:27:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:27:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:27:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:27:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:27:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:27:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:27:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:27:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:27:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:27:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:27:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:27:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:27:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:27:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:27:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:27:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:27:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:27:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:27:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:27:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:27:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:27:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:27:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:27:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:27:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:27:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:27:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:27:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:27:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:27:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:27:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:27:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:27:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:27:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:27:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:27:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:27:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:27:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:27:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:27:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:27:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:27:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:27:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:27:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:27:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:27:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:27:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:27:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:27:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:27:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:27:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:27:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:27:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:27:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:27:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:27:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:27:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:27:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:27:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:27:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:27:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:27:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:27:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:27:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:27:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:27:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:27:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:27:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:27:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:27:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:27:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:27:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:27:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:27:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:27:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:27:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:27:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:27:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:27:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:27:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:27:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:27:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:27:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:27:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:27:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:27:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:27:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:27:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:27:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:27:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:27:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:27:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:27:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:27:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:27:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:27:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:27:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:27:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:27:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:27:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:27:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:27:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:27:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:27:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:27:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:27:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:27:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:27:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:27:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:27:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:27:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:27:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:27:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:27:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:27:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:27:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:27:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:27:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:27:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:27:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:27:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:27:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:27:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:27:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:27:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:27:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:27:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:27:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:27:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:27:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:27:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:27:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:27:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:27:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:27:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:27:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:27:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:27:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:27:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:27:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:27:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:27:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:27:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:27:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:27:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:27:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:27:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:27:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:27:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:27:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:27:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:27:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:27:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:28:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:28:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:28:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:28:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:28:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:28:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:28:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:28:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:28:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:28:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:28:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:28:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:28:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:28:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:28:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:28:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:28:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:28:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:28:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:28:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:28:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:28:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:28:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:28:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:28:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:28:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:28:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:28:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:28:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:28:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:28:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:28:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:28:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:28:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:28:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:28:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:28:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:28:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:28:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:28:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:28:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:28:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:28:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:28:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:28:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:28:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:28:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:28:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:28:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:28:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:28:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:28:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:28:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:28:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:28:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:28:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:28:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:28:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:28:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:28:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:28:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:28:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:28:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:28:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:28:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:28:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:28:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:28:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:28:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:28:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:28:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:28:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:28:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:28:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:28:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:28:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:28:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:28:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:28:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:28:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:28:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:28:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:28:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:28:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:28:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:28:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:28:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:28:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:28:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:28:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:28:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:28:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:28:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:28:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:28:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:28:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:28:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:28:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:28:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:28:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:28:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:28:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:28:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:28:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:28:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:28:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:28:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:28:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:28:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:28:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:28:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:28:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:28:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:28:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:28:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:28:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:28:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:28:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:28:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:28:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:28:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:28:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:28:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:28:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:28:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:28:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:28:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:28:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:28:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:28:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:28:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:28:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:28:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:28:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:28:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:28:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:28:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:28:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:28:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:28:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:28:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:28:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:28:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:28:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:28:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:28:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:28:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:28:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:28:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:28:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:28:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:28:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:28:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:28:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:28:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:28:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:28:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:28:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:28:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:28:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:28:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:28:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:28:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:28:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:28:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:28:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:28:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:28:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:28:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:28:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:28:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:28:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:28:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:28:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:28:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:28:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:28:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:28:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:28:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:28:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:28:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:28:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:28:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:28:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:28:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:28:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:28:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:28:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:28:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:28:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:28:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:28:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:28:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:28:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:28:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:28:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:28:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:28:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:28:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:28:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:28:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:28:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:28:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:28:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:28:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:28:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:28:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:28:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:28:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:28:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:28:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:28:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:28:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:28:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:28:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:28:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:29:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:29:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:29:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:29:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:29:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:29:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:29:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:29:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:29:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:29:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:29:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:29:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:29:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:29:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:29:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:29:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:29:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:29:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:29:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:29:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:29:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:29:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:29:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:29:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:29:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:29:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:29:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:29:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:29:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:29:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:29:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:29:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:29:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:29:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:29:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:29:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:29:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:29:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:29:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:29:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:29:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:29:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:29:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:29:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:29:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:29:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:29:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:29:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:29:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:29:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:29:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:29:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:29:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:29:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:29:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:29:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:29:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:29:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:29:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:29:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:29:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:29:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:29:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:29:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:29:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:29:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:29:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:29:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:29:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:29:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:29:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:29:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:29:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:29:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:29:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:29:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:29:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:29:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:29:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:29:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:29:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:29:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:29:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:29:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:29:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:29:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:29:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:29:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:29:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:29:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:29:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:29:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:29:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:29:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:29:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:29:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:29:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:29:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:29:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:29:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:29:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:29:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:29:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:29:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:29:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:29:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:29:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:29:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:29:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:29:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:29:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:29:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:29:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:29:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:29:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:29:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:29:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:29:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:29:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:29:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:29:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:29:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:29:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:29:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:29:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:29:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:29:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:29:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:29:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:29:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:29:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:29:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:29:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:29:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:29:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:29:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:29:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:29:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:29:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:29:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:29:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:29:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:29:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:29:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:29:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:29:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:29:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:29:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:29:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:29:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:29:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:29:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:29:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:29:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:29:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:29:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:29:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:29:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:29:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:29:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:29:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:29:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:29:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:29:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:29:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:29:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:29:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:29:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:29:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:29:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:29:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:29:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:29:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:29:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:29:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:29:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:29:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:29:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:29:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:29:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:29:49 rdma06 systemd-journald[790]: Data hash table of /run/log/journal/95379d2f723847b68d9e976dee8967a9/system.journal has a fill level at 75.0 (163750 of 218332 items, 75497472 file size, 461 bytes per hash table item), suggesting rotation.
Apr 18 05:29:49 rdma06 systemd-journald[790]: /run/log/journal/95379d2f723847b68d9e976dee8967a9/system.journal: Journal header limits reached or header out-of-date, rotating.
Apr 18 05:29:49 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 18 05:29:50 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 18 05:29:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:29:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:29:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:29:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:29:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:29:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:29:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:29:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:29:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:29:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:29:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:29:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:29:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:29:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:29:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:29:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:29:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:29:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:29:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:29:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:29:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:29:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:29:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:29:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:29:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:29:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:29:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:29:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:29:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:29:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:29:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:29:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:29:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:29:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:29:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:29:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:30:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:30:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:30:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:30:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:30:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:30:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:30:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:30:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:30:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:30:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:30:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:30:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:30:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:30:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:30:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:30:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:30:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:30:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:30:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:30:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:30:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:30:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:30:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:30:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:30:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:30:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:30:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:30:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:30:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:30:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:30:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:30:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:30:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:30:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:30:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:30:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:30:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:30:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:30:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:30:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:30:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:30:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:30:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:30:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:30:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:30:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:30:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:30:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:30:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:30:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:30:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:30:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:30:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:30:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:30:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:30:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:30:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:30:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:30:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:30:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:30:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:30:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:30:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:30:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:30:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:30:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:30:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:30:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:30:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:30:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:30:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:30:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:30:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:30:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:30:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:30:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:30:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:30:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:30:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:30:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:30:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:30:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:30:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:30:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:30:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:30:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:30:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:30:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:30:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:30:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:30:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:30:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:30:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:30:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:30:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:30:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:30:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:30:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:30:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:30:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:30:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:30:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:30:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:30:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:30:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:30:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:30:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:30:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:30:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:30:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:30:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:30:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:30:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:30:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:30:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:30:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:30:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:30:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:30:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:30:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:30:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:30:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:30:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:30:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:30:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:30:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:30:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:30:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:30:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:30:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:30:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:30:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:30:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:30:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:30:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:30:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:30:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:30:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:30:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:30:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:30:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:30:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:30:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:30:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:30:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:30:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:30:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:30:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:30:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:30:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:30:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:30:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:30:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:30:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:30:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:30:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:30:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:30:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:30:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:30:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:30:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:30:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:32:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:32:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:32:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:32:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:32:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:32:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:32:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:32:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:32:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:32:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:32:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:32:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:32:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:32:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:32:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:32:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:32:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:32:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:32:55 rdma06 rsyslogd[1041]: imjournal: 5295 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 05:32:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:32:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:32:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:32:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:32:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:32:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:32:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:32:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:32:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:32:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:32:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:32:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:32:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:32:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:32:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:32:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:32:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:32:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:33:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:33:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:33:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:33:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:33:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:33:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:33:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:33:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:33:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:33:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:33:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:33:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:33:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:33:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:33:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:33:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:33:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:33:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:33:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:33:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:33:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:33:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:33:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:33:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:33:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:33:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:33:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:33:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:33:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:33:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:33:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:33:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:33:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:33:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:33:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:33:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:33:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:33:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:33:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:33:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:33:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:33:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:33:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:33:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:33:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:33:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:33:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:33:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:33:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:33:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:33:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:33:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:33:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:33:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:33:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:33:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:33:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:33:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:33:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:33:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:33:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:33:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:33:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:33:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:33:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:33:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:33:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:33:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:33:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:33:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:33:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:33:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:33:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:33:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:33:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:33:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:33:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:33:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:33:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:33:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:33:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:33:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:33:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:33:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:33:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:33:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:33:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:33:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:33:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:33:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:33:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:33:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:33:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:33:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:33:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:33:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:33:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:33:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:33:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:33:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:33:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:33:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:33:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:33:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:33:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:33:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:33:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:33:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:33:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:33:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:33:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:33:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:33:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:33:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:33:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:33:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:33:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:33:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:33:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:33:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:33:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:33:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:33:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:33:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:33:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:33:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:33:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:33:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:33:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:33:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:33:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:33:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:33:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:33:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:33:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:33:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:33:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:33:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:33:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:33:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:33:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:33:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:33:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:33:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:33:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:33:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:33:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:33:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:33:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:33:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:33:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:33:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:33:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:33:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:33:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:33:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:33:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:33:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:33:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:33:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:33:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:33:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:33:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:33:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:33:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:33:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:33:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:33:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:33:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:33:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:33:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:33:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:33:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:33:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:33:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:33:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:33:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:33:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:33:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:33:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:33:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:33:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:33:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:33:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:33:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:33:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:33:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:33:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:33:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:33:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:33:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:33:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:33:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:33:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:33:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:33:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:33:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:33:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:33:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:33:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:33:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:33:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:33:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:33:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:33:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:33:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:33:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:33:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:33:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:33:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:33:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:33:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:33:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:33:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:33:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:33:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:34:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:34:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:34:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:34:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:34:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:34:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:34:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:34:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:34:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:34:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:34:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:34:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:34:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:34:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:34:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:34:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:34:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:34:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:34:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:34:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:34:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:34:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:34:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:34:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:34:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:34:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:34:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:34:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:34:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:34:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:34:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:34:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:34:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:34:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:34:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:34:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:34:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:34:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:34:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:34:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:34:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:34:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:34:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:34:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:34:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:34:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:34:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:34:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:34:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:34:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:34:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:34:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:34:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:34:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:34:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:34:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:34:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:34:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:34:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:34:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:34:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:34:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:34:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:34:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:34:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:34:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:34:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:34:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:34:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:34:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:34:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:34:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:34:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:34:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:34:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:34:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:34:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:34:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:34:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:34:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:34:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:34:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:34:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:34:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:34:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:34:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:34:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:34:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:34:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:34:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:34:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:34:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:34:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:34:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:34:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:34:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:34:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:34:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:34:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:34:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:34:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:34:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:34:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:34:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:34:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:34:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:34:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:34:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:34:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:34:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:34:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:34:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:34:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:34:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:34:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:34:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:34:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:34:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:34:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:34:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:34:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:34:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:34:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:34:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:34:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:34:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:34:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:34:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:34:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:34:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:34:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:34:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:34:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:34:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:34:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:34:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:34:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:34:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:34:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:34:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:34:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:34:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:34:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:34:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:34:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:34:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:34:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:34:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:34:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:34:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:34:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:34:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:34:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:34:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:34:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:34:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:34:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:34:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:34:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:34:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:34:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:34:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:34:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:34:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:34:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:34:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:34:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:34:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:34:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:34:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:34:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:34:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:34:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:34:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:34:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:34:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:34:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:34:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:34:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:34:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:34:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:34:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:34:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:34:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:34:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:34:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:34:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:34:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:34:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:34:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:34:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:34:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:34:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:34:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:34:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:34:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:34:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:34:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:35:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:35:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:35:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:35:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:35:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:35:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:35:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:35:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:35:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:35:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:35:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:35:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:35:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:35:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:35:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:35:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:35:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:35:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:35:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:35:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:35:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:35:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:35:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:35:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:35:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:35:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:35:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:35:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:35:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:35:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:35:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:35:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:35:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:35:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:35:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:35:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:35:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:35:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:35:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:35:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:35:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:35:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:35:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:35:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:35:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:35:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:35:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:35:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:35:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:35:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:35:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:35:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:35:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:35:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:35:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:35:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:35:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:35:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:35:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:35:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:35:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:35:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:35:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:35:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:35:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:35:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:35:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:35:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:35:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:35:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:35:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:35:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:35:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:35:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:35:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:35:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:35:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:35:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:35:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:35:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:35:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:35:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:35:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:35:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:35:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:35:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:35:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:35:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:35:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:35:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:35:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:35:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:35:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:35:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:35:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:35:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:35:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:35:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:35:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:35:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:35:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:35:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:35:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:35:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:35:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:35:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:35:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:35:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:35:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:35:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:35:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:35:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:35:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:35:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:35:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:35:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:35:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:35:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:35:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:35:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:35:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:35:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:35:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:35:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:35:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:35:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:35:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:35:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:35:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:35:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:35:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:35:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:35:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:35:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:35:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:35:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:35:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:35:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:35:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:35:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:35:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:35:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:35:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:35:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:35:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:35:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:35:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:35:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:35:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:35:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:35:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:35:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:35:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:35:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:35:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:35:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:35:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:35:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:35:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:35:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:35:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:35:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:35:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:35:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:35:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:35:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:35:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:35:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:35:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:35:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:35:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:35:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:35:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:35:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:35:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:35:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:35:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:35:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:35:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:35:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:35:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:35:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:35:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:35:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:35:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:35:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:35:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:35:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:35:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:35:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:35:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:35:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:35:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:35:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:35:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:35:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:35:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:35:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:35:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:35:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:35:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:35:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:35:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:35:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:35:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:35:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:35:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:35:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:35:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:35:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:35:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:35:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:35:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:35:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:35:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:35:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:36:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:36:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:36:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:36:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:36:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:36:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:36:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:36:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:36:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:36:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:36:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:36:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:36:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:36:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:36:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:36:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:36:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:36:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:36:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:36:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:36:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:36:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:36:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:36:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:36:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:36:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:36:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:36:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:36:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:36:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:36:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:36:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:36:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:36:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:36:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:36:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:36:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:36:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:36:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:36:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:36:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:36:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:36:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:36:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:36:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:36:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:36:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:36:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:36:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:36:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:36:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:36:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:36:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:36:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:36:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:36:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:36:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:36:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:36:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:36:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:36:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:36:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:36:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:36:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:36:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:36:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:36:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:36:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:36:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:36:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:36:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:36:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:36:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:36:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:36:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:36:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:36:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:36:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:36:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:36:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:36:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:36:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:36:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:36:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:36:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:36:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:36:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:36:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:36:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:36:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:36:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:36:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:36:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:36:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:36:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:36:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:36:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:36:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:36:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:36:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:36:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:36:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:36:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:36:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:36:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:36:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:36:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:36:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:36:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:36:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:36:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:36:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:36:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:36:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:36:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:36:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:36:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:36:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:36:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:36:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:36:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:36:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:36:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:36:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:36:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:36:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:36:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:36:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:36:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:36:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:36:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:36:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:36:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:36:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:36:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:36:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:36:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:36:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:36:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:36:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:36:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:36:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:36:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:36:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:36:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:36:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:36:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:36:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:36:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:36:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:36:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:36:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:36:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:36:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:36:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:36:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:36:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:36:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:36:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:36:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:36:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:36:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:36:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:36:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:36:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:36:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:36:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:36:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:36:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:36:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:36:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:36:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:36:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:36:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:36:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:36:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:36:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:36:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:36:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:36:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:36:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:36:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:36:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:36:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:36:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:36:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:36:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:36:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:36:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:36:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:36:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:36:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:36:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:36:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:36:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:36:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:36:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:36:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:36:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:36:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:36:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:36:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:36:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:36:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:36:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:36:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:36:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:36:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:36:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:36:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:36:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:36:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:36:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:36:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:36:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:36:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:37:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:37:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:37:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:37:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:37:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:37:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:37:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:37:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:37:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:37:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:37:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:37:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:37:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:37:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:37:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:37:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:37:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:37:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:37:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:37:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:37:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:37:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:37:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:37:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:37:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:37:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:37:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:37:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:37:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:37:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:37:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:37:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:37:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:37:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:37:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:37:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:37:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:37:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:37:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:37:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:37:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:37:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:37:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:37:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:37:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:37:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:37:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:37:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:37:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:37:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:37:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:37:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:37:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:37:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:37:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:37:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:37:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:37:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:37:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:37:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:37:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:37:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:37:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:37:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:37:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:37:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:37:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:37:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:37:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:37:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:37:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:37:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:37:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:37:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:37:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:37:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:37:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:37:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:37:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:37:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:37:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:37:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:37:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:37:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:37:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:37:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:37:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:37:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:37:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:37:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:37:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:37:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:37:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:37:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:37:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:37:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:37:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:37:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:37:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:37:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:37:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:37:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:37:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:37:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:37:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:37:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:37:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:37:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:37:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:37:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:37:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:37:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:37:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:37:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:37:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:37:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:37:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:37:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:37:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:37:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:37:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:37:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:37:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:37:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:37:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:37:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:37:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:37:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:37:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:37:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:37:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:37:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:37:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:37:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:37:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:37:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:37:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:37:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:37:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:37:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:37:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:37:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:37:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:37:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:37:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:37:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:37:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:37:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:37:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:37:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:37:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:37:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:37:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:37:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:37:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:37:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:37:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:37:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:37:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:37:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:37:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:37:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:37:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:37:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:37:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:37:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:37:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:37:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:37:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:37:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:37:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:37:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:37:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:37:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:37:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:37:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:37:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:37:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:37:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:37:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:37:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:37:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:37:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:37:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:37:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:37:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:37:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:37:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:37:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:37:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:37:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:37:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:37:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:37:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:37:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:37:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:37:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:37:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:37:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:37:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:37:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:37:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:37:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:37:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:37:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:37:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:37:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:37:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:37:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:37:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:37:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:37:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:37:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:37:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:37:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:37:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:38:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:38:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:38:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:38:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:38:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:38:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:38:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:38:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:38:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:38:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:38:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:38:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:38:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:38:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:38:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:38:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:38:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:38:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:38:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:38:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:38:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:38:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:38:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:38:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:38:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:38:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:38:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:38:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:38:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:38:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:38:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:38:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:38:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:38:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:38:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:38:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:38:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:38:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:38:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:38:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:38:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:38:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:38:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:38:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:38:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:38:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:38:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:38:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:38:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:38:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:38:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:38:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:38:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:38:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:38:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:38:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:38:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:38:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:38:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:38:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:38:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:38:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:38:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:38:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:38:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:38:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:38:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:38:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:38:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:38:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:38:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:38:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:38:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:38:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:38:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:38:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:38:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:38:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:38:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:38:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:38:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:38:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:38:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:38:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:38:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:38:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:38:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:38:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:38:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:38:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:38:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:38:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:38:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:38:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:38:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:38:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:38:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:38:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:38:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:38:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:38:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:38:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:38:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:38:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:38:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:38:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:38:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:38:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:38:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:38:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:38:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:38:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:38:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:38:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:38:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:38:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:38:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:38:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:38:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:38:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:38:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:38:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:38:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:38:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:38:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:38:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:38:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:38:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:38:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:38:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:38:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:38:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:38:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:38:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:38:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:38:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:38:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:38:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:38:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:38:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:38:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:38:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:38:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:38:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:38:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:38:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:38:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:38:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:38:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:38:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:38:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:38:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:38:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:38:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:38:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:38:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:38:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:38:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:38:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:38:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:38:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:38:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:38:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:38:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:38:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:38:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:38:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:38:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:38:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:38:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:38:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:38:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:38:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:38:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:38:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:38:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:38:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:38:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:38:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:38:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:38:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:38:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:38:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:38:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:38:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:38:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:38:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:38:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:38:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:38:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:38:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:38:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:38:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:38:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:38:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:38:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:38:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:38:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:38:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:38:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:38:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:38:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:38:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:38:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:38:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:38:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:38:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:38:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:38:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:38:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:38:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:38:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:38:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:38:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:38:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:38:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:39:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:39:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:39:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:39:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:39:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:39:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:39:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:39:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:39:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:39:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:39:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:39:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:39:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:39:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:39:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:39:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:39:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:39:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:39:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:39:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:39:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:39:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:39:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:39:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:39:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:39:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:39:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:39:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:39:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:39:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:39:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:39:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:39:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:39:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:39:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:39:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:39:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:39:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:39:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:39:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:39:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:39:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:39:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:39:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:39:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:39:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:39:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:39:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:39:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:39:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:39:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:39:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:39:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:39:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:39:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:39:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:39:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:39:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:39:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:39:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:39:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:39:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:39:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:39:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:39:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:39:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:39:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:39:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:39:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:39:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:39:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:39:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:39:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:39:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:39:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:39:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:39:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:39:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:39:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:39:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:39:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:39:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:39:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:39:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:39:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:39:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:39:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:39:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:39:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:39:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:39:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:39:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:39:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:39:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:39:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:39:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:39:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:39:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:39:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:39:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:39:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:39:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:39:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:39:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:39:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:39:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:39:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:39:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:39:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:39:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:39:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:39:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:39:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:39:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:39:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:39:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:39:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:39:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:39:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:39:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:39:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:39:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:39:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:39:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:39:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:39:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:39:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:39:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:39:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:39:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:39:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:39:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:39:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:39:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:39:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:39:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:39:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:39:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:39:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:39:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:39:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:39:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:39:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:39:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:39:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:39:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:39:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:39:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:39:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:39:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:39:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:39:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:39:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:39:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:39:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:39:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:39:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:39:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:39:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:39:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:39:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:39:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:39:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:39:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:39:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:39:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:39:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:39:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:39:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:39:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:39:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:39:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:39:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:39:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:39:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:39:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:39:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:39:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:39:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:39:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:39:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:39:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:39:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:39:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:39:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:39:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:39:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:39:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:39:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:39:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:39:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:39:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:39:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:39:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:39:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:39:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:39:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:39:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:39:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:39:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:39:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:39:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:39:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:39:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:39:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:39:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:39:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:39:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:39:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:39:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:39:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:39:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:39:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:39:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:39:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:39:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:40:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:40:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:40:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:40:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:40:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:40:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:40:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:40:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:40:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:40:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:40:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:40:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:40:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:40:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:40:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:40:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:40:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:40:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:40:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:40:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:40:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:40:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:40:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:40:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:40:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:40:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:40:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:40:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:40:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:40:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:40:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:40:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:40:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:40:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:40:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:40:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:40:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:40:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:40:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:40:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:40:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:40:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:40:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:40:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:40:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:40:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:40:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:40:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:40:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:40:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:40:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:40:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:40:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:40:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:40:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:40:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:40:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:40:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:40:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:40:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:40:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:40:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:40:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:40:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:40:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:40:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:40:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:40:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:40:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:40:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:40:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:40:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:40:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:40:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:40:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:40:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:40:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:40:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:40:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:40:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:40:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:40:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:40:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:40:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:40:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:40:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:40:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:40:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:40:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:40:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:40:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:40:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:40:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:40:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:40:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:40:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:40:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:40:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:40:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:40:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:40:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:40:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:40:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:40:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:40:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:40:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:40:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:40:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:40:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:40:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:40:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:40:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:40:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:40:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:40:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:40:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:40:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:40:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:40:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:40:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:40:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:40:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:40:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:40:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:40:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:40:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:40:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:40:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:40:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:40:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:40:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:40:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:40:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:40:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:40:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:40:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:40:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:40:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:40:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:40:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:40:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:40:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:40:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:40:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:40:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:40:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:40:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:40:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:40:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:40:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:40:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:40:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:40:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:40:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:40:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:40:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:40:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:40:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:40:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:40:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:40:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:40:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:40:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:40:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:40:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:40:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:40:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:40:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:40:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:40:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:40:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:40:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:40:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:40:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:40:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:40:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:40:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:40:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:40:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:40:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:40:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:40:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:40:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:40:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:40:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:40:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:40:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:40:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:40:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:40:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:40:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:40:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:40:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:40:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:40:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:40:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:40:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:40:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:43:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:43:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:43:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:43:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:43:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:43:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:43:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:43:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:43:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:43:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:43:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:43:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:43:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:43:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:43:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:43:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:43:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:43:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:43:00 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 05:43:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:43:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:43:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:43:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:43:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:43:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:43:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:43:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:43:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:43:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:43:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:43:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:43:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:43:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:43:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:43:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:43:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:43:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:43:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:43:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:43:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:43:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:43:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:43:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:43:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:43:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:43:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:43:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:43:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:43:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:43:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:43:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:43:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:43:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:43:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:43:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:43:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:43:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:43:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:43:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:43:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:43:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:43:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:43:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:43:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:43:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:43:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:43:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:43:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:43:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:43:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:43:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:43:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:43:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:43:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:43:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:43:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:43:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:43:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:43:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:43:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:43:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:43:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:43:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:43:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:43:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:43:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:43:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:43:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:43:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:43:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:43:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:43:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:43:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:43:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:43:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:43:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:43:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:43:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:43:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:43:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:43:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:43:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:43:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:43:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:43:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:43:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:43:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:43:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:43:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:43:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:43:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:43:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:43:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:43:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:43:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:43:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:43:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:43:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:43:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:43:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:43:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:43:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:43:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:43:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:43:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:43:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:43:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:43:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:43:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:43:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:43:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:43:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:43:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:43:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:43:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:43:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:43:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:43:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:43:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:43:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:43:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:43:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:43:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:43:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:43:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:43:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:43:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:43:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:43:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:43:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:43:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:43:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:43:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:43:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:43:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:43:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:43:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:43:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:43:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:43:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:43:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:43:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:43:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:43:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:43:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:43:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:43:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:43:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:43:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:43:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:43:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:43:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:43:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:43:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:43:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:43:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:43:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:43:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:43:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:43:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:43:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:43:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:43:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:43:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:43:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:43:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:43:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:43:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:43:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:43:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:43:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:43:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:43:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:43:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:43:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:43:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:43:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:43:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:43:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:43:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:43:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:43:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:43:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:43:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:43:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:43:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:43:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:43:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:43:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:43:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:43:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:43:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:43:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:43:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:43:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:43:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:43:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:44:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:44:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:44:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:44:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:44:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:44:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:44:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:44:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:44:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:44:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:44:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:44:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:44:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:44:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:44:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:44:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:44:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:44:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:44:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:44:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:44:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:44:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:44:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:44:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:44:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:44:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:44:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:44:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:44:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:44:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:44:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:44:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:44:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:44:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:44:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:44:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:44:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:44:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:44:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:44:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:44:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:44:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:44:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:44:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:44:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:44:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:44:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:44:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:44:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:44:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:44:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:44:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:44:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:44:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:44:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:44:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:44:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:44:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:44:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:44:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:44:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:44:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:44:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:44:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:44:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:44:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:44:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:44:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:44:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:44:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:44:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:44:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:44:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:44:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:44:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:44:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:44:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:44:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:44:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:44:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:44:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:44:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:44:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:44:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:44:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:44:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:44:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:44:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:44:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:44:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:44:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:44:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:44:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:44:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:44:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:44:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:44:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:44:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:44:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:44:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:44:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:44:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:44:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:44:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:44:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:44:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:44:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:44:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:44:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:44:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:44:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:44:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:44:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:44:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:44:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:44:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:44:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:44:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:44:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:44:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:44:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:44:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:44:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:44:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:44:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:44:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:44:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:44:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:44:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:44:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:44:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:44:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:44:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:44:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:44:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:44:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:44:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:44:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:44:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:44:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:44:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:44:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:44:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:44:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:44:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:44:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:44:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:44:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:44:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:44:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:44:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:44:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:44:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:44:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:44:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:44:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:44:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:44:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:44:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:44:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:44:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:44:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:44:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:44:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:44:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:44:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:44:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:44:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:44:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:44:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:44:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:44:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:44:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:44:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:44:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:44:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:44:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:44:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:44:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:44:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:44:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:44:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:44:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:44:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:44:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:44:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:44:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:44:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:44:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:44:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:44:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:44:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:44:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:44:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:44:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:44:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:44:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:44:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:44:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:44:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:44:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:44:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:44:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:44:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:44:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:44:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:44:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:44:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:44:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:44:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:44:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:44:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:44:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:44:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:44:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:44:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:45:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:45:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:45:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:45:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:45:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:45:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:45:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:45:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:45:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:45:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:45:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:45:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:45:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:45:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:45:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:45:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:45:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:45:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:45:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:45:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:45:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:45:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:45:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:45:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:45:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:45:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:45:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:45:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:45:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:45:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:45:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:45:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:45:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:45:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:45:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:45:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:45:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:45:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:45:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:45:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:45:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:45:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:45:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:45:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:45:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:45:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:45:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:45:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:45:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:45:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:45:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:45:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:45:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:45:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:45:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:45:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:45:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:45:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:45:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:45:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:45:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:45:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:45:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:45:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:45:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:45:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:45:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:45:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:45:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:45:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:45:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:45:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:45:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:45:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:45:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:45:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:45:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:45:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:45:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:45:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:45:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:45:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:45:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:45:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:45:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:45:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:45:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:45:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:45:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:45:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:45:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:45:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:45:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:45:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:45:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:45:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:45:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:45:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:45:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:45:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:45:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:45:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:45:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:45:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:45:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:45:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:45:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:45:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:45:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:45:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:45:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:45:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:45:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:45:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:45:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:45:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:45:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:45:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:45:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:45:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:45:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:45:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:45:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:45:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:45:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:45:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:45:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:45:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:45:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:45:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:45:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:45:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:45:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:45:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:45:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:45:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:45:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:45:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:45:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:45:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:45:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:45:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:45:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:45:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:45:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:45:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:45:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:45:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:45:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:45:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:45:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:45:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:45:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:45:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:45:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:45:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:45:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:45:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:45:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:45:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:45:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:45:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:45:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:45:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:45:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:45:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:45:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:45:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:45:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:45:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:45:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:45:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:45:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:45:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:45:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:45:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:45:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:45:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:45:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:45:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:45:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:45:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:45:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:45:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:45:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:45:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:45:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:45:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:45:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:45:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:45:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:45:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:45:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:45:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:45:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:45:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:45:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:45:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:45:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:45:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:45:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:45:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:45:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:45:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:45:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:45:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:45:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:45:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:45:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:45:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:45:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:45:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:45:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:45:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:45:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:45:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:46:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:46:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:46:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:46:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:46:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:46:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:46:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:46:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:46:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:46:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:46:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:46:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:46:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:46:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:46:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:46:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:46:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:46:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:46:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:46:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:46:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:46:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:46:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:46:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:46:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:46:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:46:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:46:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:46:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:46:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:46:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:46:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:46:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:46:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:46:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:46:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:46:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:46:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:46:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:46:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:46:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:46:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:46:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:46:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:46:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:46:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:46:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:46:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:46:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:46:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:46:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:46:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:46:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:46:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:46:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:46:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:46:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:46:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:46:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:46:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:46:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:46:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:46:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:46:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:46:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:46:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:46:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:46:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:46:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:46:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:46:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:46:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:46:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:46:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:46:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:46:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:46:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:46:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:46:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:46:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:46:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:46:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:46:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:46:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:46:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:46:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:46:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:46:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:46:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:46:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:46:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:46:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:46:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:46:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:46:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:46:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:46:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:46:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:46:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:46:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:46:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:46:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:46:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:46:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:46:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:46:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:46:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:46:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:46:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:46:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:46:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:46:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:46:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:46:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:46:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:46:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:46:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:46:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:46:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:46:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:46:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:46:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:46:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:46:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:46:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:46:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:46:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:46:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:46:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:46:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:46:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:46:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:46:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:46:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:46:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:46:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:46:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:46:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:46:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:46:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:46:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:46:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:46:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:46:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:46:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:46:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:46:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:46:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:46:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:46:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:46:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:46:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:46:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:46:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:46:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:46:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:46:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:46:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:46:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:46:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:46:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:46:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:46:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:46:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:46:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:46:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:46:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:46:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:46:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:46:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:46:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:46:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:46:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:46:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:46:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:46:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:46:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:46:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:46:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:46:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:46:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:46:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:46:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:46:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:46:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:46:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:46:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:46:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:46:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:46:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:46:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:46:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:46:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:46:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:46:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:46:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:46:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:46:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:46:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:46:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:46:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:46:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:46:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:46:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:46:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:46:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:46:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:46:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:46:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:46:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:46:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:46:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:46:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:46:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:46:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:46:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:47:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:47:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:47:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:47:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:47:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:47:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:47:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:47:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:47:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:47:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:47:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:47:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:47:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:47:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:47:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:47:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:47:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:47:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:47:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:47:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:47:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:47:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:47:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:47:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:47:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:47:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:47:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:47:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:47:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:47:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:47:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:47:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:47:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:47:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:47:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:47:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:47:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:47:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:47:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:47:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:47:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:47:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:47:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:47:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:47:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:47:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:47:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:47:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:47:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:47:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:47:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:47:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:47:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:47:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:47:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:47:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:47:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:47:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:47:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:47:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:47:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:47:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:47:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:47:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:47:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:47:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:47:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:47:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:47:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:47:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:47:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:47:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:47:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:47:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:47:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:47:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:47:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:47:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:47:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:47:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:47:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:47:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:47:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:47:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:47:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:47:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:47:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:47:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:47:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:47:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:47:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:47:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:47:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:47:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:47:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:47:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:47:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:47:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:47:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:47:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:47:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:47:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:47:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:47:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:47:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:47:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:47:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:47:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:47:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:47:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:47:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:47:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:47:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:47:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:47:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:47:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:47:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:47:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:47:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:47:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:47:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:47:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:47:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:47:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:47:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:47:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:47:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:47:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:47:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:47:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:47:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:47:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:47:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:47:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:47:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:47:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:47:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:47:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:47:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:47:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:47:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:47:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:47:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:47:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:47:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:47:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:47:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:47:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:47:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:47:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:47:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:47:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:47:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:47:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:47:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:47:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:47:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:47:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:47:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:47:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:47:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:47:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:47:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:47:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:47:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:47:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:47:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:47:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:47:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:47:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:47:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:47:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:47:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:47:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:47:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:47:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:47:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:47:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:47:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:47:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:47:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:47:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:47:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:47:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:47:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:47:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:47:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:47:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:47:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:47:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:47:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:47:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:47:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:47:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:47:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:47:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:47:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:47:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:47:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:47:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:47:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:47:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:47:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:47:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:47:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:47:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:47:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:47:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:47:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:47:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:47:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:47:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:47:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:47:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:47:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:47:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:48:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:48:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:48:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:48:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:48:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:48:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:48:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:48:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:48:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:48:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:48:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:48:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:48:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:48:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:48:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:48:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:48:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:48:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:48:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:48:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:48:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:48:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:48:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:48:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:48:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:48:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:48:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:48:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:48:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:48:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:48:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:48:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:48:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:48:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:48:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:48:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:48:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:48:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:48:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:48:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:48:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:48:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:48:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:48:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:48:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:48:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:48:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:48:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:48:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:48:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:48:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:48:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:48:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:48:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:48:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:48:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:48:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:48:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:48:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:48:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:48:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:48:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:48:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:48:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:48:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:48:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:48:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:48:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:48:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:48:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:48:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:48:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:48:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:48:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:48:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:48:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:48:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:48:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:48:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:48:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:48:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:48:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:48:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:48:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:48:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:48:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:48:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:48:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:48:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:48:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:48:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:48:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:48:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:48:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:48:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:48:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:48:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:48:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:48:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:48:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:48:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:48:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:48:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:48:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:48:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:48:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:48:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:48:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:48:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:48:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:48:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:48:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:48:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:48:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:48:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:48:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:48:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:48:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:48:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:48:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:48:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:48:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:48:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:48:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:48:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:48:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:48:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:48:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:48:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:48:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:48:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:48:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:48:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:48:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:48:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:48:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:48:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:48:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:48:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:48:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:48:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:48:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:48:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:48:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:48:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:48:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:48:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:48:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:48:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:48:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:48:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:48:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:48:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:48:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:48:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:48:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:48:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:48:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:48:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:48:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:48:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:48:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:48:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:48:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:48:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:48:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:48:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:48:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:48:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:48:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:48:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:48:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:48:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:48:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:48:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:48:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:48:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:48:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:48:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:48:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:48:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:48:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:48:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:48:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:48:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:48:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:48:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:48:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:48:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:48:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:48:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:48:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:48:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:48:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:48:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:48:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:48:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:48:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:48:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:48:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:48:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:48:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:48:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:48:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:48:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:48:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:48:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:48:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:48:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:48:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:48:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:48:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:48:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:48:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:48:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:48:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:49:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:49:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:49:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:49:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:49:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:49:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:49:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:49:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:49:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:49:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:49:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:49:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:49:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:49:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:49:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:49:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:49:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:49:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:49:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:49:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:49:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:49:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:49:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:49:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:49:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:49:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:49:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:49:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:49:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:49:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:49:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:49:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:49:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:49:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:49:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:49:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:49:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:49:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:49:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:49:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:49:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:49:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:49:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:49:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:49:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:49:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:49:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:49:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:49:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:49:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:49:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:49:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:49:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:49:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:49:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:49:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:49:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:49:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:49:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:49:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:49:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:49:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:49:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:49:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:49:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:49:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:49:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:49:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:49:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:49:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:49:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:49:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:49:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:49:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:49:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:49:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:49:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:49:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:49:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:49:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:49:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:49:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:49:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:49:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:49:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:49:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:49:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:49:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:49:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:49:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:49:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:49:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:49:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:49:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:49:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:49:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:49:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:49:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:49:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:49:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:49:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:49:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:49:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:49:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:49:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:49:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:49:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:49:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:49:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:49:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:49:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:49:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:49:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:49:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:49:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:49:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:49:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:49:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:49:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:49:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:49:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:49:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:49:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:49:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:49:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:49:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:49:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:49:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:49:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:49:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:49:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:49:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:49:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:49:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:49:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:49:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:49:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:49:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:49:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:49:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:49:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:49:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:49:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:49:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:49:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:49:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:49:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:49:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:49:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:49:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:49:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:49:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:49:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:49:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:49:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:49:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:49:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:49:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:49:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:49:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:49:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:49:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:49:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:49:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:49:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:49:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:49:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:49:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:49:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:49:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:49:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:49:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:49:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:49:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:49:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:49:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:49:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:49:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:49:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:49:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:49:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:49:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:49:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:49:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:49:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:49:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:49:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:49:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:49:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:49:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:49:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:49:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:49:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:49:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:49:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:49:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:49:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:49:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:49:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:49:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:49:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:49:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:49:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:49:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:49:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:49:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:49:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:49:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:49:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:49:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:49:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:49:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:49:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:49:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:49:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:49:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:50:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:50:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:50:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:50:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:50:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:50:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:50:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:50:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:50:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:50:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:50:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:50:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:50:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:50:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:50:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:50:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:50:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:50:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:50:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:50:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:50:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:50:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:50:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:50:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:50:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:50:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:50:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:50:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:50:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:50:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:50:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:50:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:50:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:50:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:50:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:50:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:50:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:50:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:50:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:50:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:50:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:50:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:50:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:50:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:50:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:50:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:50:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:50:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:50:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:50:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:50:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:50:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:50:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:50:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:50:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:50:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:50:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:50:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:50:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:50:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:50:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:50:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:50:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:50:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:50:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:50:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:50:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:50:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:50:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:50:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:50:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:50:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:50:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:50:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:50:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:50:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:50:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:50:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:50:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:50:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:50:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:50:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:50:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:50:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:50:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:50:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:50:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:50:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:50:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:50:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:50:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:50:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:50:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:50:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:50:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:50:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:50:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:50:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:50:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:50:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:50:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:50:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:50:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:50:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:50:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:50:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:50:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:50:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:50:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:50:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:50:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:50:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:50:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:50:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:50:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:50:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:50:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:50:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:50:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:50:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:50:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:50:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:50:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:50:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:50:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:50:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:50:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:50:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:50:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:50:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:50:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:50:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:50:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:50:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:50:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:50:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:50:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:50:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:50:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:50:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:50:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:50:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:50:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:50:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:50:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:50:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:50:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:50:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:50:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:50:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:50:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:50:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:50:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:50:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:50:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:50:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:50:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:50:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:50:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:50:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:50:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:50:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:50:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:50:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:50:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:50:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:50:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:50:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:50:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:50:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:50:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:50:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:50:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:50:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:50:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:50:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:50:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:50:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:50:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:50:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:50:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:50:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:50:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:50:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:50:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:50:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:50:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:50:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:50:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:50:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:50:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:50:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:50:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:50:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:50:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:50:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:50:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:50:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:50:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:50:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:50:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:50:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:50:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:50:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:50:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:50:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:50:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:50:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:50:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:50:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:50:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:50:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:50:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:50:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:50:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:50:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:53:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:53:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:53:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:53:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:53:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:53:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:53:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:53:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:53:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:53:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:53:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:53:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:53:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:53:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:53:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:53:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:53:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:53:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:53:01 rdma06 rsyslogd[1041]: imjournal: 5082 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 05:53:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:53:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:53:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:53:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:53:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:53:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:53:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:53:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:53:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:53:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:53:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:53:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:53:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:53:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:53:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:53:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:53:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:53:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:53:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:53:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:53:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:53:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:53:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:53:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:53:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:53:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:53:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:53:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:53:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:53:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:53:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:53:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:53:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:53:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:53:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:53:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:53:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:53:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:53:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:53:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:53:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:53:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:53:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:53:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:53:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:53:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:53:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:53:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:53:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:53:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:53:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:53:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:53:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:53:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:53:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:53:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:53:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:53:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:53:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:53:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:53:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:53:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:53:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:53:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:53:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:53:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:53:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:53:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:53:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:53:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:53:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:53:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:53:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:53:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:53:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:53:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:53:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:53:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:53:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:53:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:53:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:53:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:53:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:53:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:53:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:53:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:53:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:53:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:53:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:53:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:53:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:53:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:53:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:53:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:53:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:53:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:53:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:53:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:53:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:53:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:53:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:53:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:53:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:53:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:53:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:53:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:53:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:53:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:53:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:53:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:53:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:53:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:53:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:53:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:53:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:53:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:53:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:53:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:53:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:53:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:53:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:53:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:53:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:53:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:53:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:53:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:53:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:53:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:53:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:53:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:53:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:53:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:53:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:53:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:53:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:53:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:53:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:53:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:53:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:53:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:53:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:53:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:53:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:53:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:53:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:53:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:53:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:53:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:53:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:53:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:53:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:53:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:53:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:53:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:53:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:53:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:53:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:53:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:53:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:53:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:53:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:53:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:53:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:53:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:53:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:53:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:53:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:53:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:53:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:53:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:53:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:53:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:53:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:53:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:53:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:53:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:53:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:53:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:53:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:53:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:53:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:53:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:53:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:53:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:53:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:53:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:53:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:53:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:53:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:53:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:53:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:53:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:53:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:53:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:53:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:53:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:53:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:53:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:54:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:54:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:54:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:54:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:54:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:54:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:54:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:54:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:54:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:54:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:54:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:54:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:54:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:54:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:54:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:54:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:54:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:54:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:54:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:54:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:54:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:54:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:54:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:54:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:54:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:54:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:54:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:54:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:54:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:54:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:54:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:54:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:54:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:54:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:54:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:54:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:54:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:54:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:54:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:54:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:54:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:54:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:54:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:54:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:54:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:54:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:54:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:54:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:54:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:54:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:54:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:54:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:54:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:54:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:54:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:54:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:54:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:54:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:54:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:54:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:54:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:54:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:54:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:54:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:54:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:54:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:54:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:54:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:54:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:54:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:54:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:54:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:54:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:54:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:54:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:54:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:54:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:54:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:54:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:54:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:54:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:54:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:54:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:54:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:54:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:54:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:54:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:54:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:54:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:54:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:54:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:54:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:54:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:54:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:54:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:54:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:54:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:54:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:54:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:54:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:54:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:54:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:54:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:54:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:54:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:54:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:54:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:54:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:54:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:54:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:54:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:54:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:54:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:54:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:54:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:54:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:54:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:54:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:54:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:54:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:54:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:54:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:54:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:54:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:54:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:54:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:54:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:54:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:54:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:54:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:54:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:54:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:54:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:54:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:54:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:54:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:54:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:54:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:54:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:54:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:54:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:54:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:54:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:54:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:54:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:54:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:54:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:54:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:54:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:54:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:54:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:54:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:54:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:54:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:54:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:54:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:54:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:54:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:54:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:54:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:54:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:54:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:54:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:54:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:54:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:54:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:54:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:54:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:54:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:54:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:54:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:54:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:54:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:54:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:54:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:54:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:54:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:54:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:54:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:54:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:54:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:54:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:54:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:54:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:54:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:54:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:54:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:54:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:54:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:54:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:54:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:54:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:54:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:54:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:54:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:54:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:54:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:54:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:54:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:54:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:54:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:54:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:54:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:54:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:54:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:54:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:54:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:54:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:54:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:54:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:54:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:54:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:54:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:54:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:54:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:54:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:55:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:55:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:55:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:55:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:55:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:55:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:55:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:55:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:55:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:55:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:55:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:55:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:55:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:55:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:55:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:55:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:55:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:55:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:55:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:55:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:55:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:55:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:55:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:55:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:55:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:55:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:55:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:55:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:55:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:55:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:55:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:55:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:55:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:55:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:55:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:55:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:55:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:55:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:55:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:55:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:55:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:55:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:55:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:55:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:55:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:55:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:55:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:55:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:55:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:55:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:55:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:55:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:55:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:55:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:55:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:55:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:55:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:55:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:55:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:55:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:55:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:55:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:55:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:55:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:55:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:55:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:55:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:55:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:55:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:55:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:55:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:55:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:55:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:55:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:55:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:55:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:55:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:55:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:55:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:55:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:55:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:55:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:55:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:55:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:55:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:55:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:55:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:55:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:55:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:55:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:55:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:55:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:55:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:55:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:55:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:55:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:55:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:55:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:55:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:55:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:55:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:55:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:55:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:55:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:55:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:55:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:55:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:55:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:55:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:55:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:55:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:55:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:55:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:55:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:55:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:55:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:55:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:55:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:55:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:55:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:55:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:55:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:55:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:55:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:55:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:55:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:55:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:55:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:55:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:55:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:55:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:55:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:55:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:55:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:55:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:55:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:55:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:55:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:55:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:55:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:55:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:55:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:55:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:55:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:55:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:55:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:55:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:55:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:55:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:55:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:55:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:55:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:55:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:55:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:55:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:55:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:55:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:55:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:55:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:55:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:55:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:55:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:55:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:55:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:55:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:55:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:55:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:55:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:55:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:55:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:55:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:55:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:55:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:55:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:55:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:55:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:55:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:55:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:55:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:55:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:55:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:55:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:55:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:55:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:55:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:55:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:55:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:55:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:55:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:55:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:55:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:55:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:55:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:55:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:55:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:55:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:55:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:55:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:55:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:55:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:55:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:55:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:55:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:55:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:55:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:55:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:55:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:55:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:55:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:55:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:55:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:55:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:55:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:55:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:55:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:55:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:56:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:56:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:56:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:56:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:56:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:56:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:56:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:56:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:56:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:56:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:56:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:56:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:56:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:56:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:56:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:56:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:56:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:56:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:56:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:56:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:56:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:56:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:56:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:56:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:56:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:56:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:56:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:56:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:56:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:56:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:56:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:56:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:56:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:56:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:56:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:56:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:56:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:56:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:56:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:56:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:56:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:56:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:56:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:56:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:56:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:56:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:56:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:56:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:56:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:56:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:56:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:56:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:56:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:56:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:56:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:56:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:56:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:56:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:56:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:56:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:56:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:56:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:56:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:56:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:56:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:56:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:56:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:56:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:56:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:56:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:56:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:56:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:56:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:56:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:56:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:56:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:56:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:56:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:56:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:56:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:56:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:56:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:56:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:56:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:56:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:56:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:56:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:56:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:56:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:56:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:56:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:56:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:56:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:56:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:56:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:56:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:56:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:56:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:56:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:56:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:56:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:56:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:56:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:56:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:56:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:56:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:56:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:56:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:56:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:56:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:56:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:56:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:56:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:56:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:56:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:56:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:56:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:56:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:56:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:56:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:56:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:56:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:56:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:56:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:56:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:56:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:56:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:56:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:56:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:56:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:56:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:56:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:56:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:56:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:56:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:56:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:56:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:56:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:56:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:56:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:56:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:56:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:56:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:56:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:56:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:56:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:56:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:56:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:56:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:56:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:56:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:56:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:56:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:56:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:56:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:56:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:56:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:56:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:56:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:56:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:56:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:56:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:56:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:56:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:56:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:56:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:56:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:56:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:56:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:56:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:56:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:56:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:56:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:56:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:56:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:56:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:56:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:56:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:56:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:56:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:56:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:56:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:56:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:56:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:56:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:56:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:56:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:56:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:56:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:56:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:56:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:56:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:56:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:56:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:56:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:56:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:56:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:56:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:56:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:56:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:56:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:56:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:56:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:56:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:56:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:56:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:56:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:56:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:56:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:56:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:56:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:56:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:56:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:56:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:56:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:56:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:57:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:57:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:57:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:57:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:57:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:57:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:57:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:57:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:57:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:57:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:57:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:57:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:57:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:57:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:57:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:57:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:57:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:57:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:57:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:57:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:57:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:57:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:57:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:57:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:57:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:57:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:57:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:57:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:57:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:57:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:57:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:57:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:57:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:57:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:57:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:57:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:57:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:57:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:57:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:57:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:57:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:57:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:57:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:57:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:57:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:57:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:57:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:57:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:57:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:57:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:57:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:57:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:57:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:57:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:57:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:57:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:57:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:57:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:57:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:57:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:57:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:57:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:57:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:57:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:57:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:57:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:57:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:57:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:57:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:57:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:57:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:57:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:57:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:57:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:57:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:57:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:57:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:57:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:57:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:57:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:57:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:57:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:57:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:57:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:57:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:57:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:57:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:57:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:57:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:57:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:57:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:57:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:57:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:57:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:57:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:57:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:57:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:57:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:57:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:57:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:57:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:57:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:57:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:57:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:57:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:57:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:57:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:57:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:57:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:57:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:57:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:57:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:57:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:57:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:57:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:57:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:57:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:57:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:57:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:57:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:57:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:57:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:57:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:57:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:57:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:57:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:57:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:57:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:57:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:57:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:57:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:57:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:57:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:57:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:57:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:57:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:57:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:57:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:57:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:57:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:57:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:57:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:57:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:57:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:57:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:57:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:57:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:57:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:57:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:57:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:57:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:57:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:57:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:57:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:57:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:57:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:57:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:57:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:57:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:57:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:57:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:57:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:57:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:57:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:57:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:57:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:57:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:57:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:57:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:57:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:57:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:57:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:57:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:57:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:57:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:57:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:57:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:57:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:57:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:57:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:57:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:57:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:57:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:57:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:57:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:57:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:57:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:57:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:57:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:57:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:57:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:57:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:57:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:57:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:57:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:57:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:57:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:57:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:57:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:57:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:57:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:57:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:57:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:57:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:57:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:57:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:57:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:57:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:57:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:57:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:57:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:57:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:57:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:57:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:57:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:57:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:58:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:58:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:58:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:58:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:58:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:58:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:58:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:58:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:58:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:58:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:58:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:58:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:58:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:58:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:58:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:58:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:58:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:58:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:58:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:58:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:58:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:58:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:58:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:58:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:58:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:58:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:58:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:58:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:58:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:58:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:58:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:58:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:58:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:58:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:58:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:58:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:58:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:58:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:58:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:58:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:58:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:58:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:58:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:58:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:58:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:58:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:58:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:58:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:58:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:58:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:58:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:58:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:58:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:58:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:58:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:58:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:58:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:58:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:58:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:58:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:58:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:58:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:58:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:58:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:58:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:58:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:58:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:58:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:58:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:58:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:58:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:58:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:58:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:58:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:58:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:58:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:58:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:58:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:58:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:58:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:58:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:58:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:58:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:58:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:58:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:58:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:58:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:58:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:58:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:58:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:58:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:58:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:58:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:58:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:58:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:58:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:58:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:58:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:58:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:58:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:58:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:58:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:58:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:58:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:58:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:58:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:58:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:58:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:58:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:58:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:58:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:58:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:58:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:58:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:58:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:58:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:58:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:58:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:58:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:58:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:58:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:58:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:58:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:58:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:58:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:58:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:58:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:58:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:58:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:58:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:58:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:58:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:58:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:58:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:58:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:58:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:58:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:58:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:58:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:58:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:58:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:58:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:58:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:58:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:58:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:58:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:58:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:58:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:58:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:58:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:58:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:58:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:58:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:58:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:58:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:58:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:58:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:58:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:58:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:58:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:58:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:58:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:58:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:58:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:58:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:58:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:58:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:58:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:58:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:58:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:58:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:58:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:58:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:58:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:58:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:58:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:58:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:58:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:58:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:58:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:58:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:58:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:58:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:58:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:58:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:58:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:58:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:58:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:58:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:58:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:58:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:58:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:58:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:58:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:58:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:58:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:58:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:58:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:58:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:58:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:58:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:58:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:58:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:58:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:58:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:58:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:58:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:58:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:58:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:58:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:58:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:58:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:58:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:58:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:58:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:58:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:59:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:59:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:59:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:59:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:59:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:59:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:59:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:59:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:59:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:59:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:59:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:59:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:59:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:59:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:59:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:59:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:59:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:59:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:59:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:59:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:59:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:59:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:59:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:59:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:59:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:59:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:59:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:59:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:59:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:59:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:59:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:59:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:59:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:59:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:59:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:59:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:59:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:59:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:59:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:59:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:59:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:59:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:59:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:59:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:59:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:59:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:59:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:59:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:59:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:59:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:59:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:59:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:59:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:59:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:59:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:59:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:59:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:59:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:59:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:59:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:59:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:59:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:59:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:59:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:59:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:59:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:59:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:59:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:59:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:59:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:59:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:59:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:59:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:59:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:59:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:59:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:59:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:59:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:59:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:59:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:59:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:59:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:59:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:59:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:59:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:59:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:59:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:59:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:59:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:59:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:59:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:59:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:59:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:59:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:59:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:59:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:59:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:59:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:59:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:59:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:59:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:59:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:59:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:59:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:59:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:59:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:59:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:59:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:59:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:59:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:59:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:59:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:59:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:59:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:59:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:59:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:59:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:59:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:59:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:59:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:59:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:59:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:59:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:59:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:59:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:59:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:59:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:59:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:59:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:59:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:59:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:59:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:59:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:59:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:59:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:59:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:59:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:59:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:59:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:59:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:59:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:59:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:59:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:59:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:59:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:59:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:59:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:59:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:59:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:59:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:59:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:59:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:59:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:59:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:59:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:59:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:59:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:59:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:59:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:59:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:59:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:59:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:59:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:59:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:59:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:59:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:59:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:59:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:59:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:59:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:59:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:59:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:59:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:59:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:59:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:59:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:59:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:59:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:59:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:59:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:59:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:59:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:59:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:59:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:59:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:59:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:59:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:59:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:59:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:59:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:59:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:59:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:59:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:59:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:59:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:59:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:59:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:59:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 05:59:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 05:59:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:59:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:59:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:59:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 05:59:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 05:59:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 05:59:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 05:59:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 05:59:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 05:59:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 05:59:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 05:59:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 05:59:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 05:59:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 05:59:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 05:59:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 05:59:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:00:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:00:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:00:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:00:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:00:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:00:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:00:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:00:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:00:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:00:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:00:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:00:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:00:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:00:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:00:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:00:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:00:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:00:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:00:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:00:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:00:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:00:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:00:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:00:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:00:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:00:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:00:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:00:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:00:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:00:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:00:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:00:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:00:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:00:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:00:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:00:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:00:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:00:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:00:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:00:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:00:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:00:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:00:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:00:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:00:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:00:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:00:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:00:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:00:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:00:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:00:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:00:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:00:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:00:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:00:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:00:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:00:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:00:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:00:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:00:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:00:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:00:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:00:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:00:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:00:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:00:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:00:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:00:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:00:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:00:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:00:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:00:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:00:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:00:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:00:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:00:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:00:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:00:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:00:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:00:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:00:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:00:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:00:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:00:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:00:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:00:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:00:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:00:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:00:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:00:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:00:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:00:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:00:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:00:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:00:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:00:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:00:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:00:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:00:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:00:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:00:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:00:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:00:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:00:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:00:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:00:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:00:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:00:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:00:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:00:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:00:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:00:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:00:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:00:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:00:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:00:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:00:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:00:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:00:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:00:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:00:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:00:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:00:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:00:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:00:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:00:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:00:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:00:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:00:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:00:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:00:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:00:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:00:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:00:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:00:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:00:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:00:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:00:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:00:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:00:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:00:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:00:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:00:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:00:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:00:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:00:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:00:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:00:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:00:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:00:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:00:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:00:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:00:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:00:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:00:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:00:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:00:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:00:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:00:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:00:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:00:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:00:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:00:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:00:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:00:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:00:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:00:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:00:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:00:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:00:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:00:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:00:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:00:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:00:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:00:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:00:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:00:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:00:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:00:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:00:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:00:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:00:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:00:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:00:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:00:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:00:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:00:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:00:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:00:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:00:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:00:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:00:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:00:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:00:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:00:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:00:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:00:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:00:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:00:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:00:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:00:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:00:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:00:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:00:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:00:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:00:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:00:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:00:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:00:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:00:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:00:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:00:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:00:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:00:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:00:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:00:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:03:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:03:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:03:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:03:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:03:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:03:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:03:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:03:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:03:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:03:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:03:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:03:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:03:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:03:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:03:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:03:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:03:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:03:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:03:06 rdma06 rsyslogd[1041]: imjournal: 5295 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 06:03:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:03:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:03:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:03:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:03:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:03:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:03:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:03:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:03:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:03:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:03:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:03:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:03:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:03:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:03:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:03:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:03:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:03:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:03:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:03:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:03:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:03:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:03:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:03:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:03:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:03:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:03:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:03:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:03:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:03:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:03:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:03:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:03:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:03:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:03:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:03:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:03:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:03:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:03:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:03:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:03:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:03:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:03:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:03:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:03:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:03:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:03:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:03:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:03:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:03:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:03:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:03:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:03:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:03:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:03:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:03:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:03:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:03:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:03:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:03:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:03:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:03:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:03:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:03:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:03:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:03:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:03:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:03:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:03:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:03:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:03:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:03:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:03:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:03:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:03:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:03:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:03:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:03:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:03:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:03:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:03:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:03:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:03:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:03:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:03:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:03:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:03:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:03:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:03:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:03:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:03:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:03:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:03:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:03:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:03:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:03:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:03:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:03:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:03:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:03:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:03:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:03:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:03:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:03:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:03:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:03:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:03:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:03:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:03:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:03:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:03:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:03:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:03:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:03:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:03:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:03:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:03:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:03:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:03:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:03:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:03:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:03:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:03:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:03:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:03:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:03:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:03:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:03:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:03:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:03:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:03:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:03:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:03:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:03:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:03:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:03:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:03:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:03:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:03:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:03:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:03:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:03:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:03:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:03:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:03:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:03:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:03:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:03:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:03:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:03:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:03:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:03:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:03:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:03:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:03:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:03:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:03:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:03:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:03:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:03:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:03:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:03:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:03:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:03:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:03:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:03:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:03:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:03:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:03:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:03:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:03:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:03:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:03:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:03:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:03:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:03:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:03:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:03:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:03:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:03:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:04:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:04:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:04:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:04:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:04:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:04:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:04:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:04:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:04:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:04:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:04:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:04:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:04:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:04:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:04:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:04:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:04:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:04:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:04:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:04:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:04:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:04:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:04:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:04:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:04:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:04:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:04:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:04:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:04:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:04:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:04:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:04:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:04:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:04:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:04:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:04:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:04:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:04:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:04:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:04:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:04:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:04:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:04:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:04:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:04:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:04:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:04:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:04:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:04:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:04:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:04:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:04:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:04:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:04:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:04:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:04:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:04:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:04:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:04:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:04:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:04:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:04:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:04:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:04:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:04:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:04:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:04:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:04:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:04:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:04:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:04:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:04:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:04:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:04:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:04:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:04:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:04:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:04:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:04:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:04:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:04:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:04:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:04:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:04:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:04:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:04:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:04:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:04:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:04:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:04:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:04:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:04:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:04:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:04:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:04:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:04:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:04:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:04:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:04:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:04:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:04:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:04:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:04:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:04:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:04:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:04:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:04:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:04:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:04:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:04:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:04:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:04:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:04:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:04:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:04:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:04:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:04:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:04:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:04:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:04:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:04:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:04:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:04:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:04:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:04:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:04:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:04:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:04:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:04:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:04:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:04:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:04:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:04:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:04:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:04:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:04:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:04:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:04:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:04:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:04:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:04:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:04:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:04:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:04:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:04:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:04:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:04:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:04:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:04:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:04:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:04:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:04:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:04:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:04:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:04:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:04:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:04:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:04:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:04:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:04:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:04:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:04:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:04:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:04:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:04:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:04:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:04:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:04:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:04:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:04:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:04:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:04:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:04:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:04:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:04:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:04:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:04:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:04:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:04:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:04:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:04:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:04:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:04:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:04:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:04:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:04:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:04:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:04:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:04:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:04:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:04:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:04:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:04:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:04:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:04:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:04:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:04:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:04:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:04:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:04:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:04:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:04:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:04:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:04:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:04:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:04:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:04:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:04:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:04:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:04:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:04:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:04:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:04:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:04:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:04:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:04:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:05:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:05:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:05:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:05:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:05:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:05:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:05:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:05:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:05:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:05:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:05:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:05:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:05:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:05:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:05:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:05:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:05:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:05:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:05:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:05:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:05:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:05:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:05:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:05:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:05:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:05:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:05:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:05:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:05:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:05:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:05:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:05:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:05:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:05:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:05:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:05:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:05:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:05:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:05:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:05:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:05:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:05:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:05:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:05:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:05:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:05:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:05:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:05:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:05:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:05:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:05:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:05:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:05:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:05:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:05:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:05:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:05:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:05:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:05:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:05:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:05:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:05:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:05:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:05:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:05:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:05:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:05:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:05:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:05:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:05:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:05:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:05:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:05:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:05:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:05:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:05:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:05:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:05:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:05:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:05:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:05:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:05:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:05:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:05:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:05:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:05:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:05:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:05:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:05:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:05:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:05:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:05:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:05:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:05:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:05:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:05:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:05:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:05:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:05:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:05:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:05:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:05:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:05:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:05:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:05:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:05:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:05:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:05:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:05:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:05:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:05:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:05:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:05:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:05:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:05:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:05:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:05:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:05:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:05:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:05:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:05:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:05:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:05:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:05:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:05:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:05:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:05:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:05:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:05:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:05:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:05:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:05:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:05:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:05:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:05:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:05:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:05:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:05:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:05:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:05:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:05:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:05:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:05:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:05:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:05:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:05:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:05:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:05:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:05:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:05:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:05:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:05:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:05:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:05:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:05:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:05:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:05:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:05:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:05:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:05:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:05:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:05:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:05:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:05:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:05:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:05:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:05:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:05:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:05:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:05:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:05:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:05:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:05:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:05:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:05:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:05:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:05:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:05:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:05:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:05:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:05:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:05:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:05:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:05:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:05:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:05:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:05:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:05:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:05:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:05:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:05:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:05:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:05:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:05:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:05:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:05:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:05:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:05:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:05:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:05:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:05:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:05:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:05:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:05:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:05:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:05:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:05:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:05:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:05:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:05:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:05:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:05:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:05:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:05:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:05:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:05:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:06:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:06:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:06:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:06:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:06:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:06:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:06:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:06:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:06:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:06:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:06:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:06:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:06:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:06:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:06:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:06:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:06:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:06:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:06:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:06:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:06:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:06:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:06:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:06:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:06:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:06:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:06:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:06:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:06:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:06:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:06:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:06:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:06:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:06:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:06:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:06:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:06:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:06:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:06:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:06:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:06:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:06:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:06:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:06:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:06:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:06:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:06:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:06:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:06:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:06:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:06:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:06:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:06:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:06:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:06:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:06:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:06:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:06:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:06:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:06:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:06:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:06:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:06:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:06:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:06:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:06:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:06:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:06:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:06:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:06:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:06:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:06:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:06:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:06:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:06:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:06:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:06:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:06:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:06:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:06:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:06:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:06:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:06:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:06:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:06:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:06:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:06:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:06:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:06:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:06:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:06:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:06:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:06:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:06:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:06:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:06:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:06:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:06:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:06:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:06:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:06:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:06:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:06:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:06:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:06:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:06:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:06:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:06:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:06:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:06:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:06:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:06:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:06:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:06:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:06:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:06:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:06:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:06:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:06:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:06:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:06:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:06:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:06:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:06:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:06:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:06:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:06:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:06:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:06:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:06:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:06:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:06:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:06:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:06:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:06:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:06:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:06:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:06:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:06:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:06:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:06:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:06:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:06:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:06:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:06:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:06:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:06:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:06:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:06:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:06:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:06:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:06:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:06:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:06:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:06:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:06:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:06:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:06:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:06:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:06:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:06:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:06:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:06:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:06:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:06:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:06:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:06:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:06:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:06:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:06:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:06:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:06:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:06:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:06:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:06:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:06:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:06:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:06:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:06:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:06:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:06:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:06:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:06:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:06:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:06:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:06:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:06:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:06:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:06:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:06:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:06:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:06:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:06:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:06:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:06:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:06:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:06:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:06:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:06:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:06:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:06:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:06:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:06:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:06:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:06:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:06:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:06:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:06:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:06:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:06:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:06:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:06:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:06:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:06:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:06:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:06:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:07:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:07:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:07:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:07:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:07:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:07:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:07:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:07:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:07:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:07:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:07:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:07:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:07:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:07:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:07:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:07:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:07:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:07:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:07:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:07:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:07:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:07:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:07:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:07:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:07:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:07:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:07:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:07:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:07:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:07:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:07:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:07:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:07:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:07:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:07:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:07:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:07:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:07:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:07:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:07:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:07:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:07:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:07:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:07:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:07:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:07:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:07:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:07:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:07:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:07:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:07:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:07:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:07:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:07:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:07:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:07:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:07:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:07:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:07:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:07:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:07:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:07:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:07:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:07:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:07:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:07:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:07:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:07:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:07:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:07:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:07:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:07:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:07:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:07:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:07:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:07:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:07:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:07:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:07:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:07:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:07:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:07:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:07:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:07:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:07:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:07:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:07:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:07:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:07:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:07:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:07:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:07:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:07:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:07:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:07:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:07:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:07:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:07:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:07:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:07:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:07:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:07:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:07:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:07:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:07:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:07:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:07:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:07:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:07:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:07:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:07:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:07:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:07:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:07:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:07:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:07:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:07:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:07:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:07:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:07:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:07:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:07:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:07:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:07:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:07:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:07:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:07:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:07:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:07:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:07:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:07:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:07:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:07:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:07:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:07:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:07:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:07:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:07:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:07:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:07:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:07:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:07:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:07:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:07:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:07:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:07:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:07:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:07:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:07:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:07:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:07:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:07:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:07:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:07:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:07:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:07:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:07:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:07:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:07:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:07:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:07:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:07:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:07:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:07:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:07:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:07:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:07:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:07:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:07:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:07:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:07:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:07:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:07:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:07:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:07:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:07:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:07:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:07:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:07:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:07:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:07:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:07:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:07:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:07:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:07:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:07:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:07:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:07:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:07:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:07:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:07:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:07:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:07:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:07:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:07:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:07:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:07:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:07:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:07:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:07:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:07:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:07:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:07:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:07:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:07:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:07:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:07:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:07:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:07:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:07:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:07:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:07:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:07:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:07:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:07:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:07:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:08:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:08:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:08:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:08:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:08:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:08:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:08:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:08:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:08:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:08:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:08:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:08:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:08:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:08:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:08:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:08:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:08:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:08:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:08:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:08:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:08:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:08:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:08:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:08:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:08:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:08:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:08:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:08:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:08:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:08:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:08:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:08:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:08:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:08:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:08:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:08:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:08:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:08:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:08:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:08:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:08:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:08:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:08:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:08:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:08:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:08:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:08:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:08:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:08:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:08:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:08:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:08:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:08:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:08:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:08:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:08:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:08:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:08:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:08:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:08:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:08:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:08:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:08:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:08:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:08:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:08:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:08:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:08:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:08:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:08:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:08:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:08:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:08:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:08:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:08:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:08:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:08:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:08:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:08:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:08:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:08:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:08:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:08:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:08:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:08:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:08:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:08:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:08:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:08:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:08:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:08:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:08:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:08:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:08:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:08:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:08:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:08:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:08:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:08:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:08:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:08:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:08:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:08:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:08:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:08:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:08:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:08:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:08:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:08:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:08:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:08:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:08:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:08:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:08:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:08:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:08:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:08:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:08:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:08:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:08:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:08:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:08:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:08:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:08:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:08:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:08:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:08:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:08:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:08:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:08:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:08:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:08:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:08:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:08:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:08:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:08:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:08:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:08:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:08:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:08:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:08:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:08:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:08:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:08:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:08:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:08:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:08:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:08:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:08:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:08:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:08:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:08:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:08:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:08:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:08:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:08:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:08:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:08:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:08:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:08:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:08:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:08:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:08:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:08:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:08:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:08:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:08:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:08:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:08:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:08:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:08:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:08:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:08:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:08:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:08:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:08:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:08:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:08:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:08:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:08:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:08:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:08:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:08:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:08:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:08:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:08:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:08:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:08:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:08:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:08:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:08:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:08:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:08:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:08:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:08:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:08:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:08:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:08:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:08:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:08:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:08:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:08:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:08:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:08:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:08:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:08:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:08:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:08:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:08:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:08:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:08:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:08:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:08:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:08:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:08:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:08:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:09:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:09:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:09:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:09:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:09:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:09:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:09:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:09:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:09:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:09:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:09:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:09:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:09:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:09:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:09:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:09:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:09:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:09:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:09:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:09:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:09:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:09:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:09:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:09:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:09:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:09:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:09:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:09:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:09:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:09:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:09:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:09:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:09:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:09:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:09:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:09:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:09:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:09:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:09:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:09:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:09:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:09:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:09:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:09:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:09:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:09:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:09:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:09:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:09:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:09:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:09:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:09:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:09:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:09:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:09:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:09:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:09:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:09:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:09:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:09:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:09:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:09:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:09:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:09:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:09:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:09:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:09:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:09:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:09:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:09:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:09:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:09:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:09:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:09:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:09:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:09:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:09:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:09:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:09:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:09:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:09:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:09:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:09:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:09:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:09:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:09:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:09:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:09:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:09:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:09:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:09:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:09:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:09:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:09:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:09:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:09:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:09:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:09:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:09:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:09:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:09:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:09:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:09:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:09:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:09:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:09:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:09:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:09:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:09:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:09:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:09:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:09:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:09:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:09:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:09:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:09:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:09:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:09:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:09:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:09:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:09:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:09:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:09:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:09:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:09:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:09:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:09:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:09:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:09:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:09:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:09:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:09:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:09:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:09:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:09:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:09:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:09:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:09:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:09:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:09:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:09:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:09:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:09:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:09:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:09:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:09:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:09:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:09:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:09:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:09:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:09:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:09:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:09:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:09:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:09:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:09:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:09:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:09:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:09:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:09:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:09:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:09:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:09:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:09:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:09:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:09:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:09:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:09:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:09:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:09:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:09:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:09:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:09:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:09:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:09:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:09:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:09:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:09:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:09:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:09:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:09:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:09:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:09:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:09:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:09:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:09:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:09:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:09:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:09:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:09:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:09:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:09:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:09:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:09:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:09:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:09:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:09:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:09:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:09:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:09:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:09:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:09:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:09:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:09:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:09:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:09:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:09:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:09:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:09:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:09:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:09:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:09:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:09:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:09:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:09:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:09:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:10:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:10:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:10:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:10:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:10:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:10:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:10:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:10:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:10:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:10:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:10:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:10:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:10:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:10:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:10:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:10:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:10:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:10:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:10:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:10:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:10:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:10:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:10:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:10:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:10:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:10:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:10:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:10:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:10:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:10:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:10:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:10:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:10:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:10:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:10:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:10:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:10:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:10:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:10:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:10:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:10:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:10:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:10:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:10:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:10:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:10:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:10:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:10:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:10:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:10:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:10:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:10:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:10:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:10:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:10:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:10:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:10:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:10:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:10:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:10:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:10:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:10:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:10:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:10:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:10:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:10:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:10:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:10:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:10:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:10:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:10:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:10:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:10:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:10:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:10:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:10:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:10:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:10:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:10:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:10:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:10:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:10:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:10:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:10:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:10:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:10:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:10:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:10:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:10:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:10:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:10:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:10:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:10:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:10:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:10:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:10:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:10:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:10:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:10:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:10:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:10:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:10:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:10:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:10:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:10:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:10:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:10:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:10:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:10:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:10:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:10:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:10:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:10:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:10:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:10:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:10:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:10:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:10:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:10:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:10:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:10:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:10:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:10:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:10:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:10:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:10:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:10:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:10:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:10:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:10:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:10:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:10:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:10:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:10:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:10:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:10:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:10:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:10:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:10:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:10:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:10:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:10:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:10:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:10:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:10:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:10:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:10:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:10:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:10:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:10:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:10:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:10:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:10:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:10:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:10:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:10:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:10:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:10:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:10:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:10:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:10:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:10:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:10:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:10:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:10:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:10:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:10:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:10:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:10:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:10:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:10:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:10:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:10:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:10:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:10:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:10:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:10:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:10:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:10:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:10:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:10:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:10:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:10:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:10:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:10:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:10:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:10:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:10:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:10:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:10:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:10:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:10:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:10:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:10:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:10:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:10:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:10:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:10:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:10:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:10:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:10:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:10:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:10:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:10:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:10:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:10:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:10:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:10:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:10:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:10:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:10:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:10:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:10:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:10:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:10:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:10:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:11:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:11:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:11:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:11:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:11:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:11:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:11:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:11:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:11:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:11:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:11:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:11:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:11:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:11:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:11:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:11:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:11:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:11:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:13:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:13:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:13:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:13:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:13:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:13:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:13:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:13:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:13:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:13:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:13:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:13:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:13:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:13:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:13:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:13:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:13:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:13:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:13:11 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 06:13:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:13:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:13:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:13:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:13:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:13:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:13:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:13:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:13:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:13:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:13:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:13:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:13:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:13:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:13:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:13:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:13:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:13:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:13:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:13:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:13:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:13:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:13:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:13:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:13:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:13:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:13:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:13:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:13:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:13:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:13:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:13:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:13:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:13:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:13:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:13:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:13:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:13:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:13:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:13:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:13:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:13:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:13:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:13:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:13:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:13:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:13:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:13:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:13:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:13:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:13:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:13:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:13:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:13:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:13:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:13:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:13:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:13:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:13:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:13:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:13:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:13:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:13:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:13:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:13:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:13:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:13:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:13:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:13:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:13:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:13:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:13:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:13:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:13:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:13:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:13:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:13:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:13:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:13:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:13:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:13:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:13:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:13:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:13:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:13:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:13:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:13:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:13:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:13:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:13:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:13:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:13:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:13:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:13:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:13:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:13:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:13:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:13:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:13:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:13:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:13:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:13:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:13:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:13:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:13:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:13:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:13:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:13:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:13:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:13:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:13:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:13:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:13:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:13:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:13:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:13:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:13:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:13:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:13:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:13:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:13:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:13:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:13:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:13:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:13:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:13:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:13:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:13:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:13:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:13:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:13:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:13:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:13:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:13:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:13:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:13:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:13:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:13:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:13:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:13:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:13:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:13:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:13:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:13:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:13:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:13:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:13:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:13:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:13:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:13:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:13:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:13:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:13:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:13:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:13:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:13:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:13:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:13:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:13:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:13:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:13:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:13:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:14:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:14:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:14:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:14:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:14:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:14:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:14:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:14:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:14:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:14:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:14:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:14:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:14:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:14:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:14:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:14:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:14:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:14:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:14:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:14:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:14:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:14:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:14:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:14:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:14:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:14:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:14:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:14:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:14:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:14:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:14:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:14:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:14:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:14:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:14:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:14:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:14:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:14:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:14:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:14:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:14:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:14:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:14:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:14:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:14:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:14:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:14:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:14:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:14:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:14:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:14:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:14:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:14:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:14:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:14:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:14:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:14:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:14:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:14:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:14:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:14:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:14:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:14:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:14:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:14:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:14:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:14:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:14:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:14:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:14:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:14:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:14:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:14:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:14:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:14:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:14:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:14:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:14:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:14:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:14:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:14:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:14:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:14:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:14:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:14:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:14:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:14:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:14:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:14:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:14:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:14:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:14:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:14:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:14:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:14:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:14:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:14:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:14:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:14:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:14:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:14:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:14:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:14:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:14:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:14:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:14:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:14:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:14:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:14:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:14:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:14:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:14:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:14:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:14:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:14:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:14:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:14:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:14:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:14:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:14:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:14:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:14:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:14:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:14:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:14:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:14:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:14:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:14:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:14:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:14:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:14:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:14:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:14:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:14:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:14:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:14:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:14:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:14:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:14:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:14:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:14:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:14:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:14:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:14:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:14:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:14:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:14:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:14:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:14:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:14:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:14:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:14:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:14:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:14:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:14:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:14:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:14:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:14:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:14:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:14:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:14:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:14:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:14:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:14:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:14:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:14:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:14:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:14:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:14:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:14:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:14:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:14:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:14:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:14:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:14:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:14:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:14:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:14:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:14:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:14:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:14:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:14:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:14:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:14:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:14:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:14:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:14:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:14:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:14:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:14:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:14:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:14:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:14:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:14:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:14:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:14:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:14:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:14:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:14:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:14:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:14:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:14:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:14:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:14:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:14:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:14:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:14:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:14:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:14:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:14:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:14:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:14:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:14:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:14:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:14:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:14:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:15:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:15:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:15:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:15:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:15:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:15:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:15:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:15:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:15:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:15:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:15:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:15:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:15:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:15:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:15:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:15:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:15:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:15:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:15:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:15:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:15:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:15:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:15:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:15:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:15:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:15:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:15:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:15:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:15:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:15:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:15:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:15:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:15:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:15:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:15:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:15:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:15:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:15:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:15:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:15:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:15:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:15:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:15:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:15:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:15:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:15:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:15:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:15:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:15:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:15:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:15:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:15:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:15:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:15:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:15:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:15:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:15:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:15:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:15:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:15:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:15:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:15:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:15:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:15:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:15:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:15:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:15:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:15:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:15:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:15:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:15:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:15:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:15:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:15:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:15:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:15:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:15:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:15:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:15:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:15:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:15:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:15:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:15:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:15:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:15:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:15:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:15:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:15:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:15:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:15:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:15:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:15:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:15:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:15:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:15:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:15:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:15:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:15:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:15:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:15:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:15:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:15:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:15:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:15:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:15:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:15:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:15:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:15:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:15:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:15:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:15:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:15:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:15:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:15:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:15:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:15:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:15:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:15:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:15:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:15:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:15:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:15:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:15:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:15:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:15:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:15:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:15:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:15:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:15:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:15:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:15:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:15:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:15:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:15:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:15:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:15:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:15:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:15:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:15:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:15:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:15:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:15:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:15:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:15:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:15:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:15:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:15:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:15:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:15:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:15:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:15:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:15:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:15:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:15:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:15:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:15:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:15:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:15:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:15:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:15:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:15:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:15:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:15:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:15:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:15:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:15:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:15:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:15:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:15:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:15:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:15:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:15:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:15:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:15:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:15:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:15:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:15:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:15:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:15:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:15:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:15:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:15:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:15:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:15:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:15:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:15:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:15:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:15:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:15:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:15:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:15:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:15:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:15:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:15:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:15:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:15:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:15:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:15:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:15:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:15:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:15:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:15:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:15:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:15:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:15:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:15:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:15:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:15:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:15:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:15:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:15:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:15:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:15:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:15:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:15:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:15:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:16:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:16:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:16:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:16:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:16:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:16:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:16:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:16:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:16:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:16:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:16:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:16:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:16:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:16:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:16:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:16:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:16:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:16:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:16:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:16:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:16:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:16:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:16:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:16:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:16:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:16:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:16:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:16:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:16:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:16:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:16:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:16:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:16:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:16:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:16:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:16:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:16:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:16:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:16:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:16:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:16:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:16:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:16:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:16:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:16:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:16:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:16:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:16:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:16:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:16:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:16:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:16:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:16:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:16:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:16:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:16:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:16:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:16:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:16:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:16:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:16:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:16:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:16:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:16:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:16:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:16:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:16:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:16:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:16:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:16:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:16:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:16:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:16:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:16:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:16:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:16:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:16:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:16:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:16:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:16:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:16:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:16:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:16:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:16:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:16:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:16:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:16:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:16:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:16:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:16:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:16:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:16:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:16:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:16:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:16:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:16:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:16:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:16:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:16:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:16:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:16:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:16:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:16:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:16:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:16:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:16:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:16:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:16:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:16:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:16:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:16:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:16:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:16:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:16:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:16:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:16:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:16:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:16:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:16:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:16:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:16:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:16:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:16:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:16:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:16:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:16:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:16:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:16:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:16:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:16:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:16:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:16:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:16:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:16:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:16:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:16:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:16:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:16:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:16:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:16:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:16:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:16:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:16:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:16:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:16:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:16:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:16:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:16:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:16:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:16:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:16:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:16:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:16:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:16:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:16:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:16:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:16:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:16:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:16:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:16:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:16:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:16:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:16:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:16:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:16:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:16:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:16:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:16:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:16:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:16:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:16:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:16:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:16:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:16:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:16:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:16:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:16:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:16:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:16:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:16:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:16:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:16:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:16:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:16:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:16:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:16:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:16:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:16:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:16:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:16:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:16:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:16:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:16:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:16:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:16:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:16:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:16:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:16:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:16:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:16:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:16:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:16:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:16:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:16:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:16:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:16:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:16:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:16:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:16:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:16:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:16:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:16:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:16:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:16:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:16:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:16:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:17:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:17:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:17:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:17:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:17:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:17:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:17:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:17:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:17:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:17:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:17:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:17:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:17:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:17:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:17:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:17:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:17:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:17:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:17:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:17:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:17:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:17:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:17:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:17:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:17:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:17:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:17:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:17:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:17:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:17:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:17:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:17:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:17:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:17:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:17:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:17:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:17:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:17:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:17:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:17:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:17:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:17:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:17:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:17:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:17:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:17:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:17:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:17:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:17:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:17:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:17:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:17:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:17:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:17:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:17:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:17:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:17:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:17:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:17:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:17:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:17:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:17:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:17:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:17:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:17:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:17:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:17:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:17:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:17:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:17:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:17:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:17:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:17:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:17:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:17:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:17:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:17:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:17:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:17:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:17:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:17:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:17:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:17:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:17:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:17:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:17:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:17:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:17:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:17:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:17:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:17:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:17:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:17:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:17:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:17:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:17:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:17:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:17:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:17:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:17:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:17:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:17:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:17:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:17:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:17:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:17:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:17:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:17:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:17:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:17:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:17:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:17:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:17:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:17:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:17:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:17:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:17:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:17:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:17:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:17:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:17:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:17:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:17:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:17:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:17:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:17:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:17:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:17:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:17:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:17:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:17:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:17:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:17:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:17:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:17:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:17:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:17:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:17:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:17:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:17:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:17:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:17:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:17:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:17:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:17:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:17:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:17:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:17:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:17:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:17:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:17:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:17:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:17:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:17:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:17:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:17:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:17:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:17:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:17:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:17:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:17:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:17:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:17:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:17:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:17:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:17:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:17:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:17:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:17:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:17:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:17:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:17:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:17:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:17:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:17:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:17:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:17:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:17:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:17:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:17:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:17:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:17:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:17:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:17:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:17:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:17:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:17:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:17:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:17:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:17:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:17:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:17:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:17:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:17:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:17:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:17:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:17:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:17:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:17:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:17:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:17:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:17:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:17:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:17:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:17:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:17:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:17:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:17:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:17:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:17:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:17:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:17:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:17:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:17:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:17:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:17:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:18:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:18:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:18:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:18:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:18:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:18:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:18:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:18:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:18:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:18:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:18:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:18:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:18:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:18:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:18:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:18:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:18:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:18:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:18:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:18:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:18:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:18:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:18:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:18:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:18:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:18:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:18:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:18:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:18:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:18:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:18:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:18:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:18:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:18:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:18:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:18:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:18:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:18:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:18:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:18:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:18:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:18:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:18:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:18:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:18:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:18:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:18:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:18:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:18:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:18:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:18:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:18:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:18:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:18:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:18:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:18:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:18:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:18:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:18:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:18:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:18:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:18:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:18:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:18:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:18:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:18:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:18:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:18:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:18:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:18:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:18:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:18:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:18:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:18:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:18:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:18:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:18:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:18:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:18:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:18:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:18:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:18:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:18:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:18:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:18:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:18:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:18:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:18:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:18:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:18:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:18:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:18:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:18:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:18:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:18:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:18:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:18:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:18:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:18:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:18:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:18:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:18:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:18:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:18:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:18:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:18:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:18:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:18:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:18:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:18:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:18:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:18:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:18:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:18:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:18:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:18:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:18:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:18:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:18:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:18:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:18:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:18:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:18:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:18:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:18:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:18:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:18:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:18:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:18:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:18:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:18:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:18:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:18:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:18:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:18:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:18:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:18:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:18:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:18:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:18:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:18:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:18:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:18:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:18:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:18:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:18:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:18:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:18:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:18:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:18:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:18:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:18:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:18:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:18:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:18:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:18:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:18:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:18:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:18:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:18:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:18:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:18:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:18:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:18:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:18:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:18:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:18:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:18:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:18:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:18:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:18:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:18:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:18:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:18:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:18:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:18:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:18:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:18:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:18:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:18:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:18:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:18:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:18:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:18:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:18:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:18:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:18:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:18:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:18:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:18:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:18:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:18:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:18:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:18:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:18:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:18:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:18:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:18:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:18:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:18:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:18:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:18:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:18:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:18:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:18:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:18:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:18:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:18:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:18:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:18:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:18:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:18:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:18:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:18:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:18:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:18:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:19:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:19:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:19:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:19:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:19:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:19:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:19:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:19:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:19:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:19:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:19:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:19:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:19:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:19:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:19:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:19:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:19:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:19:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:19:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:19:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:19:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:19:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:19:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:19:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:19:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:19:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:19:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:19:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:19:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:19:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:19:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:19:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:19:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:19:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:19:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:19:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:19:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:19:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:19:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:19:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:19:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:19:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:19:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:19:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:19:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:19:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:19:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:19:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:19:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:19:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:19:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:19:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:19:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:19:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:19:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:19:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:19:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:19:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:19:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:19:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:19:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:19:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:19:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:19:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:19:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:19:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:19:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:19:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:19:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:19:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:19:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:19:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:19:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:19:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:19:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:19:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:19:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:19:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:19:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:19:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:19:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:19:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:19:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:19:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:19:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:19:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:19:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:19:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:19:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:19:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:19:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:19:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:19:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:19:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:19:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:19:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:19:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:19:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:19:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:19:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:19:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:19:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:19:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:19:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:19:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:19:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:19:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:19:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:19:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:19:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:19:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:19:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:19:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:19:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:19:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:19:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:19:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:19:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:19:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:19:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:19:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:19:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:19:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:19:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:19:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:19:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:19:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:19:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:19:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:19:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:19:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:19:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:19:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:19:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:19:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:19:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:19:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:19:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:19:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:19:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:19:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:19:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:19:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:19:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:19:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:19:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:19:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:19:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:19:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:19:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:19:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:19:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:19:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:19:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:19:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:19:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:19:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:19:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:19:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:19:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:19:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:19:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:19:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:19:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:19:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:19:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:19:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:19:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:19:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:19:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:19:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:19:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:19:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:19:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:19:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:19:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:19:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:19:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:19:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:19:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:19:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:19:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:19:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:19:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:19:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:19:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:19:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:19:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:19:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:19:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:19:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:19:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:19:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:19:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:19:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:19:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:19:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:19:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:19:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:19:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:19:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:19:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:19:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:19:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:19:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:19:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:19:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:19:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:19:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:19:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:19:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:19:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:19:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:19:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:19:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:19:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:20:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:20:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:20:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:20:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:20:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:20:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:20:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:20:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:20:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:20:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:20:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:20:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:20:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:20:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:20:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:20:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:20:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:20:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:20:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:20:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:20:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:20:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:20:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:20:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:20:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:20:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:20:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:20:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:20:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:20:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:20:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:20:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:20:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:20:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:20:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:20:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:20:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:20:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:20:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:20:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:20:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:20:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:20:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:20:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:20:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:20:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:20:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:20:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:20:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:20:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:20:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:20:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:20:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:20:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:20:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:20:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:20:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:20:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:20:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:20:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:20:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:20:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:20:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:20:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:20:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:20:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:20:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:20:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:20:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:20:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:20:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:20:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:20:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:20:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:20:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:20:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:20:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:20:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:20:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:20:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:20:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:20:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:20:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:20:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:20:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:20:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:20:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:20:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:20:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:20:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:20:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:20:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:20:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:20:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:20:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:20:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:20:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:20:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:20:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:20:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:20:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:20:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:20:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:20:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:20:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:20:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:20:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:20:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:20:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:20:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:20:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:20:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:20:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:20:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:20:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:20:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:20:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:20:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:20:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:20:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:20:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:20:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:20:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:20:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:20:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:20:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:20:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:20:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:20:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:20:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:20:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:20:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:20:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:20:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:20:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:20:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:20:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:20:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:20:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:20:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:20:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:20:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:20:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:20:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:20:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:20:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:20:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:20:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:20:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:20:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:20:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:20:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:20:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:20:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:20:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:20:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:20:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:20:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:20:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:20:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:20:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:20:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:20:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:20:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:20:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:20:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:20:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:20:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:20:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:20:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:20:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:20:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:20:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:20:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:20:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:20:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:20:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:20:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:20:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:20:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:20:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:20:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:20:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:20:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:20:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:20:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:20:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:20:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:20:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:20:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:20:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:20:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:20:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:20:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:20:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:20:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:20:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:20:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:20:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:20:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:20:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:20:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:20:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:20:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:20:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:20:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:20:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:20:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:20:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:20:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:20:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:20:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:20:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:20:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:20:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:20:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:21:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:21:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:21:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:21:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:21:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:21:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:21:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:21:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:21:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:21:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:21:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:21:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:21:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:21:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:21:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:21:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:21:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:21:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:21:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:21:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:21:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:21:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:21:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:21:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:21:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:21:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:21:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:21:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:21:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:21:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:21:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:21:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:21:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:21:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:21:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:21:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:23:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:23:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:23:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:23:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:23:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:23:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:23:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:23:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:23:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:23:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:23:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:23:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:23:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:23:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:23:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:23:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:23:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:23:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:23:12 rdma06 rsyslogd[1041]: imjournal: 5082 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 06:23:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:23:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:23:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:23:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:23:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:23:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:23:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:23:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:23:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:23:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:23:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:23:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:23:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:23:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:23:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:23:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:23:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:23:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:23:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:23:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:23:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:23:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:23:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:23:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:23:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:23:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:23:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:23:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:23:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:23:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:23:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:23:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:23:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:23:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:23:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:23:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:23:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:23:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:23:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:23:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:23:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:23:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:23:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:23:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:23:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:23:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:23:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:23:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:23:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:23:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:23:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:23:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:23:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:23:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:23:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:23:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:23:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:23:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:23:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:23:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:23:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:23:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:23:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:23:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:23:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:23:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:23:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:23:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:23:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:23:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:23:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:23:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:23:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:23:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:23:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:23:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:23:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:23:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:23:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:23:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:23:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:23:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:23:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:23:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:23:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:23:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:23:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:23:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:23:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:23:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:23:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:23:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:23:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:23:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:23:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:23:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:23:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:23:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:23:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:23:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:23:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:23:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:23:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:23:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:23:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:23:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:23:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:23:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:23:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:23:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:23:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:23:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:23:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:23:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:23:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:23:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:23:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:23:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:23:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:23:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:23:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:23:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:23:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:23:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:23:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:23:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:23:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:23:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:23:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:23:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:23:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:23:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:23:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:23:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:23:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:23:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:23:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:23:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:23:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:23:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:23:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:23:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:23:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:23:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:23:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:23:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:23:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:23:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:23:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:23:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:23:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:23:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:23:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:23:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:23:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:23:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:23:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:23:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:23:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:23:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:23:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:23:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:24:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:24:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:24:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:24:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:24:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:24:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:24:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:24:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:24:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:24:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:24:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:24:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:24:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:24:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:24:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:24:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:24:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:24:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:24:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:24:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:24:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:24:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:24:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:24:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:24:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:24:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:24:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:24:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:24:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:24:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:24:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:24:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:24:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:24:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:24:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:24:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:24:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:24:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:24:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:24:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:24:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:24:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:24:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:24:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:24:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:24:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:24:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:24:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:24:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:24:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:24:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:24:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:24:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:24:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:24:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:24:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:24:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:24:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:24:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:24:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:24:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:24:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:24:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:24:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:24:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:24:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:24:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:24:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:24:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:24:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:24:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:24:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:24:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:24:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:24:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:24:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:24:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:24:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:24:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:24:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:24:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:24:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:24:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:24:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:24:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:24:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:24:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:24:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:24:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:24:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:24:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:24:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:24:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:24:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:24:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:24:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:24:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:24:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:24:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:24:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:24:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:24:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:24:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:24:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:24:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:24:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:24:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:24:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:24:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:24:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:24:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:24:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:24:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:24:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:24:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:24:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:24:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:24:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:24:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:24:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:24:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:24:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:24:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:24:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:24:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:24:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:24:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:24:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:24:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:24:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:24:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:24:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:24:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:24:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:24:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:24:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:24:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:24:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:24:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:24:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:24:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:24:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:24:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:24:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:24:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:24:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:24:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:24:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:24:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:24:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:24:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:24:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:24:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:24:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:24:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:24:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:24:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:24:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:24:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:24:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:24:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:24:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:24:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:24:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:24:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:24:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:24:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:24:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:24:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:24:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:24:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:24:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:24:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:24:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:24:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:24:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:24:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:24:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:24:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:24:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:24:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:24:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:24:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:24:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:24:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:24:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:24:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:24:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:24:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:24:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:24:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:24:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:24:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:24:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:24:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:24:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:24:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:24:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:24:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:24:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:24:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:24:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:24:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:24:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:24:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:24:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:24:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:24:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:24:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:24:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:24:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:24:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:24:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:24:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:24:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:24:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:25:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:25:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:25:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:25:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:25:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:25:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:25:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:25:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:25:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:25:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:25:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:25:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:25:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:25:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:25:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:25:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:25:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:25:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:25:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:25:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:25:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:25:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:25:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:25:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:25:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:25:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:25:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:25:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:25:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:25:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:25:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:25:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:25:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:25:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:25:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:25:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:25:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:25:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:25:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:25:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:25:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:25:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:25:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:25:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:25:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:25:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:25:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:25:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:25:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:25:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:25:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:25:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:25:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:25:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:25:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:25:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:25:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:25:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:25:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:25:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:25:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:25:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:25:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:25:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:25:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:25:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:25:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:25:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:25:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:25:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:25:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:25:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:25:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:25:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:25:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:25:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:25:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:25:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:25:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:25:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:25:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:25:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:25:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:25:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:25:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:25:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:25:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:25:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:25:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:25:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:25:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:25:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:25:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:25:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:25:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:25:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:25:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:25:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:25:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:25:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:25:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:25:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:25:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:25:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:25:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:25:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:25:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:25:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:25:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:25:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:25:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:25:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:25:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:25:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:25:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:25:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:25:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:25:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:25:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:25:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:25:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:25:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:25:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:25:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:25:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:25:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:25:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:25:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:25:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:25:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:25:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:25:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:25:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:25:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:25:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:25:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:25:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:25:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:25:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:25:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:25:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:25:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:25:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:25:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:25:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:25:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:25:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:25:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:25:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:25:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:25:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:25:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:25:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:25:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:25:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:25:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:25:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:25:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:25:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:25:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:25:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:25:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:25:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:25:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:25:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:25:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:25:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:25:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:25:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:25:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:25:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:25:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:25:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:25:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:25:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:25:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:25:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:25:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:25:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:25:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:25:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:25:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:25:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:25:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:25:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:25:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:25:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:25:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:25:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:25:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:25:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:25:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:25:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:25:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:25:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:25:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:25:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:25:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:25:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:25:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:25:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:25:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:25:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:25:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:25:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:25:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:25:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:25:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:25:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:25:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:25:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:25:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:25:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:25:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:25:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:25:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:26:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:26:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:26:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:26:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:26:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:26:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:26:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:26:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:26:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:26:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:26:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:26:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:26:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:26:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:26:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:26:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:26:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:26:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:26:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:26:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:26:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:26:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:26:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:26:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:26:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:26:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:26:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:26:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:26:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:26:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:26:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:26:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:26:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:26:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:26:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:26:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:26:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:26:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:26:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:26:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:26:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:26:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:26:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:26:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:26:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:26:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:26:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:26:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:26:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:26:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:26:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:26:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:26:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:26:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:26:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:26:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:26:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:26:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:26:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:26:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:26:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:26:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:26:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:26:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:26:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:26:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:26:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:26:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:26:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:26:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:26:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:26:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:26:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:26:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:26:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:26:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:26:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:26:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:26:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:26:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:26:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:26:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:26:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:26:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:26:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:26:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:26:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:26:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:26:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:26:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:26:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:26:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:26:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:26:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:26:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:26:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:26:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:26:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:26:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:26:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:26:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:26:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:26:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:26:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:26:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:26:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:26:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:26:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:26:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:26:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:26:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:26:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:26:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:26:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:26:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:26:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:26:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:26:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:26:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:26:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:26:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:26:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:26:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:26:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:26:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:26:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:26:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:26:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:26:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:26:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:26:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:26:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:26:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:26:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:26:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:26:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:26:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:26:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:26:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:26:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:26:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:26:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:26:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:26:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:26:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:26:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:26:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:26:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:26:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:26:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:26:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:26:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:26:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:26:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:26:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:26:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:26:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:26:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:26:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:26:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:26:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:26:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:26:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:26:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:26:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:26:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:26:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:26:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:26:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:26:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:26:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:26:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:26:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:26:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:26:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:26:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:26:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:26:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:26:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:26:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:26:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:26:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:26:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:26:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:26:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:26:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:26:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:26:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:26:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:26:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:26:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:26:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:26:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:26:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:26:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:26:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:26:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:26:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:26:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:26:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:26:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:26:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:26:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:26:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:26:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:26:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:26:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:26:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:26:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:26:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:26:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:26:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:26:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:26:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:26:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:26:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:27:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:27:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:27:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:27:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:27:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:27:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:27:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:27:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:27:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:27:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:27:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:27:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:27:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:27:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:27:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:27:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:27:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:27:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:27:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:27:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:27:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:27:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:27:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:27:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:27:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:27:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:27:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:27:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:27:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:27:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:27:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:27:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:27:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:27:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:27:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:27:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:27:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:27:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:27:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:27:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:27:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:27:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:27:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:27:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:27:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:27:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:27:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:27:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:27:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:27:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:27:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:27:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:27:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:27:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:27:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:27:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:27:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:27:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:27:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:27:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:27:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:27:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:27:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:27:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:27:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:27:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:27:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:27:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:27:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:27:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:27:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:27:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:27:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:27:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:27:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:27:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:27:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:27:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:27:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:27:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:27:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:27:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:27:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:27:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:27:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:27:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:27:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:27:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:27:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:27:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:27:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:27:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:27:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:27:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:27:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:27:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:27:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:27:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:27:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:27:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:27:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:27:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:27:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:27:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:27:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:27:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:27:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:27:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:27:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:27:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:27:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:27:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:27:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:27:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:27:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:27:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:27:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:27:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:27:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:27:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:27:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:27:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:27:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:27:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:27:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:27:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:27:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:27:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:27:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:27:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:27:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:27:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:27:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:27:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:27:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:27:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:27:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:27:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:27:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:27:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:27:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:27:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:27:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:27:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:27:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:27:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:27:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:27:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:27:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:27:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:27:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:27:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:27:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:27:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:27:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:27:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:27:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:27:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:27:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:27:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:27:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:27:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:27:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:27:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:27:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:27:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:27:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:27:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:27:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:27:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:27:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:27:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:27:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:27:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:27:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:27:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:27:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:27:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:27:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:27:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:27:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:27:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:27:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:27:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:27:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:27:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:27:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:27:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:27:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:27:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:27:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:27:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:27:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:27:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:27:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:27:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:27:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:27:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:27:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:27:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:27:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:27:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:27:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:27:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:27:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:27:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:27:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:27:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:27:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:27:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:27:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:27:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:27:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:27:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:27:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:27:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:28:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:28:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:28:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:28:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:28:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:28:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:28:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:28:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:28:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:28:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:28:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:28:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:28:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:28:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:28:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:28:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:28:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:28:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:28:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:28:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:28:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:28:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:28:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:28:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:28:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:28:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:28:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:28:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:28:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:28:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:28:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:28:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:28:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:28:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:28:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:28:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:28:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:28:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:28:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:28:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:28:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:28:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:28:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:28:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:28:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:28:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:28:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:28:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:28:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:28:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:28:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:28:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:28:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:28:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:28:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:28:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:28:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:28:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:28:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:28:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:28:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:28:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:28:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:28:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:28:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:28:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:28:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:28:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:28:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:28:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:28:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:28:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:28:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:28:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:28:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:28:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:28:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:28:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:28:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:28:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:28:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:28:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:28:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:28:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:28:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:28:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:28:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:28:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:28:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:28:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:28:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:28:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:28:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:28:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:28:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:28:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:28:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:28:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:28:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:28:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:28:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:28:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:28:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:28:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:28:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:28:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:28:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:28:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:28:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:28:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:28:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:28:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:28:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:28:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:28:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:28:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:28:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:28:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:28:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:28:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:28:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:28:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:28:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:28:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:28:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:28:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:28:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:28:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:28:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:28:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:28:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:28:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:28:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:28:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:28:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:28:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:28:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:28:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:28:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:28:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:28:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:28:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:28:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:28:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:28:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:28:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:28:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:28:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:28:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:28:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:28:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:28:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:28:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:28:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:28:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:28:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:28:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:28:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:28:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:28:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:28:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:28:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:28:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:28:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:28:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:28:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:28:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:28:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:28:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:28:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:28:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:28:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:28:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:28:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:28:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:28:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:28:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:28:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:28:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:28:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:28:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:28:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:28:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:28:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:28:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:28:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:28:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:28:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:28:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:28:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:28:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:28:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:28:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:28:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:28:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:28:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:28:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:28:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:28:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:28:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:28:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:28:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:28:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:28:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:28:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:28:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:28:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:28:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:28:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:28:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:28:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:28:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:28:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:28:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:28:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:28:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:29:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:29:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:29:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:29:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:29:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:29:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:29:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:29:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:29:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:29:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:29:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:29:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:29:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:29:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:29:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:29:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:29:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:29:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:29:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:29:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:29:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:29:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:29:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:29:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:29:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:29:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:29:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:29:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:29:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:29:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:29:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:29:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:29:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:29:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:29:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:29:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:29:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:29:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:29:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:29:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:29:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:29:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:29:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:29:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:29:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:29:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:29:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:29:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:29:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:29:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:29:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:29:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:29:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:29:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:29:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:29:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:29:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:29:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:29:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:29:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:29:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:29:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:29:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:29:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:29:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:29:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:29:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:29:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:29:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:29:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:29:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:29:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:29:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:29:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:29:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:29:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:29:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:29:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:29:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:29:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:29:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:29:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:29:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:29:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:29:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:29:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:29:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:29:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:29:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:29:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:29:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:29:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:29:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:29:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:29:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:29:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:29:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:29:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:29:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:29:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:29:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:29:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:29:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:29:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:29:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:29:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:29:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:29:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:29:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:29:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:29:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:29:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:29:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:29:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:29:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:29:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:29:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:29:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:29:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:29:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:29:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:29:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:29:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:29:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:29:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:29:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:29:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:29:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:29:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:29:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:29:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:29:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:29:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:29:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:29:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:29:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:29:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:29:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:29:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:29:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:29:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:29:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:29:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:29:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:29:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:29:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:29:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:29:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:29:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:29:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:29:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:29:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:29:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:29:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:29:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:29:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:29:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:29:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:29:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:29:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:29:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:29:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:29:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:29:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:29:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:29:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:29:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:29:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:29:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:29:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:29:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:29:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:29:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:29:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:29:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:29:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:29:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:29:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:29:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:29:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:29:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:29:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:29:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:29:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:29:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:29:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:29:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:29:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:29:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:29:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:29:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:29:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:29:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:29:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:29:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:29:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:29:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:29:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:29:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:29:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:29:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:29:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:29:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:29:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:29:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:29:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:29:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:29:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:29:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:29:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:29:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:29:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:29:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:29:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:29:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:29:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:30:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:30:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:30:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:30:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:30:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:30:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:30:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:30:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:30:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:30:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:30:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:30:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:30:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:30:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:30:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:30:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:30:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:30:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:30:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:30:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:30:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:30:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:30:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:30:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:30:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:30:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:30:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:30:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:30:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:30:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:30:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:30:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:30:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:30:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:30:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:30:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:30:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:30:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:30:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:30:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:30:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:30:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:30:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:30:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:30:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:30:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:30:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:30:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:30:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:30:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:30:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:30:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:30:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:30:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:30:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:30:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:30:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:30:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:30:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:30:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:30:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:30:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:30:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:30:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:30:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:30:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:30:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:30:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:30:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:30:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:30:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:30:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:30:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:30:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:30:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:30:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:30:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:30:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:30:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:30:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:30:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:30:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:30:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:30:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:30:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:30:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:30:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:30:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:30:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:30:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:30:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:30:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:30:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:30:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:30:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:30:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:30:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:30:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:30:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:30:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:30:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:30:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:30:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:30:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:30:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:30:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:30:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:30:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:30:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:30:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:30:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:30:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:30:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:30:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:30:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:30:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:30:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:30:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:30:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:30:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:30:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:30:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:30:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:30:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:30:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:30:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:30:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:30:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:30:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:30:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:30:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:30:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:30:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:30:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:30:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:30:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:30:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:30:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:30:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:30:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:30:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:30:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:30:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:30:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:30:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:30:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:30:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:30:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:30:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:30:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:30:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:30:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:30:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:30:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:30:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:30:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:30:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:30:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:30:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:30:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:30:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:30:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:30:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:30:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:30:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:30:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:30:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:30:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:30:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:30:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:30:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:30:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:30:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:30:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:30:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:30:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:30:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:30:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:30:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:30:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:30:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:30:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:30:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:30:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:30:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:30:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:30:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:30:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:30:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:30:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:30:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:30:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:30:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:30:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:30:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:30:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:30:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:30:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:30:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:30:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:30:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:30:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:30:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:30:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:30:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:30:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:30:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:30:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:30:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:30:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:30:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:30:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:30:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:30:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:30:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:30:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:31:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:31:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:31:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:31:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:31:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:31:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:31:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:31:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:31:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:31:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:31:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:31:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:31:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:31:01 rdma06 systemd-journald[790]: Data hash table of /run/log/journal/95379d2f723847b68d9e976dee8967a9/system.journal has a fill level at 75.0 (163751 of 218332 items, 75497472 file size, 461 bytes per hash table item), suggesting rotation.
Apr 18 06:31:01 rdma06 systemd-journald[790]: /run/log/journal/95379d2f723847b68d9e976dee8967a9/system.journal: Journal header limits reached or header out-of-date, rotating.
Apr 18 06:31:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:31:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:31:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:31:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:31:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:31:01 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 18 06:31:01 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 18 06:31:01 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 18 06:31:02 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 18 06:31:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:31:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:31:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:31:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:31:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:31:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:31:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:31:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:31:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:31:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:31:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:31:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:31:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:31:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:31:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:31:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:31:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:31:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:33:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:33:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:33:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:33:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:33:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:33:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:33:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:33:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:33:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:33:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:33:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:33:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:33:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:33:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:33:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:33:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:33:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:33:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:33:17 rdma06 rsyslogd[1041]: imjournal: 5297 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 06:33:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:33:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:33:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:33:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:33:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:33:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:33:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:33:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:33:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:33:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:33:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:33:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:33:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:33:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:33:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:33:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:33:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:33:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:33:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:33:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:33:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:33:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:33:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:33:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:33:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:33:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:33:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:33:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:33:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:33:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:33:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:33:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:33:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:33:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:33:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:33:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:33:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:33:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:33:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:33:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:33:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:33:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:33:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:33:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:33:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:33:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:33:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:33:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:33:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:33:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:33:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:33:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:33:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:33:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:33:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:33:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:33:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:33:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:33:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:33:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:33:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:33:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:33:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:33:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:33:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:33:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:33:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:33:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:33:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:33:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:33:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:33:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:33:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:33:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:33:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:33:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:33:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:33:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:33:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:33:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:33:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:33:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:33:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:33:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:33:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:33:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:33:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:33:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:33:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:33:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:33:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:33:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:33:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:33:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:33:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:33:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:33:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:33:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:33:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:33:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:33:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:33:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:33:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:33:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:33:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:33:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:33:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:33:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:33:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:33:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:33:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:33:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:33:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:33:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:33:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:33:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:33:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:33:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:33:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:33:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:33:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:33:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:33:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:33:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:33:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:33:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:33:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:33:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:33:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:33:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:33:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:33:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:33:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:33:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:33:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:33:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:33:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:33:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:33:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:33:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:33:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:33:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:33:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:33:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:34:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:34:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:34:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:34:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:34:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:34:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:34:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:34:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:34:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:34:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:34:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:34:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:34:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:34:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:34:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:34:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:34:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:34:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:34:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:34:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:34:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:34:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:34:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:34:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:34:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:34:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:34:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:34:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:34:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:34:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:34:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:34:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:34:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:34:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:34:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:34:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:34:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:34:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:34:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:34:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:34:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:34:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:34:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:34:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:34:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:34:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:34:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:34:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:34:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:34:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:34:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:34:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:34:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:34:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:34:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:34:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:34:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:34:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:34:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:34:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:34:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:34:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:34:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:34:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:34:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:34:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:34:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:34:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:34:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:34:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:34:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:34:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:34:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:34:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:34:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:34:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:34:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:34:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:34:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:34:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:34:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:34:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:34:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:34:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:34:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:34:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:34:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:34:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:34:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:34:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:34:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:34:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:34:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:34:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:34:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:34:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:34:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:34:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:34:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:34:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:34:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:34:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:34:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:34:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:34:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:34:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:34:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:34:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:34:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:34:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:34:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:34:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:34:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:34:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:34:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:34:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:34:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:34:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:34:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:34:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:34:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:34:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:34:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:34:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:34:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:34:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:34:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:34:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:34:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:34:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:34:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:34:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:34:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:34:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:34:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:34:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:34:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:34:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:34:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:34:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:34:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:34:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:34:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:34:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:34:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:34:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:34:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:34:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:34:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:34:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:34:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:34:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:34:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:34:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:34:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:34:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:34:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:34:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:34:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:34:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:34:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:34:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:34:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:34:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:34:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:34:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:34:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:34:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:34:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:34:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:34:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:34:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:34:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:34:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:34:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:34:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:34:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:34:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:34:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:34:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:34:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:34:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:34:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:34:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:34:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:34:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:34:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:34:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:34:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:34:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:34:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:34:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:34:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:34:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:34:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:34:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:34:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:34:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:34:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:34:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:34:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:34:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:34:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:34:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:34:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:34:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:34:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:34:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:34:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:34:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:34:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:34:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:34:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:34:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:34:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:34:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:35:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:35:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:35:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:35:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:35:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:35:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:35:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:35:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:35:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:35:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:35:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:35:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:35:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:35:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:35:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:35:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:35:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:35:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:35:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:35:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:35:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:35:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:35:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:35:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:35:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:35:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:35:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:35:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:35:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:35:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:35:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:35:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:35:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:35:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:35:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:35:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:35:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:35:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:35:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:35:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:35:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:35:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:35:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:35:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:35:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:35:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:35:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:35:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:35:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:35:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:35:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:35:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:35:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:35:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:35:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:35:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:35:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:35:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:35:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:35:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:35:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:35:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:35:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:35:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:35:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:35:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:35:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:35:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:35:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:35:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:35:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:35:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:35:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:35:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:35:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:35:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:35:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:35:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:35:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:35:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:35:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:35:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:35:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:35:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:35:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:35:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:35:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:35:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:35:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:35:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:35:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:35:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:35:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:35:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:35:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:35:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:35:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:35:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:35:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:35:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:35:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:35:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:35:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:35:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:35:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:35:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:35:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:35:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:35:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:35:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:35:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:35:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:35:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:35:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:35:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:35:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:35:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:35:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:35:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:35:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:35:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:35:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:35:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:35:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:35:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:35:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:35:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:35:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:35:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:35:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:35:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:35:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:35:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:35:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:35:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:35:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:35:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:35:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:35:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:35:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:35:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:35:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:35:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:35:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:35:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:35:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:35:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:35:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:35:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:35:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:35:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:35:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:35:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:35:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:35:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:35:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:35:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:35:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:35:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:35:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:35:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:35:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:35:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:35:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:35:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:35:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:35:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:35:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:35:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:35:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:35:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:35:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:35:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:35:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:35:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:35:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:35:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:35:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:35:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:35:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:35:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:35:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:35:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:35:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:35:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:35:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:35:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:35:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:35:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:35:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:35:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:35:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:35:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:35:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:35:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:35:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:35:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:35:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:35:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:35:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:35:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:35:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:35:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:35:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:35:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:35:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:35:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:35:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:35:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:35:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:35:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:35:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:35:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:35:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:35:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:35:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:36:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:36:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:36:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:36:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:36:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:36:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:36:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:36:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:36:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:36:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:36:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:36:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:36:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:36:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:36:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:36:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:36:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:36:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:36:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:36:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:36:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:36:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:36:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:36:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:36:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:36:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:36:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:36:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:36:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:36:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:36:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:36:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:36:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:36:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:36:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:36:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:36:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:36:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:36:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:36:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:36:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:36:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:36:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:36:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:36:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:36:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:36:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:36:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:36:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:36:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:36:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:36:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:36:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:36:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:36:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:36:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:36:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:36:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:36:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:36:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:36:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:36:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:36:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:36:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:36:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:36:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:36:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:36:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:36:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:36:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:36:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:36:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:36:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:36:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:36:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:36:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:36:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:36:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:36:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:36:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:36:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:36:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:36:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:36:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:36:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:36:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:36:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:36:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:36:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:36:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:36:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:36:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:36:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:36:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:36:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:36:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:36:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:36:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:36:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:36:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:36:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:36:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:36:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:36:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:36:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:36:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:36:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:36:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:36:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:36:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:36:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:36:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:36:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:36:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:36:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:36:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:36:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:36:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:36:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:36:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:36:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:36:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:36:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:36:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:36:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:36:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:36:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:36:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:36:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:36:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:36:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:36:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:36:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:36:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:36:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:36:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:36:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:36:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:36:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:36:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:36:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:36:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:36:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:36:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:36:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:36:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:36:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:36:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:36:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:36:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:36:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:36:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:36:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:36:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:36:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:36:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:36:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:36:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:36:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:36:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:36:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:36:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:36:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:36:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:36:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:36:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:36:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:36:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:36:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:36:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:36:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:36:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:36:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:36:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:36:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:36:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:36:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:36:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:36:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:36:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:36:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:36:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:36:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:36:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:36:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:36:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:36:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:36:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:36:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:36:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:36:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:36:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:36:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:36:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:36:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:36:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:36:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:36:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:36:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:36:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:36:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:36:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:36:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:36:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:36:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:36:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:36:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:36:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:36:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:36:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:36:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:36:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:36:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:36:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:36:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:36:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:37:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:37:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:37:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:37:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:37:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:37:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:37:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:37:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:37:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:37:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:37:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:37:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:37:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:37:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:37:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:37:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:37:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:37:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:37:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:37:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:37:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:37:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:37:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:37:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:37:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:37:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:37:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:37:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:37:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:37:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:37:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:37:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:37:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:37:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:37:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:37:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:37:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:37:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:37:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:37:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:37:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:37:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:37:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:37:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:37:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:37:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:37:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:37:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:37:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:37:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:37:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:37:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:37:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:37:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:37:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:37:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:37:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:37:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:37:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:37:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:37:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:37:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:37:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:37:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:37:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:37:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:37:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:37:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:37:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:37:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:37:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:37:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:37:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:37:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:37:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:37:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:37:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:37:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:37:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:37:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:37:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:37:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:37:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:37:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:37:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:37:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:37:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:37:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:37:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:37:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:37:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:37:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:37:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:37:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:37:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:37:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:37:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:37:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:37:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:37:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:37:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:37:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:37:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:37:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:37:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:37:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:37:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:37:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:37:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:37:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:37:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:37:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:37:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:37:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:37:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:37:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:37:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:37:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:37:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:37:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:37:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:37:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:37:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:37:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:37:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:37:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:37:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:37:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:37:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:37:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:37:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:37:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:37:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:37:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:37:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:37:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:37:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:37:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:37:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:37:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:37:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:37:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:37:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:37:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:37:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:37:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:37:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:37:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:37:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:37:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:37:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:37:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:37:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:37:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:37:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:37:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:37:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:37:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:37:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:37:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:37:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:37:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:37:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:37:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:37:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:37:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:37:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:37:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:37:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:37:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:37:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:37:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:37:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:37:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:37:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:37:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:37:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:37:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:37:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:37:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:37:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:37:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:37:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:37:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:37:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:37:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:37:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:37:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:37:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:37:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:37:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:37:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:37:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:37:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:37:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:37:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:37:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:37:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:37:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:37:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:37:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:37:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:37:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:37:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:37:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:37:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:37:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:37:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:37:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:37:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:37:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:37:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:37:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:37:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:37:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:37:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:38:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:38:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:38:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:38:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:38:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:38:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:38:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:38:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:38:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:38:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:38:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:38:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:38:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:38:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:38:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:38:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:38:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:38:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:38:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:38:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:38:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:38:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:38:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:38:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:38:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:38:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:38:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:38:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:38:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:38:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:38:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:38:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:38:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:38:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:38:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:38:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:38:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:38:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:38:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:38:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:38:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:38:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:38:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:38:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:38:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:38:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:38:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:38:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:38:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:38:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:38:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:38:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:38:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:38:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:38:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:38:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:38:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:38:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:38:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:38:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:38:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:38:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:38:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:38:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:38:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:38:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:38:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:38:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:38:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:38:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:38:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:38:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:38:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:38:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:38:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:38:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:38:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:38:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:38:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:38:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:38:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:38:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:38:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:38:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:38:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:38:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:38:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:38:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:38:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:38:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:38:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:38:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:38:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:38:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:38:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:38:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:38:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:38:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:38:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:38:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:38:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:38:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:38:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:38:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:38:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:38:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:38:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:38:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:38:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:38:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:38:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:38:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:38:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:38:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:38:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:38:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:38:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:38:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:38:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:38:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:38:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:38:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:38:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:38:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:38:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:38:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:38:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:38:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:38:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:38:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:38:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:38:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:38:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:38:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:38:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:38:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:38:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:38:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:38:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:38:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:38:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:38:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:38:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:38:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:38:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:38:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:38:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:38:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:38:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:38:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:38:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:38:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:38:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:38:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:38:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:38:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:38:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:38:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:38:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:38:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:38:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:38:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:38:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:38:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:38:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:38:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:38:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:38:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:38:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:38:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:38:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:38:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:38:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:38:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:38:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:38:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:38:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:38:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:38:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:38:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:38:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:38:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:38:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:38:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:38:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:38:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:38:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:38:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:38:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:38:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:38:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:38:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:38:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:38:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:38:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:38:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:38:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:38:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:38:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:38:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:38:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:38:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:38:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:38:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:38:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:38:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:38:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:38:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:38:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:38:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:38:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:38:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:38:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:38:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:38:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:38:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:39:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:39:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:39:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:39:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:39:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:39:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:39:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:39:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:39:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:39:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:39:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:39:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:39:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:39:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:39:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:39:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:39:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:39:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:39:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:39:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:39:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:39:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:39:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:39:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:39:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:39:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:39:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:39:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:39:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:39:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:39:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:39:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:39:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:39:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:39:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:39:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:39:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:39:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:39:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:39:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:39:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:39:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:39:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:39:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:39:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:39:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:39:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:39:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:39:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:39:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:39:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:39:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:39:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:39:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:39:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:39:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:39:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:39:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:39:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:39:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:39:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:39:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:39:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:39:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:39:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:39:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:39:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:39:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:39:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:39:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:39:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:39:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:39:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:39:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:39:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:39:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:39:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:39:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:39:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:39:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:39:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:39:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:39:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:39:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:39:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:39:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:39:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:39:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:39:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:39:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:39:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:39:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:39:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:39:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:39:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:39:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:39:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:39:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:39:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:39:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:39:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:39:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:39:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:39:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:39:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:39:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:39:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:39:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:39:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:39:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:39:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:39:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:39:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:39:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:39:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:39:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:39:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:39:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:39:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:39:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:39:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:39:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:39:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:39:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:39:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:39:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:39:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:39:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:39:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:39:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:39:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:39:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:39:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:39:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:39:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:39:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:39:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:39:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:39:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:39:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:39:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:39:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:39:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:39:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:39:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:39:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:39:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:39:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:39:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:39:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:39:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:39:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:39:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:39:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:39:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:39:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:39:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:39:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:39:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:39:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:39:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:39:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:39:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:39:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:39:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:39:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:39:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:39:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:39:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:39:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:39:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:39:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:39:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:39:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:39:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:39:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:39:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:39:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:39:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:39:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:39:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:39:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:39:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:39:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:39:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:39:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:39:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:39:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:39:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:39:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:39:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:39:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:39:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:39:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:39:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:39:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:39:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:39:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:39:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:39:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:39:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:39:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:39:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:39:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:39:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:39:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:39:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:39:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:39:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:39:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:39:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:39:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:39:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:39:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:39:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:39:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:40:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:40:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:40:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:40:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:40:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:40:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:40:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:40:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:40:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:40:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:40:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:40:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:40:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:40:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:40:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:40:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:40:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:40:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:40:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:40:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:40:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:40:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:40:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:40:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:40:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:40:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:40:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:40:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:40:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:40:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:40:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:40:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:40:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:40:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:40:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:40:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:40:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:40:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:40:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:40:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:40:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:40:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:40:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:40:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:40:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:40:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:40:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:40:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:40:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:40:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:40:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:40:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:40:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:40:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:40:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:40:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:40:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:40:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:40:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:40:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:40:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:40:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:40:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:40:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:40:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:40:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:40:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:40:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:40:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:40:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:40:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:40:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:40:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:40:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:40:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:40:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:40:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:40:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:40:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:40:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:40:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:40:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:40:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:40:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:40:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:40:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:40:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:40:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:40:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:40:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:40:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:40:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:40:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:40:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:40:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:40:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:40:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:40:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:40:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:40:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:40:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:40:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:40:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:40:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:40:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:40:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:40:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:40:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:40:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:40:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:40:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:40:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:40:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:40:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:40:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:40:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:40:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:40:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:40:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:40:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:40:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:40:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:40:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:40:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:40:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:40:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:40:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:40:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:40:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:40:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:40:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:40:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:40:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:40:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:40:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:40:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:40:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:40:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:40:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:40:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:40:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:40:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:40:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:40:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:40:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:40:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:40:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:40:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:40:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:40:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:40:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:40:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:40:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:40:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:40:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:40:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:40:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:40:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:40:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:40:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:40:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:40:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:40:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:40:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:40:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:40:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:40:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:40:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:40:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:40:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:40:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:40:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:40:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:40:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:40:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:40:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:40:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:40:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:40:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:40:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:40:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:40:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:40:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:40:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:40:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:40:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:40:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:40:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:40:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:40:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:40:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:40:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:40:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:40:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:40:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:40:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:40:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:40:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:40:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:40:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:40:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:40:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:40:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:40:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:40:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:40:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:40:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:40:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:40:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:40:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:40:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:40:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:40:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:40:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:40:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:40:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:41:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:41:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:41:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:41:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:41:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:41:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:41:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:41:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:41:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:41:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:41:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:41:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:41:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:41:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:41:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:41:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:41:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:41:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:41:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:41:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:41:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:41:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:41:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:41:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:41:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:41:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:41:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:41:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:41:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:41:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:41:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:41:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:41:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:41:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:41:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:41:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:41:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:41:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:41:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:41:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:41:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:41:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:41:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:41:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:41:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:41:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:41:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:41:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:41:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:41:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:41:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:41:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:41:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:41:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:43:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:43:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:43:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:43:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:43:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:43:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:43:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:43:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:43:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:43:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:43:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:43:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:43:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:43:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:43:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:43:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:43:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:43:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:43:22 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 06:43:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:43:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:43:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:43:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:43:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:43:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:43:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:43:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:43:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:43:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:43:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:43:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:43:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:43:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:43:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:43:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:43:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:43:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:43:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:43:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:43:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:43:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:43:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:43:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:43:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:43:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:43:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:43:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:43:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:43:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:43:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:43:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:43:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:43:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:43:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:43:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:43:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:43:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:43:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:43:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:43:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:43:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:43:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:43:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:43:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:43:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:43:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:43:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:43:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:43:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:43:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:43:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:43:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:43:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:43:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:43:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:43:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:43:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:43:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:43:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:43:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:43:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:43:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:43:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:43:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:43:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:43:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:43:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:43:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:43:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:43:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:43:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:43:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:43:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:43:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:43:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:43:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:43:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:43:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:43:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:43:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:43:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:43:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:43:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:43:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:43:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:43:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:43:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:43:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:43:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:43:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:43:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:43:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:43:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:43:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:43:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:43:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:43:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:43:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:43:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:43:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:43:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:43:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:43:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:43:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:43:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:43:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:43:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:43:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:43:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:43:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:43:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:43:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:43:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:43:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:43:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:43:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:43:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:43:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:43:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:43:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:43:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:43:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:43:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:43:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:43:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:44:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:44:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:44:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:44:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:44:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:44:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:44:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:44:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:44:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:44:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:44:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:44:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:44:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:44:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:44:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:44:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:44:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:44:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:44:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:44:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:44:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:44:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:44:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:44:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:44:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:44:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:44:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:44:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:44:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:44:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:44:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:44:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:44:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:44:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:44:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:44:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:44:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:44:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:44:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:44:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:44:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:44:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:44:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:44:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:44:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:44:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:44:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:44:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:44:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:44:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:44:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:44:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:44:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:44:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:44:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:44:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:44:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:44:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:44:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:44:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:44:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:44:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:44:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:44:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:44:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:44:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:44:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:44:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:44:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:44:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:44:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:44:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:44:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:44:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:44:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:44:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:44:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:44:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:44:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:44:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:44:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:44:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:44:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:44:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:44:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:44:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:44:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:44:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:44:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:44:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:44:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:44:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:44:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:44:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:44:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:44:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:44:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:44:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:44:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:44:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:44:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:44:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:44:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:44:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:44:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:44:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:44:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:44:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:44:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:44:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:44:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:44:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:44:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:44:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:44:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:44:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:44:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:44:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:44:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:44:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:44:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:44:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:44:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:44:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:44:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:44:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:44:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:44:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:44:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:44:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:44:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:44:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:44:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:44:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:44:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:44:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:44:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:44:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:44:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:44:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:44:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:44:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:44:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:44:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:44:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:44:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:44:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:44:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:44:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:44:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:44:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:44:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:44:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:44:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:44:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:44:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:44:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:44:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:44:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:44:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:44:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:44:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:44:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:44:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:44:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:44:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:44:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:44:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:44:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:44:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:44:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:44:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:44:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:44:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:44:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:44:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:44:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:44:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:44:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:44:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:44:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:44:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:44:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:44:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:44:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:44:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:44:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:44:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:44:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:44:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:44:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:44:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:44:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:44:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:44:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:44:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:44:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:44:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:44:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:44:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:44:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:44:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:44:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:44:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:44:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:44:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:44:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:44:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:44:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:44:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:44:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:44:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:44:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:44:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:44:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:44:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:45:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:45:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:45:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:45:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:45:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:45:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:45:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:45:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:45:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:45:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:45:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:45:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:45:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:45:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:45:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:45:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:45:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:45:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:45:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:45:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:45:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:45:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:45:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:45:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:45:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:45:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:45:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:45:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:45:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:45:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:45:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:45:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:45:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:45:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:45:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:45:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:45:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:45:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:45:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:45:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:45:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:45:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:45:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:45:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:45:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:45:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:45:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:45:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:45:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:45:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:45:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:45:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:45:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:45:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:45:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:45:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:45:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:45:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:45:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:45:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:45:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:45:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:45:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:45:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:45:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:45:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:45:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:45:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:45:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:45:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:45:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:45:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:45:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:45:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:45:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:45:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:45:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:45:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:45:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:45:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:45:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:45:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:45:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:45:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:45:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:45:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:45:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:45:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:45:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:45:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:45:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:45:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:45:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:45:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:45:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:45:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:45:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:45:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:45:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:45:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:45:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:45:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:45:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:45:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:45:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:45:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:45:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:45:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:45:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:45:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:45:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:45:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:45:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:45:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:45:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:45:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:45:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:45:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:45:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:45:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:45:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:45:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:45:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:45:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:45:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:45:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:45:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:45:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:45:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:45:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:45:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:45:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:45:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:45:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:45:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:45:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:45:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:45:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:45:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:45:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:45:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:45:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:45:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:45:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:45:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:45:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:45:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:45:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:45:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:45:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:45:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:45:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:45:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:45:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:45:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:45:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:45:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:45:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:45:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:45:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:45:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:45:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:45:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:45:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:45:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:45:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:45:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:45:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:45:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:45:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:45:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:45:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:45:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:45:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:45:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:45:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:45:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:45:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:45:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:45:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:45:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:45:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:45:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:45:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:45:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:45:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:45:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:45:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:45:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:45:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:45:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:45:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:45:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:45:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:45:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:45:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:45:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:45:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:45:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:45:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:45:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:45:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:45:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:45:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:45:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:45:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:45:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:45:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:45:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:45:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:45:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:45:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:45:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:45:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:45:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:45:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:46:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:46:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:46:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:46:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:46:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:46:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:46:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:46:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:46:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:46:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:46:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:46:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:46:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:46:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:46:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:46:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:46:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:46:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:46:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:46:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:46:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:46:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:46:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:46:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:46:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:46:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:46:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:46:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:46:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:46:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:46:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:46:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:46:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:46:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:46:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:46:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:46:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:46:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:46:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:46:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:46:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:46:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:46:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:46:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:46:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:46:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:46:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:46:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:46:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:46:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:46:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:46:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:46:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:46:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:46:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:46:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:46:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:46:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:46:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:46:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:46:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:46:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:46:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:46:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:46:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:46:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:46:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:46:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:46:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:46:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:46:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:46:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:46:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:46:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:46:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:46:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:46:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:46:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:46:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:46:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:46:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:46:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:46:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:46:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:46:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:46:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:46:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:46:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:46:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:46:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:46:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:46:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:46:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:46:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:46:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:46:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:46:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:46:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:46:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:46:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:46:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:46:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:46:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:46:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:46:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:46:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:46:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:46:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:46:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:46:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:46:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:46:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:46:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:46:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:46:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:46:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:46:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:46:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:46:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:46:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:46:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:46:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:46:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:46:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:46:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:46:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:46:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:46:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:46:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:46:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:46:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:46:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:46:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:46:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:46:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:46:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:46:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:46:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:46:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:46:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:46:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:46:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:46:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:46:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:46:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:46:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:46:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:46:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:46:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:46:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:46:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:46:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:46:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:46:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:46:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:46:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:46:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:46:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:46:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:46:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:46:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:46:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:46:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:46:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:46:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:46:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:46:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:46:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:46:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:46:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:46:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:46:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:46:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:46:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:46:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:46:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:46:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:46:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:46:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:46:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:46:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:46:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:46:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:46:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:46:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:46:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:46:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:46:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:46:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:46:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:46:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:46:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:46:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:46:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:46:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:46:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:46:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:46:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:46:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:46:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:46:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:46:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:46:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:46:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:46:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:46:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:46:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:46:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:46:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:46:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:46:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:46:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:46:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:46:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:46:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:46:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:47:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:47:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:47:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:47:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:47:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:47:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:47:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:47:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:47:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:47:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:47:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:47:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:47:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:47:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:47:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:47:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:47:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:47:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:47:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:47:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:47:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:47:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:47:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:47:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:47:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:47:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:47:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:47:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:47:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:47:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:47:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:47:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:47:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:47:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:47:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:47:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:47:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:47:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:47:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:47:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:47:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:47:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:47:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:47:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:47:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:47:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:47:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:47:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:47:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:47:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:47:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:47:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:47:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:47:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:47:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:47:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:47:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:47:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:47:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:47:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:47:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:47:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:47:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:47:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:47:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:47:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:47:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:47:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:47:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:47:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:47:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:47:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:47:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:47:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:47:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:47:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:47:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:47:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:47:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:47:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:47:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:47:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:47:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:47:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:47:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:47:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:47:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:47:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:47:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:47:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:47:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:47:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:47:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:47:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:47:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:47:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:47:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:47:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:47:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:47:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:47:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:47:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:47:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:47:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:47:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:47:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:47:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:47:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:47:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:47:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:47:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:47:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:47:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:47:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:47:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:47:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:47:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:47:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:47:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:47:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:47:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:47:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:47:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:47:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:47:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:47:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:47:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:47:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:47:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:47:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:47:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:47:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:47:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:47:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:47:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:47:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:47:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:47:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:47:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:47:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:47:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:47:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:47:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:47:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:47:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:47:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:47:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:47:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:47:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:47:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:47:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:47:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:47:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:47:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:47:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:47:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:47:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:47:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:47:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:47:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:47:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:47:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:47:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:47:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:47:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:47:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:47:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:47:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:47:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:47:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:47:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:47:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:47:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:47:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:47:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:47:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:47:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:47:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:47:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:47:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:47:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:47:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:47:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:47:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:47:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:47:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:47:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:47:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:47:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:47:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:47:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:47:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:47:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:47:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:47:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:47:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:47:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:47:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:47:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:47:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:47:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:47:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:47:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:47:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:47:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:47:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:47:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:47:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:47:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:47:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:47:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:47:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:47:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:47:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:47:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:47:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:48:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:48:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:48:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:48:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:48:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:48:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:48:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:48:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:48:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:48:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:48:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:48:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:48:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:48:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:48:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:48:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:48:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:48:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:48:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:48:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:48:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:48:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:48:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:48:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:48:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:48:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:48:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:48:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:48:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:48:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:48:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:48:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:48:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:48:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:48:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:48:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:48:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:48:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:48:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:48:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:48:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:48:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:48:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:48:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:48:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:48:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:48:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:48:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:48:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:48:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:48:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:48:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:48:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:48:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:48:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:48:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:48:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:48:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:48:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:48:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:48:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:48:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:48:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:48:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:48:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:48:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:48:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:48:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:48:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:48:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:48:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:48:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:48:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:48:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:48:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:48:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:48:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:48:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:48:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:48:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:48:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:48:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:48:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:48:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:48:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:48:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:48:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:48:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:48:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:48:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:48:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:48:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:48:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:48:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:48:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:48:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:48:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:48:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:48:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:48:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:48:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:48:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:48:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:48:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:48:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:48:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:48:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:48:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:48:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:48:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:48:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:48:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:48:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:48:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:48:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:48:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:48:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:48:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:48:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:48:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:48:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:48:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:48:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:48:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:48:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:48:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:48:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:48:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:48:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:48:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:48:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:48:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:48:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:48:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:48:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:48:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:48:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:48:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:48:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:48:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:48:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:48:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:48:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:48:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:48:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:48:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:48:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:48:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:48:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:48:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:48:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:48:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:48:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:48:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:48:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:48:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:48:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:48:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:48:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:48:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:48:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:48:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:48:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:48:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:48:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:48:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:48:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:48:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:48:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:48:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:48:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:48:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:48:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:48:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:48:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:48:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:48:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:48:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:48:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:48:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:48:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:48:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:48:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:48:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:48:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:48:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:48:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:48:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:48:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:48:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:48:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:48:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:48:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:48:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:48:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:48:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:48:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:48:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:48:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:48:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:48:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:48:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:48:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:48:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:48:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:48:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:48:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:48:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:48:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:48:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:48:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:48:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:48:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:48:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:48:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:48:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:49:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:49:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:49:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:49:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:49:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:49:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:49:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:49:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:49:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:49:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:49:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:49:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:49:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:49:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:49:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:49:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:49:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:49:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:49:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:49:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:49:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:49:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:49:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:49:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:49:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:49:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:49:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:49:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:49:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:49:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:49:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:49:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:49:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:49:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:49:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:49:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:49:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:49:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:49:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:49:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:49:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:49:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:49:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:49:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:49:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:49:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:49:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:49:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:49:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:49:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:49:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:49:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:49:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:49:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:49:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:49:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:49:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:49:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:49:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:49:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:49:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:49:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:49:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:49:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:49:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:49:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:49:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:49:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:49:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:49:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:49:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:49:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:49:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:49:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:49:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:49:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:49:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:49:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:49:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:49:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:49:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:49:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:49:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:49:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:49:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:49:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:49:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:49:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:49:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:49:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:49:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:49:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:49:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:49:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:49:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:49:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:49:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:49:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:49:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:49:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:49:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:49:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:49:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:49:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:49:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:49:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:49:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:49:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:49:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:49:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:49:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:49:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:49:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:49:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:49:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:49:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:49:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:49:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:49:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:49:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:49:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:49:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:49:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:49:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:49:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:49:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:49:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:49:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:49:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:49:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:49:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:49:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:49:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:49:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:49:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:49:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:49:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:49:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:49:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:49:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:49:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:49:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:49:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:49:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:49:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:49:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:49:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:49:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:49:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:49:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:49:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:49:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:49:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:49:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:49:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:49:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:49:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:49:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:49:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:49:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:49:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:49:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:49:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:49:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:49:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:49:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:49:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:49:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:49:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:49:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:49:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:49:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:49:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:49:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:49:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:49:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:49:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:49:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:49:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:49:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:49:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:49:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:49:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:49:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:49:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:49:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:49:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:49:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:49:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:49:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:49:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:49:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:49:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:49:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:49:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:49:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:49:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:49:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:49:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:49:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:49:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:49:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:49:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:49:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:49:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:49:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:49:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:49:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:49:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:49:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:49:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:49:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:49:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:49:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:49:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:49:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:50:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:50:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:50:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:50:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:50:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:50:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:50:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:50:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:50:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:50:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:50:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:50:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:50:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:50:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:50:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:50:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:50:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:50:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:50:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:50:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:50:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:50:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:50:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:50:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:50:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:50:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:50:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:50:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:50:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:50:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:50:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:50:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:50:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:50:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:50:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:50:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:50:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:50:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:50:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:50:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:50:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:50:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:50:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:50:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:50:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:50:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:50:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:50:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:50:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:50:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:50:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:50:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:50:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:50:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:50:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:50:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:50:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:50:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:50:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:50:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:50:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:50:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:50:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:50:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:50:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:50:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:50:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:50:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:50:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:50:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:50:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:50:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:50:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:50:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:50:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:50:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:50:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:50:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:50:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:50:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:50:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:50:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:50:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:50:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:50:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:50:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:50:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:50:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:50:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:50:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:50:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:50:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:50:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:50:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:50:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:50:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:50:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:50:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:50:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:50:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:50:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:50:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:50:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:50:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:50:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:50:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:50:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:50:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:50:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:50:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:50:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:50:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:50:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:50:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:50:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:50:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:50:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:50:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:50:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:50:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:50:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:50:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:50:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:50:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:50:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:50:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:50:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:50:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:50:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:50:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:50:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:50:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:50:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:50:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:50:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:50:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:50:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:50:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:50:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:50:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:50:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:50:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:50:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:50:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:50:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:50:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:50:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:50:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:50:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:50:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:50:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:50:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:50:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:50:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:50:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:50:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:50:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:50:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:50:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:50:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:50:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:50:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:50:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:50:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:50:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:50:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:50:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:50:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:50:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:50:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:50:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:50:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:50:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:50:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:50:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:50:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:50:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:50:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:50:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:50:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:50:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:50:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:50:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:50:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:50:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:50:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:50:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:50:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:50:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:50:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:50:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:50:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:50:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:50:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:50:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:50:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:50:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:50:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:50:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:50:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:50:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:50:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:50:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:50:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:50:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:50:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:50:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:50:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:50:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:50:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:50:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:50:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:50:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:50:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:50:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:50:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:51:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:51:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:51:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:51:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:51:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:51:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:51:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:51:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:51:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:51:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:51:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:51:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:51:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:51:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:51:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:51:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:51:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:51:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:51:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:51:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:51:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:51:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:51:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:51:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:51:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:51:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:51:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:51:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:51:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:51:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:51:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:51:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:51:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:51:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:51:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:51:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:51:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:51:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:51:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:51:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:51:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:51:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:51:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:51:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:51:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:51:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:51:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:51:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:51:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:51:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:51:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:51:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:51:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:51:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:51:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:51:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:51:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:51:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:51:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:51:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:51:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:51:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:51:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:51:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:51:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:51:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:51:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:51:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:51:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:51:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:51:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:51:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:53:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:53:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:53:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:53:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:53:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:53:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:53:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:53:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:53:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:53:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:53:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:53:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:53:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:53:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:53:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:53:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:53:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:53:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:53:23 rdma06 rsyslogd[1041]: imjournal: 5082 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 06:53:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:53:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:53:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:53:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:53:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:53:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:53:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:53:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:53:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:53:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:53:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:53:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:53:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:53:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:53:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:53:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:53:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:53:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:53:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:53:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:53:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:53:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:53:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:53:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:53:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:53:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:53:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:53:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:53:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:53:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:53:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:53:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:53:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:53:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:53:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:53:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:53:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:53:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:53:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:53:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:53:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:53:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:53:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:53:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:53:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:53:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:53:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:53:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:53:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:53:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:53:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:53:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:53:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:53:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:53:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:53:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:53:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:53:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:53:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:53:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:53:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:53:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:53:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:53:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:53:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:53:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:53:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:53:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:53:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:53:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:53:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:53:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:53:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:53:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:53:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:53:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:53:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:53:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:53:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:53:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:53:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:53:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:53:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:53:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:53:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:53:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:53:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:53:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:53:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:53:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:53:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:53:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:53:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:53:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:53:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:53:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:53:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:53:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:53:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:53:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:53:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:53:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:53:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:53:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:53:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:53:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:53:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:53:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:53:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:53:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:53:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:53:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:53:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:53:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:53:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:53:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:53:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:53:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:53:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:53:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:53:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:53:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:53:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:53:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:53:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:53:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:54:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:54:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:54:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:54:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:54:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:54:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:54:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:54:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:54:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:54:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:54:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:54:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:54:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:54:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:54:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:54:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:54:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:54:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:54:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:54:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:54:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:54:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:54:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:54:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:54:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:54:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:54:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:54:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:54:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:54:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:54:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:54:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:54:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:54:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:54:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:54:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:54:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:54:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:54:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:54:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:54:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:54:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:54:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:54:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:54:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:54:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:54:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:54:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:54:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:54:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:54:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:54:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:54:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:54:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:54:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:54:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:54:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:54:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:54:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:54:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:54:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:54:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:54:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:54:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:54:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:54:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:54:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:54:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:54:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:54:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:54:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:54:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:54:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:54:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:54:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:54:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:54:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:54:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:54:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:54:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:54:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:54:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:54:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:54:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:54:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:54:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:54:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:54:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:54:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:54:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:54:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:54:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:54:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:54:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:54:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:54:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:54:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:54:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:54:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:54:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:54:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:54:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:54:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:54:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:54:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:54:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:54:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:54:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:54:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:54:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:54:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:54:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:54:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:54:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:54:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:54:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:54:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:54:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:54:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:54:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:54:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:54:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:54:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:54:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:54:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:54:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:54:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:54:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:54:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:54:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:54:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:54:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:54:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:54:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:54:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:54:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:54:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:54:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:54:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:54:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:54:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:54:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:54:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:54:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:54:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:54:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:54:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:54:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:54:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:54:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:54:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:54:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:54:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:54:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:54:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:54:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:54:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:54:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:54:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:54:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:54:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:54:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:54:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:54:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:54:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:54:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:54:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:54:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:54:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:54:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:54:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:54:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:54:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:54:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:54:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:54:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:54:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:54:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:54:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:54:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:54:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:54:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:54:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:54:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:54:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:54:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:54:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:54:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:54:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:54:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:54:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:54:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:54:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:54:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:54:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:54:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:54:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:54:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:54:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:54:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:54:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:54:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:54:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:54:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:54:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:54:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:54:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:54:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:54:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:54:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:54:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:54:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:54:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:54:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:54:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:54:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:55:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:55:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:55:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:55:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:55:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:55:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:55:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:55:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:55:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:55:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:55:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:55:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:55:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:55:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:55:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:55:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:55:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:55:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:55:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:55:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:55:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:55:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:55:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:55:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:55:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:55:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:55:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:55:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:55:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:55:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:55:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:55:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:55:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:55:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:55:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:55:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:55:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:55:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:55:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:55:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:55:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:55:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:55:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:55:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:55:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:55:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:55:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:55:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:55:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:55:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:55:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:55:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:55:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:55:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:55:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:55:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:55:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:55:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:55:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:55:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:55:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:55:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:55:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:55:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:55:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:55:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:55:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:55:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:55:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:55:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:55:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:55:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:55:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:55:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:55:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:55:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:55:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:55:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:55:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:55:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:55:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:55:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:55:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:55:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:55:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:55:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:55:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:55:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:55:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:55:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:55:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:55:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:55:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:55:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:55:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:55:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:55:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:55:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:55:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:55:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:55:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:55:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:55:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:55:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:55:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:55:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:55:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:55:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:55:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:55:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:55:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:55:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:55:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:55:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:55:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:55:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:55:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:55:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:55:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:55:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:55:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:55:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:55:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:55:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:55:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:55:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:55:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:55:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:55:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:55:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:55:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:55:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:55:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:55:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:55:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:55:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:55:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:55:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:55:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:55:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:55:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:55:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:55:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:55:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:55:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:55:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:55:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:55:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:55:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:55:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:55:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:55:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:55:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:55:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:55:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:55:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:55:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:55:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:55:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:55:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:55:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:55:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:55:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:55:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:55:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:55:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:55:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:55:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:55:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:55:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:55:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:55:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:55:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:55:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:55:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:55:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:55:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:55:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:55:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:55:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:55:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:55:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:55:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:55:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:55:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:55:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:55:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:55:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:55:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:55:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:55:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:55:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:55:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:55:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:55:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:55:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:55:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:55:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:55:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:55:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:55:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:55:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:55:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:55:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:55:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:55:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:55:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:55:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:55:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:55:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:55:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:55:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:55:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:55:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:55:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:55:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:56:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:56:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:56:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:56:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:56:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:56:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:56:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:56:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:56:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:56:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:56:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:56:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:56:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:56:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:56:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:56:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:56:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:56:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:56:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:56:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:56:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:56:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:56:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:56:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:56:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:56:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:56:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:56:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:56:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:56:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:56:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:56:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:56:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:56:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:56:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:56:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:56:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:56:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:56:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:56:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:56:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:56:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:56:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:56:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:56:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:56:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:56:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:56:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:56:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:56:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:56:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:56:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:56:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:56:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:56:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:56:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:56:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:56:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:56:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:56:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:56:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:56:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:56:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:56:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:56:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:56:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:56:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:56:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:56:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:56:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:56:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:56:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:56:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:56:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:56:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:56:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:56:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:56:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:56:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:56:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:56:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:56:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:56:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:56:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:56:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:56:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:56:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:56:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:56:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:56:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:56:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:56:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:56:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:56:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:56:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:56:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:56:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:56:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:56:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:56:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:56:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:56:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:56:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:56:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:56:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:56:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:56:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:56:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:56:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:56:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:56:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:56:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:56:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:56:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:56:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:56:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:56:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:56:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:56:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:56:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:56:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:56:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:56:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:56:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:56:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:56:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:56:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:56:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:56:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:56:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:56:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:56:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:56:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:56:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:56:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:56:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:56:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:56:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:56:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:56:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:56:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:56:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:56:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:56:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:56:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:56:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:56:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:56:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:56:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:56:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:56:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:56:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:56:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:56:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:56:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:56:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:56:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:56:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:56:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:56:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:56:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:56:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:56:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:56:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:56:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:56:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:56:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:56:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:56:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:56:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:56:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:56:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:56:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:56:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:56:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:56:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:56:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:56:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:56:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:56:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:56:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:56:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:56:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:56:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:56:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:56:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:56:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:56:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:56:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:56:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:56:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:56:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:56:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:56:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:56:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:56:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:56:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:56:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:56:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:56:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:56:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:56:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:56:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:56:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:56:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:56:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:56:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:56:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:56:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:56:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:56:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:56:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:56:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:56:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:56:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:56:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:57:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:57:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:57:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:57:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:57:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:57:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:57:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:57:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:57:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:57:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:57:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:57:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:57:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:57:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:57:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:57:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:57:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:57:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:57:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:57:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:57:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:57:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:57:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:57:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:57:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:57:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:57:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:57:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:57:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:57:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:57:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:57:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:57:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:57:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:57:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:57:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:57:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:57:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:57:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:57:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:57:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:57:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:57:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:57:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:57:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:57:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:57:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:57:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:57:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:57:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:57:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:57:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:57:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:57:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:57:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:57:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:57:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:57:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:57:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:57:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:57:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:57:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:57:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:57:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:57:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:57:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:57:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:57:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:57:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:57:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:57:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:57:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:57:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:57:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:57:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:57:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:57:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:57:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:57:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:57:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:57:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:57:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:57:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:57:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:57:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:57:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:57:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:57:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:57:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:57:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:57:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:57:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:57:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:57:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:57:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:57:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:57:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:57:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:57:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:57:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:57:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:57:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:57:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:57:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:57:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:57:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:57:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:57:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:57:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:57:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:57:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:57:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:57:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:57:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:57:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:57:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:57:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:57:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:57:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:57:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:57:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:57:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:57:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:57:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:57:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:57:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:57:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:57:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:57:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:57:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:57:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:57:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:57:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:57:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:57:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:57:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:57:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:57:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:57:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:57:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:57:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:57:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:57:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:57:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:57:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:57:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:57:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:57:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:57:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:57:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:57:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:57:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:57:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:57:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:57:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:57:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:57:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:57:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:57:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:57:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:57:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:57:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:57:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:57:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:57:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:57:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:57:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:57:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:57:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:57:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:57:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:57:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:57:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:57:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:57:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:57:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:57:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:57:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:57:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:57:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:57:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:57:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:57:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:57:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:57:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:57:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:57:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:57:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:57:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:57:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:57:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:57:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:57:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:57:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:57:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:57:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:57:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:57:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:57:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:57:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:57:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:57:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:57:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:57:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:57:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:57:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:57:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:57:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:57:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:57:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:57:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:57:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:57:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:57:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:57:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:57:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:58:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:58:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:58:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:58:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:58:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:58:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:58:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:58:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:58:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:58:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:58:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:58:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:58:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:58:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:58:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:58:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:58:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:58:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:58:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:58:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:58:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:58:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:58:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:58:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:58:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:58:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:58:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:58:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:58:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:58:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:58:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:58:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:58:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:58:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:58:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:58:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:58:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:58:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:58:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:58:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:58:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:58:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:58:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:58:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:58:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:58:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:58:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:58:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:58:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:58:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:58:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:58:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:58:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:58:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:58:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:58:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:58:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:58:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:58:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:58:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:58:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:58:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:58:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:58:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:58:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:58:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:58:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:58:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:58:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:58:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:58:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:58:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:58:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:58:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:58:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:58:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:58:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:58:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:58:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:58:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:58:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:58:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:58:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:58:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:58:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:58:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:58:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:58:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:58:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:58:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:58:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:58:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:58:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:58:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:58:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:58:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:58:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:58:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:58:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:58:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:58:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:58:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:58:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:58:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:58:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:58:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:58:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:58:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:58:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:58:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:58:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:58:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:58:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:58:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:58:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:58:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:58:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:58:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:58:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:58:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:58:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:58:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:58:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:58:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:58:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:58:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:58:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:58:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:58:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:58:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:58:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:58:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:58:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:58:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:58:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:58:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:58:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:58:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:58:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:58:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:58:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:58:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:58:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:58:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:58:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:58:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:58:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:58:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:58:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:58:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:58:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:58:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:58:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:58:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:58:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:58:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:58:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:58:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:58:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:58:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:58:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:58:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:58:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:58:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:58:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:58:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:58:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:58:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:58:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:58:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:58:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:58:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:58:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:58:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:58:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:58:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:58:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:58:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:58:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:58:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:58:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:58:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:58:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:58:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:58:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:58:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:58:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:58:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:58:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:58:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:58:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:58:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:58:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:58:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:58:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:58:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:58:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:58:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:58:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:58:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:58:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:58:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:58:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:58:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:58:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:58:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:58:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:58:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:58:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:58:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:58:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:58:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:58:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:58:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:58:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:58:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:59:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:59:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:59:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:59:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:59:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:59:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:59:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:59:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:59:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:59:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:59:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:59:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:59:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:59:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:59:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:59:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:59:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:59:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:59:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:59:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:59:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:59:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:59:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:59:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:59:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:59:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:59:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:59:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:59:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:59:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:59:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:59:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:59:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:59:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:59:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:59:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:59:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:59:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:59:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:59:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:59:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:59:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:59:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:59:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:59:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:59:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:59:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:59:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:59:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:59:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:59:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:59:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:59:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:59:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:59:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:59:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:59:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:59:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:59:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:59:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:59:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:59:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:59:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:59:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:59:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:59:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:59:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:59:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:59:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:59:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:59:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:59:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:59:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:59:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:59:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:59:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:59:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:59:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:59:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:59:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:59:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:59:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:59:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:59:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:59:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:59:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:59:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:59:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:59:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:59:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:59:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:59:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:59:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:59:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:59:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:59:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:59:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:59:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:59:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:59:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:59:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:59:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:59:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:59:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:59:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:59:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:59:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:59:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:59:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:59:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:59:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:59:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:59:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:59:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:59:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:59:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:59:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:59:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:59:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:59:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:59:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:59:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:59:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:59:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:59:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:59:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:59:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:59:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:59:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:59:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:59:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:59:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:59:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:59:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:59:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:59:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:59:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:59:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:59:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:59:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:59:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:59:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:59:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:59:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:59:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:59:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:59:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:59:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:59:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:59:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:59:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:59:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:59:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:59:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:59:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:59:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:59:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:59:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:59:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:59:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:59:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:59:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:59:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:59:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:59:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:59:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:59:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:59:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:59:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:59:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:59:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:59:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:59:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:59:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:59:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:59:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:59:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:59:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:59:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:59:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:59:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:59:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:59:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:59:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:59:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:59:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:59:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:59:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:59:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:59:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:59:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:59:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:59:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:59:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:59:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:59:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:59:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:59:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 06:59:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 06:59:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:59:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:59:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:59:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 06:59:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 06:59:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 06:59:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 06:59:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 06:59:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 06:59:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 06:59:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 06:59:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 06:59:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 06:59:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 06:59:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 06:59:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 06:59:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:00:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:00:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:00:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:00:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:00:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:00:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:00:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:00:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:00:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:00:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:00:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:00:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:00:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:00:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:00:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:00:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:00:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:00:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:00:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:00:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:00:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:00:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:00:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:00:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:00:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:00:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:00:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:00:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:00:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:00:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:00:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:00:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:00:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:00:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:00:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:00:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:00:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:00:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:00:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:00:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:00:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:00:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:00:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:00:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:00:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:00:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:00:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:00:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:00:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:00:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:00:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:00:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:00:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:00:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:00:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:00:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:00:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:00:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:00:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:00:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:00:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:00:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:00:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:00:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:00:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:00:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:00:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:00:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:00:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:00:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:00:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:00:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:00:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:00:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:00:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:00:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:00:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:00:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:00:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:00:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:00:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:00:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:00:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:00:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:00:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:00:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:00:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:00:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:00:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:00:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:00:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:00:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:00:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:00:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:00:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:00:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:00:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:00:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:00:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:00:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:00:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:00:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:00:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:00:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:00:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:00:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:00:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:00:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:00:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:00:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:00:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:00:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:00:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:00:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:00:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:00:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:00:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:00:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:00:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:00:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:00:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:00:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:00:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:00:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:00:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:00:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:00:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:00:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:00:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:00:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:00:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:00:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:00:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:00:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:00:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:00:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:00:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:00:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:00:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:00:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:00:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:00:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:00:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:00:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:00:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:00:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:00:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:00:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:00:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:00:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:00:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:00:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:00:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:00:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:00:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:00:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:00:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:00:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:00:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:00:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:00:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:00:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:00:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:00:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:00:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:00:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:00:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:00:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:00:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:00:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:00:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:00:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:00:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:00:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:00:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:00:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:00:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:00:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:00:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:00:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:00:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:00:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:00:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:00:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:00:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:00:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:00:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:00:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:00:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:00:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:00:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:00:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:00:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:00:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:00:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:00:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:00:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:00:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:00:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:00:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:00:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:00:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:00:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:00:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:00:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:00:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:00:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:00:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:00:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:00:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:00:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:00:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:00:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:00:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:00:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:00:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:01:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:01:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:01:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:01:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:01:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:01:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:01:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:01:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:01:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:01:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:01:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:01:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:01:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:01:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:01:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:01:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:01:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:01:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:01:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:01:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:01:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:01:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:01:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:01:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:01:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:01:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:01:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:01:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:01:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:01:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:01:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:01:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:01:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:01:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:01:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:01:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:01:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:01:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:01:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:01:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:01:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:01:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:01:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:01:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:01:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:01:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:01:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:01:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:01:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:01:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:01:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:01:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:01:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:01:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:01:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:01:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:01:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:01:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:01:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:01:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:01:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:01:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:01:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:01:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:01:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:01:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:01:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:01:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:01:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:01:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:01:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:01:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:03:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:03:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:03:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:03:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:03:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:03:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:03:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:03:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:03:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:03:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:03:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:03:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:03:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:03:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:03:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:03:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:03:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:03:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:03:28 rdma06 rsyslogd[1041]: imjournal: 5295 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 07:03:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:03:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:03:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:03:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:03:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:03:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:03:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:03:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:03:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:03:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:03:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:03:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:03:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:03:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:03:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:03:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:03:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:03:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:03:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:03:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:03:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:03:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:03:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:03:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:03:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:03:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:03:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:03:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:03:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:03:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:03:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:03:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:03:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:03:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:03:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:03:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:03:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:03:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:03:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:03:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:03:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:03:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:03:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:03:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:03:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:03:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:03:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:03:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:03:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:03:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:03:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:03:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:03:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:03:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:03:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:03:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:03:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:03:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:03:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:03:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:03:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:03:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:03:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:03:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:03:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:03:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:03:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:03:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:03:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:03:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:03:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:03:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:03:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:03:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:03:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:03:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:03:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:03:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:03:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:03:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:03:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:03:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:03:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:03:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:03:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:03:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:03:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:03:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:03:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:03:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:03:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:03:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:03:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:03:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:03:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:03:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:03:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:03:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:03:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:03:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:03:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:03:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:03:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:03:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:03:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:03:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:03:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:03:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:04:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:04:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:04:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:04:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:04:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:04:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:04:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:04:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:04:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:04:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:04:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:04:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:04:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:04:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:04:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:04:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:04:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:04:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:04:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:04:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:04:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:04:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:04:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:04:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:04:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:04:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:04:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:04:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:04:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:04:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:04:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:04:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:04:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:04:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:04:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:04:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:04:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:04:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:04:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:04:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:04:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:04:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:04:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:04:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:04:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:04:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:04:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:04:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:04:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:04:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:04:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:04:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:04:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:04:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:04:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:04:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:04:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:04:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:04:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:04:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:04:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:04:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:04:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:04:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:04:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:04:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:04:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:04:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:04:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:04:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:04:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:04:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:04:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:04:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:04:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:04:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:04:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:04:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:04:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:04:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:04:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:04:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:04:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:04:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:04:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:04:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:04:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:04:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:04:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:04:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:04:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:04:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:04:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:04:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:04:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:04:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:04:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:04:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:04:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:04:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:04:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:04:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:04:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:04:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:04:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:04:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:04:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:04:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:04:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:04:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:04:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:04:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:04:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:04:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:04:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:04:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:04:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:04:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:04:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:04:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:04:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:04:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:04:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:04:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:04:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:04:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:04:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:04:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:04:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:04:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:04:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:04:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:04:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:04:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:04:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:04:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:04:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:04:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:04:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:04:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:04:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:04:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:04:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:04:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:04:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:04:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:04:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:04:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:04:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:04:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:04:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:04:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:04:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:04:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:04:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:04:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:04:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:04:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:04:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:04:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:04:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:04:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:04:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:04:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:04:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:04:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:04:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:04:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:04:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:04:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:04:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:04:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:04:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:04:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:04:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:04:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:04:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:04:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:04:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:04:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:04:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:04:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:04:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:04:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:04:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:04:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:04:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:04:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:04:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:04:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:04:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:04:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:04:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:04:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:04:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:04:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:04:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:04:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:04:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:04:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:04:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:04:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:04:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:04:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:04:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:04:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:04:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:04:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:04:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:04:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:04:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:04:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:04:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:04:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:04:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:04:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:05:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:05:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:05:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:05:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:05:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:05:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:05:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:05:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:05:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:05:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:05:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:05:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:05:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:05:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:05:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:05:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:05:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:05:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:05:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:05:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:05:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:05:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:05:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:05:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:05:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:05:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:05:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:05:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:05:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:05:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:05:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:05:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:05:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:05:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:05:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:05:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:05:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:05:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:05:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:05:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:05:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:05:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:05:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:05:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:05:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:05:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:05:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:05:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:05:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:05:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:05:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:05:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:05:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:05:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:05:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:05:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:05:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:05:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:05:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:05:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:05:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:05:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:05:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:05:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:05:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:05:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:05:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:05:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:05:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:05:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:05:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:05:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:05:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:05:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:05:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:05:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:05:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:05:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:05:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:05:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:05:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:05:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:05:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:05:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:05:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:05:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:05:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:05:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:05:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:05:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:05:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:05:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:05:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:05:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:05:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:05:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:05:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:05:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:05:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:05:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:05:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:05:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:05:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:05:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:05:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:05:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:05:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:05:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:05:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:05:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:05:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:05:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:05:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:05:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:05:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:05:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:05:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:05:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:05:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:05:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:05:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:05:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:05:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:05:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:05:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:05:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:05:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:05:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:05:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:05:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:05:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:05:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:05:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:05:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:05:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:05:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:05:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:05:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:05:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:05:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:05:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:05:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:05:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:05:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:05:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:05:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:05:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:05:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:05:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:05:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:05:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:05:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:05:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:05:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:05:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:05:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:05:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:05:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:05:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:05:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:05:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:05:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:05:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:05:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:05:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:05:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:05:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:05:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:05:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:05:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:05:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:05:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:05:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:05:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:05:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:05:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:05:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:05:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:05:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:05:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:05:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:05:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:05:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:05:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:05:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:05:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:05:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:05:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:05:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:05:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:05:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:05:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:05:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:05:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:05:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:05:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:05:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:05:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:05:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:05:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:05:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:05:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:05:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:05:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:05:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:05:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:05:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:05:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:05:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:05:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:05:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:05:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:05:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:05:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:05:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:05:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:06:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:06:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:06:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:06:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:06:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:06:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:06:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:06:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:06:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:06:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:06:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:06:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:06:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:06:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:06:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:06:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:06:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:06:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:06:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:06:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:06:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:06:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:06:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:06:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:06:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:06:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:06:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:06:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:06:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:06:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:06:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:06:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:06:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:06:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:06:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:06:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:06:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:06:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:06:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:06:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:06:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:06:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:06:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:06:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:06:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:06:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:06:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:06:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:06:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:06:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:06:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:06:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:06:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:06:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:06:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:06:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:06:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:06:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:06:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:06:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:06:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:06:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:06:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:06:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:06:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:06:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:06:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:06:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:06:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:06:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:06:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:06:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:06:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:06:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:06:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:06:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:06:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:06:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:06:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:06:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:06:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:06:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:06:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:06:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:06:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:06:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:06:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:06:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:06:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:06:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:06:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:06:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:06:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:06:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:06:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:06:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:06:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:06:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:06:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:06:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:06:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:06:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:06:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:06:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:06:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:06:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:06:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:06:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:06:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:06:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:06:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:06:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:06:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:06:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:06:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:06:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:06:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:06:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:06:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:06:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:06:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:06:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:06:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:06:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:06:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:06:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:06:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:06:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:06:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:06:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:06:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:06:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:06:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:06:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:06:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:06:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:06:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:06:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:06:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:06:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:06:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:06:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:06:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:06:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:06:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:06:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:06:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:06:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:06:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:06:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:06:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:06:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:06:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:06:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:06:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:06:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:06:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:06:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:06:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:06:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:06:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:06:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:06:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:06:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:06:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:06:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:06:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:06:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:06:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:06:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:06:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:06:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:06:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:06:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:06:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:06:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:06:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:06:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:06:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:06:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:06:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:06:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:06:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:06:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:06:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:06:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:06:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:06:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:06:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:06:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:06:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:06:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:06:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:06:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:06:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:06:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:06:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:06:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:06:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:06:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:06:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:06:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:06:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:06:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:06:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:06:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:06:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:06:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:06:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:06:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:06:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:06:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:06:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:06:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:06:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:06:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:07:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:07:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:07:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:07:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:07:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:07:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:07:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:07:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:07:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:07:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:07:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:07:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:07:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:07:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:07:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:07:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:07:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:07:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:07:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:07:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:07:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:07:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:07:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:07:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:07:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:07:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:07:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:07:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:07:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:07:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:07:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:07:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:07:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:07:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:07:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:07:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:07:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:07:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:07:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:07:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:07:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:07:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:07:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:07:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:07:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:07:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:07:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:07:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:07:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:07:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:07:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:07:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:07:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:07:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:07:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:07:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:07:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:07:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:07:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:07:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:07:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:07:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:07:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:07:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:07:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:07:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:07:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:07:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:07:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:07:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:07:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:07:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:07:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:07:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:07:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:07:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:07:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:07:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:07:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:07:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:07:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:07:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:07:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:07:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:07:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:07:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:07:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:07:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:07:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:07:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:07:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:07:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:07:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:07:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:07:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:07:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:07:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:07:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:07:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:07:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:07:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:07:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:07:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:07:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:07:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:07:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:07:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:07:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:07:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:07:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:07:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:07:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:07:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:07:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:07:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:07:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:07:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:07:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:07:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:07:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:07:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:07:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:07:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:07:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:07:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:07:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:07:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:07:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:07:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:07:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:07:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:07:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:07:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:07:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:07:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:07:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:07:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:07:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:07:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:07:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:07:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:07:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:07:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:07:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:07:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:07:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:07:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:07:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:07:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:07:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:07:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:07:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:07:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:07:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:07:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:07:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:07:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:07:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:07:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:07:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:07:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:07:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:07:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:07:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:07:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:07:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:07:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:07:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:07:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:07:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:07:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:07:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:07:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:07:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:07:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:07:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:07:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:07:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:07:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:07:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:07:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:07:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:07:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:07:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:07:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:07:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:07:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:07:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:07:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:07:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:07:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:07:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:07:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:07:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:07:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:07:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:07:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:07:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:07:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:07:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:07:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:07:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:07:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:07:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:07:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:07:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:07:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:07:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:07:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:07:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:07:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:07:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:07:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:07:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:07:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:07:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:08:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:08:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:08:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:08:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:08:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:08:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:08:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:08:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:08:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:08:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:08:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:08:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:08:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:08:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:08:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:08:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:08:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:08:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:08:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:08:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:08:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:08:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:08:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:08:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:08:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:08:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:08:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:08:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:08:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:08:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:08:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:08:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:08:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:08:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:08:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:08:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:08:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:08:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:08:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:08:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:08:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:08:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:08:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:08:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:08:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:08:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:08:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:08:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:08:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:08:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:08:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:08:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:08:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:08:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:08:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:08:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:08:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:08:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:08:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:08:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:08:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:08:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:08:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:08:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:08:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:08:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:08:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:08:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:08:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:08:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:08:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:08:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:08:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:08:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:08:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:08:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:08:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:08:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:08:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:08:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:08:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:08:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:08:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:08:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:08:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:08:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:08:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:08:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:08:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:08:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:08:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:08:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:08:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:08:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:08:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:08:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:08:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:08:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:08:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:08:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:08:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:08:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:08:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:08:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:08:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:08:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:08:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:08:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:08:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:08:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:08:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:08:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:08:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:08:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:08:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:08:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:08:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:08:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:08:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:08:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:08:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:08:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:08:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:08:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:08:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:08:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:08:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:08:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:08:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:08:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:08:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:08:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:08:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:08:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:08:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:08:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:08:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:08:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:08:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:08:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:08:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:08:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:08:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:08:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:08:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:08:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:08:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:08:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:08:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:08:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:08:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:08:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:08:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:08:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:08:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:08:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:08:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:08:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:08:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:08:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:08:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:08:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:08:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:08:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:08:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:08:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:08:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:08:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:08:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:08:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:08:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:08:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:08:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:08:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:08:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:08:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:08:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:08:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:08:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:08:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:08:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:08:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:08:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:08:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:08:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:08:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:08:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:08:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:08:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:08:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:08:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:08:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:08:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:08:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:08:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:08:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:08:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:08:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:08:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:08:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:08:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:08:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:08:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:08:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:08:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:08:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:08:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:08:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:08:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:08:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:08:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:08:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:08:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:08:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:08:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:08:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:09:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:09:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:09:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:09:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:09:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:09:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:09:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:09:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:09:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:09:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:09:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:09:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:09:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:09:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:09:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:09:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:09:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:09:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:09:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:09:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:09:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:09:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:09:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:09:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:09:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:09:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:09:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:09:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:09:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:09:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:09:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:09:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:09:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:09:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:09:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:09:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:09:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:09:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:09:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:09:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:09:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:09:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:09:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:09:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:09:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:09:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:09:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:09:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:09:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:09:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:09:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:09:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:09:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:09:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:09:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:09:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:09:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:09:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:09:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:09:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:09:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:09:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:09:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:09:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:09:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:09:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:09:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:09:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:09:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:09:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:09:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:09:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:09:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:09:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:09:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:09:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:09:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:09:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:09:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:09:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:09:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:09:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:09:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:09:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:09:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:09:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:09:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:09:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:09:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:09:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:09:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:09:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:09:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:09:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:09:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:09:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:09:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:09:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:09:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:09:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:09:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:09:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:09:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:09:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:09:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:09:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:09:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:09:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:09:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:09:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:09:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:09:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:09:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:09:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:09:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:09:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:09:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:09:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:09:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:09:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:09:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:09:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:09:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:09:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:09:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:09:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:09:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:09:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:09:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:09:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:09:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:09:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:09:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:09:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:09:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:09:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:09:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:09:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:09:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:09:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:09:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:09:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:09:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:09:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:09:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:09:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:09:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:09:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:09:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:09:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:09:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:09:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:09:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:09:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:09:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:09:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:09:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:09:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:09:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:09:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:09:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:09:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:09:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:09:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:09:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:09:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:09:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:09:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:09:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:09:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:09:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:09:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:09:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:09:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:09:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:09:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:09:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:09:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:09:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:09:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:09:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:09:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:09:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:09:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:09:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:09:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:09:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:09:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:09:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:09:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:09:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:09:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:09:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:09:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:09:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:09:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:09:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:09:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:09:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:09:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:09:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:09:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:09:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:09:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:09:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:09:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:09:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:09:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:09:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:09:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:09:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:09:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:09:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:09:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:09:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:09:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:10:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:10:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:10:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:10:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:10:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:10:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:10:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:10:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:10:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:10:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:10:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:10:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:10:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:10:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:10:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:10:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:10:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:10:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:10:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:10:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:10:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:10:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:10:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:10:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:10:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:10:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:10:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:10:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:10:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:10:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:10:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:10:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:10:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:10:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:10:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:10:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:10:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:10:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:10:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:10:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:10:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:10:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:10:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:10:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:10:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:10:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:10:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:10:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:10:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:10:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:10:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:10:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:10:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:10:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:10:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:10:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:10:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:10:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:10:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:10:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:10:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:10:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:10:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:10:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:10:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:10:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:10:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:10:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:10:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:10:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:10:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:10:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:10:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:10:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:10:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:10:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:10:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:10:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:10:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:10:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:10:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:10:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:10:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:10:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:10:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:10:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:10:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:10:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:10:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:10:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:10:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:10:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:10:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:10:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:10:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:10:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:10:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:10:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:10:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:10:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:10:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:10:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:10:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:10:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:10:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:10:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:10:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:10:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:10:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:10:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:10:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:10:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:10:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:10:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:10:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:10:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:10:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:10:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:10:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:10:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:10:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:10:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:10:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:10:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:10:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:10:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:10:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:10:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:10:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:10:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:10:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:10:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:10:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:10:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:10:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:10:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:10:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:10:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:10:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:10:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:10:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:10:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:10:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:10:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:10:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:10:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:10:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:10:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:10:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:10:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:10:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:10:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:10:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:10:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:10:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:10:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:10:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:10:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:10:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:10:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:10:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:10:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:10:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:10:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:10:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:10:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:10:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:10:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:10:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:10:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:10:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:10:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:10:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:10:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:10:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:10:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:10:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:10:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:10:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:10:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:10:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:10:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:10:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:10:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:10:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:10:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:10:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:10:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:10:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:10:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:10:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:10:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:10:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:10:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:10:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:10:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:10:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:10:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:10:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:10:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:10:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:10:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:10:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:10:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:10:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:10:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:10:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:10:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:10:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:10:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:10:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:10:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:10:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:10:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:10:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:10:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:11:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:11:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:11:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:11:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:11:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:11:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:11:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:11:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:11:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:11:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:11:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:11:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:11:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:11:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:11:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:11:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:11:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:11:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:11:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:11:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:11:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:11:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:11:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:11:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:11:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:11:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:11:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:11:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:11:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:11:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:11:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:11:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:11:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:11:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:11:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:11:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:11:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:11:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:11:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:11:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:11:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:11:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:11:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:11:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:11:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:11:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:11:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:11:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:11:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:11:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:11:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:11:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:11:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:11:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:11:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:11:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:11:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:11:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:11:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:11:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:11:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:11:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:11:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:11:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:11:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:11:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:11:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:11:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:11:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:11:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:11:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:11:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:11:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:11:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:11:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:11:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:11:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:11:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:11:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:11:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:11:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:11:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:11:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:11:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:11:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:11:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:11:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:11:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:11:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:11:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:13:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:13:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:13:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:13:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:13:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:13:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:13:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:13:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:13:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:13:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:13:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:13:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:13:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:13:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:13:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:13:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:13:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:13:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:13:33 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 07:13:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:13:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:13:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:13:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:13:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:13:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:13:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:13:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:13:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:13:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:13:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:13:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:13:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:13:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:13:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:13:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:13:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:13:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:13:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:13:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:13:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:13:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:13:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:13:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:13:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:13:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:13:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:13:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:13:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:13:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:13:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:13:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:13:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:13:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:13:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:13:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:13:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:13:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:13:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:13:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:13:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:13:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:13:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:13:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:13:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:13:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:13:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:13:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:13:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:13:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:13:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:13:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:13:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:13:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:13:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:13:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:13:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:13:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:13:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:13:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:13:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:13:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:13:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:13:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:13:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:13:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:13:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:13:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:13:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:13:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:13:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:13:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:13:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:13:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:13:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:13:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:13:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:13:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:13:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:13:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:13:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:13:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:13:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:13:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:13:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:13:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:13:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:13:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:13:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:13:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:14:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:14:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:14:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:14:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:14:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:14:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:14:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:14:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:14:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:14:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:14:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:14:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:14:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:14:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:14:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:14:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:14:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:14:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:14:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:14:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:14:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:14:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:14:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:14:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:14:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:14:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:14:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:14:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:14:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:14:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:14:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:14:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:14:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:14:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:14:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:14:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:14:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:14:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:14:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:14:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:14:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:14:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:14:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:14:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:14:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:14:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:14:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:14:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:14:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:14:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:14:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:14:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:14:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:14:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:14:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:14:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:14:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:14:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:14:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:14:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:14:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:14:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:14:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:14:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:14:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:14:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:14:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:14:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:14:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:14:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:14:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:14:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:14:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:14:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:14:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:14:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:14:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:14:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:14:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:14:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:14:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:14:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:14:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:14:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:14:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:14:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:14:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:14:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:14:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:14:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:14:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:14:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:14:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:14:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:14:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:14:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:14:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:14:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:14:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:14:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:14:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:14:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:14:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:14:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:14:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:14:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:14:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:14:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:14:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:14:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:14:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:14:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:14:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:14:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:14:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:14:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:14:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:14:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:14:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:14:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:14:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:14:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:14:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:14:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:14:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:14:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:14:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:14:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:14:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:14:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:14:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:14:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:14:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:14:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:14:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:14:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:14:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:14:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:14:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:14:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:14:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:14:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:14:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:14:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:14:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:14:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:14:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:14:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:14:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:14:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:14:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:14:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:14:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:14:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:14:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:14:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:14:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:14:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:14:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:14:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:14:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:14:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:14:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:14:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:14:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:14:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:14:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:14:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:14:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:14:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:14:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:14:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:14:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:14:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:14:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:14:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:14:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:14:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:14:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:14:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:14:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:14:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:14:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:14:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:14:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:14:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:14:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:14:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:14:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:14:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:14:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:14:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:14:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:14:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:14:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:14:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:14:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:14:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:14:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:14:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:14:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:14:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:14:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:14:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:14:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:14:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:14:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:14:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:14:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:14:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:14:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:14:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:14:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:14:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:14:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:14:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:15:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:15:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:15:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:15:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:15:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:15:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:15:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:15:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:15:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:15:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:15:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:15:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:15:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:15:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:15:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:15:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:15:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:15:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:15:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:15:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:15:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:15:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:15:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:15:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:15:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:15:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:15:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:15:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:15:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:15:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:15:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:15:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:15:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:15:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:15:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:15:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:15:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:15:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:15:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:15:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:15:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:15:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:15:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:15:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:15:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:15:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:15:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:15:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:15:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:15:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:15:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:15:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:15:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:15:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:15:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:15:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:15:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:15:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:15:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:15:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:15:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:15:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:15:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:15:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:15:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:15:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:15:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:15:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:15:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:15:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:15:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:15:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:15:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:15:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:15:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:15:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:15:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:15:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:15:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:15:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:15:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:15:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:15:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:15:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:15:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:15:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:15:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:15:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:15:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:15:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:15:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:15:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:15:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:15:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:15:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:15:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:15:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:15:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:15:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:15:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:15:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:15:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:15:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:15:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:15:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:15:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:15:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:15:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:15:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:15:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:15:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:15:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:15:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:15:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:15:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:15:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:15:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:15:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:15:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:15:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:15:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:15:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:15:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:15:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:15:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:15:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:15:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:15:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:15:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:15:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:15:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:15:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:15:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:15:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:15:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:15:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:15:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:15:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:15:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:15:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:15:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:15:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:15:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:15:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:15:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:15:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:15:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:15:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:15:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:15:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:15:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:15:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:15:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:15:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:15:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:15:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:15:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:15:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:15:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:15:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:15:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:15:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:15:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:15:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:15:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:15:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:15:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:15:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:15:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:15:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:15:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:15:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:15:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:15:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:15:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:15:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:15:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:15:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:15:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:15:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:15:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:15:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:15:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:15:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:15:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:15:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:15:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:15:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:15:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:15:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:15:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:15:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:15:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:15:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:15:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:15:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:15:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:15:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:15:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:15:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:15:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:15:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:15:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:15:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:15:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:15:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:15:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:15:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:15:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:15:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:15:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:15:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:15:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:15:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:15:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:15:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:16:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:16:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:16:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:16:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:16:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:16:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:16:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:16:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:16:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:16:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:16:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:16:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:16:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:16:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:16:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:16:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:16:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:16:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:16:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:16:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:16:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:16:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:16:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:16:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:16:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:16:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:16:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:16:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:16:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:16:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:16:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:16:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:16:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:16:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:16:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:16:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:16:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:16:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:16:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:16:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:16:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:16:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:16:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:16:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:16:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:16:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:16:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:16:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:16:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:16:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:16:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:16:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:16:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:16:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:16:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:16:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:16:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:16:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:16:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:16:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:16:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:16:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:16:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:16:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:16:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:16:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:16:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:16:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:16:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:16:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:16:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:16:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:16:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:16:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:16:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:16:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:16:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:16:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:16:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:16:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:16:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:16:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:16:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:16:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:16:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:16:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:16:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:16:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:16:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:16:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:16:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:16:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:16:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:16:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:16:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:16:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:16:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:16:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:16:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:16:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:16:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:16:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:16:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:16:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:16:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:16:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:16:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:16:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:16:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:16:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:16:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:16:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:16:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:16:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:16:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:16:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:16:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:16:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:16:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:16:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:16:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:16:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:16:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:16:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:16:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:16:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:16:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:16:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:16:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:16:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:16:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:16:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:16:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:16:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:16:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:16:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:16:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:16:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:16:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:16:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:16:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:16:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:16:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:16:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:16:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:16:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:16:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:16:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:16:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:16:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:16:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:16:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:16:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:16:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:16:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:16:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:16:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:16:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:16:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:16:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:16:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:16:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:16:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:16:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:16:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:16:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:16:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:16:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:16:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:16:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:16:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:16:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:16:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:16:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:16:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:16:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:16:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:16:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:16:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:16:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:16:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:16:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:16:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:16:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:16:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:16:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:16:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:16:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:16:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:16:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:16:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:16:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:16:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:16:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:16:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:16:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:16:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:16:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:16:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:16:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:16:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:16:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:16:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:16:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:16:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:16:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:16:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:16:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:16:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:16:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:16:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:16:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:16:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:16:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:16:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:16:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:17:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:17:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:17:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:17:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:17:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:17:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:17:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:17:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:17:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:17:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:17:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:17:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:17:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:17:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:17:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:17:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:17:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:17:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:17:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:17:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:17:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:17:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:17:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:17:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:17:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:17:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:17:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:17:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:17:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:17:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:17:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:17:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:17:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:17:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:17:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:17:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:17:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:17:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:17:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:17:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:17:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:17:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:17:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:17:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:17:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:17:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:17:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:17:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:17:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:17:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:17:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:17:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:17:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:17:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:17:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:17:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:17:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:17:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:17:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:17:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:17:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:17:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:17:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:17:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:17:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:17:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:17:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:17:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:17:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:17:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:17:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:17:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:17:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:17:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:17:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:17:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:17:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:17:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:17:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:17:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:17:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:17:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:17:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:17:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:17:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:17:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:17:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:17:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:17:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:17:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:17:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:17:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:17:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:17:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:17:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:17:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:17:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:17:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:17:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:17:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:17:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:17:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:17:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:17:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:17:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:17:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:17:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:17:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:17:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:17:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:17:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:17:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:17:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:17:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:17:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:17:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:17:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:17:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:17:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:17:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:17:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:17:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:17:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:17:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:17:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:17:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:17:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:17:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:17:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:17:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:17:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:17:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:17:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:17:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:17:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:17:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:17:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:17:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:17:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:17:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:17:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:17:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:17:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:17:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:17:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:17:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:17:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:17:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:17:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:17:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:17:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:17:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:17:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:17:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:17:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:17:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:17:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:17:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:17:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:17:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:17:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:17:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:17:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:17:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:17:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:17:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:17:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:17:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:17:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:17:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:17:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:17:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:17:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:17:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:17:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:17:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:17:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:17:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:17:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:17:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:17:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:17:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:17:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:17:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:17:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:17:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:17:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:17:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:17:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:17:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:17:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:17:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:17:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:17:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:17:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:17:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:17:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:17:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:17:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:17:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:17:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:17:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:17:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:17:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:17:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:17:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:17:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:17:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:17:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:17:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:17:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:17:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:17:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:17:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:17:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:17:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:18:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:18:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:18:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:18:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:18:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:18:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:18:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:18:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:18:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:18:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:18:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:18:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:18:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:18:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:18:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:18:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:18:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:18:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:18:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:18:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:18:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:18:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:18:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:18:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:18:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:18:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:18:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:18:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:18:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:18:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:18:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:18:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:18:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:18:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:18:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:18:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:18:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:18:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:18:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:18:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:18:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:18:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:18:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:18:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:18:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:18:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:18:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:18:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:18:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:18:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:18:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:18:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:18:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:18:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:18:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:18:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:18:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:18:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:18:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:18:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:18:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:18:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:18:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:18:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:18:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:18:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:18:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:18:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:18:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:18:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:18:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:18:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:18:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:18:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:18:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:18:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:18:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:18:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:18:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:18:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:18:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:18:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:18:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:18:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:18:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:18:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:18:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:18:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:18:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:18:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:18:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:18:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:18:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:18:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:18:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:18:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:18:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:18:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:18:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:18:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:18:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:18:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:18:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:18:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:18:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:18:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:18:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:18:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:18:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:18:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:18:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:18:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:18:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:18:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:18:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:18:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:18:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:18:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:18:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:18:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:18:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:18:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:18:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:18:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:18:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:18:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:18:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:18:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:18:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:18:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:18:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:18:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:18:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:18:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:18:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:18:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:18:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:18:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:18:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:18:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:18:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:18:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:18:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:18:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:18:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:18:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:18:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:18:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:18:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:18:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:18:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:18:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:18:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:18:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:18:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:18:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:18:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:18:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:18:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:18:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:18:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:18:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:18:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:18:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:18:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:18:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:18:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:18:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:18:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:18:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:18:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:18:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:18:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:18:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:18:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:18:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:18:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:18:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:18:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:18:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:18:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:18:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:18:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:18:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:18:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:18:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:18:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:18:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:18:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:18:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:18:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:18:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:18:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:18:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:18:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:18:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:18:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:18:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:18:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:18:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:18:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:18:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:18:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:18:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:18:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:18:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:18:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:18:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:18:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:18:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:18:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:18:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:18:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:18:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:18:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:18:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:19:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:19:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:19:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:19:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:19:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:19:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:19:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:19:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:19:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:19:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:19:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:19:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:19:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:19:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:19:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:19:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:19:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:19:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:19:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:19:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:19:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:19:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:19:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:19:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:19:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:19:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:19:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:19:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:19:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:19:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:19:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:19:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:19:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:19:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:19:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:19:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:19:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:19:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:19:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:19:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:19:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:19:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:19:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:19:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:19:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:19:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:19:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:19:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:19:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:19:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:19:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:19:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:19:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:19:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:19:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:19:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:19:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:19:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:19:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:19:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:19:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:19:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:19:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:19:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:19:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:19:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:19:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:19:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:19:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:19:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:19:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:19:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:19:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:19:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:19:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:19:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:19:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:19:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:19:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:19:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:19:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:19:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:19:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:19:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:19:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:19:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:19:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:19:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:19:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:19:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:19:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:19:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:19:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:19:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:19:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:19:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:19:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:19:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:19:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:19:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:19:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:19:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:19:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:19:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:19:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:19:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:19:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:19:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:19:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:19:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:19:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:19:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:19:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:19:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:19:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:19:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:19:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:19:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:19:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:19:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:19:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:19:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:19:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:19:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:19:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:19:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:19:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:19:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:19:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:19:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:19:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:19:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:19:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:19:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:19:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:19:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:19:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:19:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:19:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:19:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:19:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:19:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:19:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:19:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:19:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:19:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:19:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:19:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:19:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:19:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:19:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:19:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:19:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:19:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:19:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:19:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:19:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:19:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:19:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:19:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:19:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:19:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:19:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:19:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:19:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:19:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:19:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:19:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:19:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:19:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:19:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:19:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:19:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:19:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:19:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:19:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:19:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:19:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:19:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:19:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:19:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:19:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:19:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:19:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:19:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:19:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:19:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:19:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:19:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:19:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:19:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:19:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:19:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:19:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:19:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:19:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:19:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:19:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:19:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:19:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:19:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:19:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:19:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:19:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:19:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:19:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:19:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:19:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:19:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:19:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:19:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:19:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:19:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:19:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:19:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:19:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:20:00 rdma06 systemd[1]: Starting dnf makecache...
Apr 18 07:20:01 rdma06 dnf[1183004]: Updating Subscription Management repositories.
Apr 18 07:20:01 rdma06 dnf[1183004]: Unable to read consumer identity
Apr 18 07:20:01 rdma06 dnf[1183004]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 18 07:20:02 rdma06 dnf[1183004]: Copr repo for fmf owned by psss                 2.3 kB/s | 3.3 kB     00:01
Apr 18 07:20:02 rdma06 dnf[1183004]: beaker-AppStream                                 55 kB/s | 168  B     00:00
Apr 18 07:20:02 rdma06 dnf[1183004]: Errors during downloading metadata for repository 'beaker-AppStream':
Apr 18 07:20:02 rdma06 dnf[1183004]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 07:20:02 rdma06 dnf[1183004]: Error: Failed to download metadata for repo 'beaker-AppStream': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 07:20:02 rdma06 dnf[1183004]: beaker-AppStream-debuginfo                       56 kB/s | 168  B     00:00
Apr 18 07:20:02 rdma06 dnf[1183004]: Errors during downloading metadata for repository 'beaker-AppStream-debuginfo':
Apr 18 07:20:02 rdma06 dnf[1183004]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 07:20:02 rdma06 dnf[1183004]: Error: Failed to download metadata for repo 'beaker-AppStream-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 07:20:02 rdma06 dnf[1183004]: beaker-BaseOS                                    60 kB/s | 168  B     00:00
Apr 18 07:20:02 rdma06 dnf[1183004]: Errors during downloading metadata for repository 'beaker-BaseOS':
Apr 18 07:20:02 rdma06 dnf[1183004]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 07:20:02 rdma06 dnf[1183004]: Error: Failed to download metadata for repo 'beaker-BaseOS': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 07:20:02 rdma06 dnf[1183004]: beaker-BaseOS-debuginfo                          63 kB/s | 168  B     00:00
Apr 18 07:20:02 rdma06 dnf[1183004]: Errors during downloading metadata for repository 'beaker-BaseOS-debuginfo':
Apr 18 07:20:02 rdma06 dnf[1183004]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 07:20:02 rdma06 dnf[1183004]: Error: Failed to download metadata for repo 'beaker-BaseOS-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 07:20:02 rdma06 dnf[1183004]: beaker-CRB                                       61 kB/s | 168  B     00:00
Apr 18 07:20:02 rdma06 dnf[1183004]: Errors during downloading metadata for repository 'beaker-CRB':
Apr 18 07:20:02 rdma06 dnf[1183004]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 07:20:02 rdma06 dnf[1183004]: Error: Failed to download metadata for repo 'beaker-CRB': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 07:20:02 rdma06 dnf[1183004]: beaker-CRB-debuginfo                             62 kB/s | 168  B     00:00
Apr 18 07:20:02 rdma06 dnf[1183004]: Errors during downloading metadata for repository 'beaker-CRB-debuginfo':
Apr 18 07:20:02 rdma06 dnf[1183004]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 07:20:02 rdma06 dnf[1183004]: Error: Failed to download metadata for repo 'beaker-CRB-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 07:20:02 rdma06 dnf[1183004]: beaker-HighAvailability                          65 kB/s | 168  B     00:00
Apr 18 07:20:02 rdma06 dnf[1183004]: Errors during downloading metadata for repository 'beaker-HighAvailability':
Apr 18 07:20:02 rdma06 dnf[1183004]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 07:20:02 rdma06 dnf[1183004]: Error: Failed to download metadata for repo 'beaker-HighAvailability': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 07:20:02 rdma06 dnf[1183004]: beaker-HighAvailability-debuginfo                64 kB/s | 168  B     00:00
Apr 18 07:20:02 rdma06 dnf[1183004]: Errors during downloading metadata for repository 'beaker-HighAvailability-debuginfo':
Apr 18 07:20:02 rdma06 dnf[1183004]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 07:20:02 rdma06 dnf[1183004]: Error: Failed to download metadata for repo 'beaker-HighAvailability-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 07:20:02 rdma06 dnf[1183004]: beaker-NFV                                       65 kB/s | 168  B     00:00
Apr 18 07:20:02 rdma06 dnf[1183004]: Errors during downloading metadata for repository 'beaker-NFV':
Apr 18 07:20:02 rdma06 dnf[1183004]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 07:20:02 rdma06 dnf[1183004]: Error: Failed to download metadata for repo 'beaker-NFV': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 07:20:02 rdma06 dnf[1183004]: beaker-NFV-debuginfo                             64 kB/s | 168  B     00:00
Apr 18 07:20:02 rdma06 dnf[1183004]: Errors during downloading metadata for repository 'beaker-NFV-debuginfo':
Apr 18 07:20:02 rdma06 dnf[1183004]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 07:20:02 rdma06 dnf[1183004]: Error: Failed to download metadata for repo 'beaker-NFV-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 07:20:02 rdma06 dnf[1183004]: beaker-RT                                        61 kB/s | 168  B     00:00
Apr 18 07:20:02 rdma06 dnf[1183004]: Errors during downloading metadata for repository 'beaker-RT':
Apr 18 07:20:02 rdma06 dnf[1183004]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 07:20:02 rdma06 dnf[1183004]: Error: Failed to download metadata for repo 'beaker-RT': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 07:20:02 rdma06 dnf[1183004]: beaker-RT-debuginfo                              66 kB/s | 168  B     00:00
Apr 18 07:20:02 rdma06 dnf[1183004]: Errors during downloading metadata for repository 'beaker-RT-debuginfo':
Apr 18 07:20:02 rdma06 dnf[1183004]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 07:20:02 rdma06 dnf[1183004]: Error: Failed to download metadata for repo 'beaker-RT-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 07:20:02 rdma06 dnf[1183004]: beaker-ResilientStorage                          68 kB/s | 168  B     00:00
Apr 18 07:20:02 rdma06 dnf[1183004]: Errors during downloading metadata for repository 'beaker-ResilientStorage':
Apr 18 07:20:02 rdma06 dnf[1183004]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 07:20:02 rdma06 dnf[1183004]: Error: Failed to download metadata for repo 'beaker-ResilientStorage': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 07:20:02 rdma06 dnf[1183004]: beaker-ResilientStorage-debuginfo                65 kB/s | 168  B     00:00
Apr 18 07:20:02 rdma06 dnf[1183004]: Errors during downloading metadata for repository 'beaker-ResilientStorage-debuginfo':
Apr 18 07:20:02 rdma06 dnf[1183004]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 07:20:02 rdma06 dnf[1183004]: Error: Failed to download metadata for repo 'beaker-ResilientStorage-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 07:20:02 rdma06 dnf[1183004]: beaker-SAP                                       59 kB/s | 168  B     00:00
Apr 18 07:20:02 rdma06 dnf[1183004]: Errors during downloading metadata for repository 'beaker-SAP':
Apr 18 07:20:02 rdma06 dnf[1183004]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 07:20:02 rdma06 dnf[1183004]: Error: Failed to download metadata for repo 'beaker-SAP': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 07:20:02 rdma06 dnf[1183004]: beaker-SAP-debuginfo                             57 kB/s | 168  B     00:00
Apr 18 07:20:02 rdma06 dnf[1183004]: Errors during downloading metadata for repository 'beaker-SAP-debuginfo':
Apr 18 07:20:02 rdma06 dnf[1183004]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 07:20:02 rdma06 dnf[1183004]: Error: Failed to download metadata for repo 'beaker-SAP-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 07:20:02 rdma06 dnf[1183004]: beaker-SAPHANA                                   64 kB/s | 168  B     00:00
Apr 18 07:20:02 rdma06 dnf[1183004]: Errors during downloading metadata for repository 'beaker-SAPHANA':
Apr 18 07:20:02 rdma06 dnf[1183004]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 07:20:02 rdma06 dnf[1183004]: Error: Failed to download metadata for repo 'beaker-SAPHANA': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 07:20:02 rdma06 dnf[1183004]: beaker-SAPHANA-debuginfo                         56 kB/s | 168  B     00:00
Apr 18 07:20:02 rdma06 dnf[1183004]: Errors during downloading metadata for repository 'beaker-SAPHANA-debuginfo':
Apr 18 07:20:02 rdma06 dnf[1183004]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 07:20:02 rdma06 dnf[1183004]: Error: Failed to download metadata for repo 'beaker-SAPHANA-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 07:20:03 rdma06 dnf[1183004]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr 18 07:20:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:20:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:20:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:20:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:20:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:20:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:20:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:20:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:20:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:20:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:20:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:20:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:20:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:20:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:20:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:20:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:20:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:20:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:20:04 rdma06 dnf[1183004]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr 18 07:20:04 rdma06 dnf[1183004]: Ignoring repositories: beaker-AppStream, beaker-AppStream-debuginfo, beaker-BaseOS, beaker-BaseOS-debuginfo, beaker-CRB, beaker-CRB-debuginfo, beaker-HighAvailability, beaker-HighAvailability-debuginfo, beaker-NFV, beaker-NFV-debuginfo, beaker-RT, beaker-RT-debuginfo, beaker-ResilientStorage, beaker-ResilientStorage-debuginfo, beaker-SAP, beaker-SAP-debuginfo, beaker-SAPHANA, beaker-SAPHANA-debuginfo
Apr 18 07:20:04 rdma06 dnf[1183004]: Metadata cache created.
Apr 18 07:20:04 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 18 07:20:04 rdma06 systemd[1]: Finished dnf makecache.
Apr 18 07:20:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:20:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:20:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:20:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:20:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:20:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:20:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:20:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:20:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:20:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:20:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:20:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:20:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:20:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:20:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:20:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:20:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:20:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:20:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:20:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:20:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:20:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:20:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:20:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:20:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:20:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:20:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:20:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:20:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:20:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:20:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:20:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:20:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:20:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:20:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:20:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:20:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:20:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:20:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:20:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:20:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:20:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:20:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:20:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:20:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:20:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:20:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:20:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:20:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:20:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:20:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:20:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:20:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:20:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:20:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:20:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:20:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:20:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:20:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:20:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:20:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:20:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:20:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:20:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:20:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:20:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:20:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:20:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:20:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:20:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:20:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:20:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:20:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:20:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:20:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:20:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:20:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:20:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:20:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:20:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:20:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:20:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:20:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:20:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:20:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:20:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:20:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:20:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:20:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:20:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:20:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:20:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:20:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:20:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:20:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:20:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:20:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:20:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:20:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:20:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:20:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:20:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:20:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:20:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:20:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:20:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:20:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:20:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:20:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:20:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:20:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:20:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:20:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:20:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:20:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:20:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:20:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:20:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:20:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:20:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:20:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:20:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:20:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:20:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:20:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:20:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:20:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:20:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:20:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:20:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:20:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:20:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:20:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:20:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:20:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:20:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:20:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:20:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:20:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:20:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:20:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:20:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:20:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:20:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:20:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:20:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:20:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:20:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:20:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:20:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:20:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:20:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:20:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:20:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:20:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:20:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:20:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:20:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:20:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:20:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:20:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:20:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:20:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:20:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:20:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:20:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:20:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:20:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:20:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:20:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:20:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:20:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:20:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:20:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:20:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:20:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:20:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:20:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:20:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:20:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:20:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:20:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:20:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:20:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:20:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:20:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:20:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:20:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:20:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:20:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:20:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:20:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:20:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:20:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:20:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:20:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:20:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:20:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:21:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:21:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:21:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:21:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:21:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:21:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:21:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:21:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:21:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:21:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:21:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:21:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:21:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:21:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:21:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:21:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:21:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:21:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:21:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:21:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:21:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:21:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:21:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:21:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:21:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:21:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:21:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:21:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:21:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:21:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:21:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:21:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:21:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:21:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:21:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:21:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:21:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:21:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:21:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:21:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:21:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:21:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:21:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:21:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:21:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:21:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:21:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:21:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:21:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:21:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:21:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:21:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:21:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:21:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:21:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:21:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:21:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:21:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:21:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:21:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:21:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:21:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:21:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:21:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:21:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:21:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:21:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:21:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:21:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:21:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:21:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:21:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:21:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:21:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:21:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:21:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:21:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:21:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:21:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:21:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:21:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:21:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:21:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:21:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:21:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:21:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:21:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:21:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:21:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:21:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:21:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:21:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:21:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:21:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:21:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:21:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:21:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:21:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:21:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:21:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:21:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:21:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:21:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:21:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:21:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:23:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:23:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:23:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:23:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:23:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:23:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:23:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:23:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:23:39 rdma06 rsyslogd[1041]: imjournal: 5374 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 07:23:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:23:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:23:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:23:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:23:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:23:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:23:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:23:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:23:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:23:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:23:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:23:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:23:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:23:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:23:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:23:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:23:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:23:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:23:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:23:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:23:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:23:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:23:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:23:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:23:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:23:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:23:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:23:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:23:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:23:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:23:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:23:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:24:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:24:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:24:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:24:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:24:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:24:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:24:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:24:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:24:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:24:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:24:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:24:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:24:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:24:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:24:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:24:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:24:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:24:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:24:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:24:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:24:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:24:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:24:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:24:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:24:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:24:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:24:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:24:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:24:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:24:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:24:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:24:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:24:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:24:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:24:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:24:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:24:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:24:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:24:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:24:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:24:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:24:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:24:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:24:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:24:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:24:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:24:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:24:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:24:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:24:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:24:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:24:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:24:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:24:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:24:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:24:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:24:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:24:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:24:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:24:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:24:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:24:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:24:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:24:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:24:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:24:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:24:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:24:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:24:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:24:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:24:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:24:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:24:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:24:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:24:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:24:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:24:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:24:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:24:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:24:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:24:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:24:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:24:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:24:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:24:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:24:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:24:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:24:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:24:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:24:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:24:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:24:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:24:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:24:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:24:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:24:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:24:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:24:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:24:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:24:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:24:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:24:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:24:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:24:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:24:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:24:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:24:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:24:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:24:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:24:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:24:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:24:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:24:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:24:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:24:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:24:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:24:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:24:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:24:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:24:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:24:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:24:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:24:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:24:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:24:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:24:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:24:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:24:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:24:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:24:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:24:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:24:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:24:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:24:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:24:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:24:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:24:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:24:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:24:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:24:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:24:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:24:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:24:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:24:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:24:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:24:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:24:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:24:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:24:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:24:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:24:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:24:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:24:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:24:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:24:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:24:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:24:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:24:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:24:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:24:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:24:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:24:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:24:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:24:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:24:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:24:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:24:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:24:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:24:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:24:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:24:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:24:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:24:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:24:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:24:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:24:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:24:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:24:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:24:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:24:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:24:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:24:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:24:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:24:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:24:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:24:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:24:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:24:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:24:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:24:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:24:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:24:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:24:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:24:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:24:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:24:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:25:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:25:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:25:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:25:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:25:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:25:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:25:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:25:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:25:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:25:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:25:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:25:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:25:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:25:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:25:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:25:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:25:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:25:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:25:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:25:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:25:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:25:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:25:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:25:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:25:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:25:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:25:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:25:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:25:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:25:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:25:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:25:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:25:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:25:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:25:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:25:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:25:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:25:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:25:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:25:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:25:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:25:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:25:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:25:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:25:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:25:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:25:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:25:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:25:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:25:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:25:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:25:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:25:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:25:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:25:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:25:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:25:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:25:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:25:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:25:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:25:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:25:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:25:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:25:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:25:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:25:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:25:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:25:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:25:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:25:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:25:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:25:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:25:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:25:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:25:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:25:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:25:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:25:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:25:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:25:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:25:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:25:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:25:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:25:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:25:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:25:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:25:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:25:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:25:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:25:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:25:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:25:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:25:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:25:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:25:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:25:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:25:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:25:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:25:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:25:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:25:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:25:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:25:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:25:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:25:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:25:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:25:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:25:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:25:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:25:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:25:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:25:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:25:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:25:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:25:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:25:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:25:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:25:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:25:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:25:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:25:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:25:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:25:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:25:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:25:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:25:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:25:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:25:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:25:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:25:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:25:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:25:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:25:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:25:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:25:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:25:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:25:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:25:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:25:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:25:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:25:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:25:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:25:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:25:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:25:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:25:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:25:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:25:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:25:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:25:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:25:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:25:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:25:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:25:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:25:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:25:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:25:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:25:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:25:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:25:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:25:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:25:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:25:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:25:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:25:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:25:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:26:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:26:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:26:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:26:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:26:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:26:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:26:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:26:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:26:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:26:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:26:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:26:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:26:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:26:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:26:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:26:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:26:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:26:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:26:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:26:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:26:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:26:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:26:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:26:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:26:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:26:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:26:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:26:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:26:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:26:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:26:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:26:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:26:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:26:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:26:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:26:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:26:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:26:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:26:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:26:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:26:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:26:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:26:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:26:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:26:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:26:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:26:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:26:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:26:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:26:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:26:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:26:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:26:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:26:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:26:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:26:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:26:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:26:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:26:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:26:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:26:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:26:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:26:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:26:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:26:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:26:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:26:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:26:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:26:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:26:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:26:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:26:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:26:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:26:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:26:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:26:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:26:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:26:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:26:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:26:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:26:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:26:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:26:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:26:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:26:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:26:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:26:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:26:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:26:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:26:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:26:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:26:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:26:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:26:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:26:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:26:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:26:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:26:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:26:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:26:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:26:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:26:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:26:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:26:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:26:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:26:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:26:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:26:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:26:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:26:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:26:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:26:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:26:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:26:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:26:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:26:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:26:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:26:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:26:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:26:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:26:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:26:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:26:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:26:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:26:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:26:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:26:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:26:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:26:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:26:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:26:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:26:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:26:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:26:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:26:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:26:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:26:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:26:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:26:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:26:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:26:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:26:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:26:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:26:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:26:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:26:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:26:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:26:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:26:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:26:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:26:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:26:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:26:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:26:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:26:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:26:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:26:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:26:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:26:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:26:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:26:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:26:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:26:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:26:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:26:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:26:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:26:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:26:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:26:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:26:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:26:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:26:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:26:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:26:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:26:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:26:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:26:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:26:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:26:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:26:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:26:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:26:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:26:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:26:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:26:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:26:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:26:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:26:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:26:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:26:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:26:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:26:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:26:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:26:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:26:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:26:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:26:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:26:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:26:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:26:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:26:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:26:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:26:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:26:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:26:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:26:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:26:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:26:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:26:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:26:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:26:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:26:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:26:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:26:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:26:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:26:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:27:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:27:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:27:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:27:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:27:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:27:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:27:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:27:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:27:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:27:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:27:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:27:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:27:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:27:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:27:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:27:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:27:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:27:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:27:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:27:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:27:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:27:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:27:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:27:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:27:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:27:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:27:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:27:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:27:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:27:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:27:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:27:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:27:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:27:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:27:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:27:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:27:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:27:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:27:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:27:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:27:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:27:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:27:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:27:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:27:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:27:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:27:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:27:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:27:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:27:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:27:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:27:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:27:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:27:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:27:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:27:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:27:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:27:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:27:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:27:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:27:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:27:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:27:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:27:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:27:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:27:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:27:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:27:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:27:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:27:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:27:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:27:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:27:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:27:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:27:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:27:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:27:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:27:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:27:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:27:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:27:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:27:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:27:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:27:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:27:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:27:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:27:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:27:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:27:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:27:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:27:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:27:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:27:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:27:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:27:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:27:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:27:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:27:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:27:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:27:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:27:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:27:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:27:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:27:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:27:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:27:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:27:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:27:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:27:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:27:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:27:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:27:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:27:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:27:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:27:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:27:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:27:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:27:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:27:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:27:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:27:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:27:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:27:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:27:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:27:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:27:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:27:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:27:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:27:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:27:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:27:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:27:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:27:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:27:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:27:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:27:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:27:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:27:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:27:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:27:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:27:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:27:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:27:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:27:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:27:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:27:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:27:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:27:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:27:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:27:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:27:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:27:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:27:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:27:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:27:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:27:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:27:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:27:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:27:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:27:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:27:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:27:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:27:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:27:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:27:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:27:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:27:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:27:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:27:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:27:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:27:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:27:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:27:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:27:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:27:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:27:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:27:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:27:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:27:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:27:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:27:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:27:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:27:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:27:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:27:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:27:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:27:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:27:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:27:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:27:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:27:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:27:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:27:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:27:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:27:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:27:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:27:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:27:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:27:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:27:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:27:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:27:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:27:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:27:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:27:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:27:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:27:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:27:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:27:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:27:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:27:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:27:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:27:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:27:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:27:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:27:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:28:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:28:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:28:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:28:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:28:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:28:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:28:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:28:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:28:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:28:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:28:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:28:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:28:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:28:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:28:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:28:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:28:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:28:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:28:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:28:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:28:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:28:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:28:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:28:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:28:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:28:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:28:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:28:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:28:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:28:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:28:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:28:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:28:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:28:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:28:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:28:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:28:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:28:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:28:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:28:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:28:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:28:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:28:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:28:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:28:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:28:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:28:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:28:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:28:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:28:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:28:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:28:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:28:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:28:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:28:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:28:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:28:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:28:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:28:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:28:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:28:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:28:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:28:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:28:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:28:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:28:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:28:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:28:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:28:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:28:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:28:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:28:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:28:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:28:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:28:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:28:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:28:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:28:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:28:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:28:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:28:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:28:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:28:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:28:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:28:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:28:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:28:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:28:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:28:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:28:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:28:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:28:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:28:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:28:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:28:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:28:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:28:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:28:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:28:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:28:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:28:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:28:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:28:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:28:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:28:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:28:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:28:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:28:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:28:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:28:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:28:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:28:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:28:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:28:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:28:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:28:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:28:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:28:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:28:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:28:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:28:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:28:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:28:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:28:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:28:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:28:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:28:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:28:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:28:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:28:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:28:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:28:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:28:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:28:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:28:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:28:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:28:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:28:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:28:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:28:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:28:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:28:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:28:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:28:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:28:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:28:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:28:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:28:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:28:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:28:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:28:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:28:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:28:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:28:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:28:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:28:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:28:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:28:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:28:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:28:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:28:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:28:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:28:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:28:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:28:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:28:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:28:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:28:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:28:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:28:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:28:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:28:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:28:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:28:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:28:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:28:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:28:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:28:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:28:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:28:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:28:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:28:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:28:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:28:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:28:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:28:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:28:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:28:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:28:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:28:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:28:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:28:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:28:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:28:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:28:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:28:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:28:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:28:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:28:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:28:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:28:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:28:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:28:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:28:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:28:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:28:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:28:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:28:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:28:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:28:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:28:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:28:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:28:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:28:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:28:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:28:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:29:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:29:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:29:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:29:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:29:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:29:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:29:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:29:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:29:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:29:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:29:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:29:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:29:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:29:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:29:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:29:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:29:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:29:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:29:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:29:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:29:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:29:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:29:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:29:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:29:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:29:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:29:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:29:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:29:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:29:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:29:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:29:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:29:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:29:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:29:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:29:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:29:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:29:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:29:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:29:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:29:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:29:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:29:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:29:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:29:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:29:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:29:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:29:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:29:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:29:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:29:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:29:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:29:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:29:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:29:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:29:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:29:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:29:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:29:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:29:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:29:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:29:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:29:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:29:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:29:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:29:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:29:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:29:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:29:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:29:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:29:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:29:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:29:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:29:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:29:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:29:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:29:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:29:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:29:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:29:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:29:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:29:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:29:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:29:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:29:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:29:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:29:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:29:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:29:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:29:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:29:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:29:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:29:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:29:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:29:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:29:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:29:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:29:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:29:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:29:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:29:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:29:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:29:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:29:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:29:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:29:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:29:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:29:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:29:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:29:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:29:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:29:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:29:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:29:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:29:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:29:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:29:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:29:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:29:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:29:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:29:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:29:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:29:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:29:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:29:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:29:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:29:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:29:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:29:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:29:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:29:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:29:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:29:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:29:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:29:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:29:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:29:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:29:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:29:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:29:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:29:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:29:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:29:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:29:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:29:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:29:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:29:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:29:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:29:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:29:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:29:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:29:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:29:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:29:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:29:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:29:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:29:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:29:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:29:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:29:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:29:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:29:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:29:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:29:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:29:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:29:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:29:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:29:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:29:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:29:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:29:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:29:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:29:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:29:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:29:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:29:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:29:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:29:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:29:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:29:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:29:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:29:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:29:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:29:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:29:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:29:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:29:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:29:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:29:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:29:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:29:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:29:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:29:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:29:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:29:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:29:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:29:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:29:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:29:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:29:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:29:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:29:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:29:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:29:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:29:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:29:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:29:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:29:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:29:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:29:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:29:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:29:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:29:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:29:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:29:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:29:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:30:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:30:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:30:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:30:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:30:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:30:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:30:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:30:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:30:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:30:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:30:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:30:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:30:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:30:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:30:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:30:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:30:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:30:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:30:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:30:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:30:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:30:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:30:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:30:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:30:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:30:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:30:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:30:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:30:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:30:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:30:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:30:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:30:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:30:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:30:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:30:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:30:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:30:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:30:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:30:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:30:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:30:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:30:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:30:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:30:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:30:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:30:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:30:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:30:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:30:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:30:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:30:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:30:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:30:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:30:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:30:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:30:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:30:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:30:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:30:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:30:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:30:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:30:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:30:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:30:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:30:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:30:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:30:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:30:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:30:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:30:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:30:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:30:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:30:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:30:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:30:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:30:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:30:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:30:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:30:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:30:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:30:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:30:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:30:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:30:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:30:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:30:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:30:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:30:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:30:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:30:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:30:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:30:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:30:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:30:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:30:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:30:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:30:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:30:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:30:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:30:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:30:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:30:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:30:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:30:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:30:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:30:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:30:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:30:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:30:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:30:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:30:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:30:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:30:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:30:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:30:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:31:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:31:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:31:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:31:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:31:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:31:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:31:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:31:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:31:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:31:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:31:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:31:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:31:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:31:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:31:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:31:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:31:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:31:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:31:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:31:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:31:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:31:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:31:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:31:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:31:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:31:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:31:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:31:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:31:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:31:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:31:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:31:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:31:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:31:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:31:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:31:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:31:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:31:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:31:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:31:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:31:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:31:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:31:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:31:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:31:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:31:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:31:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:31:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:31:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:31:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:31:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:31:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:31:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:31:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:31:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:31:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:31:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:31:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:31:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:31:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:31:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:31:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:31:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:31:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:31:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:31:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:31:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:31:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:31:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:31:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:31:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:31:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:31:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:31:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:31:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:31:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:31:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:31:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:31:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:31:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:31:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:31:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:31:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:31:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:31:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:31:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:31:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:31:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:31:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:31:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:31:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:31:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:31:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:31:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:31:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:31:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:31:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:31:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:31:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:31:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:31:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:31:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:31:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:31:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:31:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:31:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:31:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:31:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:31:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:31:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:31:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:31:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:31:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:31:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:31:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:31:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:31:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:31:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:31:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:31:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:31:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:31:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:31:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:33:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:33:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:33:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:33:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:33:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:33:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:33:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:33:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:33:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:33:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:33:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:33:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:33:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:33:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:33:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:33:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:33:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:33:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:33:44 rdma06 rsyslogd[1041]: imjournal: 5296 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 07:33:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:33:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:33:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:33:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:33:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:33:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:33:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:33:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:33:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:33:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:33:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:33:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:33:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:33:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:33:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:33:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:33:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:33:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:33:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:33:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:33:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:33:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:33:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:33:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:33:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:33:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:33:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:33:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:33:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:33:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:33:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:33:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:33:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:33:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:33:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:33:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:33:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:33:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:33:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:33:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:33:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:33:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:33:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:33:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:33:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:33:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:33:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:33:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:33:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:33:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:33:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:33:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:33:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:33:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:34:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:34:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:34:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:34:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:34:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:34:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:34:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:34:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:34:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:34:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:34:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:34:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:34:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:34:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:34:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:34:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:34:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:34:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:34:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:34:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:34:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:34:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:34:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:34:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:34:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:34:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:34:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:34:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:34:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:34:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:34:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:34:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:34:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:34:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:34:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:34:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:34:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:34:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:34:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:34:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:34:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:34:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:34:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:34:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:34:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:34:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:34:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:34:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:34:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:34:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:34:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:34:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:34:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:34:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:34:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:34:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:34:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:34:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:34:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:34:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:34:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:34:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:34:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:34:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:34:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:34:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:34:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:34:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:34:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:34:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:34:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:34:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:34:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:34:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:34:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:34:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:34:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:34:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:34:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:34:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:34:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:34:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:34:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:34:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:34:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:34:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:34:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:34:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:34:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:34:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:34:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:34:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:34:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:34:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:34:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:34:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:34:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:34:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:34:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:34:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:34:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:34:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:34:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:34:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:34:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:34:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:34:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:34:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:34:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:34:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:34:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:34:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:34:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:34:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:34:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:34:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:34:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:34:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:34:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:34:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:34:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:34:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:34:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:34:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:34:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:34:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:34:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:34:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:34:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:34:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:34:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:34:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:34:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:34:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:34:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:34:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:34:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:34:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:34:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:34:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:34:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:34:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:34:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:34:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:34:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:34:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:34:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:34:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:34:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:34:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:34:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:34:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:34:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:34:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:34:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:34:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:34:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:34:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:34:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:34:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:34:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:34:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:34:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:34:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:34:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:34:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:34:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:34:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:34:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:34:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:34:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:34:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:34:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:34:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:34:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:34:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:34:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:34:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:34:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:34:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:34:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:34:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:34:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:34:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:34:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:34:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:34:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:34:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:34:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:34:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:34:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:34:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:34:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:34:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:34:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:34:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:34:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:34:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:34:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:34:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:34:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:34:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:34:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:34:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:34:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:34:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:34:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:34:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:34:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:34:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:34:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:34:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:34:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:34:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:34:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:34:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:35:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:35:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:35:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:35:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:35:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:35:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:35:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:35:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:35:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:35:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:35:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:35:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:35:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:35:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:35:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:35:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:35:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:35:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:35:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:35:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:35:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:35:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:35:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:35:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:35:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:35:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:35:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:35:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:35:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:35:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:35:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:35:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:35:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:35:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:35:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:35:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:35:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:35:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:35:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:35:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:35:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:35:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:35:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:35:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:35:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:35:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:35:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:35:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:35:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:35:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:35:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:35:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:35:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:35:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:35:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:35:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:35:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:35:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:35:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:35:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:35:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:35:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:35:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:35:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:35:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:35:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:35:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:35:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:35:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:35:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:35:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:35:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:35:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:35:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:35:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:35:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:35:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:35:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:35:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:35:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:35:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:35:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:35:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:35:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:35:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:35:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:35:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:35:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:35:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:35:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:35:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:35:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:35:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:35:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:35:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:35:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:35:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:35:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:35:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:35:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:35:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:35:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:35:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:35:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:35:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:35:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:35:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:35:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:35:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:35:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:35:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:35:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:35:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:35:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:35:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:35:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:35:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:35:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:35:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:35:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:35:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:35:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:35:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:35:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:35:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:35:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:35:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:35:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:35:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:35:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:35:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:35:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:35:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:35:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:35:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:35:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:35:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:35:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:35:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:35:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:35:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:35:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:35:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:35:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:35:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:35:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:35:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:35:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:35:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:35:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:35:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:35:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:35:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:35:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:35:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:35:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:35:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:35:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:35:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:35:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:35:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:35:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:35:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:35:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:35:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:35:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:35:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:35:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:35:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:35:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:35:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:35:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:35:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:35:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:35:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:35:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:35:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:35:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:35:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:35:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:35:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:35:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:35:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:35:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:35:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:35:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:35:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:35:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:35:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:35:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:35:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:35:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:35:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:35:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:35:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:35:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:35:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:35:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:35:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:35:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:35:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:35:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:35:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:35:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:35:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:35:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:35:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:35:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:35:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:35:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:35:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:35:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:35:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:35:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:35:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:35:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:36:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:36:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:36:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:36:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:36:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:36:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:36:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:36:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:36:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:36:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:36:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:36:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:36:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:36:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:36:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:36:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:36:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:36:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:36:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:36:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:36:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:36:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:36:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:36:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:36:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:36:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:36:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:36:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:36:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:36:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:36:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:36:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:36:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:36:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:36:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:36:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:36:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:36:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:36:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:36:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:36:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:36:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:36:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:36:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:36:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:36:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:36:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:36:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:36:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:36:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:36:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:36:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:36:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:36:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:36:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:36:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:36:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:36:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:36:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:36:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:36:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:36:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:36:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:36:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:36:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:36:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:36:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:36:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:36:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:36:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:36:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:36:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:36:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:36:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:36:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:36:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:36:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:36:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:36:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:36:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:36:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:36:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:36:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:36:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:36:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:36:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:36:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:36:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:36:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:36:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:36:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:36:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:36:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:36:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:36:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:36:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:36:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:36:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:36:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:36:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:36:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:36:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:36:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:36:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:36:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:36:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:36:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:36:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:36:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:36:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:36:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:36:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:36:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:36:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:36:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:36:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:36:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:36:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:36:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:36:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:36:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:36:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:36:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:36:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:36:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:36:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:36:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:36:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:36:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:36:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:36:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:36:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:36:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:36:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:36:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:36:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:36:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:36:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:36:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:36:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:36:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:36:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:36:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:36:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:36:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:36:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:36:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:36:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:36:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:36:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:36:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:36:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:36:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:36:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:36:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:36:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:36:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:36:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:36:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:36:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:36:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:36:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:36:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:36:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:36:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:36:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:36:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:36:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:36:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:36:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:36:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:36:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:36:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:36:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:36:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:36:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:36:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:36:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:36:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:36:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:36:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:36:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:36:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:36:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:36:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:36:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:36:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:36:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:36:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:36:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:36:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:36:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:36:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:36:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:36:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:36:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:36:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:36:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:36:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:36:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:36:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:36:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:36:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:36:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:36:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:36:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:36:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:36:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:36:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:36:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:36:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:36:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:36:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:36:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:36:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:36:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:37:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:37:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:37:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:37:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:37:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:37:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:37:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:37:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:37:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:37:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:37:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:37:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:37:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:37:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:37:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:37:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:37:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:37:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:37:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:37:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:37:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:37:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:37:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:37:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:37:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:37:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:37:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:37:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:37:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:37:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:37:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:37:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:37:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:37:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:37:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:37:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:37:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:37:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:37:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:37:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:37:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:37:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:37:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:37:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:37:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:37:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:37:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:37:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:37:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:37:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:37:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:37:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:37:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:37:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:37:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:37:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:37:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:37:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:37:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:37:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:37:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:37:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:37:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:37:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:37:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:37:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:37:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:37:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:37:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:37:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:37:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:37:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:37:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:37:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:37:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:37:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:37:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:37:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:37:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:37:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:37:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:37:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:37:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:37:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:37:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:37:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:37:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:37:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:37:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:37:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:37:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:37:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:37:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:37:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:37:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:37:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:37:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:37:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:37:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:37:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:37:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:37:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:37:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:37:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:37:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:37:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:37:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:37:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:37:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:37:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:37:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:37:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:37:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:37:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:37:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:37:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:37:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:37:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:37:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:37:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:37:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:37:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:37:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:37:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:37:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:37:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:37:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:37:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:37:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:37:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:37:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:37:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:37:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:37:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:37:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:37:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:37:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:37:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:37:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:37:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:37:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:37:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:37:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:37:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:37:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:37:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:37:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:37:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:37:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:37:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:37:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:37:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:37:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:37:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:37:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:37:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:37:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:37:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:37:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:37:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:37:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:37:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:37:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:37:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:37:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:37:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:37:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:37:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:37:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:37:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:37:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:37:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:37:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:37:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:37:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:37:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:37:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:37:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:37:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:37:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:37:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:37:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:37:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:37:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:37:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:37:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:37:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:37:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:37:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:37:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:37:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:37:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:37:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:37:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:37:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:37:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:37:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:37:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:37:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:37:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:37:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:37:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:37:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:37:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:37:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:37:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:37:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:37:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:37:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:37:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:37:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:37:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:37:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:37:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:37:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:37:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:38:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:38:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:38:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:38:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:38:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:38:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:38:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:38:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:38:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:38:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:38:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:38:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:38:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:38:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:38:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:38:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:38:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:38:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:38:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:38:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:38:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:38:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:38:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:38:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:38:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:38:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:38:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:38:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:38:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:38:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:38:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:38:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:38:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:38:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:38:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:38:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:38:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:38:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:38:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:38:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:38:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:38:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:38:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:38:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:38:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:38:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:38:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:38:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:38:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:38:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:38:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:38:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:38:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:38:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:38:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:38:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:38:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:38:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:38:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:38:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:38:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:38:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:38:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:38:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:38:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:38:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:38:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:38:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:38:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:38:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:38:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:38:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:38:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:38:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:38:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:38:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:38:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:38:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:38:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:38:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:38:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:38:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:38:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:38:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:38:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:38:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:38:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:38:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:38:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:38:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:38:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:38:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:38:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:38:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:38:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:38:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:38:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:38:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:38:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:38:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:38:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:38:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:38:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:38:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:38:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:38:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:38:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:38:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:38:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:38:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:38:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:38:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:38:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:38:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:38:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:38:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:38:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:38:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:38:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:38:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:38:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:38:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:38:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:38:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:38:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:38:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:38:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:38:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:38:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:38:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:38:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:38:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:38:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:38:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:38:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:38:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:38:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:38:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:38:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:38:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:38:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:38:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:38:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:38:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:38:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:38:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:38:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:38:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:38:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:38:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:38:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:38:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:38:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:38:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:38:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:38:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:38:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:38:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:38:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:38:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:38:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:38:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:38:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:38:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:38:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:38:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:38:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:38:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:38:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:38:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:38:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:38:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:38:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:38:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:38:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:38:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:38:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:38:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:38:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:38:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:38:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:38:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:38:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:38:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:38:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:38:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:38:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:38:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:38:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:38:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:38:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:38:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:38:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:38:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:38:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:38:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:38:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:38:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:38:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:38:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:38:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:38:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:38:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:38:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:38:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:38:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:38:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:38:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:38:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:38:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:38:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:38:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:38:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:38:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:38:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:38:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:39:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:39:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:39:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:39:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:39:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:39:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:39:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:39:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:39:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:39:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:39:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:39:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:39:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:39:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:39:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:39:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:39:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:39:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:39:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:39:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:39:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:39:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:39:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:39:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:39:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:39:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:39:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:39:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:39:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:39:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:39:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:39:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:39:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:39:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:39:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:39:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:39:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:39:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:39:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:39:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:39:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:39:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:39:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:39:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:39:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:39:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:39:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:39:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:39:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:39:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:39:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:39:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:39:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:39:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:39:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:39:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:39:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:39:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:39:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:39:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:39:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:39:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:39:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:39:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:39:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:39:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:39:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:39:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:39:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:39:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:39:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:39:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:39:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:39:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:39:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:39:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:39:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:39:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:39:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:39:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:39:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:39:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:39:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:39:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:39:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:39:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:39:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:39:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:39:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:39:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:39:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:39:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:39:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:39:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:39:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:39:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:39:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:39:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:39:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:39:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:39:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:39:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:39:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:39:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:39:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:39:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:39:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:39:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:39:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:39:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:39:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:39:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:39:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:39:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:39:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:39:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:39:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:39:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:39:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:39:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:39:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:39:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:39:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:39:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:39:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:39:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:39:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:39:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:39:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:39:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:39:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:39:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:39:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:39:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:39:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:39:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:39:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:39:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:39:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:39:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:39:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:39:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:39:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:39:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:39:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:39:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:39:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:39:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:39:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:39:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:39:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:39:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:39:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:39:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:39:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:39:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:40:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:40:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:40:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:40:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:40:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:40:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:40:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:40:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:40:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:40:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:40:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:40:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:40:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:40:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:40:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:40:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:40:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:40:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:40:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:40:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:40:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:40:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:40:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:40:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:40:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:40:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:40:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:40:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:40:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:40:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:40:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:40:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:40:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:40:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:40:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:40:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:40:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:40:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:40:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:40:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:40:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:40:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:40:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:40:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:40:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:40:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:40:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:40:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:40:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:40:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:40:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:40:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:40:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:40:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:40:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:40:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:40:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:40:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:40:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:40:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:40:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:40:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:40:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:40:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:40:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:40:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:40:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:40:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:40:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:40:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:40:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:40:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:40:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:40:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:40:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:40:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:40:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:40:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:40:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:40:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:40:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:40:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:40:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:40:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:40:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:40:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:40:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:40:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:40:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:40:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:40:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:40:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:40:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:40:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:40:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:40:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:40:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:40:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:40:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:40:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:40:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:40:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:40:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:40:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:40:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:40:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:40:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:40:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:40:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:40:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:40:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:40:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:40:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:40:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:40:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:40:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:40:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:40:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:40:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:40:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:40:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:40:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:40:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:40:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:40:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:40:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:40:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:40:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:40:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:40:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:40:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:40:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:40:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:40:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:40:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:40:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:40:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:40:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:40:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:40:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:40:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:40:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:40:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:40:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:40:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:40:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:40:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:40:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:40:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:40:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:40:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:40:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:40:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:40:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:40:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:40:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:40:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:40:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:40:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:40:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:40:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:40:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:40:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:40:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:40:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:40:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:40:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:40:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:40:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:40:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:40:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:40:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:40:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:40:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:40:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:40:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:40:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:40:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:40:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:40:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:40:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:40:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:40:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:40:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:40:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:40:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:40:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:40:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:40:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:40:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:40:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:40:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:40:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:40:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:40:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:40:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:40:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:40:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:40:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:40:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:40:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:40:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:40:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:40:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:40:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:40:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:40:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:40:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:40:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:40:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:40:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:40:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:40:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:40:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:40:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:40:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:41:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:41:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:41:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:41:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:41:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:41:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:41:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:41:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:41:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:41:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:41:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:41:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:41:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:41:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:41:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:41:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:41:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:41:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:41:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:41:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:41:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:41:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:41:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:41:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:41:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:41:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:41:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:41:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:41:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:41:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:41:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:41:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:41:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:41:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:41:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:41:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:41:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:41:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:41:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:41:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:41:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:41:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:41:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:41:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:41:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:41:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:41:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:41:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:41:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:41:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:41:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:41:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:41:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:41:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:41:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:41:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:41:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:41:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:41:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:41:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:41:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:41:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:41:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:41:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:41:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:41:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:41:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:41:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:41:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:41:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:41:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:41:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:41:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:41:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:41:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:41:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:41:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:41:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:41:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:41:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:41:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:41:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:41:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:41:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:41:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:41:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:41:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:41:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:41:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:41:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:41:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:41:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:41:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:41:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:41:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:41:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:41:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:41:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:41:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:41:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:41:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:41:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:41:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:41:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:41:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:41:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:41:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:41:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:41:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:41:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:41:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:41:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:41:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:41:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:41:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:41:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:41:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:41:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:41:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:41:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:41:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:41:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:41:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:41:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:41:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:41:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:41:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:41:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:41:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:41:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:41:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:41:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:41:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:41:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:41:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:41:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:41:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:41:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:41:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:41:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:41:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:41:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:41:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:41:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:43:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:43:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:43:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:43:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:43:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:43:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:43:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:43:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:43:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:43:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:43:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:43:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:43:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:43:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:43:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:43:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:43:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:43:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:43:49 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 07:43:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:43:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:43:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:43:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:43:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:43:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:43:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:43:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:43:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:43:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:43:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:43:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:43:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:43:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:43:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:43:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:43:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:43:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:43:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:43:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:43:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:43:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:43:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:43:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:43:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:43:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:43:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:43:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:43:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:43:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:43:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:43:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:43:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:43:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:43:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:43:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:44:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:44:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:44:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:44:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:44:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:44:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:44:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:44:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:44:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:44:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:44:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:44:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:44:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:44:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:44:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:44:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:44:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:44:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:44:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:44:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:44:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:44:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:44:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:44:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:44:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:44:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:44:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:44:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:44:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:44:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:44:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:44:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:44:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:44:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:44:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:44:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:44:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:44:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:44:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:44:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:44:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:44:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:44:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:44:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:44:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:44:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:44:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:44:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:44:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:44:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:44:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:44:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:44:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:44:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:44:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:44:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:44:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:44:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:44:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:44:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:44:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:44:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:44:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:44:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:44:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:44:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:44:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:44:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:44:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:44:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:44:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:44:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:44:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:44:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:44:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:44:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:44:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:44:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:44:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:44:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:44:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:44:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:44:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:44:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:44:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:44:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:44:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:44:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:44:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:44:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:44:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:44:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:44:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:44:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:44:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:44:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:44:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:44:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:44:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:44:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:44:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:44:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:44:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:44:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:44:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:44:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:44:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:44:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:44:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:44:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:44:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:44:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:44:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:44:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:44:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:44:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:44:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:44:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:44:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:44:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:44:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:44:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:44:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:44:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:44:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:44:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:44:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:44:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:44:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:44:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:44:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:44:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:44:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:44:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:44:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:44:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:44:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:44:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:44:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:44:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:44:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:44:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:44:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:44:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:44:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:44:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:44:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:44:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:44:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:44:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:44:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:44:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:44:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:44:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:44:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:44:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:44:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:44:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:44:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:44:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:44:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:44:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:44:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:44:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:44:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:44:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:44:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:44:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:44:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:44:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:44:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:44:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:44:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:44:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:44:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:44:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:44:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:44:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:44:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:44:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:44:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:44:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:44:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:44:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:44:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:44:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:44:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:44:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:44:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:44:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:44:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:44:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:44:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:44:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:44:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:44:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:44:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:44:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:44:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:44:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:44:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:44:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:44:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:44:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:44:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:44:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:44:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:44:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:44:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:44:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:44:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:44:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:44:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:44:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:44:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:44:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:45:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:45:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:45:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:45:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:45:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:45:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:45:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:45:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:45:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:45:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:45:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:45:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:45:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:45:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:45:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:45:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:45:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:45:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:45:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:45:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:45:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:45:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:45:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:45:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:45:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:45:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:45:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:45:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:45:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:45:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:45:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:45:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:45:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:45:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:45:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:45:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:45:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:45:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:45:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:45:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:45:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:45:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:45:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:45:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:45:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:45:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:45:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:45:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:45:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:45:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:45:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:45:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:45:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:45:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:45:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:45:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:45:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:45:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:45:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:45:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:45:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:45:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:45:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:45:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:45:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:45:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:45:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:45:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:45:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:45:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:45:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:45:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:45:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:45:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:45:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:45:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:45:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:45:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:45:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:45:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:45:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:45:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:45:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:45:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:45:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:45:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:45:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:45:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:45:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:45:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:45:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:45:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:45:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:45:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:45:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:45:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:45:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:45:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:45:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:45:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:45:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:45:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:45:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:45:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:45:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:45:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:45:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:45:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:45:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:45:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:45:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:45:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:45:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:45:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:45:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:45:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:45:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:45:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:45:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:45:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:45:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:45:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:45:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:45:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:45:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:45:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:45:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:45:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:45:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:45:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:45:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:45:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:45:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:45:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:45:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:45:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:45:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:45:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:45:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:45:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:45:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:45:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:45:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:45:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:45:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:45:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:45:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:45:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:45:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:45:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:45:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:45:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:45:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:45:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:45:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:45:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:45:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:45:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:45:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:45:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:45:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:45:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:45:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:45:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:45:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:45:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:45:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:45:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:45:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:45:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:45:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:45:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:45:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:45:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:45:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:45:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:45:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:45:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:45:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:45:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:45:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:45:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:45:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:45:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:45:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:45:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:45:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:45:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:45:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:45:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:45:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:45:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:45:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:45:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:45:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:45:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:45:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:45:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:45:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:45:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:45:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:45:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:45:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:45:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:45:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:45:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:45:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:45:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:45:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:45:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:45:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:45:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:45:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:45:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:45:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:45:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:46:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:46:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:46:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:46:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:46:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:46:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:46:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:46:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:46:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:46:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:46:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:46:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:46:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:46:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:46:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:46:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:46:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:46:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:46:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:46:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:46:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:46:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:46:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:46:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:46:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:46:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:46:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:46:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:46:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:46:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:46:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:46:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:46:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:46:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:46:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:46:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:46:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:46:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:46:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:46:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:46:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:46:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:46:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:46:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:46:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:46:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:46:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:46:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:46:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:46:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:46:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:46:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:46:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:46:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:46:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:46:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:46:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:46:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:46:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:46:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:46:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:46:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:46:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:46:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:46:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:46:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:46:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:46:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:46:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:46:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:46:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:46:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:46:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:46:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:46:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:46:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:46:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:46:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:46:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:46:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:46:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:46:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:46:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:46:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:46:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:46:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:46:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:46:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:46:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:46:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:46:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:46:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:46:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:46:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:46:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:46:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:46:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:46:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:46:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:46:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:46:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:46:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:46:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:46:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:46:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:46:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:46:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:46:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:46:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:46:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:46:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:46:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:46:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:46:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:46:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:46:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:46:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:46:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:46:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:46:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:46:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:46:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:46:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:46:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:46:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:46:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:46:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:46:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:46:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:46:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:46:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:46:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:46:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:46:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:46:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:46:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:46:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:46:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:46:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:46:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:46:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:46:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:46:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:46:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:46:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:46:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:46:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:46:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:46:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:46:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:46:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:46:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:46:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:46:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:46:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:46:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:46:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:46:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:46:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:46:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:46:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:46:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:46:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:46:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:46:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:46:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:46:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:46:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:46:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:46:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:46:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:46:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:46:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:46:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:46:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:46:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:46:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:46:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:46:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:46:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:46:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:46:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:46:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:46:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:46:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:46:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:46:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:46:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:46:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:46:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:46:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:46:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:46:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:46:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:46:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:46:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:46:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:46:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:46:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:46:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:46:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:46:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:46:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:46:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:46:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:46:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:46:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:46:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:46:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:46:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:46:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:46:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:46:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:46:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:46:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:46:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:47:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:47:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:47:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:47:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:47:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:47:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:47:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:47:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:47:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:47:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:47:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:47:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:47:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:47:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:47:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:47:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:47:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:47:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:47:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:47:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:47:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:47:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:47:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:47:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:47:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:47:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:47:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:47:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:47:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:47:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:47:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:47:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:47:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:47:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:47:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:47:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:47:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:47:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:47:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:47:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:47:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:47:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:47:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:47:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:47:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:47:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:47:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:47:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:47:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:47:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:47:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:47:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:47:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:47:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:47:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:47:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:47:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:47:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:47:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:47:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:47:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:47:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:47:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:47:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:47:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:47:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:47:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:47:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:47:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:47:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:47:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:47:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:47:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:47:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:47:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:47:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:47:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:47:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:47:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:47:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:47:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:47:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:47:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:47:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:47:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:47:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:47:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:47:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:47:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:47:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:47:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:47:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:47:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:47:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:47:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:47:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:47:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:47:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:47:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:47:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:47:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:47:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:47:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:47:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:47:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:47:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:47:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:47:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:47:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:47:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:47:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:47:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:47:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:47:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:47:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:47:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:47:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:47:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:47:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:47:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:47:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:47:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:47:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:47:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:47:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:47:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:47:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:47:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:47:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:47:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:47:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:47:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:47:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:47:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:47:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:47:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:47:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:47:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:47:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:47:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:47:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:47:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:47:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:47:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:47:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:47:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:47:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:47:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:47:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:47:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:47:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:47:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:47:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:47:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:47:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:47:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:47:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:47:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:47:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:47:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:47:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:47:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:47:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:47:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:47:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:47:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:47:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:47:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:47:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:47:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:47:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:47:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:47:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:47:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:47:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:47:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:48:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:48:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:48:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:48:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:48:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:48:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:48:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:48:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:48:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:48:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:48:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:48:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:48:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:48:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:48:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:48:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:48:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:48:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:48:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:48:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:48:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:48:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:48:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:48:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:48:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:48:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:48:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:48:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:48:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:48:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:48:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:48:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:48:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:48:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:48:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:48:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:48:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:48:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:48:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:48:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:48:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:48:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:48:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:48:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:48:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:48:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:48:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:48:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:48:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:48:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:48:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:48:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:48:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:48:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:48:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:48:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:48:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:48:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:48:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:48:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:48:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:48:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:48:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:48:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:48:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:48:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:48:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:48:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:48:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:48:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:48:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:48:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:48:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:48:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:48:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:48:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:48:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:48:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:48:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:48:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:48:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:48:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:48:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:48:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:48:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:48:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:48:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:48:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:48:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:48:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:48:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:48:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:48:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:48:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:48:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:48:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:48:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:48:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:48:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:48:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:48:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:48:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:48:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:48:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:48:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:48:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:48:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:48:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:48:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:48:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:48:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:48:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:48:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:48:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:48:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:48:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:48:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:48:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:48:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:48:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:48:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:48:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:48:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:48:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:48:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:48:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:48:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:48:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:48:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:48:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:48:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:48:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:48:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:48:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:48:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:48:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:48:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:48:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:48:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:48:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:48:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:48:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:48:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:48:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:48:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:48:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:48:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:48:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:48:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:48:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:48:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:48:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:48:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:48:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:48:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:48:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:48:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:48:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:48:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:48:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:48:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:48:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:48:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:48:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:48:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:48:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:48:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:48:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:48:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:48:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:48:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:48:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:48:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:48:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:48:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:48:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:48:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:48:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:48:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:48:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:48:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:48:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:48:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:48:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:48:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:48:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:48:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:48:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:48:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:48:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:48:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:48:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:48:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:48:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:48:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:48:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:48:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:48:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:48:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:48:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:48:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:48:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:48:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:48:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:48:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:48:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:48:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:48:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:48:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:48:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:48:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:48:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:48:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:48:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:48:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:48:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:49:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:49:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:49:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:49:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:49:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:49:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:49:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:49:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:49:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:49:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:49:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:49:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:49:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:49:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:49:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:49:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:49:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:49:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:49:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:49:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:49:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:49:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:49:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:49:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:49:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:49:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:49:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:49:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:49:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:49:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:49:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:49:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:49:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:49:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:49:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:49:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:49:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:49:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:49:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:49:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:49:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:49:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:49:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:49:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:49:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:49:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:49:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:49:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:49:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:49:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:49:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:49:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:49:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:49:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:49:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:49:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:49:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:49:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:49:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:49:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:49:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:49:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:49:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:49:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:49:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:49:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:49:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:49:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:49:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:49:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:49:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:49:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:49:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:49:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:49:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:49:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:49:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:49:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:49:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:49:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:49:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:49:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:49:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:49:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:49:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:49:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:49:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:49:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:49:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:49:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:49:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:49:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:49:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:49:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:49:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:49:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:49:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:49:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:49:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:49:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:49:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:49:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:49:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:49:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:49:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:49:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:49:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:49:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:49:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:49:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:49:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:49:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:49:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:49:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:49:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:49:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:49:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:49:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:49:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:49:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:49:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:49:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:49:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:49:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:49:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:49:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:49:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:49:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:49:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:49:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:49:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:49:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:49:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:49:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:49:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:49:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:49:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:49:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:49:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:49:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:49:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:49:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:49:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:49:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:49:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:49:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:49:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:49:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:49:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:49:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:49:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:49:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:49:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:49:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:49:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:49:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:49:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:49:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:49:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:49:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:49:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:49:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:49:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:49:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:49:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:49:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:49:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:49:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:49:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:49:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:49:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:49:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:49:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:49:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:49:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:49:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:49:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:49:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:49:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:49:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:49:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:49:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:49:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:49:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:49:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:49:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:49:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:49:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:49:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:49:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:49:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:49:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:49:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:49:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:49:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:49:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:49:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:49:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:49:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:49:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:49:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:49:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:49:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:49:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:49:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:49:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:50:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:50:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:50:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:50:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:50:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:50:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:50:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:50:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:50:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:50:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:50:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:50:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:50:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:50:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:50:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:50:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:50:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:50:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:50:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:50:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:50:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:50:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:50:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:50:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:50:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:50:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:50:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:50:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:50:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:50:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:50:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:50:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:50:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:50:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:50:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:50:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:50:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:50:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:50:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:50:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:50:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:50:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:50:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:50:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:50:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:50:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:50:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:50:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:50:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:50:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:50:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:50:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:50:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:50:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:50:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:50:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:50:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:50:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:50:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:50:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:50:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:50:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:50:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:50:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:50:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:50:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:50:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:50:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:50:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:50:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:50:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:50:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:50:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:50:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:50:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:50:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:50:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:50:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:50:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:50:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:50:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:50:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:50:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:50:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:50:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:50:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:50:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:50:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:50:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:50:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:50:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:50:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:50:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:50:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:50:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:50:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:50:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:50:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:50:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:50:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:50:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:50:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:50:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:50:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:50:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:50:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:50:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:50:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:50:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:50:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:50:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:50:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:50:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:50:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:50:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:50:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:50:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:50:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:50:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:50:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:50:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:50:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:50:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:50:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:50:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:50:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:50:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:50:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:50:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:50:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:50:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:50:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:50:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:50:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:50:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:50:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:50:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:50:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:50:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:50:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:50:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:50:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:50:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:50:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:50:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:50:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:50:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:50:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:50:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:50:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:50:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:50:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:50:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:50:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:50:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:50:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:50:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:50:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:50:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:50:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:50:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:50:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:50:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:50:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:50:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:50:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:50:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:50:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:50:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:50:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:50:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:50:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:50:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:50:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:50:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:50:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:50:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:50:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:50:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:50:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:50:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:50:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:50:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:50:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:50:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:50:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:50:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:50:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:50:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:50:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:50:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:50:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:50:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:50:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:50:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:50:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:50:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:50:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:50:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:50:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:50:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:50:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:50:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:50:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:50:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:50:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:50:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:50:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:50:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:50:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:50:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:50:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:50:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:50:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:50:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:50:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:51:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:51:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:51:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:51:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:51:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:51:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:51:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:51:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:51:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:51:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:51:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:51:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:51:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:51:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:51:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:51:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:51:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:51:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:51:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:51:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:51:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:51:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:51:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:51:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:51:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:51:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:51:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:51:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:51:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:51:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:51:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:51:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:51:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:51:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:51:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:51:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:51:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:51:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:51:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:51:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:51:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:51:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:51:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:51:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:51:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:51:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:51:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:51:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:51:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:51:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:51:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:51:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:51:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:51:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:51:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:51:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:51:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:51:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:51:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:51:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:51:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:51:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:51:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:51:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:51:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:51:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:51:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:51:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:51:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:51:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:51:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:51:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:51:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:51:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:51:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:51:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:51:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:51:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:51:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:51:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:51:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:51:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:51:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:51:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:51:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:51:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:51:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:51:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:51:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:51:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:51:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:51:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:51:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:51:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:51:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:51:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:51:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:51:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:51:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:51:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:51:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:51:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:51:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:51:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:51:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:51:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:51:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:51:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:51:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:51:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:51:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:51:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:51:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:51:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:51:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:51:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:51:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:51:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:51:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:51:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:51:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:51:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:51:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:51:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:51:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:51:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:51:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:51:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:51:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:51:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:51:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:51:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:51:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:51:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:51:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:51:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:51:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:51:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:51:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:51:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:51:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:51:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:51:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:51:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:51:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:51:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:51:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:51:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:51:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:51:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:51:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:51:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:51:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:51:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:51:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:51:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:51:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:51:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:51:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:51:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:51:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:51:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:53:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:53:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:53:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:53:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:53:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:53:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:53:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:53:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:53:50 rdma06 rsyslogd[1041]: imjournal: 5082 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 07:53:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:53:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:53:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:53:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:53:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:53:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:53:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:53:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:53:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:53:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:53:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:53:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:53:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:53:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:53:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:53:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:54:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:54:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:54:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:54:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:54:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:54:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:54:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:54:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:54:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:54:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:54:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:54:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:54:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:54:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:54:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:54:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:54:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:54:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:54:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:54:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:54:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:54:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:54:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:54:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:54:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:54:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:54:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:54:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:54:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:54:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:54:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:54:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:54:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:54:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:54:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:54:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:54:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:54:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:54:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:54:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:54:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:54:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:54:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:54:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:54:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:54:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:54:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:54:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:54:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:54:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:54:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:54:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:54:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:54:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:54:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:54:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:54:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:54:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:54:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:54:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:54:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:54:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:54:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:54:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:54:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:54:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:54:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:54:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:54:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:54:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:54:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:54:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:54:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:54:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:54:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:54:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:54:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:54:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:54:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:54:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:54:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:54:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:54:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:54:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:54:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:54:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:54:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:54:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:54:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:54:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:54:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:54:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:54:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:54:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:54:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:54:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:54:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:54:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:54:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:54:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:54:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:54:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:54:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:54:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:54:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:54:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:54:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:54:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:54:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:54:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:54:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:54:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:54:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:54:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:54:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:54:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:54:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:54:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:54:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:54:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:54:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:54:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:54:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:54:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:54:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:54:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:54:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:54:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:54:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:54:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:54:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:54:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:54:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:54:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:54:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:54:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:54:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:54:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:54:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:54:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:54:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:54:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:54:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:54:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:54:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:54:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:54:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:54:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:54:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:54:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:54:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:54:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:54:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:54:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:54:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:54:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:54:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:54:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:54:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:54:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:54:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:54:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:54:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:54:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:54:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:54:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:54:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:54:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:54:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:54:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:54:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:54:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:54:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:54:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:54:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:54:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:55:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:55:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:55:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:55:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:55:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:55:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:55:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:55:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:55:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:55:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:55:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:55:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:55:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:55:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:55:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:55:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:55:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:55:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:55:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:55:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:55:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:55:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:55:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:55:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:55:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:55:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:55:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:55:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:55:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:55:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:55:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:55:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:55:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:55:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:55:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:55:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:55:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:55:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:55:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:55:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:55:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:55:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:55:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:55:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:55:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:55:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:55:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:55:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:55:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:55:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:55:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:55:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:55:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:55:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:55:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:55:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:55:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:55:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:55:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:55:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:55:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:55:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:55:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:55:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:55:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:55:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:55:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:55:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:55:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:55:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:55:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:55:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:55:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:55:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:55:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:55:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:55:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:55:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:55:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:55:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:55:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:55:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:55:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:55:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:55:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:55:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:55:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:55:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:55:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:55:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:55:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:55:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:55:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:55:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:55:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:55:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:55:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:55:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:55:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:55:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:55:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:55:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:55:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:55:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:55:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:55:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:55:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:55:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:55:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:55:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:55:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:55:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:55:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:55:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:55:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:55:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:55:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:55:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:55:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:55:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:55:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:55:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:55:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:55:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:55:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:55:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:55:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:55:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:55:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:55:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:55:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:55:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:55:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:55:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:55:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:55:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:55:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:55:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:55:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:55:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:55:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:55:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:55:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:55:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:55:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:55:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:55:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:55:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:55:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:55:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:55:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:55:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:55:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:55:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:55:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:55:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:55:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:55:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:55:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:55:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:55:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:55:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:55:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:55:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:55:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:55:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:55:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:55:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:55:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:55:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:55:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:55:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:55:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:55:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:55:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:55:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:55:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:55:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:55:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:55:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:55:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:55:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:55:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:55:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:55:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:55:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:55:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:55:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:55:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:55:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:55:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:55:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:55:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:55:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:55:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:55:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:55:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:55:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:55:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:55:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:55:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:55:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:55:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:55:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:55:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:55:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:55:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:55:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:55:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:55:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:55:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:55:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:55:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:55:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:55:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:55:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:56:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:56:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:56:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:56:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:56:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:56:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:56:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:56:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:56:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:56:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:56:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:56:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:56:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:56:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:56:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:56:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:56:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:56:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:56:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:56:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:56:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:56:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:56:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:56:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:56:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:56:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:56:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:56:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:56:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:56:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:56:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:56:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:56:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:56:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:56:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:56:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:56:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:56:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:56:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:56:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:56:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:56:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:56:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:56:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:56:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:56:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:56:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:56:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:56:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:56:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:56:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:56:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:56:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:56:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:56:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:56:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:56:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:56:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:56:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:56:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:56:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:56:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:56:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:56:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:56:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:56:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:56:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:56:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:56:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:56:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:56:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:56:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:56:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:56:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:56:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:56:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:56:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:56:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:56:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:56:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:56:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:56:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:56:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:56:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:56:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:56:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:56:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:56:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:56:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:56:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:56:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:56:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:56:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:56:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:56:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:56:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:56:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:56:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:56:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:56:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:56:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:56:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:56:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:56:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:56:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:56:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:56:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:56:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:56:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:56:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:56:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:56:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:56:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:56:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:56:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:56:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:56:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:56:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:56:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:56:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:56:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:56:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:56:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:56:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:56:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:56:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:56:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:56:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:56:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:56:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:56:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:56:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:56:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:56:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:56:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:56:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:56:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:56:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:56:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:56:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:56:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:56:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:56:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:56:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:56:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:56:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:56:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:56:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:56:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:56:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:56:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:56:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:56:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:56:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:56:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:56:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:56:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:56:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:56:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:56:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:56:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:56:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:56:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:56:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:56:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:56:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:56:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:56:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:56:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:56:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:56:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:56:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:56:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:56:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:56:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:56:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:56:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:56:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:56:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:56:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:56:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:56:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:56:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:56:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:56:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:56:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:56:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:56:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:56:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:56:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:56:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:56:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:56:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:56:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:56:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:56:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:56:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:56:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:56:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:56:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:56:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:56:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:56:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:56:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:56:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:56:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:56:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:56:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:56:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:56:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:56:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:56:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:56:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:56:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:56:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:56:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:57:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:57:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:57:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:57:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:57:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:57:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:57:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:57:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:57:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:57:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:57:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:57:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:57:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:57:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:57:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:57:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:57:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:57:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:57:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:57:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:57:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:57:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:57:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:57:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:57:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:57:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:57:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:57:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:57:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:57:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:57:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:57:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:57:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:57:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:57:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:57:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:57:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:57:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:57:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:57:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:57:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:57:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:57:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:57:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:57:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:57:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:57:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:57:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:57:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:57:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:57:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:57:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:57:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:57:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:57:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:57:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:57:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:57:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:57:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:57:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:57:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:57:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:57:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:57:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:57:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:57:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:57:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:57:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:57:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:57:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:57:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:57:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:57:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:57:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:57:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:57:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:57:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:57:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:57:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:57:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:57:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:57:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:57:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:57:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:57:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:57:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:57:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:57:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:57:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:57:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:57:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:57:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:57:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:57:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:57:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:57:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:57:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:57:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:57:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:57:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:57:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:57:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:57:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:57:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:57:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:57:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:57:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:57:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:57:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:57:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:57:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:57:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:57:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:57:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:57:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:57:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:57:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:57:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:57:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:57:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:57:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:57:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:57:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:57:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:57:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:57:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:57:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:57:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:57:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:57:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:57:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:57:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:57:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:57:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:57:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:57:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:57:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:57:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:57:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:57:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:57:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:57:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:57:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:57:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:57:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:57:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:57:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:57:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:57:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:57:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:57:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:57:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:57:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:57:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:57:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:57:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:57:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:57:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:57:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:57:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:57:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:57:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:57:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:57:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:57:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:57:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:57:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:57:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:57:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:57:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:57:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:57:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:57:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:57:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:57:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:57:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:57:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:57:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:57:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:57:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:57:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:57:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:57:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:57:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:57:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:57:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:57:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:57:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:57:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:57:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:57:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:57:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:57:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:57:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:57:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:57:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:57:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:57:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:57:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:57:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:57:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:57:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:57:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:57:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:57:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:57:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:57:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:57:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:57:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:57:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:57:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:57:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:57:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:57:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:57:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:57:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:58:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:58:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:58:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:58:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:58:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:58:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:58:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:58:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:58:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:58:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:58:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:58:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:58:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:58:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:58:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:58:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:58:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:58:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:58:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:58:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:58:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:58:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:58:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:58:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:58:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:58:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:58:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:58:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:58:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:58:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:58:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:58:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:58:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:58:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:58:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:58:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:58:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:58:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:58:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:58:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:58:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:58:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:58:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:58:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:58:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:58:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:58:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:58:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:58:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:58:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:58:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:58:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:58:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:58:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:58:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:58:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:58:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:58:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:58:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:58:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:58:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:58:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:58:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:58:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:58:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:58:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:58:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:58:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:58:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:58:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:58:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:58:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:58:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:58:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:58:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:58:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:58:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:58:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:58:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:58:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:58:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:58:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:58:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:58:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:58:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:58:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:58:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:58:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:58:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:58:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:58:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:58:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:58:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:58:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:58:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:58:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:58:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:58:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:58:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:58:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:58:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:58:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:58:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:58:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:58:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:58:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:58:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:58:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:58:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:58:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:58:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:58:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:58:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:58:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:58:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:58:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:58:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:58:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:58:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:58:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:58:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:58:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:58:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:58:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:58:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:58:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:58:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:58:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:58:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:58:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:58:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:58:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:58:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:58:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:58:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:58:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:58:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:58:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:58:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:58:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:58:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:58:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:58:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:58:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:58:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:58:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:58:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:58:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:58:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:58:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:58:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:58:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:58:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:58:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:58:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:58:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:58:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:58:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:58:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:58:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:58:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:58:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:58:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:58:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:58:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:58:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:58:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:58:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:58:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:58:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:58:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:58:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:58:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:58:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:58:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:58:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:58:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:58:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:58:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:58:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:58:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:58:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:58:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:58:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:58:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:58:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:58:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:58:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:58:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:58:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:58:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:58:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:58:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:58:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:58:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:58:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:58:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:58:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:58:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:58:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:58:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:58:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:58:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:58:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:58:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:58:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:58:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:58:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:58:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:58:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:58:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:58:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:58:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:58:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:58:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:58:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:59:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:59:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:59:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:59:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:59:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:59:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:59:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:59:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:59:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:59:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:59:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:59:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:59:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:59:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:59:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:59:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:59:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:59:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:59:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:59:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:59:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:59:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:59:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:59:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:59:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:59:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:59:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:59:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:59:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:59:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:59:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:59:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:59:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:59:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:59:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:59:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:59:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:59:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:59:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:59:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:59:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:59:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:59:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:59:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:59:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:59:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:59:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:59:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:59:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:59:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:59:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:59:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:59:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:59:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:59:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:59:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:59:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:59:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:59:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:59:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:59:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:59:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:59:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:59:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:59:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:59:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:59:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:59:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:59:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:59:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:59:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:59:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:59:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:59:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:59:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:59:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:59:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:59:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:59:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:59:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:59:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:59:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:59:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:59:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:59:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:59:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:59:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:59:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:59:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:59:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:59:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:59:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:59:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:59:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:59:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:59:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:59:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:59:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:59:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:59:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:59:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:59:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:59:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:59:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:59:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:59:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:59:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:59:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:59:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:59:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:59:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:59:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:59:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:59:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:59:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:59:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:59:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:59:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:59:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:59:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:59:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:59:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:59:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:59:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:59:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:59:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:59:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:59:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:59:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:59:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:59:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:59:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:59:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:59:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:59:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:59:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:59:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:59:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 07:59:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 07:59:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:59:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:59:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:59:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 07:59:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 07:59:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 07:59:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 07:59:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 07:59:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 07:59:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 07:59:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 07:59:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 07:59:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 07:59:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 07:59:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 07:59:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 07:59:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:00:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:00:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:00:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:00:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:00:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:00:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:00:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:00:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:00:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:00:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:00:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:00:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:00:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:00:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:00:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:00:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:00:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:00:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:00:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:00:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:00:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:00:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:00:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:00:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:00:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:00:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:00:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:00:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:00:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:00:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:00:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:00:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:00:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:00:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:00:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:00:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:00:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:00:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:00:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:00:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:00:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:00:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:00:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:00:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:00:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:00:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:00:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:00:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:00:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:00:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:00:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:00:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:00:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:00:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:00:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:00:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:00:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:00:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:00:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:00:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:00:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:00:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:00:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:00:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:00:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:00:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:00:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:00:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:00:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:00:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:00:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:00:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:00:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:00:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:00:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:00:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:00:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:00:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:00:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:00:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:00:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:00:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:00:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:00:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:00:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:00:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:00:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:00:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:00:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:00:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:00:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:00:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:00:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:00:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:00:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:00:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:00:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:00:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:00:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:00:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:00:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:00:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:00:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:00:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:00:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:00:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:00:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:00:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:00:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:00:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:00:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:00:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:00:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:00:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:00:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:00:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:00:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:00:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:00:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:00:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:00:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:00:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:00:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:00:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:00:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:00:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:00:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:00:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:00:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:00:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:00:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:00:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:00:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:00:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:00:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:00:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:00:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:00:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:00:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:00:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:00:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:00:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:00:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:00:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:00:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:00:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:00:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:00:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:00:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:00:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:00:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:00:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:00:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:00:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:00:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:00:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:00:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:00:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:00:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:00:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:00:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:00:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:00:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:00:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:00:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:00:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:00:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:00:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:00:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:00:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:00:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:00:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:00:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:00:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:00:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:00:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:00:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:00:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:00:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:00:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:00:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:00:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:00:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:00:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:00:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:00:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:01:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:01:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:01:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:01:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:01:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:01:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:01:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:01:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:01:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:01:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:01:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:01:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:01:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:01:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:01:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:01:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:01:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:01:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:01:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:01:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:01:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:01:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:01:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:01:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:01:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:01:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:01:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:01:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:01:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:01:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:01:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:01:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:01:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:01:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:01:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:01:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:01:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:01:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:01:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:01:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:01:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:01:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:01:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:01:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:01:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:01:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:01:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:01:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:01:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:01:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:01:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:01:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:01:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:01:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:01:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:01:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:01:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:01:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:01:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:01:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:01:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:01:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:01:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:01:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:01:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:01:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:01:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:01:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:01:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:01:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:01:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:01:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:01:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:01:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:01:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:01:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:01:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:01:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:01:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:01:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:01:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:01:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:01:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:01:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:01:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:01:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:01:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:01:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:01:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:01:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:01:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:01:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:01:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:01:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:01:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:01:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:01:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:01:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:01:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:01:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:01:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:01:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:01:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:01:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:01:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:01:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:01:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:01:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:01:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:03:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:03:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:03:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:03:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:03:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:03:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:03:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:03:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:03:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:03:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:03:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:03:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:03:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:03:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:03:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:03:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:03:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:03:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:03:55 rdma06 rsyslogd[1041]: imjournal: 5295 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 08:04:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:04:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:04:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:04:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:04:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:04:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:04:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:04:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:04:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:04:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:04:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:04:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:04:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:04:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:04:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:04:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:04:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:04:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:04:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:04:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:04:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:04:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:04:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:04:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:04:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:04:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:04:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:04:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:04:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:04:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:04:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:04:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:04:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:04:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:04:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:04:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:04:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:04:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:04:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:04:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:04:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:04:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:04:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:04:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:04:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:04:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:04:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:04:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:04:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:04:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:04:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:04:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:04:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:04:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:04:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:04:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:04:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:04:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:04:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:04:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:04:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:04:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:04:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:04:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:04:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:04:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:04:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:04:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:04:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:04:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:04:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:04:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:04:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:04:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:04:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:04:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:04:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:04:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:04:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:04:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:04:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:04:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:04:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:04:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:04:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:04:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:04:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:04:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:04:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:04:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:04:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:04:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:04:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:04:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:04:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:04:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:04:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:04:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:04:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:04:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:04:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:04:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:04:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:04:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:04:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:04:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:04:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:04:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:04:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:04:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:04:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:04:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:04:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:04:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:04:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:04:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:04:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:04:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:04:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:04:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:04:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:04:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:04:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:04:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:04:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:04:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:04:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:04:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:04:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:04:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:04:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:04:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:04:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:04:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:04:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:04:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:04:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:04:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:04:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:04:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:04:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:04:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:04:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:04:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:04:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:04:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:04:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:04:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:04:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:04:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:04:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:04:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:04:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:04:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:04:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:04:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:04:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:04:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:04:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:04:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:04:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:04:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:04:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:04:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:04:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:04:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:04:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:04:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:04:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:04:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:04:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:04:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:04:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:04:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:04:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:04:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:04:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:04:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:04:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:04:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:04:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:04:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:04:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:04:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:04:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:04:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:04:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:04:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:04:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:04:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:04:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:04:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:04:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:04:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:04:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:04:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:04:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:04:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:04:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:04:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:04:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:04:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:04:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:04:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:04:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:04:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:04:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:04:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:04:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:04:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:04:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:04:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:04:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:04:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:04:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:04:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:05:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:05:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:05:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:05:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:05:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:05:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:05:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:05:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:05:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:05:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:05:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:05:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:05:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:05:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:05:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:05:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:05:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:05:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:05:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:05:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:05:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:05:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:05:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:05:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:05:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:05:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:05:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:05:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:05:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:05:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:05:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:05:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:05:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:05:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:05:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:05:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:05:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:05:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:05:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:05:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:05:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:05:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:05:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:05:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:05:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:05:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:05:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:05:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:05:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:05:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:05:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:05:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:05:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:05:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:05:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:05:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:05:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:05:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:05:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:05:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:05:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:05:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:05:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:05:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:05:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:05:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:05:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:05:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:05:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:05:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:05:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:05:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:05:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:05:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:05:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:05:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:05:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:05:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:05:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:05:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:05:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:05:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:05:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:05:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:05:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:05:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:05:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:05:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:05:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:05:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:05:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:05:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:05:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:05:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:05:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:05:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:05:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:05:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:05:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:05:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:05:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:05:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:05:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:05:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:05:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:05:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:05:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:05:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:05:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:05:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:05:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:05:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:05:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:05:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:05:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:05:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:05:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:05:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:05:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:05:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:05:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:05:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:05:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:05:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:05:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:05:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:05:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:05:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:05:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:05:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:05:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:05:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:05:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:05:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:05:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:05:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:05:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:05:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:05:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:05:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:05:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:05:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:05:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:05:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:05:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:05:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:05:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:05:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:05:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:05:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:05:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:05:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:05:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:05:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:05:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:05:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:05:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:05:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:05:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:05:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:05:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:05:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:05:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:05:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:05:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:05:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:05:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:05:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:05:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:05:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:05:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:05:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:05:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:05:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:05:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:05:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:05:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:05:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:05:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:05:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:05:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:05:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:05:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:05:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:05:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:05:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:05:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:05:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:05:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:05:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:05:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:05:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:05:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:05:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:05:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:05:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:05:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:05:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:05:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:05:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:05:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:05:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:05:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:05:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:05:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:05:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:05:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:05:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:05:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:05:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:05:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:05:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:05:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:05:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:05:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:05:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:06:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:06:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:06:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:06:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:06:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:06:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:06:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:06:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:06:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:06:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:06:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:06:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:06:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:06:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:06:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:06:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:06:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:06:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:06:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:06:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:06:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:06:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:06:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:06:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:06:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:06:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:06:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:06:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:06:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:06:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:06:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:06:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:06:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:06:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:06:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:06:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:06:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:06:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:06:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:06:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:06:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:06:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:06:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:06:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:06:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:06:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:06:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:06:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:06:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:06:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:06:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:06:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:06:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:06:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:06:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:06:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:06:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:06:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:06:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:06:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:06:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:06:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:06:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:06:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:06:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:06:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:06:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:06:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:06:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:06:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:06:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:06:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:06:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:06:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:06:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:06:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:06:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:06:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:06:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:06:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:06:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:06:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:06:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:06:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:06:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:06:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:06:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:06:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:06:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:06:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:06:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:06:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:06:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:06:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:06:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:06:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:06:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:06:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:06:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:06:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:06:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:06:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:06:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:06:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:06:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:06:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:06:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:06:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:06:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:06:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:06:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:06:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:06:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:06:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:06:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:06:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:06:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:06:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:06:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:06:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:06:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:06:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:06:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:06:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:06:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:06:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:06:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:06:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:06:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:06:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:06:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:06:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:06:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:06:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:06:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:06:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:06:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:06:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:06:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:06:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:06:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:06:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:06:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:06:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:06:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:06:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:06:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:06:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:06:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:06:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:06:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:06:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:06:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:06:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:06:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:06:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:06:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:06:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:06:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:06:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:06:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:06:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:06:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:06:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:06:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:06:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:06:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:06:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:06:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:06:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:06:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:06:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:06:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:06:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:06:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:06:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:06:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:06:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:06:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:06:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:06:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:06:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:06:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:06:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:06:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:06:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:06:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:06:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:06:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:06:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:06:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:06:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:06:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:06:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:06:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:06:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:06:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:06:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:06:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:06:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:06:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:06:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:06:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:06:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:06:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:06:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:06:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:06:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:06:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:06:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:06:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:06:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:06:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:06:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:06:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:06:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:07:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:07:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:07:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:07:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:07:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:07:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:07:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:07:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:07:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:07:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:07:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:07:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:07:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:07:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:07:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:07:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:07:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:07:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:07:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:07:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:07:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:07:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:07:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:07:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:07:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:07:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:07:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:07:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:07:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:07:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:07:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:07:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:07:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:07:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:07:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:07:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:07:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:07:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:07:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:07:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:07:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:07:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:07:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:07:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:07:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:07:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:07:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:07:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:07:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:07:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:07:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:07:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:07:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:07:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:07:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:07:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:07:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:07:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:07:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:07:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:07:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:07:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:07:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:07:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:07:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:07:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:07:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:07:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:07:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:07:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:07:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:07:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:07:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:07:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:07:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:07:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:07:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:07:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:07:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:07:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:07:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:07:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:07:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:07:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:07:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:07:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:07:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:07:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:07:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:07:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:07:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:07:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:07:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:07:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:07:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:07:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:07:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:07:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:07:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:07:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:07:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:07:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:07:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:07:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:07:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:07:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:07:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:07:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:07:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:07:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:07:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:07:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:07:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:07:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:07:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:07:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:07:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:07:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:07:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:07:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:07:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:07:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:07:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:07:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:07:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:07:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:07:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:07:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:07:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:07:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:07:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:07:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:07:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:07:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:07:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:07:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:07:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:07:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:07:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:07:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:07:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:07:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:07:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:07:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:07:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:07:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:07:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:07:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:07:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:07:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:07:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:07:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:07:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:07:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:07:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:07:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:07:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:07:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:07:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:07:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:07:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:07:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:07:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:07:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:07:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:07:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:07:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:07:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:07:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:07:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:07:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:07:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:07:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:07:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:07:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:07:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:07:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:07:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:07:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:07:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:07:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:07:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:07:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:07:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:07:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:07:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:07:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:07:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:07:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:07:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:07:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:07:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:07:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:07:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:07:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:07:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:07:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:07:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:07:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:07:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:07:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:07:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:07:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:07:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:07:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:07:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:07:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:07:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:07:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:07:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:07:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:07:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:07:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:07:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:07:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:07:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:08:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:08:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:08:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:08:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:08:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:08:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:08:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:08:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:08:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:08:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:08:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:08:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:08:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:08:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:08:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:08:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:08:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:08:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:08:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:08:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:08:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:08:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:08:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:08:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:08:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:08:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:08:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:08:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:08:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:08:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:08:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:08:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:08:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:08:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:08:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:08:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:08:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:08:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:08:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:08:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:08:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:08:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:08:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:08:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:08:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:08:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:08:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:08:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:08:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:08:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:08:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:08:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:08:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:08:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:08:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:08:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:08:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:08:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:08:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:08:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:08:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:08:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:08:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:08:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:08:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:08:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:08:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:08:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:08:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:08:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:08:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:08:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:08:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:08:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:08:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:08:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:08:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:08:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:08:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:08:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:08:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:08:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:08:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:08:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:08:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:08:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:08:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:08:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:08:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:08:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:08:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:08:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:08:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:08:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:08:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:08:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:08:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:08:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:08:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:08:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:08:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:08:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:08:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:08:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:08:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:08:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:08:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:08:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:08:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:08:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:08:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:08:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:08:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:08:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:08:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:08:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:08:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:08:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:08:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:08:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:08:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:08:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:08:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:08:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:08:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:08:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:08:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:08:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:08:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:08:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:08:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:08:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:08:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:08:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:08:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:08:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:08:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:08:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:08:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:08:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:08:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:08:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:08:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:08:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:08:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:08:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:08:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:08:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:08:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:08:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:08:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:08:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:08:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:08:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:08:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:08:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:08:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:08:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:08:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:08:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:08:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:08:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:08:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:08:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:08:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:08:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:08:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:08:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:08:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:08:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:08:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:08:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:08:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:08:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:08:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:08:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:08:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:08:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:08:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:08:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:08:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:08:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:08:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:08:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:08:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:08:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:08:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:08:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:08:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:08:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:08:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:08:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:08:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:08:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:08:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:08:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:08:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:08:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:08:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:08:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:08:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:08:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:08:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:08:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:08:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:08:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:08:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:08:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:08:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:08:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:08:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:08:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:08:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:08:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:08:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:08:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:09:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:09:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:09:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:09:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:09:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:09:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:09:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:09:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:09:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:09:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:09:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:09:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:09:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:09:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:09:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:09:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:09:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:09:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:09:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:09:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:09:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:09:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:09:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:09:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:09:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:09:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:09:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:09:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:09:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:09:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:09:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:09:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:09:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:09:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:09:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:09:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:09:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:09:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:09:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:09:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:09:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:09:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:09:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:09:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:09:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:09:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:09:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:09:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:09:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:09:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:09:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:09:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:09:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:09:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:09:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:09:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:09:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:09:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:09:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:09:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:09:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:09:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:09:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:09:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:09:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:09:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:09:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:09:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:09:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:09:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:09:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:09:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:09:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:09:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:09:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:09:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:09:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:09:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:09:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:09:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:09:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:09:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:09:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:09:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:09:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:09:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:09:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:09:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:09:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:09:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:09:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:09:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:09:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:09:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:09:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:09:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:09:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:09:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:09:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:09:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:09:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:09:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:09:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:09:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:09:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:09:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:09:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:09:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:09:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:09:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:09:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:09:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:09:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:09:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:09:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:09:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:09:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:09:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:09:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:09:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:09:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:09:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:09:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:09:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:09:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:09:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:09:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:09:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:09:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:09:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:09:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:09:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:09:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:09:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:09:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:09:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:09:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:09:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:09:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:09:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:09:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:09:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:09:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:09:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:09:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:09:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:09:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:09:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:09:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:09:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:09:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:09:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:09:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:09:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:09:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:09:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:09:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:09:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:09:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:09:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:09:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:09:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:09:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:09:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:09:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:09:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:09:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:09:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:09:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:09:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:09:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:09:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:09:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:09:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:09:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:09:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:09:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:09:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:09:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:09:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:09:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:09:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:09:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:09:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:09:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:09:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:09:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:09:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:09:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:09:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:09:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:09:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:09:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:09:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:09:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:09:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:09:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:09:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:09:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:09:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:09:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:09:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:09:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:09:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:09:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:09:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:09:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:09:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:09:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:09:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:09:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:09:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:09:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:09:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:09:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:09:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:10:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:10:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:10:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:10:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:10:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:10:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:10:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:10:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:10:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:10:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:10:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:10:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:10:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:10:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:10:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:10:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:10:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:10:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:10:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:10:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:10:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:10:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:10:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:10:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:10:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:10:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:10:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:10:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:10:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:10:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:10:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:10:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:10:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:10:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:10:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:10:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:10:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:10:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:10:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:10:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:10:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:10:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:10:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:10:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:10:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:10:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:10:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:10:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:10:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:10:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:10:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:10:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:10:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:10:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:10:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:10:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:10:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:10:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:10:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:10:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:10:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:10:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:10:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:10:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:10:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:10:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:10:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:10:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:10:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:10:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:10:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:10:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:10:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:10:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:10:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:10:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:10:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:10:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:10:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:10:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:10:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:10:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:10:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:10:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:10:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:10:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:10:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:10:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:10:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:10:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:10:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:10:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:10:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:10:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:10:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:10:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:10:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:10:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:10:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:10:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:10:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:10:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:10:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:10:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:10:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:10:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:10:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:10:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:10:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:10:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:10:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:10:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:10:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:10:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:10:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:10:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:10:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:10:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:10:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:10:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:10:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:10:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:10:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:10:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:10:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:10:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:10:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:10:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:10:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:10:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:10:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:10:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:10:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:10:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:10:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:10:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:10:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:10:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:10:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:10:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:10:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:10:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:10:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:10:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:10:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:10:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:10:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:10:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:10:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:10:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:10:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:10:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:10:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:10:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:10:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:10:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:10:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:10:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:10:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:10:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:10:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:10:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:10:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:10:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:10:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:10:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:10:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:10:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:10:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:10:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:10:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:10:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:10:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:10:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:10:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:10:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:10:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:10:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:10:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:10:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:10:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:10:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:10:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:10:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:10:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:10:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:10:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:10:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:10:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:10:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:10:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:10:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:10:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:10:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:10:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:10:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:10:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:10:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:10:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:10:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:10:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:10:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:10:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:10:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:10:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:10:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:10:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:10:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:10:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:10:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:10:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:10:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:10:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:10:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:10:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:10:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:11:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:11:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:11:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:11:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:11:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:11:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:11:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:11:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:11:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:11:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:11:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:11:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:11:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:11:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:11:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:11:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:11:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:11:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:11:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:11:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:11:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:11:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:11:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:11:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:11:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:11:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:11:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:11:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:11:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:11:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:11:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:11:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:11:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:11:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:11:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:11:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:11:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:11:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:11:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:11:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:11:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:11:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:11:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:11:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:11:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:11:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:11:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:11:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:11:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:11:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:11:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:11:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:11:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:11:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:11:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:11:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:11:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:11:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:11:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:11:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:11:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:11:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:11:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:11:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:11:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:11:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:11:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:11:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:11:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:11:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:11:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:11:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:11:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:11:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:11:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:11:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:11:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:11:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:11:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:11:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:11:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:11:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:11:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:11:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:11:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:11:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:11:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:11:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:11:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:11:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:11:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:11:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:11:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:11:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:11:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:11:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:11:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:11:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:11:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:11:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:11:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:11:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:11:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:11:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:11:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:11:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:11:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:11:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:11:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:11:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:11:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:11:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:11:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:11:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:11:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:11:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:11:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:11:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:11:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:11:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:11:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:11:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:11:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:11:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:11:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:11:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:11:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:11:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:11:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:11:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:11:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:11:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:11:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:11:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:11:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:11:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:11:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:11:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:11:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:11:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:11:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:11:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:11:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:11:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:11:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:11:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:11:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:11:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:11:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:11:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:11:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:11:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:11:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:11:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:11:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:11:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:11:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:11:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:11:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:11:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:11:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:11:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:11:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:11:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:11:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:11:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:11:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:11:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:14:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:14:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:14:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:14:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:14:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:14:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:14:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:14:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:14:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:14:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:14:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:14:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:14:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:14:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:14:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:14:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:14:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:14:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:14:00 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 08:14:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:14:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:14:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:14:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:14:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:14:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:14:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:14:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:14:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:14:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:14:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:14:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:14:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:14:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:14:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:14:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:14:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:14:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:14:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:14:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:14:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:14:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:14:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:14:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:14:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:14:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:14:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:14:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:14:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:14:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:14:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:14:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:14:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:14:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:14:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:14:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:14:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:14:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:14:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:14:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:14:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:14:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:14:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:14:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:14:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:14:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:14:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:14:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:14:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:14:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:14:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:14:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:14:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:14:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:14:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:14:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:14:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:14:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:14:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:14:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:14:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:14:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:14:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:14:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:14:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:14:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:14:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:14:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:14:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:14:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:14:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:14:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:14:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:14:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:14:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:14:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:14:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:14:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:14:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:14:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:14:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:14:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:14:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:14:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:14:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:14:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:14:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:14:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:14:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:14:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:14:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:14:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:14:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:14:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:14:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:14:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:14:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:14:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:14:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:14:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:14:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:14:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:14:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:14:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:14:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:14:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:14:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:14:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:14:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:14:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:14:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:14:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:14:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:14:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:14:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:14:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:14:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:14:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:14:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:14:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:14:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:14:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:14:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:14:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:14:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:14:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:14:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:14:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:14:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:14:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:14:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:14:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:14:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:14:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:14:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:14:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:14:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:14:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:14:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:14:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:14:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:14:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:14:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:14:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:14:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:14:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:14:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:14:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:14:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:14:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:14:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:14:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:14:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:14:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:14:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:14:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:14:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:14:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:14:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:14:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:14:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:14:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:14:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:14:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:14:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:14:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:14:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:14:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:14:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:14:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:14:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:14:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:14:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:14:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:14:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:14:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:14:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:14:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:14:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:14:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:14:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:14:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:14:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:14:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:14:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:14:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:14:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:14:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:14:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:14:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:14:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:14:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:14:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:14:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:14:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:14:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:14:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:14:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:15:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:15:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:15:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:15:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:15:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:15:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:15:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:15:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:15:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:15:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:15:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:15:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:15:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:15:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:15:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:15:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:15:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:15:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:15:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:15:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:15:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:15:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:15:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:15:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:15:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:15:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:15:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:15:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:15:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:15:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:15:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:15:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:15:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:15:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:15:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:15:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:15:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:15:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:15:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:15:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:15:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:15:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:15:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:15:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:15:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:15:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:15:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:15:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:15:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:15:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:15:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:15:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:15:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:15:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:15:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:15:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:15:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:15:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:15:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:15:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:15:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:15:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:15:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:15:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:15:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:15:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:15:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:15:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:15:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:15:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:15:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:15:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:15:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:15:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:15:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:15:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:15:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:15:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:15:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:15:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:15:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:15:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:15:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:15:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:15:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:15:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:15:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:15:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:15:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:15:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:15:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:15:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:15:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:15:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:15:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:15:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:15:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:15:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:15:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:15:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:15:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:15:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:15:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:15:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:15:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:15:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:15:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:15:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:15:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:15:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:15:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:15:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:15:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:15:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:15:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:15:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:15:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:15:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:15:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:15:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:15:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:15:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:15:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:15:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:15:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:15:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:15:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:15:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:15:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:15:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:15:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:15:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:15:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:15:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:15:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:15:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:15:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:15:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:15:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:15:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:15:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:15:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:15:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:15:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:15:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:15:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:15:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:15:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:15:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:15:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:15:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:15:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:15:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:15:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:15:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:15:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:15:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:15:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:15:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:15:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:15:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:15:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:15:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:15:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:15:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:15:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:15:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:15:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:15:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:15:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:15:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:15:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:15:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:15:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:15:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:15:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:15:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:15:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:15:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:15:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:15:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:15:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:15:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:15:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:15:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:15:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:15:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:15:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:15:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:15:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:15:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:15:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:15:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:15:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:15:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:15:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:15:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:15:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:15:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:15:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:15:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:15:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:15:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:15:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:15:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:15:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:15:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:15:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:15:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:15:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:15:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:15:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:15:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:15:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:15:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:15:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:16:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:16:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:16:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:16:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:16:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:16:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:16:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:16:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:16:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:16:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:16:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:16:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:16:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:16:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:16:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:16:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:16:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:16:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:16:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:16:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:16:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:16:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:16:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:16:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:16:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:16:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:16:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:16:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:16:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:16:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:16:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:16:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:16:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:16:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:16:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:16:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:16:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:16:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:16:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:16:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:16:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:16:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:16:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:16:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:16:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:16:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:16:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:16:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:16:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:16:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:16:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:16:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:16:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:16:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:16:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:16:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:16:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:16:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:16:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:16:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:16:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:16:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:16:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:16:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:16:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:16:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:16:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:16:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:16:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:16:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:16:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:16:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:16:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:16:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:16:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:16:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:16:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:16:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:16:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:16:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:16:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:16:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:16:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:16:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:16:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:16:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:16:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:16:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:16:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:16:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:16:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:16:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:16:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:16:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:16:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:16:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:16:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:16:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:16:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:16:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:16:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:16:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:16:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:16:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:16:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:16:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:16:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:16:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:16:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:16:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:16:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:16:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:16:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:16:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:16:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:16:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:16:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:16:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:16:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:16:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:16:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:16:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:16:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:16:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:16:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:16:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:16:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:16:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:16:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:16:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:16:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:16:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:16:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:16:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:16:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:16:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:16:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:16:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:16:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:16:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:16:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:16:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:16:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:16:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:16:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:16:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:16:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:16:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:16:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:16:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:16:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:16:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:16:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:16:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:16:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:16:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:16:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:16:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:16:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:16:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:16:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:16:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:16:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:16:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:16:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:16:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:16:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:16:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:16:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:16:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:16:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:16:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:16:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:16:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:16:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:16:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:16:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:16:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:16:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:16:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:16:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:16:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:16:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:16:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:16:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:16:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:16:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:16:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:16:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:16:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:16:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:16:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:16:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:16:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:16:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:16:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:16:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:16:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:16:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:16:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:16:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:16:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:16:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:16:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:16:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:16:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:16:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:16:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:16:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:16:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:16:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:16:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:16:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:16:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:16:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:16:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:17:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:17:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:17:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:17:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:17:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:17:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:17:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:17:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:17:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:17:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:17:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:17:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:17:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:17:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:17:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:17:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:17:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:17:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:17:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:17:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:17:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:17:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:17:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:17:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:17:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:17:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:17:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:17:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:17:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:17:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:17:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:17:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:17:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:17:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:17:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:17:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:17:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:17:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:17:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:17:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:17:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:17:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:17:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:17:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:17:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:17:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:17:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:17:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:17:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:17:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:17:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:17:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:17:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:17:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:17:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:17:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:17:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:17:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:17:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:17:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:17:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:17:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:17:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:17:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:17:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:17:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:17:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:17:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:17:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:17:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:17:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:17:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:17:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:17:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:17:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:17:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:17:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:17:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:17:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:17:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:17:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:17:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:17:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:17:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:17:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:17:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:17:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:17:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:17:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:17:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:17:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:17:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:17:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:17:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:17:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:17:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:17:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:17:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:17:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:17:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:17:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:17:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:17:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:17:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:17:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:17:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:17:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:17:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:17:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:17:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:17:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:17:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:17:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:17:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:17:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:17:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:17:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:17:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:17:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:17:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:17:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:17:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:17:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:17:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:17:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:17:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:17:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:17:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:17:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:17:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:17:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:17:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:17:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:17:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:17:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:17:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:17:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:17:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:17:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:17:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:17:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:17:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:17:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:17:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:17:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:17:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:17:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:17:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:17:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:17:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:17:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:17:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:17:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:17:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:17:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:17:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:17:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:17:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:17:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:17:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:17:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:17:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:17:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:17:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:17:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:17:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:17:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:17:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:17:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:17:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:17:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:17:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:17:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:17:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:17:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:17:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:17:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:17:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:17:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:17:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:17:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:17:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:17:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:17:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:17:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:17:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:17:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:17:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:17:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:17:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:17:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:17:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:17:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:17:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:17:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:17:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:17:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:17:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:17:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:17:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:17:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:17:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:17:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:17:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:17:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:17:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:17:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:17:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:17:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:17:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:17:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:17:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:17:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:17:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:17:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:17:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:18:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:18:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:18:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:18:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:18:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:18:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:18:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:18:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:18:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:18:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:18:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:18:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:18:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:18:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:18:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:18:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:18:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:18:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:18:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:18:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:18:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:18:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:18:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:18:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:18:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:18:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:18:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:18:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:18:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:18:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:18:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:18:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:18:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:18:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:18:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:18:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:18:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:18:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:18:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:18:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:18:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:18:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:18:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:18:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:18:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:18:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:18:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:18:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:18:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:18:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:18:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:18:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:18:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:18:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:18:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:18:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:18:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:18:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:18:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:18:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:18:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:18:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:18:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:18:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:18:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:18:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:18:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:18:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:18:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:18:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:18:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:18:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:18:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:18:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:18:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:18:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:18:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:18:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:18:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:18:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:18:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:18:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:18:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:18:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:18:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:18:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:18:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:18:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:18:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:18:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:18:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:18:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:18:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:18:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:18:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:18:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:18:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:18:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:18:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:18:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:18:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:18:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:18:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:18:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:18:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:18:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:18:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:18:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:18:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:18:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:18:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:18:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:18:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:18:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:18:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:18:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:18:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:18:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:18:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:18:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:18:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:18:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:18:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:18:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:18:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:18:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:18:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:18:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:18:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:18:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:18:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:18:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:18:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:18:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:18:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:18:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:18:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:18:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:18:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:18:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:18:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:18:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:18:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:18:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:18:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:18:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:18:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:18:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:18:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:18:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:18:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:18:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:18:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:18:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:18:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:18:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:18:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:18:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:18:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:18:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:18:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:18:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:18:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:18:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:18:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:18:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:18:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:18:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:18:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:18:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:18:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:18:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:18:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:18:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:18:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:18:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:18:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:18:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:18:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:18:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:18:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:18:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:18:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:18:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:18:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:18:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:18:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:18:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:18:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:18:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:18:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:18:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:18:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:18:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:18:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:18:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:18:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:18:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:18:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:18:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:18:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:18:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:18:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:18:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:18:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:18:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:18:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:18:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:18:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:18:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:18:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:18:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:18:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:18:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:18:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:18:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:19:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:19:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:19:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:19:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:19:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:19:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:19:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:19:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:19:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:19:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:19:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:19:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:19:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:19:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:19:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:19:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:19:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:19:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:19:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:19:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:19:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:19:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:19:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:19:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:19:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:19:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:19:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:19:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:19:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:19:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:19:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:19:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:19:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:19:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:19:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:19:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:19:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:19:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:19:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:19:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:19:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:19:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:19:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:19:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:19:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:19:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:19:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:19:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:19:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:19:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:19:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:19:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:19:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:19:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:19:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:19:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:19:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:19:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:19:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:19:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:19:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:19:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:19:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:19:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:19:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:19:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:19:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:19:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:19:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:19:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:19:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:19:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:19:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:19:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:19:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:19:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:19:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:19:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:19:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:19:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:19:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:19:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:19:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:19:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:19:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:19:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:19:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:19:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:19:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:19:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:19:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:19:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:19:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:19:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:19:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:19:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:19:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:19:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:19:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:19:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:19:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:19:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:19:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:19:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:19:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:19:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:19:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:19:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:19:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:19:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:19:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:19:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:19:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:19:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:19:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:19:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:19:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:19:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:19:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:19:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:19:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:19:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:19:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:19:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:19:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:19:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:19:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:19:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:19:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:19:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:19:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:19:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:19:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:19:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:19:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:19:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:19:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:19:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:19:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:19:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:19:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:19:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:19:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:19:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:19:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:19:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:19:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:19:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:19:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:19:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:19:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:19:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:19:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:19:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:19:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:19:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:19:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:19:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:19:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:19:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:19:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:19:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:19:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:19:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:19:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:19:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:19:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:19:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:19:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:19:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:19:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:19:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:19:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:19:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:19:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:19:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:19:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:19:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:19:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:19:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:19:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:19:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:19:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:19:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:19:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:19:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:19:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:19:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:19:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:19:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:19:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:19:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:19:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:19:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:19:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:19:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:19:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:19:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:19:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:19:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:19:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:19:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:19:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:19:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:19:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:19:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:19:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:19:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:19:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:19:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:19:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:19:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:19:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:19:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:19:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:19:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:20:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:20:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:20:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:20:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:20:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:20:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:20:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:20:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:20:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:20:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:20:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:20:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:20:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:20:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:20:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:20:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:20:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:20:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:20:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:20:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:20:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:20:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:20:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:20:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:20:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:20:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:20:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:20:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:20:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:20:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:20:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:20:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:20:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:20:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:20:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:20:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:20:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:20:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:20:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:20:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:20:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:20:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:20:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:20:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:20:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:20:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:20:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:20:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:20:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:20:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:20:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:20:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:20:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:20:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:20:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:20:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:20:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:20:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:20:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:20:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:20:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:20:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:20:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:20:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:20:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:20:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:20:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:20:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:20:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:20:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:20:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:20:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:20:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:20:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:20:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:20:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:20:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:20:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:20:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:20:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:20:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:20:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:20:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:20:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:20:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:20:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:20:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:20:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:20:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:20:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:20:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:20:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:20:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:20:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:20:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:20:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:20:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:20:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:20:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:20:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:20:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:20:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:20:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:20:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:20:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:20:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:20:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:20:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:20:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:20:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:20:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:20:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:20:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:20:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:20:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:20:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:20:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:20:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:20:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:20:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:20:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:20:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:20:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:20:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:20:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:20:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:20:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:20:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:20:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:20:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:20:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:20:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:20:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:20:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:20:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:20:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:20:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:20:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:20:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:20:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:20:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:20:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:20:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:20:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:20:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:20:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:20:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:20:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:20:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:20:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:20:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:20:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:20:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:20:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:20:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:20:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:20:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:20:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:20:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:20:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:20:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:20:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:20:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:20:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:20:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:20:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:20:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:20:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:20:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:20:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:20:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:20:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:20:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:20:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:20:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:20:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:20:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:20:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:20:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:20:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:20:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:20:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:20:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:20:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:20:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:20:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:20:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:20:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:20:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:20:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:20:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:20:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:20:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:20:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:20:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:20:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:20:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:20:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:20:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:20:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:20:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:20:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:20:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:20:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:20:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:20:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:20:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:20:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:20:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:20:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:20:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:20:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:20:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:20:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:20:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:20:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:21:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:21:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:21:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:21:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:21:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:21:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:21:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:21:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:21:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:21:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:21:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:21:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:21:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:21:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:21:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:21:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:21:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:21:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:21:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:21:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:21:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:21:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:21:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:21:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:21:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:21:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:21:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:21:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:21:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:21:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:21:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:21:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:21:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:21:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:21:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:21:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:21:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:21:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:21:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:21:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:21:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:21:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:21:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:21:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:21:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:21:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:21:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:21:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:21:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:21:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:21:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:21:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:21:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:21:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:21:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:21:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:21:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:21:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:21:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:21:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:21:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:21:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:21:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:21:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:21:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:21:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:21:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:21:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:21:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:21:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:21:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:21:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:21:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:21:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:21:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:21:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:21:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:21:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:21:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:21:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:21:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:21:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:21:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:21:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:21:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:21:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:21:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:21:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:21:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:21:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:21:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:21:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:21:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:21:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:21:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:21:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:21:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:21:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:21:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:21:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:21:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:21:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:21:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:21:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:21:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:21:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:21:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:21:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:21:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:21:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:21:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:21:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:21:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:21:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:21:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:21:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:21:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:21:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:21:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:21:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:21:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:21:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:21:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:21:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:21:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:21:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:21:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:21:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:21:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:21:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:21:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:21:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:21:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:21:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:21:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:21:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:21:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:21:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:21:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:21:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:21:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:21:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:21:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:21:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:21:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:21:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:21:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:21:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:21:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:21:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:21:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:21:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:21:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:21:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:21:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:21:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:21:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:21:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:21:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:21:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:21:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:21:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:21:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:21:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:21:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:21:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:21:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:21:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:21:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:21:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:21:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:21:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:21:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:21:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:21:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:21:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:21:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:21:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:21:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:21:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:21:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:21:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:21:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:21:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:21:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:21:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:21:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:21:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:21:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:21:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:21:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:21:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:21:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:21:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:21:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:21:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:21:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:21:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:21:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:21:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:21:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:21:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:21:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:21:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:21:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:21:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:21:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:21:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:21:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:21:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:21:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:21:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:21:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:21:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:21:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:21:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:24:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:24:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:24:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:24:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:24:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:24:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:24:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:24:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:24:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:24:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:24:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:24:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:24:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:24:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:24:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:24:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:24:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:24:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:24:01 rdma06 rsyslogd[1041]: imjournal: 5082 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 08:24:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:24:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:24:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:24:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:24:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:24:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:24:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:24:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:24:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:24:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:24:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:24:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:24:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:24:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:24:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:24:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:24:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:24:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:24:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:24:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:24:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:24:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:24:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:24:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:24:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:24:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:24:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:24:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:24:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:24:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:24:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:24:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:24:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:24:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:24:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:24:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:24:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:24:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:24:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:24:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:24:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:24:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:24:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:24:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:24:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:24:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:24:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:24:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:24:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:24:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:24:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:24:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:24:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:24:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:24:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:24:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:24:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:24:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:24:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:24:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:24:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:24:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:24:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:24:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:24:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:24:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:24:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:24:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:24:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:24:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:24:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:24:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:24:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:24:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:24:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:24:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:24:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:24:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:24:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:24:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:24:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:24:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:24:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:24:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:24:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:24:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:24:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:24:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:24:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:24:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:24:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:24:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:24:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:24:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:24:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:24:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:24:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:24:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:24:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:24:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:24:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:24:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:24:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:24:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:24:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:24:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:24:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:24:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:24:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:24:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:24:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:24:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:24:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:24:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:24:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:24:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:24:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:24:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:24:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:24:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:24:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:24:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:24:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:24:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:24:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:24:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:24:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:24:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:24:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:24:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:24:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:24:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:24:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:24:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:24:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:24:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:24:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:24:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:24:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:24:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:24:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:24:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:24:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:24:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:24:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:24:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:24:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:24:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:24:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:24:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:24:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:24:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:24:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:24:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:24:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:24:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:24:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:24:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:24:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:24:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:24:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:24:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:24:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:24:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:24:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:24:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:24:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:24:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:24:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:24:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:24:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:24:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:24:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:24:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:24:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:24:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:24:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:24:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:24:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:24:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:24:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:24:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:24:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:24:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:24:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:24:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:24:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:24:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:24:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:24:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:24:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:24:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:24:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:24:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:24:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:24:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:24:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:24:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:25:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:25:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:25:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:25:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:25:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:25:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:25:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:25:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:25:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:25:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:25:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:25:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:25:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:25:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:25:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:25:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:25:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:25:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:25:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:25:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:25:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:25:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:25:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:25:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:25:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:25:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:25:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:25:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:25:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:25:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:25:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:25:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:25:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:25:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:25:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:25:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:25:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:25:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:25:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:25:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:25:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:25:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:25:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:25:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:25:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:25:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:25:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:25:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:25:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:25:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:25:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:25:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:25:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:25:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:25:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:25:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:25:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:25:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:25:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:25:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:25:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:25:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:25:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:25:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:25:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:25:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:25:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:25:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:25:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:25:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:25:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:25:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:25:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:25:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:25:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:25:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:25:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:25:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:25:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:25:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:25:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:25:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:25:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:25:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:25:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:25:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:25:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:25:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:25:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:25:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:25:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:25:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:25:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:25:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:25:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:25:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:25:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:25:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:25:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:25:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:25:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:25:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:25:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:25:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:25:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:25:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:25:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:25:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:25:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:25:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:25:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:25:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:25:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:25:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:25:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:25:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:25:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:25:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:25:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:25:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:25:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:25:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:25:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:25:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:25:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:25:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:25:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:25:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:25:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:25:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:25:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:25:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:25:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:25:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:25:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:25:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:25:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:25:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:25:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:25:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:25:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:25:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:25:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:25:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:25:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:25:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:25:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:25:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:25:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:25:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:25:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:25:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:25:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:25:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:25:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:25:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:25:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:25:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:25:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:25:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:25:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:25:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:25:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:25:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:25:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:25:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:25:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:25:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:25:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:25:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:25:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:25:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:25:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:25:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:25:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:25:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:25:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:25:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:25:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:25:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:25:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:25:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:25:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:25:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:25:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:25:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:25:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:25:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:25:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:25:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:25:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:25:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:25:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:25:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:25:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:25:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:25:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:25:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:25:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:25:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:25:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:25:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:25:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:25:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:25:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:25:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:25:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:25:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:25:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:25:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:25:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:25:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:25:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:25:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:25:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:25:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:26:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:26:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:26:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:26:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:26:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:26:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:26:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:26:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:26:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:26:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:26:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:26:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:26:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:26:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:26:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:26:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:26:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:26:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:26:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:26:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:26:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:26:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:26:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:26:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:26:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:26:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:26:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:26:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:26:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:26:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:26:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:26:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:26:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:26:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:26:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:26:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:26:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:26:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:26:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:26:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:26:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:26:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:26:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:26:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:26:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:26:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:26:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:26:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:26:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:26:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:26:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:26:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:26:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:26:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:26:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:26:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:26:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:26:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:26:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:26:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:26:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:26:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:26:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:26:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:26:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:26:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:26:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:26:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:26:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:26:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:26:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:26:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:26:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:26:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:26:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:26:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:26:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:26:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:26:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:26:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:26:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:26:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:26:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:26:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:26:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:26:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:26:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:26:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:26:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:26:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:26:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:26:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:26:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:26:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:26:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:26:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:26:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:26:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:26:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:26:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:26:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:26:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:26:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:26:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:26:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:26:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:26:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:26:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:26:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:26:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:26:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:26:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:26:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:26:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:26:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:26:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:26:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:26:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:26:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:26:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:26:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:26:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:26:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:26:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:26:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:26:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:26:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:26:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:26:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:26:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:26:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:26:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:26:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:26:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:26:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:26:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:26:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:26:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:26:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:26:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:26:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:26:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:26:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:26:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:26:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:26:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:26:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:26:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:26:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:26:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:26:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:26:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:26:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:26:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:26:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:26:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:26:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:26:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:26:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:26:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:26:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:26:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:26:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:26:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:26:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:26:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:26:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:26:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:26:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:26:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:26:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:26:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:26:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:26:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:26:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:26:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:26:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:26:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:26:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:26:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:26:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:26:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:26:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:26:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:26:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:26:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:26:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:26:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:26:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:26:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:26:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:26:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:26:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:26:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:26:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:26:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:26:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:26:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:26:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:26:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:26:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:26:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:26:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:26:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:26:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:26:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:26:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:26:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:26:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:26:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:26:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:26:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:26:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:26:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:26:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:26:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:27:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:27:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:27:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:27:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:27:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:27:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:27:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:27:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:27:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:27:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:27:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:27:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:27:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:27:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:27:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:27:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:27:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:27:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:27:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:27:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:27:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:27:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:27:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:27:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:27:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:27:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:27:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:27:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:27:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:27:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:27:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:27:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:27:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:27:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:27:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:27:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:27:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:27:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:27:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:27:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:27:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:27:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:27:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:27:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:27:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:27:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:27:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:27:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:27:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:27:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:27:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:27:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:27:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:27:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:27:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:27:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:27:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:27:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:27:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:27:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:27:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:27:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:27:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:27:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:27:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:27:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:27:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:27:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:27:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:27:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:27:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:27:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:27:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:27:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:27:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:27:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:27:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:27:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:27:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:27:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:27:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:27:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:27:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:27:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:27:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:27:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:27:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:27:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:27:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:27:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:27:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:27:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:27:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:27:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:27:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:27:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:27:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:27:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:27:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:27:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:27:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:27:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:27:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:27:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:27:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:27:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:27:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:27:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:27:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:27:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:27:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:27:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:27:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:27:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:27:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:27:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:27:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:27:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:27:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:27:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:27:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:27:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:27:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:27:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:27:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:27:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:27:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:27:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:27:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:27:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:27:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:27:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:27:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:27:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:27:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:27:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:27:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:27:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:27:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:27:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:27:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:27:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:27:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:27:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:27:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:27:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:27:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:27:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:27:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:27:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:27:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:27:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:27:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:27:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:27:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:27:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:27:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:27:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:27:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:27:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:27:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:27:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:27:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:27:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:27:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:27:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:27:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:27:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:27:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:27:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:27:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:27:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:27:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:27:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:27:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:27:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:27:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:27:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:27:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:27:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:27:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:27:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:27:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:27:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:27:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:27:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:27:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:27:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:27:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:27:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:27:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:27:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:27:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:27:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:27:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:27:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:27:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:27:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:27:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:27:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:27:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:27:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:27:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:27:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:27:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:27:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:27:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:27:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:27:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:27:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:27:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:27:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:27:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:27:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:27:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:27:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:28:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:28:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:28:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:28:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:28:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:28:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:28:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:28:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:28:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:28:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:28:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:28:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:28:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:28:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:28:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:28:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:28:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:28:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:28:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:28:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:28:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:28:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:28:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:28:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:28:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:28:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:28:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:28:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:28:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:28:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:28:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:28:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:28:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:28:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:28:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:28:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:28:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:28:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:28:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:28:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:28:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:28:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:28:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:28:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:28:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:28:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:28:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:28:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:28:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:28:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:28:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:28:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:28:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:28:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:28:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:28:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:28:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:28:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:28:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:28:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:28:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:28:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:28:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:28:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:28:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:28:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:28:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:28:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:28:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:28:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:28:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:28:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:28:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:28:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:28:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:28:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:28:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:28:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:28:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:28:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:28:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:28:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:28:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:28:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:28:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:28:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:28:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:28:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:28:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:28:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:28:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:28:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:28:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:28:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:28:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:28:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:28:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:28:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:28:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:28:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:28:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:28:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:28:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:28:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:28:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:28:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:28:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:28:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:28:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:28:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:28:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:28:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:28:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:28:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:28:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:28:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:28:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:28:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:28:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:28:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:28:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:28:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:28:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:28:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:28:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:28:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:28:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:28:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:28:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:28:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:28:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:28:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:28:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:28:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:28:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:28:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:28:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:28:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:28:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:28:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:28:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:28:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:28:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:28:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:28:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:28:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:28:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:28:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:28:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:28:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:28:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:28:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:28:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:28:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:28:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:28:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:28:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:28:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:28:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:28:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:28:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:28:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:28:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:28:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:28:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:28:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:28:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:28:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:28:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:28:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:28:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:28:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:28:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:28:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:28:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:28:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:28:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:28:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:28:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:28:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:28:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:28:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:28:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:28:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:28:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:28:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:28:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:28:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:28:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:28:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:28:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:28:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:28:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:28:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:28:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:28:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:28:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:28:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:28:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:28:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:28:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:28:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:28:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:28:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:28:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:28:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:28:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:28:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:28:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:28:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:28:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:28:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:28:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:28:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:28:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:28:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:29:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:29:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:29:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:29:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:29:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:29:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:29:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:29:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:29:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:29:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:29:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:29:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:29:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:29:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:29:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:29:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:29:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:29:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:29:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:29:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:29:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:29:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:29:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:29:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:29:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:29:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:29:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:29:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:29:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:29:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:29:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:29:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:29:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:29:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:29:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:29:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:29:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:29:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:29:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:29:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:29:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:29:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:29:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:29:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:29:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:29:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:29:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:29:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:29:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:29:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:29:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:29:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:29:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:29:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:29:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:29:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:29:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:29:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:29:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:29:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:29:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:29:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:29:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:29:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:29:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:29:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:29:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:29:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:29:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:29:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:29:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:29:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:29:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:29:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:29:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:29:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:29:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:29:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:29:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:29:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:29:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:29:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:29:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:29:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:29:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:29:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:29:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:29:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:29:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:29:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:29:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:29:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:29:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:29:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:29:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:29:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:29:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:29:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:29:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:29:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:29:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:29:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:29:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:29:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:29:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:29:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:29:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:29:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:29:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:29:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:29:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:29:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:29:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:29:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:29:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:29:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:29:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:29:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:29:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:29:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:29:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:29:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:29:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:29:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:29:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:29:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:29:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:29:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:29:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:29:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:29:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:29:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:29:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:29:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:29:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:29:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:29:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:29:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:29:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:29:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:29:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:29:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:29:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:29:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:29:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:29:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:29:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:29:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:29:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:29:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:29:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:29:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:29:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:29:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:29:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:29:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:29:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:29:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:29:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:29:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:29:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:29:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:29:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:29:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:29:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:29:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:29:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:29:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:29:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:29:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:29:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:29:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:29:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:29:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:29:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:29:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:29:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:29:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:29:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:29:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:29:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:29:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:29:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:29:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:29:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:29:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:29:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:29:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:29:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:29:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:29:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:29:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:29:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:29:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:29:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:29:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:29:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:29:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:29:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:29:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:29:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:29:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:29:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:29:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:29:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:29:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:29:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:29:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:29:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:29:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:29:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:29:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:29:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:29:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:29:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:29:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:30:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:30:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:30:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:30:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:30:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:30:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:30:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:30:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:30:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:30:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:30:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:30:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:30:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:30:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:30:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:30:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:30:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:30:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:30:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:30:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:30:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:30:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:30:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:30:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:30:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:30:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:30:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:30:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:30:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:30:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:30:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:30:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:30:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:30:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:30:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:30:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:30:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:30:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:30:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:30:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:30:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:30:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:30:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:30:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:30:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:30:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:30:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:30:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:30:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:30:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:30:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:30:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:30:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:30:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:30:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:30:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:30:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:30:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:30:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:30:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:30:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:30:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:30:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:30:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:30:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:30:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:30:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:30:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:30:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:30:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:30:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:30:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:30:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:30:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:30:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:30:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:30:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:30:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:30:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:30:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:30:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:30:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:30:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:30:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:30:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:30:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:30:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:30:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:30:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:30:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:30:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:30:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:30:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:30:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:30:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:30:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:30:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:30:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:30:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:30:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:30:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:30:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:30:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:30:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:30:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:30:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:30:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:30:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:30:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:30:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:30:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:30:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:30:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:30:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:30:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:30:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:30:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:30:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:30:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:30:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:30:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:30:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:30:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:30:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:30:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:30:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:30:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:30:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:30:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:30:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:30:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:30:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:30:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:30:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:30:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:30:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:30:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:30:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:30:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:30:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:30:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:30:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:30:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:30:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:30:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:30:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:30:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:30:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:30:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:30:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:30:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:30:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:30:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:30:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:30:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:30:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:30:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:30:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:30:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:30:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:30:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:30:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:30:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:30:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:30:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:30:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:30:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:30:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:30:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:30:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:30:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:30:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:30:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:30:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:30:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:30:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:30:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:30:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:30:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:30:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:30:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:30:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:30:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:30:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:30:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:30:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:30:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:30:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:30:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:30:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:30:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:30:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:30:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:30:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:30:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:30:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:30:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:30:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:30:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:30:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:30:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:30:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:30:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:30:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:30:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:30:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:30:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:30:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:30:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:30:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:30:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:30:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:30:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:30:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:30:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:30:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:31:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:31:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:31:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:31:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:31:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:31:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:31:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:31:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:31:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:31:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:31:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:31:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:31:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:31:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:31:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:31:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:31:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:31:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:31:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:31:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:31:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:31:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:31:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:31:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:31:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:31:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:31:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:31:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:31:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:31:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:31:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:31:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:31:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:31:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:31:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:31:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:31:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:31:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:31:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:31:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:31:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:31:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:31:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:31:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:31:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:31:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:31:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:31:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:31:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:31:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:31:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:31:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:31:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:31:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:31:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:31:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:31:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:31:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:31:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:31:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:31:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:31:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:31:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:31:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:31:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:31:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:31:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:31:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:31:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:31:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:31:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:31:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:31:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:31:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:31:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:31:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:31:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:31:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:31:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:31:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:31:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:31:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:31:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:31:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:31:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:31:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:31:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:31:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:31:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:31:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:31:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:31:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:31:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:31:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:31:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:31:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:31:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:31:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:31:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:31:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:31:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:31:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:31:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:31:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:31:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:31:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:31:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:31:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:31:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:31:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:31:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:31:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:31:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:31:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:31:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:31:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:31:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:31:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:31:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:31:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:31:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:31:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:31:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:31:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:31:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:31:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:31:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:31:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:31:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:31:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:31:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:31:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:31:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:31:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:31:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:31:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:31:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:31:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:31:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:31:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:31:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:31:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:31:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:31:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:31:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:31:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:31:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:31:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:31:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:31:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:31:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:31:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:31:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:31:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:31:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:31:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:31:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:31:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:31:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:31:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:31:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:31:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:31:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:31:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:31:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:31:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:34:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:34:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:34:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:34:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:34:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:34:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:34:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:34:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:34:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:34:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:34:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:34:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:34:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:34:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:34:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:34:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:34:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:34:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:34:06 rdma06 rsyslogd[1041]: imjournal: 5296 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 08:34:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:34:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:34:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:34:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:34:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:34:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:34:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:34:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:34:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:34:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:34:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:34:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:34:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:34:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:34:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:34:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:34:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:34:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:34:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:34:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:34:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:34:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:34:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:34:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:34:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:34:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:34:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:34:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:34:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:34:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:34:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:34:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:34:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:34:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:34:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:34:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:34:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:34:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:34:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:34:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:34:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:34:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:34:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:34:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:34:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:34:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:34:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:34:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:34:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:34:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:34:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:34:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:34:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:34:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:34:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:34:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:34:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:34:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:34:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:34:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:34:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:34:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:34:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:34:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:34:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:34:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:34:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:34:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:34:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:34:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:34:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:34:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:34:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:34:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:34:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:34:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:34:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:34:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:34:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:34:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:34:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:34:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:34:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:34:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:34:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:34:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:34:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:34:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:34:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:34:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:34:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:34:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:34:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:34:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:34:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:34:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:34:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:34:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:34:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:34:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:34:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:34:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:34:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:34:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:34:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:34:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:34:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:34:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:34:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:34:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:34:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:34:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:34:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:34:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:34:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:34:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:34:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:34:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:34:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:34:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:34:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:34:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:34:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:34:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:34:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:34:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:34:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:34:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:34:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:34:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:34:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:34:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:34:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:34:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:34:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:34:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:34:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:34:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:34:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:34:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:34:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:34:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:34:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:34:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:34:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:34:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:34:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:34:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:34:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:34:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:34:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:34:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:34:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:34:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:34:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:34:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:34:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:34:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:34:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:34:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:34:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:34:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:34:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:34:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:34:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:34:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:34:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:34:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:34:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:34:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:34:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:34:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:34:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:34:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:34:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:34:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:34:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:34:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:34:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:34:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:35:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:35:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:35:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:35:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:35:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:35:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:35:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:35:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:35:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:35:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:35:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:35:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:35:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:35:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:35:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:35:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:35:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:35:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:35:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:35:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:35:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:35:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:35:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:35:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:35:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:35:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:35:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:35:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:35:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:35:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:35:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:35:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:35:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:35:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:35:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:35:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:35:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:35:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:35:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:35:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:35:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:35:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:35:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:35:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:35:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:35:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:35:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:35:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:35:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:35:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:35:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:35:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:35:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:35:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:35:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:35:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:35:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:35:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:35:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:35:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:35:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:35:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:35:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:35:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:35:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:35:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:35:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:35:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:35:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:35:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:35:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:35:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:35:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:35:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:35:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:35:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:35:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:35:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:35:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:35:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:35:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:35:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:35:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:35:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:35:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:35:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:35:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:35:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:35:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:35:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:35:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:35:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:35:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:35:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:35:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:35:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:35:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:35:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:35:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:35:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:35:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:35:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:35:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:35:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:35:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:35:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:35:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:35:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:35:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:35:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:35:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:35:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:35:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:35:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:35:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:35:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:35:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:35:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:35:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:35:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:35:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:35:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:35:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:35:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:35:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:35:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:35:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:35:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:35:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:35:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:35:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:35:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:35:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:35:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:35:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:35:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:35:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:35:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:35:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:35:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:35:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:35:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:35:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:35:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:35:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:35:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:35:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:35:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:35:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:35:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:35:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:35:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:35:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:35:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:35:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:35:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:35:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:35:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:35:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:35:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:35:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:35:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:35:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:35:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:35:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:35:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:35:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:35:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:35:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:35:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:35:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:35:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:35:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:35:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:35:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:35:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:35:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:35:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:35:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:35:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:35:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:35:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:35:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:35:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:35:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:35:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:35:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:35:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:35:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:35:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:35:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:35:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:35:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:35:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:35:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:35:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:35:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:35:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:35:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:35:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:35:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:35:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:35:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:35:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:35:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:35:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:35:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:35:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:35:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:35:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:35:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:35:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:35:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:35:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:35:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:35:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:36:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:36:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:36:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:36:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:36:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:36:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:36:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:36:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:36:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:36:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:36:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:36:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:36:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:36:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:36:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:36:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:36:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:36:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:36:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:36:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:36:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:36:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:36:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:36:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:36:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:36:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:36:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:36:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:36:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:36:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:36:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:36:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:36:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:36:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:36:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:36:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:36:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:36:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:36:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:36:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:36:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:36:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:36:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:36:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:36:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:36:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:36:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:36:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:36:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:36:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:36:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:36:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:36:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:36:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:36:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:36:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:36:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:36:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:36:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:36:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:36:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:36:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:36:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:36:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:36:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:36:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:36:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:36:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:36:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:36:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:36:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:36:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:36:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:36:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:36:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:36:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:36:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:36:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:36:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:36:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:36:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:36:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:36:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:36:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:36:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:36:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:36:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:36:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:36:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:36:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:36:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:36:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:36:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:36:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:36:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:36:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:36:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:36:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:36:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:36:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:36:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:36:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:36:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:36:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:36:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:36:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:36:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:36:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:36:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:36:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:36:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:36:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:36:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:36:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:36:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:36:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:36:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:36:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:36:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:36:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:36:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:36:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:36:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:36:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:36:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:36:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:36:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:36:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:36:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:36:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:36:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:36:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:36:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:36:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:36:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:36:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:36:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:36:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:36:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:36:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:36:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:36:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:36:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:36:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:36:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:36:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:36:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:36:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:36:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:36:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:36:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:36:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:36:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:36:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:36:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:36:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:36:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:36:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:36:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:36:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:36:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:36:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:36:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:36:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:36:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:36:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:36:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:36:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:36:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:36:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:36:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:36:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:36:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:36:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:36:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:36:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:36:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:36:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:36:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:36:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:36:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:36:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:36:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:36:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:36:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:36:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:36:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:36:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:36:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:36:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:36:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:36:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:36:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:36:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:36:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:36:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:36:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:36:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:36:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:36:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:36:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:36:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:36:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:36:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:36:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:36:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:36:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:36:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:36:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:36:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:36:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:36:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:36:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:36:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:36:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:36:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:37:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:37:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:37:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:37:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:37:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:37:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:37:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:37:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:37:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:37:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:37:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:37:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:37:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:37:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:37:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:37:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:37:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:37:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:37:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:37:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:37:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:37:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:37:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:37:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:37:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:37:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:37:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:37:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:37:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:37:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:37:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:37:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:37:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:37:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:37:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:37:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:37:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:37:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:37:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:37:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:37:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:37:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:37:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:37:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:37:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:37:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:37:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:37:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:37:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:37:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:37:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:37:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:37:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:37:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:37:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:37:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:37:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:37:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:37:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:37:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:37:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:37:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:37:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:37:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:37:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:37:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:37:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:37:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:37:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:37:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:37:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:37:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:37:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:37:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:37:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:37:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:37:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:37:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:37:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:37:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:37:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:37:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:37:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:37:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:37:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:37:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:37:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:37:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:37:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:37:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:37:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:37:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:37:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:37:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:37:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:37:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:37:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:37:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:37:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:37:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:37:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:37:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:37:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:37:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:37:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:37:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:37:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:37:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:37:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:37:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:37:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:37:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:37:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:37:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:37:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:37:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:37:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:37:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:37:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:37:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:37:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:37:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:37:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:37:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:37:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:37:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:37:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:37:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:37:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:37:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:37:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:37:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:37:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:37:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:37:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:37:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:37:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:37:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:37:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:37:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:37:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:37:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:37:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:37:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:37:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:37:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:37:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:37:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:37:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:37:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:37:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:37:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:37:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:37:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:37:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:37:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:37:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:37:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:37:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:37:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:37:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:37:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:37:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:37:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:37:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:37:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:37:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:37:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:37:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:37:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:37:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:37:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:37:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:37:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:37:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:37:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:37:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:37:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:37:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:37:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:37:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:37:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:37:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:37:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:37:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:37:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:37:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:37:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:37:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:37:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:37:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:37:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:37:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:37:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:37:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:37:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:37:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:37:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:37:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:37:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:37:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:37:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:37:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:37:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:37:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:37:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:37:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:37:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:37:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:37:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:37:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:37:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:37:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:37:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:37:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:37:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:38:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:38:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:38:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:38:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:38:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:38:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:38:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:38:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:38:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:38:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:38:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:38:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:38:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:38:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:38:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:38:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:38:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:38:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:38:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:38:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:38:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:38:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:38:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:38:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:38:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:38:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:38:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:38:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:38:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:38:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:38:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:38:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:38:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:38:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:38:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:38:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:38:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:38:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:38:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:38:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:38:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:38:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:38:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:38:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:38:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:38:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:38:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:38:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:38:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:38:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:38:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:38:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:38:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:38:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:38:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:38:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:38:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:38:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:38:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:38:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:38:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:38:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:38:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:38:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:38:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:38:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:38:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:38:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:38:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:38:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:38:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:38:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:38:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:38:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:38:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:38:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:38:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:38:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:38:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:38:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:38:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:38:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:38:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:38:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:38:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:38:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:38:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:38:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:38:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:38:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:38:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:38:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:38:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:38:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:38:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:38:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:38:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:38:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:38:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:38:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:38:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:38:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:38:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:38:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:38:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:38:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:38:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:38:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:38:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:38:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:38:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:38:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:38:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:38:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:38:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:38:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:38:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:38:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:38:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:38:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:38:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:38:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:38:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:38:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:38:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:38:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:38:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:38:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:38:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:38:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:38:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:38:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:38:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:38:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:38:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:38:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:38:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:38:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:38:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:38:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:38:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:38:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:38:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:38:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:38:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:38:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:38:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:38:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:38:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:38:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:38:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:38:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:38:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:38:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:38:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:38:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:38:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:38:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:38:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:38:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:38:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:38:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:38:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:38:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:38:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:38:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:38:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:38:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:38:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:38:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:38:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:38:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:38:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:38:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:38:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:38:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:38:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:38:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:38:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:38:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:38:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:38:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:38:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:38:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:38:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:38:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:38:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:38:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:38:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:38:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:38:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:38:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:38:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:38:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:38:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:38:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:38:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:38:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:38:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:38:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:38:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:38:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:38:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:38:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:38:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:38:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:38:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:38:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:38:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:38:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:38:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:38:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:38:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:38:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:38:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:38:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:39:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:39:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:39:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:39:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:39:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:39:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:39:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:39:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:39:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:39:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:39:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:39:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:39:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:39:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:39:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:39:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:39:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:39:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:39:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:39:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:39:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:39:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:39:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:39:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:39:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:39:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:39:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:39:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:39:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:39:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:39:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:39:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:39:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:39:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:39:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:39:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:39:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:39:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:39:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:39:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:39:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:39:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:39:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:39:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:39:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:39:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:39:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:39:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:39:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:39:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:39:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:39:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:39:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:39:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:39:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:39:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:39:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:39:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:39:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:39:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:39:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:39:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:39:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:39:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:39:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:39:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:39:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:39:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:39:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:39:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:39:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:39:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:39:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:39:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:39:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:39:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:39:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:39:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:39:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:39:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:39:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:39:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:39:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:39:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:39:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:39:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:39:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:39:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:39:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:39:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:39:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:39:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:39:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:39:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:39:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:39:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:39:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:39:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:39:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:39:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:39:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:39:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:39:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:39:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:39:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:39:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:39:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:39:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:39:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:39:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:39:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:39:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:39:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:39:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:39:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:39:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:39:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:39:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:39:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:39:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:39:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:39:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:39:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:39:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:39:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:39:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:39:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:39:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:39:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:39:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:39:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:39:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:39:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:39:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:39:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:39:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:39:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:39:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:39:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:39:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:39:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:39:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:39:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:39:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:39:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:39:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:39:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:39:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:39:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:39:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:39:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:39:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:39:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:39:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:39:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:39:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:39:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:39:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:39:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:39:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:39:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:39:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:39:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:39:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:39:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:39:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:39:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:39:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:39:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:39:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:39:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:39:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:39:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:39:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:39:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:39:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:39:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:39:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:39:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:39:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:39:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:39:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:39:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:39:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:39:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:39:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:39:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:39:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:39:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:39:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:39:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:39:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:39:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:39:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:39:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:39:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:39:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:39:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:39:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:39:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:39:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:39:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:39:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:39:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:39:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:39:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:39:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:39:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:39:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:39:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:39:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:39:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:39:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:39:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:39:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:39:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:40:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:40:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:40:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:40:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:40:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:40:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:40:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:40:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:40:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:40:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:40:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:40:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:40:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:40:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:40:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:40:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:40:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:40:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:40:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:40:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:40:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:40:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:40:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:40:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:40:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:40:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:40:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:40:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:40:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:40:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:40:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:40:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:40:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:40:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:40:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:40:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:40:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:40:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:40:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:40:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:40:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:40:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:40:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:40:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:40:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:40:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:40:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:40:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:40:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:40:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:40:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:40:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:40:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:40:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:40:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:40:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:40:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:40:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:40:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:40:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:40:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:40:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:40:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:40:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:40:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:40:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:40:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:40:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:40:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:40:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:40:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:40:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:40:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:40:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:40:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:40:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:40:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:40:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:40:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:40:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:40:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:40:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:40:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:40:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:40:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:40:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:40:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:40:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:40:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:40:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:40:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:40:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:40:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:40:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:40:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:40:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:40:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:40:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:40:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:40:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:40:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:40:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:40:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:40:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:40:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:40:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:40:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:40:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:40:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:40:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:40:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:40:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:40:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:40:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:40:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:40:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:40:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:40:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:40:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:40:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:40:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:40:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:40:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:40:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:40:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:40:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:40:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:40:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:40:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:40:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:40:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:40:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:40:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:40:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:40:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:40:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:40:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:40:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:40:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:40:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:40:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:40:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:40:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:40:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:40:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:40:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:40:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:40:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:40:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:40:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:40:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:40:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:40:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:40:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:40:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:40:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:40:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:40:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:40:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:40:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:40:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:40:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:40:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:40:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:40:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:40:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:40:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:40:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:40:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:40:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:40:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:40:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:40:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:40:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:40:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:40:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:40:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:40:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:40:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:40:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:40:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:40:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:40:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:40:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:40:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:40:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:40:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:40:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:40:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:40:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:40:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:40:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:40:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:40:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:40:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:40:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:40:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:40:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:40:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:40:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:40:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:40:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:40:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:40:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:40:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:40:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:40:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:40:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:40:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:40:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:40:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:40:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:40:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:40:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:40:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:40:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:41:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:41:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:41:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:41:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:41:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:41:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:41:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:41:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:41:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:41:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:41:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:41:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:41:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:41:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:41:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:41:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:41:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:41:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:41:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:41:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:41:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:41:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:41:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:41:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:41:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:41:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:41:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:41:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:41:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:41:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:41:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:41:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:41:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:41:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:41:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:41:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:41:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:41:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:41:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:41:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:41:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:41:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:41:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:41:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:41:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:41:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:41:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:41:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:41:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:41:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:41:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:41:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:41:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:41:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:41:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:41:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:41:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:41:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:41:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:41:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:41:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:41:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:41:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:41:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:41:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:41:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:41:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:41:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:41:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:41:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:41:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:41:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:41:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:41:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:41:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:41:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:41:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:41:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:41:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:41:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:41:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:41:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:41:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:41:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:41:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:41:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:41:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:41:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:41:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:41:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:41:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:41:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:41:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:41:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:41:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:41:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:41:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:41:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:41:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:41:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:41:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:41:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:41:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:41:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:41:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:41:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:41:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:41:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:41:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:41:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:41:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:41:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:41:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:41:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:41:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:41:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:41:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:41:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:41:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:41:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:41:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:41:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:41:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:41:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:41:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:41:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:41:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:41:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:41:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:41:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:41:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:41:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:41:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:41:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:41:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:41:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:41:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:41:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:41:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:41:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:41:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:41:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:41:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:41:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:41:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:41:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:41:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:41:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:41:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:41:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:41:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:41:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:41:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:41:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:41:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:41:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:41:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:41:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:41:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:41:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:41:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:41:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:41:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:41:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:41:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:41:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:41:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:41:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:41:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:41:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:41:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:41:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:41:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:41:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:41:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:41:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:41:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:41:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:41:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:41:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:41:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:41:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:41:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:41:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:41:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:41:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:41:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:41:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:41:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:41:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:41:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:41:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:41:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:41:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:41:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:41:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:41:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:41:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:41:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:41:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:41:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:41:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:41:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:41:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:41:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:41:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:41:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:41:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:41:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:41:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:41:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:41:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:41:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:41:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:41:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:41:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:42:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:42:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:42:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:42:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:42:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:42:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:42:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:42:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:42:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:42:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:42:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:42:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:42:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:42:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:42:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:42:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:42:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:42:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:44:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:44:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:44:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:44:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:44:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:44:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:44:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:44:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:44:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:44:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:44:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:44:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:44:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:44:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:44:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:44:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:44:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:44:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:44:11 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 08:44:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:44:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:44:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:44:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:44:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:44:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:44:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:44:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:44:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:44:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:44:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:44:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:44:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:44:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:44:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:44:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:44:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:44:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:44:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:44:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:44:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:44:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:44:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:44:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:44:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:44:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:44:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:44:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:44:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:44:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:44:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:44:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:44:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:44:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:44:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:44:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:44:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:44:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:44:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:44:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:44:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:44:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:44:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:44:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:44:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:44:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:44:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:44:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:44:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:44:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:44:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:44:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:44:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:44:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:44:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:44:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:44:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:44:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:44:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:44:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:44:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:44:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:44:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:44:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:44:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:44:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:44:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:44:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:44:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:44:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:44:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:44:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:44:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:44:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:44:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:44:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:44:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:44:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:44:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:44:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:44:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:44:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:44:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:44:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:44:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:44:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:44:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:44:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:44:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:44:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:44:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:44:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:44:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:44:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:44:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:44:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:44:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:44:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:44:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:44:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:44:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:44:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:44:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:44:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:44:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:44:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:44:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:44:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:44:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:44:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:44:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:44:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:44:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:44:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:44:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:44:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:44:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:44:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:44:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:44:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:44:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:44:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:44:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:44:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:44:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:44:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:44:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:44:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:44:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:44:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:44:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:44:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:44:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:44:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:44:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:44:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:44:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:44:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:44:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:44:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:44:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:44:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:44:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:44:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:44:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:44:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:44:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:44:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:44:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:44:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:44:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:44:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:44:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:44:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:44:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:44:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:44:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:44:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:44:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:44:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:44:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:44:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:45:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:45:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:45:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:45:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:45:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:45:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:45:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:45:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:45:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:45:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:45:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:45:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:45:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:45:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:45:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:45:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:45:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:45:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:45:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:45:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:45:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:45:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:45:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:45:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:45:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:45:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:45:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:45:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:45:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:45:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:45:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:45:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:45:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:45:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:45:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:45:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:45:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:45:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:45:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:45:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:45:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:45:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:45:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:45:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:45:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:45:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:45:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:45:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:45:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:45:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:45:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:45:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:45:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:45:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:45:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:45:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:45:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:45:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:45:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:45:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:45:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:45:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:45:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:45:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:45:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:45:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:45:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:45:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:45:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:45:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:45:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:45:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:45:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:45:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:45:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:45:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:45:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:45:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:45:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:45:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:45:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:45:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:45:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:45:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:45:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:45:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:45:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:45:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:45:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:45:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:45:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:45:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:45:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:45:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:45:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:45:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:45:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:45:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:45:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:45:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:45:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:45:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:45:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:45:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:45:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:45:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:45:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:45:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:45:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:45:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:45:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:45:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:45:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:45:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:45:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:45:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:45:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:45:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:45:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:45:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:45:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:45:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:45:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:45:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:45:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:45:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:45:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:45:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:45:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:45:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:45:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:45:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:45:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:45:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:45:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:45:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:45:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:45:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:45:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:45:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:45:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:45:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:45:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:45:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:45:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:45:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:45:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:45:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:45:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:45:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:45:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:45:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:45:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:45:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:45:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:45:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:45:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:45:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:45:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:45:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:45:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:45:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:45:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:45:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:45:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:45:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:45:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:45:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:45:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:45:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:45:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:45:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:45:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:45:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:45:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:45:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:45:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:45:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:45:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:45:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:45:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:45:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:45:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:45:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:45:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:45:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:45:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:45:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:45:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:45:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:45:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:45:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:45:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:45:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:45:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:45:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:45:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:45:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:45:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:45:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:45:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:45:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:45:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:45:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:45:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:45:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:45:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:45:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:45:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:45:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:45:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:45:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:45:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:45:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:45:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:45:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:46:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:46:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:46:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:46:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:46:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:46:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:46:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:46:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:46:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:46:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:46:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:46:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:46:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:46:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:46:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:46:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:46:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:46:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:46:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:46:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:46:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:46:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:46:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:46:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:46:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:46:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:46:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:46:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:46:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:46:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:46:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:46:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:46:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:46:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:46:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:46:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:46:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:46:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:46:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:46:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:46:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:46:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:46:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:46:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:46:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:46:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:46:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:46:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:46:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:46:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:46:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:46:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:46:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:46:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:46:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:46:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:46:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:46:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:46:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:46:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:46:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:46:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:46:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:46:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:46:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:46:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:46:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:46:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:46:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:46:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:46:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:46:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:46:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:46:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:46:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:46:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:46:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:46:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:46:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:46:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:46:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:46:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:46:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:46:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:46:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:46:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:46:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:46:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:46:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:46:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:46:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:46:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:46:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:46:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:46:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:46:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:46:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:46:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:46:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:46:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:46:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:46:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:46:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:46:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:46:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:46:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:46:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:46:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:46:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:46:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:46:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:46:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:46:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:46:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:46:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:46:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:46:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:46:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:46:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:46:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:46:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:46:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:46:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:46:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:46:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:46:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:46:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:46:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:46:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:46:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:46:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:46:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:46:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:46:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:46:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:46:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:46:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:46:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:46:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:46:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:46:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:46:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:46:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:46:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:46:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:46:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:46:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:46:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:46:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:46:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:46:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:46:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:46:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:46:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:46:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:46:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:46:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:46:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:46:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:46:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:46:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:46:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:46:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:46:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:46:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:46:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:46:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:46:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:46:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:46:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:46:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:46:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:46:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:46:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:46:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:46:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:46:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:46:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:46:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:46:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:46:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:46:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:46:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:46:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:46:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:46:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:46:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:46:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:46:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:46:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:46:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:46:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:46:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:46:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:46:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:46:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:46:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:46:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:46:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:46:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:46:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:46:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:46:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:46:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:46:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:46:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:46:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:46:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:46:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:46:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:46:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:46:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:46:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:46:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:46:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:46:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:47:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:47:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:47:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:47:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:47:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:47:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:47:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:47:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:47:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:47:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:47:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:47:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:47:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:47:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:47:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:47:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:47:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:47:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:47:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:47:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:47:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:47:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:47:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:47:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:47:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:47:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:47:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:47:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:47:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:47:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:47:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:47:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:47:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:47:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:47:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:47:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:47:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:47:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:47:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:47:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:47:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:47:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:47:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:47:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:47:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:47:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:47:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:47:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:47:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:47:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:47:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:47:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:47:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:47:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:47:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:47:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:47:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:47:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:47:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:47:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:47:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:47:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:47:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:47:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:47:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:47:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:47:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:47:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:47:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:47:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:47:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:47:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:47:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:47:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:47:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:47:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:47:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:47:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:47:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:47:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:47:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:47:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:47:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:47:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:47:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:47:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:47:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:47:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:47:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:47:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:47:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:47:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:47:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:47:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:47:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:47:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:47:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:47:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:47:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:47:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:47:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:47:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:47:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:47:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:47:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:47:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:47:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:47:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:47:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:47:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:47:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:47:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:47:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:47:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:47:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:47:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:47:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:47:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:47:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:47:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:47:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:47:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:47:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:47:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:47:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:47:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:47:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:47:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:47:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:47:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:47:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:47:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:47:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:47:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:47:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:47:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:47:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:47:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:47:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:47:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:47:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:47:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:47:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:47:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:47:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:47:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:47:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:47:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:47:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:47:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:47:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:47:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:47:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:47:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:47:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:47:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:47:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:47:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:47:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:47:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:47:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:47:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:47:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:47:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:47:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:47:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:47:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:47:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:47:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:47:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:47:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:47:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:47:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:47:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:47:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:47:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:47:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:47:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:47:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:47:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:47:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:47:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:47:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:47:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:47:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:47:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:47:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:47:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:47:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:47:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:47:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:47:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:47:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:47:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:47:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:47:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:47:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:47:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:47:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:47:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:47:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:47:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:47:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:47:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:47:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:47:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:47:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:47:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:47:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:47:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:47:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:47:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:47:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:47:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:47:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:47:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:48:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:48:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:48:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:48:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:48:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:48:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:48:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:48:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:48:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:48:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:48:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:48:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:48:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:48:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:48:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:48:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:48:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:48:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:48:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:48:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:48:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:48:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:48:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:48:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:48:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:48:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:48:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:48:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:48:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:48:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:48:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:48:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:48:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:48:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:48:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:48:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:48:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:48:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:48:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:48:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:48:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:48:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:48:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:48:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:48:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:48:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:48:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:48:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:48:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:48:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:48:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:48:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:48:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:48:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:48:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:48:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:48:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:48:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:48:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:48:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:48:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:48:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:48:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:48:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:48:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:48:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:48:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:48:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:48:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:48:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:48:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:48:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:48:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:48:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:48:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:48:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:48:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:48:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:48:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:48:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:48:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:48:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:48:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:48:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:48:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:48:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:48:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:48:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:48:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:48:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:48:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:48:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:48:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:48:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:48:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:48:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:48:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:48:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:48:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:48:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:48:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:48:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:48:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:48:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:48:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:48:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:48:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:48:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:48:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:48:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:48:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:48:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:48:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:48:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:48:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:48:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:48:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:48:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:48:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:48:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:48:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:48:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:48:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:48:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:48:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:48:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:48:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:48:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:48:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:48:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:48:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:48:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:48:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:48:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:48:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:48:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:48:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:48:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:48:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:48:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:48:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:48:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:48:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:48:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:48:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:48:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:48:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:48:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:48:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:48:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:48:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:48:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:48:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:48:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:48:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:48:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:48:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:48:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:48:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:48:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:48:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:48:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:48:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:48:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:48:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:48:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:48:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:48:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:48:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:48:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:48:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:48:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:48:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:48:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:48:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:48:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:48:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:48:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:48:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:48:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:48:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:48:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:48:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:48:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:48:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:48:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:48:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:48:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:48:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:48:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:48:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:48:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:48:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:48:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:48:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:48:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:48:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:48:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:48:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:48:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:48:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:48:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:48:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:48:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:48:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:48:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:48:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:48:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:48:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:48:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:48:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:48:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:48:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:48:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:48:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:48:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:49:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:49:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:49:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:49:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:49:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:49:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:49:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:49:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:49:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:49:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:49:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:49:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:49:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:49:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:49:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:49:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:49:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:49:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:49:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:49:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:49:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:49:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:49:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:49:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:49:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:49:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:49:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:49:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:49:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:49:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:49:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:49:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:49:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:49:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:49:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:49:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:49:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:49:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:49:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:49:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:49:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:49:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:49:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:49:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:49:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:49:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:49:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:49:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:49:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:49:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:49:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:49:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:49:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:49:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:49:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:49:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:49:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:49:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:49:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:49:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:49:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:49:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:49:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:49:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:49:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:49:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:49:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:49:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:49:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:49:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:49:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:49:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:49:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:49:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:49:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:49:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:49:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:49:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:49:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:49:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:49:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:49:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:49:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:49:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:49:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:49:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:49:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:49:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:49:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:49:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:49:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:49:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:49:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:49:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:49:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:49:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:49:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:49:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:49:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:49:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:49:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:49:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:49:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:49:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:49:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:49:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:49:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:49:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:49:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:49:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:49:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:49:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:49:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:49:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:49:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:49:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:49:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:49:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:49:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:49:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:49:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:49:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:49:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:49:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:49:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:49:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:49:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:49:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:49:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:49:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:49:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:49:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:49:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:49:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:49:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:49:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:49:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:49:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:49:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:49:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:49:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:49:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:49:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:49:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:49:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:49:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:49:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:49:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:49:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:49:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:49:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:49:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:49:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:49:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:49:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:49:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:49:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:49:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:49:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:49:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:49:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:49:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:49:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:49:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:49:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:49:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:49:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:49:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:49:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:49:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:49:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:49:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:49:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:49:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:49:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:49:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:49:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:49:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:49:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:49:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:49:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:49:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:49:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:49:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:49:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:49:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:49:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:49:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:49:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:49:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:49:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:49:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:49:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:49:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:49:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:49:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:49:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:49:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:49:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:49:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:49:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:49:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:49:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:49:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:49:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:49:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:49:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:49:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:49:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:49:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:49:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:49:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:49:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:49:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:49:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:49:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:50:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:50:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:50:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:50:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:50:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:50:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:50:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:50:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:50:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:50:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:50:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:50:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:50:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:50:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:50:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:50:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:50:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:50:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:50:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:50:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:50:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:50:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:50:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:50:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:50:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:50:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:50:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:50:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:50:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:50:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:50:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:50:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:50:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:50:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:50:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:50:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:50:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:50:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:50:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:50:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:50:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:50:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:50:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:50:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:50:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:50:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:50:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:50:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:50:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:50:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:50:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:50:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:50:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:50:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:50:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:50:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:50:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:50:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:50:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:50:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:50:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:50:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:50:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:50:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:50:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:50:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:50:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:50:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:50:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:50:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:50:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:50:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:50:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:50:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:50:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:50:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:50:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:50:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:50:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:50:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:50:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:50:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:50:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:50:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:50:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:50:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:50:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:50:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:50:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:50:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:50:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:50:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:50:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:50:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:50:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:50:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:50:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:50:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:50:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:50:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:50:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:50:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:50:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:50:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:50:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:50:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:50:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:50:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:50:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:50:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:50:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:50:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:50:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:50:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:50:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:50:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:50:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:50:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:50:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:50:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:50:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:50:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:50:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:50:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:50:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:50:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:50:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:50:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:50:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:50:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:50:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:50:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:50:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:50:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:50:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:50:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:50:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:50:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:50:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:50:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:50:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:50:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:50:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:50:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:50:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:50:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:50:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:50:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:50:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:50:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:50:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:50:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:50:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:50:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:50:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:50:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:50:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:50:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:50:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:50:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:50:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:50:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:50:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:50:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:50:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:50:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:50:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:50:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:50:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:50:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:50:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:50:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:50:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:50:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:50:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:50:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:50:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:50:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:50:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:50:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:50:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:50:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:50:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:50:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:50:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:50:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:50:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:50:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:50:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:50:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:50:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:50:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:50:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:50:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:50:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:50:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:50:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:50:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:50:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:50:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:50:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:50:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:50:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:50:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:50:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:50:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:50:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:50:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:50:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:50:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:50:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:50:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:50:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:50:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:50:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:50:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:51:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:51:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:51:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:51:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:51:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:51:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:51:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:51:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:51:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:51:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:51:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:51:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:51:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:51:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:51:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:51:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:51:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:51:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:51:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:51:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:51:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:51:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:51:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:51:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:51:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:51:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:51:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:51:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:51:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:51:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:51:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:51:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:51:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:51:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:51:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:51:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:51:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:51:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:51:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:51:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:51:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:51:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:51:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:51:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:51:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:51:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:51:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:51:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:51:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:51:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:51:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:51:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:51:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:51:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:51:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:51:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:51:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:51:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:51:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:51:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:51:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:51:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:51:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:51:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:51:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:51:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:51:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:51:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:51:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:51:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:51:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:51:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:51:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:51:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:51:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:51:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:51:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:51:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:51:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:51:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:51:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:51:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:51:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:51:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:51:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:51:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:51:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:51:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:51:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:51:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:51:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:51:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:51:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:51:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:51:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:51:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:51:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:51:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:51:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:51:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:51:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:51:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:51:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:51:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:51:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:51:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:51:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:51:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:51:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:51:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:51:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:51:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:51:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:51:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:51:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:51:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:51:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:51:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:51:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:51:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:51:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:51:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:51:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:51:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:51:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:51:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:51:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:51:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:51:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:51:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:51:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:51:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:51:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:51:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:51:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:51:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:51:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:51:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:51:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:51:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:51:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:51:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:51:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:51:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:51:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:51:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:51:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:51:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:51:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:51:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:51:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:51:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:51:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:51:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:51:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:51:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:51:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:51:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:51:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:51:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:51:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:51:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:51:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:51:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:51:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:51:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:51:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:51:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:51:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:51:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:51:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:51:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:51:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:51:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:51:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:51:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:51:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:51:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:51:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:51:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:51:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:51:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:51:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:51:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:51:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:51:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:51:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:51:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:51:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:51:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:51:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:51:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:51:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:51:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:51:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:51:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:51:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:51:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:51:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:51:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:51:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:51:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:51:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:51:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:51:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:51:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:51:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:51:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:51:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:51:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:51:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:51:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:51:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:51:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:51:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:51:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:52:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:52:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:52:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:52:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:52:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:52:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:52:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:52:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:52:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:52:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:52:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:52:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:52:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:52:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:52:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:52:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:52:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:52:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:52:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:52:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:52:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:52:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:52:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:52:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:52:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:52:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:52:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:52:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:52:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:52:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:52:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:52:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:52:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:52:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:52:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:52:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:54:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:54:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:54:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:54:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:54:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:54:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:54:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:54:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:54:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:54:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:54:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:54:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:54:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:54:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:54:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:54:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:54:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:54:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:54:12 rdma06 rsyslogd[1041]: imjournal: 5082 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 08:54:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:54:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:54:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:54:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:54:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:54:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:54:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:54:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:54:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:54:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:54:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:54:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:54:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:54:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:54:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:54:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:54:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:54:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:54:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:54:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:54:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:54:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:54:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:54:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:54:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:54:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:54:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:54:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:54:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:54:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:54:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:54:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:54:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:54:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:54:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:54:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:54:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:54:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:54:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:54:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:54:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:54:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:54:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:54:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:54:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:54:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:54:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:54:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:54:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:54:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:54:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:54:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:54:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:54:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:54:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:54:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:54:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:54:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:54:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:54:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:54:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:54:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:54:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:54:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:54:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:54:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:54:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:54:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:54:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:54:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:54:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:54:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:54:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:54:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:54:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:54:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:54:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:54:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:54:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:54:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:54:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:54:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:54:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:54:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:54:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:54:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:54:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:54:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:54:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:54:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:54:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:54:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:54:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:54:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:54:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:54:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:54:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:54:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:54:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:54:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:54:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:54:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:54:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:54:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:54:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:54:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:54:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:54:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:54:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:54:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:54:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:54:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:54:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:54:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:54:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:54:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:54:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:54:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:54:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:54:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:54:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:54:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:54:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:54:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:54:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:54:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:54:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:54:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:54:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:54:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:54:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:54:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:54:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:54:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:54:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:54:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:54:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:54:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:54:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:54:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:54:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:54:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:54:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:54:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:54:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:54:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:54:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:54:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:54:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:54:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:54:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:54:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:54:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:54:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:54:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:54:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:54:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:54:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:54:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:54:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:54:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:54:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:55:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:55:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:55:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:55:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:55:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:55:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:55:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:55:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:55:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:55:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:55:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:55:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:55:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:55:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:55:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:55:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:55:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:55:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:55:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:55:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:55:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:55:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:55:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:55:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:55:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:55:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:55:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:55:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:55:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:55:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:55:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:55:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:55:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:55:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:55:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:55:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:55:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:55:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:55:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:55:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:55:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:55:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:55:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:55:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:55:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:55:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:55:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:55:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:55:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:55:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:55:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:55:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:55:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:55:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:55:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:55:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:55:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:55:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:55:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:55:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:55:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:55:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:55:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:55:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:55:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:55:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:55:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:55:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:55:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:55:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:55:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:55:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:55:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:55:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:55:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:55:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:55:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:55:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:55:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:55:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:55:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:55:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:55:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:55:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:55:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:55:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:55:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:55:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:55:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:55:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:55:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:55:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:55:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:55:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:55:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:55:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:55:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:55:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:55:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:55:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:55:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:55:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:55:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:55:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:55:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:55:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:55:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:55:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:55:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:55:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:55:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:55:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:55:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:55:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:55:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:55:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:55:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:55:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:55:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:55:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:55:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:55:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:55:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:55:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:55:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:55:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:55:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:55:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:55:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:55:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:55:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:55:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:55:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:55:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:55:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:55:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:55:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:55:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:55:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:55:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:55:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:55:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:55:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:55:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:55:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:55:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:55:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:55:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:55:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:55:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:55:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:55:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:55:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:55:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:55:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:55:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:55:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:55:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:55:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:55:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:55:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:55:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:55:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:55:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:55:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:55:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:55:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:55:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:55:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:55:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:55:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:55:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:55:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:55:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:55:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:55:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:55:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:55:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:55:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:55:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:55:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:55:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:55:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:55:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:55:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:55:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:55:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:55:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:55:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:55:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:55:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:55:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:55:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:55:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:55:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:55:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:55:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:55:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:55:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:55:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:55:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:55:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:55:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:55:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:55:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:55:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:55:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:55:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:55:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:55:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:55:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:55:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:55:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:55:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:55:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:55:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:56:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:56:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:56:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:56:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:56:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:56:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:56:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:56:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:56:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:56:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:56:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:56:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:56:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:56:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:56:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:56:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:56:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:56:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:56:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:56:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:56:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:56:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:56:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:56:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:56:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:56:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:56:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:56:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:56:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:56:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:56:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:56:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:56:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:56:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:56:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:56:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:56:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:56:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:56:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:56:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:56:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:56:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:56:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:56:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:56:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:56:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:56:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:56:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:56:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:56:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:56:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:56:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:56:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:56:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:56:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:56:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:56:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:56:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:56:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:56:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:56:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:56:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:56:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:56:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:56:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:56:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:56:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:56:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:56:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:56:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:56:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:56:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:56:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:56:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:56:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:56:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:56:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:56:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:56:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:56:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:56:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:56:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:56:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:56:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:56:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:56:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:56:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:56:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:56:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:56:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:56:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:56:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:56:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:56:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:56:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:56:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:56:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:56:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:56:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:56:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:56:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:56:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:56:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:56:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:56:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:56:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:56:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:56:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:56:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:56:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:56:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:56:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:56:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:56:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:56:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:56:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:56:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:56:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:56:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:56:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:56:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:56:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:56:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:56:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:56:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:56:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:56:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:56:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:56:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:56:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:56:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:56:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:56:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:56:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:56:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:56:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:56:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:56:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:56:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:56:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:56:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:56:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:56:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:56:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:56:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:56:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:56:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:56:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:56:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:56:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:56:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:56:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:56:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:56:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:56:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:56:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:56:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:56:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:56:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:56:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:56:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:56:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:56:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:56:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:56:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:56:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:56:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:56:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:56:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:56:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:56:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:56:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:56:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:56:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:56:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:56:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:56:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:56:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:56:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:56:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:56:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:56:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:56:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:56:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:56:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:56:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:56:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:56:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:56:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:56:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:56:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:56:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:56:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:56:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:56:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:56:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:56:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:56:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:56:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:56:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:56:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:56:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:56:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:56:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:56:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:56:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:56:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:56:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:56:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:56:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:56:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:56:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:56:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:56:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:56:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:56:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:57:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:57:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:57:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:57:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:57:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:57:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:57:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:57:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:57:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:57:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:57:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:57:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:57:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:57:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:57:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:57:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:57:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:57:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:57:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:57:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:57:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:57:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:57:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:57:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:57:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:57:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:57:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:57:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:57:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:57:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:57:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:57:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:57:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:57:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:57:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:57:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:57:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:57:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:57:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:57:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:57:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:57:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:57:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:57:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:57:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:57:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:57:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:57:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:57:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:57:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:57:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:57:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:57:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:57:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:57:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:57:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:57:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:57:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:57:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:57:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:57:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:57:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:57:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:57:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:57:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:57:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:57:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:57:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:57:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:57:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:57:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:57:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:57:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:57:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:57:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:57:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:57:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:57:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:57:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:57:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:57:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:57:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:57:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:57:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:57:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:57:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:57:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:57:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:57:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:57:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:57:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:57:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:57:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:57:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:57:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:57:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:57:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:57:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:57:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:57:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:57:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:57:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:57:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:57:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:57:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:57:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:57:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:57:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:57:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:57:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:57:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:57:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:57:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:57:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:57:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:57:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:57:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:57:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:57:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:57:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:57:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:57:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:57:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:57:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:57:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:57:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:57:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:57:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:57:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:57:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:57:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:57:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:57:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:57:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:57:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:57:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:57:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:57:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:57:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:57:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:57:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:57:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:57:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:57:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:57:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:57:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:57:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:57:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:57:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:57:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:57:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:57:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:57:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:57:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:57:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:57:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:57:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:57:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:57:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:57:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:57:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:57:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:57:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:57:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:57:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:57:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:57:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:57:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:57:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:57:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:57:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:57:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:57:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:57:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:57:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:57:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:57:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:57:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:57:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:57:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:57:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:57:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:57:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:57:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:57:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:57:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:57:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:57:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:57:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:57:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:57:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:57:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:57:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:57:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:57:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:57:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:57:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:57:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:57:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:57:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:57:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:57:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:57:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:57:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:57:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:57:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:57:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:57:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:57:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:57:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:57:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:57:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:57:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:57:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:57:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:57:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:58:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:58:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:58:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:58:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:58:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:58:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:58:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:58:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:58:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:58:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:58:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:58:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:58:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:58:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:58:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:58:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:58:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:58:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:58:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:58:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:58:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:58:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:58:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:58:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:58:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:58:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:58:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:58:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:58:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:58:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:58:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:58:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:58:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:58:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:58:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:58:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:58:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:58:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:58:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:58:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:58:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:58:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:58:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:58:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:58:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:58:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:58:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:58:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:58:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:58:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:58:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:58:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:58:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:58:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:58:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:58:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:58:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:58:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:58:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:58:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:58:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:58:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:58:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:58:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:58:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:58:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:58:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:58:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:58:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:58:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:58:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:58:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:58:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:58:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:58:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:58:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:58:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:58:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:58:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:58:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:58:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:58:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:58:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:58:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:58:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:58:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:58:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:58:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:58:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:58:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:58:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:58:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:58:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:58:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:58:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:58:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:58:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:58:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:58:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:58:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:58:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:58:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:58:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:58:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:58:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:58:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:58:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:58:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:58:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:58:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:58:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:58:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:58:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:58:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:58:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:58:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:58:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:58:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:58:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:58:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:58:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:58:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:58:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:58:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:58:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:58:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:58:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:58:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:58:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:58:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:58:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:58:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:58:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:58:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:58:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:58:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:58:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:58:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:58:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:58:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:58:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:58:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:58:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:58:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:58:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:58:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:58:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:58:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:58:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:58:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:58:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:58:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:58:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:58:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:58:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:58:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:58:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:58:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:58:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:58:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:58:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:58:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:58:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:58:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:58:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:58:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:58:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:58:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:58:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:58:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:58:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:58:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:58:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:58:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:58:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:58:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:58:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:58:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:58:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:58:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:58:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:58:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:58:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:58:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:58:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:58:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:58:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:58:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:58:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:58:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:58:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:58:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:58:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:58:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:58:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:58:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:58:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:58:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:58:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:58:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:58:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:58:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:58:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:58:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:58:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:58:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:58:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:58:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:58:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:58:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:58:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:58:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:58:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:58:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:58:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:58:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:59:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:59:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:59:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:59:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:59:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:59:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:59:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:59:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:59:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:59:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:59:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:59:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:59:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:59:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:59:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:59:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:59:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:59:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:59:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:59:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:59:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:59:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:59:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:59:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:59:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:59:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:59:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:59:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:59:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:59:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:59:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:59:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:59:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:59:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:59:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:59:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:59:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:59:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:59:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:59:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:59:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:59:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:59:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:59:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:59:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:59:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:59:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:59:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:59:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:59:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:59:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:59:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:59:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:59:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:59:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:59:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:59:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:59:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:59:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:59:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:59:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:59:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:59:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:59:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:59:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:59:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:59:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:59:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:59:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:59:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:59:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:59:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:59:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:59:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:59:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:59:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:59:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:59:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:59:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:59:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:59:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:59:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:59:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:59:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:59:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:59:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:59:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:59:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:59:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:59:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:59:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:59:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:59:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:59:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:59:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:59:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:59:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:59:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:59:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:59:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:59:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:59:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:59:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:59:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:59:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:59:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:59:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:59:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:59:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:59:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:59:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:59:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:59:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:59:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:59:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:59:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:59:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:59:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:59:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:59:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:59:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:59:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:59:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:59:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:59:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:59:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:59:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:59:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:59:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:59:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:59:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:59:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:59:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:59:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:59:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:59:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:59:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:59:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:59:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:59:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:59:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:59:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:59:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:59:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:59:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:59:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:59:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:59:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:59:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:59:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:59:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:59:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:59:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:59:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:59:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:59:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:59:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:59:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:59:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:59:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:59:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:59:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:59:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:59:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:59:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:59:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:59:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:59:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:59:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:59:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:59:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:59:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:59:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:59:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:59:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:59:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:59:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:59:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:59:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:59:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:59:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:59:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:59:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:59:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:59:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:59:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:59:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:59:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:59:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:59:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:59:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:59:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:59:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:59:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:59:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:59:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:59:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:59:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 08:59:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 08:59:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:59:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:59:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:59:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 08:59:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 08:59:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 08:59:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 08:59:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 08:59:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 08:59:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 08:59:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 08:59:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 08:59:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 08:59:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 08:59:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 08:59:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 08:59:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:00:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:00:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:00:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:00:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:00:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:00:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:00:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:00:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:00:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:00:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:00:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:00:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:00:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:00:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:00:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:00:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:00:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:00:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:00:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:00:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:00:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:00:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:00:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:00:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:00:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:00:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:00:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:00:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:00:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:00:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:00:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:00:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:00:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:00:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:00:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:00:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:00:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:00:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:00:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:00:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:00:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:00:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:00:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:00:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:00:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:00:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:00:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:00:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:00:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:00:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:00:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:00:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:00:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:00:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:00:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:00:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:00:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:00:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:00:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:00:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:00:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:00:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:00:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:00:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:00:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:00:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:00:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:00:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:00:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:00:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:00:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:00:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:00:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:00:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:00:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:00:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:00:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:00:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:00:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:00:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:00:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:00:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:00:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:00:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:00:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:00:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:00:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:00:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:00:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:00:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:00:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:00:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:00:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:00:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:00:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:00:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:00:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:00:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:00:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:00:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:00:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:00:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:00:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:00:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:00:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:00:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:00:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:00:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:00:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:00:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:00:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:00:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:00:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:00:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:00:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:00:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:00:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:00:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:00:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:00:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:00:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:00:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:00:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:00:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:00:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:00:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:00:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:00:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:00:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:00:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:00:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:00:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:00:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:00:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:00:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:00:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:00:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:00:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:00:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:00:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:00:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:00:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:00:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:00:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:00:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:00:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:00:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:00:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:00:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:00:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:00:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:00:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:00:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:00:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:00:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:00:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:00:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:00:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:00:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:00:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:00:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:00:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:00:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:00:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:00:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:00:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:00:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:00:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:00:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:00:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:00:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:00:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:00:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:00:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:00:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:00:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:00:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:00:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:00:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:00:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:00:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:00:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:00:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:00:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:00:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:00:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:00:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:00:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:00:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:00:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:00:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:00:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:00:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:00:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:00:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:00:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:00:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:00:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:00:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:00:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:00:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:00:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:00:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:00:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:00:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:00:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:00:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:00:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:00:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:00:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:00:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:00:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:00:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:00:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:00:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:00:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:01:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:01:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:01:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:01:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:01:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:01:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:01:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:01:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:01:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:01:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:01:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:01:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:01:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:01:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:01:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:01:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:01:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:01:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:01:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:01:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:01:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:01:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:01:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:01:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:01:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:01:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:01:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:01:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:01:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:01:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:01:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:01:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:01:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:01:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:01:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:01:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:01:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:01:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:01:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:01:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:01:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:01:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:01:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:01:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:01:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:01:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:01:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:01:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:01:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:01:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:01:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:01:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:01:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:01:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:01:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:01:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:01:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:01:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:01:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:01:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:01:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:01:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:01:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:01:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:01:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:01:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:01:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:01:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:01:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:01:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:01:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:01:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:01:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:01:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:01:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:01:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:01:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:01:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:01:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:01:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:01:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:01:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:01:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:01:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:01:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:01:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:01:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:01:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:01:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:01:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:01:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:01:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:01:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:01:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:01:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:01:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:01:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:01:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:01:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:01:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:01:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:01:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:01:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:01:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:01:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:01:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:01:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:01:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:01:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:01:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:01:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:01:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:01:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:01:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:01:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:01:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:01:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:01:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:01:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:01:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:01:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:01:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:01:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:01:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:01:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:01:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:01:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:01:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:01:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:01:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:01:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:01:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:01:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:01:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:01:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:01:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:01:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:01:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:01:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:01:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:01:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:01:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:01:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:01:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:01:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:01:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:01:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:01:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:01:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:01:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:01:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:01:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:01:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:01:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:01:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:01:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:01:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:01:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:01:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:01:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:01:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:01:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:01:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:01:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:01:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:01:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:01:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:01:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:01:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:01:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:01:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:01:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:01:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:01:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:01:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:01:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:01:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:01:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:01:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:01:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:01:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:01:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:01:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:01:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:01:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:01:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:01:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:01:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:01:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:01:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:01:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:01:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:01:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:01:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:01:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:01:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:01:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:01:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:01:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:01:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:01:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:01:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:01:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:01:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:01:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:01:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:01:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:01:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:01:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:01:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:01:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:01:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:01:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:01:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:01:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:01:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:02:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:02:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:02:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:02:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:02:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:02:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:02:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:02:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:02:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:02:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:02:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:02:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:02:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:02:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:02:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:02:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:02:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:02:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:02:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:02:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:02:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:02:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:02:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:02:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:02:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:02:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:02:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:02:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:02:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:02:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:02:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:02:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:02:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:02:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:02:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:02:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:04:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:04:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:04:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:04:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:04:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:04:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:04:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:04:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:04:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:04:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:04:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:04:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:04:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:04:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:04:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:04:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:04:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:04:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:04:17 rdma06 rsyslogd[1041]: imjournal: 5295 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 09:04:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:04:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:04:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:04:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:04:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:04:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:04:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:04:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:04:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:04:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:04:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:04:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:04:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:04:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:04:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:04:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:04:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:04:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:04:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:04:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:04:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:04:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:04:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:04:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:04:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:04:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:04:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:04:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:04:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:04:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:04:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:04:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:04:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:04:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:04:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:04:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:04:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:04:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:04:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:04:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:04:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:04:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:04:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:04:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:04:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:04:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:04:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:04:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:04:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:04:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:04:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:04:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:04:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:04:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:04:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:04:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:04:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:04:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:04:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:04:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:04:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:04:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:04:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:04:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:04:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:04:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:04:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:04:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:04:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:04:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:04:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:04:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:04:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:04:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:04:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:04:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:04:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:04:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:04:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:04:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:04:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:04:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:04:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:04:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:04:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:04:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:04:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:04:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:04:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:04:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:04:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:04:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:04:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:04:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:04:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:04:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:04:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:04:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:04:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:04:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:04:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:04:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:04:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:04:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:04:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:04:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:04:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:04:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:04:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:04:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:04:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:04:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:04:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:04:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:04:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:04:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:04:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:04:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:04:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:04:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:04:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:04:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:04:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:04:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:04:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:04:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:04:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:04:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:04:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:04:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:04:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:04:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:04:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:04:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:04:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:04:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:04:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:04:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:04:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:04:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:04:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:04:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:04:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:04:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:05:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:05:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:05:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:05:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:05:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:05:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:05:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:05:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:05:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:05:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:05:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:05:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:05:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:05:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:05:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:05:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:05:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:05:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:05:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:05:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:05:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:05:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:05:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:05:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:05:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:05:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:05:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:05:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:05:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:05:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:05:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:05:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:05:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:05:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:05:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:05:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:05:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:05:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:05:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:05:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:05:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:05:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:05:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:05:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:05:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:05:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:05:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:05:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:05:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:05:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:05:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:05:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:05:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:05:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:05:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:05:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:05:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:05:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:05:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:05:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:05:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:05:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:05:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:05:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:05:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:05:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:05:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:05:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:05:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:05:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:05:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:05:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:05:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:05:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:05:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:05:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:05:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:05:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:05:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:05:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:05:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:05:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:05:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:05:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:05:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:05:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:05:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:05:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:05:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:05:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:05:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:05:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:05:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:05:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:05:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:05:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:05:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:05:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:05:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:05:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:05:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:05:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:05:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:05:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:05:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:05:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:05:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:05:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:05:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:05:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:05:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:05:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:05:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:05:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:05:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:05:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:05:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:05:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:05:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:05:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:05:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:05:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:05:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:05:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:05:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:05:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:05:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:05:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:05:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:05:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:05:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:05:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:05:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:05:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:05:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:05:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:05:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:05:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:05:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:05:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:05:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:05:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:05:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:05:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:05:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:05:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:05:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:05:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:05:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:05:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:05:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:05:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:05:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:05:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:05:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:05:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:05:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:05:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:05:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:05:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:05:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:05:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:05:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:05:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:05:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:05:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:05:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:05:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:05:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:05:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:05:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:05:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:05:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:05:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:05:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:05:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:05:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:05:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:05:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:05:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:05:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:05:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:05:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:05:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:05:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:05:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:05:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:05:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:05:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:05:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:05:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:05:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:05:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:05:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:05:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:05:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:05:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:05:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:05:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:05:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:05:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:05:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:05:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:05:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:05:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:05:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:05:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:05:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:05:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:05:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:05:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:05:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:05:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:05:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:05:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:05:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:06:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:06:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:06:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:06:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:06:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:06:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:06:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:06:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:06:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:06:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:06:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:06:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:06:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:06:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:06:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:06:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:06:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:06:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:06:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:06:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:06:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:06:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:06:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:06:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:06:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:06:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:06:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:06:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:06:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:06:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:06:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:06:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:06:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:06:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:06:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:06:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:06:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:06:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:06:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:06:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:06:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:06:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:06:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:06:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:06:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:06:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:06:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:06:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:06:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:06:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:06:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:06:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:06:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:06:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:06:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:06:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:06:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:06:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:06:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:06:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:06:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:06:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:06:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:06:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:06:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:06:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:06:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:06:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:06:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:06:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:06:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:06:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:06:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:06:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:06:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:06:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:06:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:06:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:06:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:06:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:06:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:06:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:06:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:06:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:06:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:06:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:06:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:06:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:06:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:06:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:06:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:06:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:06:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:06:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:06:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:06:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:06:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:06:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:06:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:06:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:06:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:06:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:06:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:06:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:06:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:06:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:06:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:06:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:06:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:06:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:06:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:06:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:06:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:06:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:06:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:06:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:06:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:06:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:06:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:06:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:06:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:06:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:06:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:06:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:06:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:06:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:06:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:06:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:06:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:06:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:06:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:06:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:06:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:06:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:06:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:06:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:06:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:06:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:06:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:06:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:06:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:06:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:06:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:06:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:06:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:06:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:06:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:06:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:06:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:06:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:06:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:06:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:06:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:06:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:06:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:06:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:06:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:06:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:06:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:06:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:06:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:06:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:06:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:06:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:06:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:06:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:06:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:06:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:06:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:06:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:06:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:06:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:06:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:06:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:06:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:06:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:06:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:06:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:06:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:06:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:06:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:06:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:06:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:06:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:06:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:06:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:06:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:06:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:06:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:06:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:06:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:06:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:06:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:06:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:06:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:06:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:06:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:06:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:06:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:06:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:06:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:06:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:06:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:06:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:06:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:06:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:06:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:06:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:06:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:06:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:06:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:06:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:06:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:06:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:06:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:06:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:07:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:07:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:07:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:07:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:07:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:07:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:07:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:07:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:07:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:07:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:07:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:07:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:07:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:07:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:07:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:07:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:07:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:07:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:07:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:07:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:07:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:07:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:07:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:07:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:07:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:07:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:07:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:07:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:07:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:07:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:07:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:07:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:07:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:07:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:07:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:07:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:07:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:07:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:07:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:07:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:07:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:07:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:07:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:07:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:07:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:07:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:07:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:07:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:07:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:07:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:07:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:07:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:07:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:07:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:07:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:07:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:07:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:07:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:07:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:07:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:07:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:07:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:07:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:07:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:07:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:07:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:07:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:07:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:07:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:07:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:07:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:07:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:07:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:07:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:07:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:07:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:07:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:07:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:07:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:07:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:07:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:07:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:07:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:07:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:07:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:07:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:07:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:07:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:07:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:07:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:07:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:07:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:07:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:07:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:07:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:07:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:07:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:07:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:07:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:07:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:07:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:07:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:07:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:07:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:07:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:07:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:07:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:07:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:07:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:07:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:07:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:07:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:07:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:07:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:07:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:07:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:07:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:07:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:07:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:07:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:07:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:07:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:07:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:07:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:07:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:07:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:07:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:07:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:07:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:07:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:07:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:07:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:07:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:07:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:07:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:07:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:07:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:07:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:07:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:07:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:07:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:07:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:07:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:07:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:07:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:07:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:07:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:07:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:07:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:07:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:07:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:07:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:07:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:07:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:07:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:07:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:07:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:07:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:07:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:07:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:07:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:07:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:07:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:07:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:07:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:07:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:07:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:07:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:07:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:07:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:07:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:07:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:07:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:07:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:07:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:07:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:07:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:07:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:07:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:07:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:07:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:07:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:07:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:07:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:07:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:07:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:07:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:07:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:07:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:07:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:07:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:07:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:07:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:07:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:07:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:07:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:07:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:07:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:07:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:07:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:07:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:07:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:07:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:07:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:07:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:07:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:07:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:07:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:07:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:07:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:07:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:07:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:07:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:07:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:07:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:07:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:08:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:08:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:08:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:08:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:08:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:08:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:08:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:08:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:08:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:08:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:08:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:08:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:08:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:08:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:08:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:08:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:08:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:08:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:08:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:08:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:08:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:08:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:08:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:08:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:08:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:08:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:08:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:08:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:08:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:08:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:08:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:08:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:08:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:08:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:08:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:08:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:08:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:08:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:08:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:08:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:08:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:08:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:08:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:08:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:08:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:08:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:08:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:08:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:08:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:08:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:08:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:08:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:08:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:08:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:08:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:08:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:08:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:08:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:08:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:08:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:08:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:08:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:08:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:08:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:08:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:08:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:08:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:08:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:08:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:08:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:08:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:08:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:08:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:08:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:08:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:08:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:08:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:08:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:08:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:08:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:08:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:08:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:08:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:08:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:08:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:08:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:08:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:08:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:08:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:08:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:08:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:08:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:08:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:08:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:08:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:08:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:08:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:08:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:08:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:08:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:08:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:08:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:08:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:08:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:08:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:08:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:08:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:08:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:08:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:08:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:08:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:08:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:08:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:08:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:08:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:08:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:08:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:08:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:08:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:08:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:08:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:08:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:08:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:08:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:08:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:08:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:08:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:08:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:08:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:08:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:08:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:08:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:08:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:08:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:08:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:08:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:08:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:08:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:08:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:08:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:08:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:08:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:08:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:08:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:08:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:08:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:08:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:08:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:08:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:08:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:08:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:08:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:08:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:08:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:08:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:08:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:08:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:08:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:08:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:08:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:08:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:08:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:08:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:08:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:08:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:08:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:08:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:08:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:08:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:08:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:08:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:08:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:08:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:08:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:08:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:08:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:08:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:08:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:08:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:08:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:08:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:08:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:08:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:08:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:08:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:08:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:08:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:08:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:08:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:08:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:08:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:08:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:08:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:08:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:08:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:08:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:08:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:08:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:08:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:08:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:08:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:08:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:08:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:08:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:08:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:08:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:08:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:08:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:08:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:08:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:08:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:08:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:08:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:08:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:08:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:08:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:09:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:09:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:09:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:09:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:09:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:09:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:09:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:09:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:09:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:09:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:09:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:09:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:09:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:09:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:09:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:09:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:09:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:09:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:09:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:09:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:09:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:09:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:09:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:09:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:09:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:09:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:09:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:09:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:09:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:09:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:09:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:09:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:09:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:09:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:09:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:09:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:09:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:09:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:09:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:09:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:09:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:09:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:09:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:09:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:09:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:09:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:09:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:09:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:09:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:09:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:09:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:09:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:09:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:09:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:09:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:09:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:09:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:09:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:09:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:09:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:09:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:09:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:09:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:09:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:09:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:09:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:09:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:09:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:09:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:09:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:09:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:09:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:09:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:09:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:09:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:09:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:09:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:09:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:09:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:09:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:09:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:09:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:09:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:09:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:09:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:09:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:09:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:09:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:09:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:09:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:09:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:09:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:09:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:09:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:09:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:09:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:09:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:09:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:09:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:09:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:09:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:09:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:09:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:09:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:09:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:09:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:09:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:09:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:09:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:09:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:09:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:09:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:09:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:09:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:09:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:09:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:09:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:09:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:09:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:09:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:09:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:09:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:09:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:09:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:09:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:09:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:09:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:09:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:09:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:09:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:09:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:09:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:09:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:09:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:09:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:09:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:09:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:09:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:09:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:09:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:09:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:09:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:09:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:09:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:09:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:09:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:09:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:09:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:09:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:09:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:09:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:09:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:09:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:09:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:09:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:09:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:09:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:09:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:09:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:09:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:09:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:09:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:09:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:09:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:09:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:09:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:09:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:09:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:09:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:09:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:09:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:09:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:09:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:09:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:09:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:09:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:09:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:09:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:09:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:09:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:09:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:09:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:09:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:09:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:09:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:09:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:09:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:09:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:09:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:09:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:09:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:09:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:09:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:09:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:09:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:09:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:09:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:09:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:09:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:09:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:09:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:09:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:09:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:09:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:09:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:09:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:09:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:09:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:09:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:09:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:09:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:09:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:09:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:09:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:09:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:09:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:10:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:10:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:10:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:10:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:10:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:10:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:10:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:10:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:10:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:10:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:10:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:10:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:10:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:10:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:10:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:10:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:10:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:10:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:10:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:10:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:10:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:10:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:10:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:10:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:10:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:10:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:10:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:10:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:10:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:10:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:10:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:10:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:10:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:10:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:10:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:10:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:10:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:10:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:10:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:10:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:10:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:10:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:10:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:10:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:10:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:10:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:10:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:10:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:10:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:10:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:10:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:10:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:10:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:10:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:10:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:10:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:10:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:10:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:10:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:10:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:10:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:10:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:10:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:10:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:10:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:10:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:10:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:10:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:10:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:10:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:10:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:10:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:10:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:10:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:10:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:10:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:10:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:10:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:10:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:10:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:10:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:10:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:10:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:10:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:10:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:10:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:10:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:10:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:10:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:10:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:10:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:10:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:10:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:10:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:10:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:10:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:10:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:10:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:10:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:10:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:10:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:10:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:10:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:10:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:10:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:10:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:10:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:10:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:10:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:10:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:10:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:10:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:10:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:10:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:10:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:10:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:10:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:10:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:10:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:10:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:10:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:10:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:10:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:10:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:10:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:10:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:10:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:10:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:10:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:10:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:10:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:10:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:10:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:10:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:10:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:10:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:10:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:10:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:10:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:10:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:10:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:10:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:10:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:10:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:10:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:10:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:10:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:10:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:10:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:10:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:10:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:10:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:10:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:10:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:10:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:10:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:10:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:10:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:10:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:10:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:10:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:10:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:10:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:10:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:10:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:10:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:10:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:10:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:10:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:10:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:10:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:10:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:10:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:10:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:10:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:10:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:10:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:10:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:10:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:10:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:10:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:10:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:10:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:10:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:10:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:10:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:10:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:10:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:10:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:10:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:10:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:10:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:10:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:10:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:10:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:10:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:10:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:10:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:10:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:10:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:10:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:10:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:10:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:10:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:10:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:10:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:10:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:10:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:10:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:10:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:10:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:10:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:10:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:10:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:10:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:10:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:11:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:11:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:11:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:11:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:11:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:11:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:11:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:11:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:11:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:11:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:11:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:11:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:11:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:11:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:11:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:11:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:11:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:11:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:11:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:11:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:11:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:11:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:11:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:11:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:11:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:11:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:11:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:11:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:11:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:11:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:11:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:11:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:11:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:11:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:11:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:11:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:11:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:11:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:11:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:11:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:11:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:11:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:11:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:11:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:11:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:11:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:11:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:11:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:11:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:11:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:11:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:11:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:11:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:11:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:11:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:11:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:11:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:11:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:11:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:11:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:11:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:11:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:11:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:11:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:11:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:11:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:11:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:11:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:11:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:11:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:11:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:11:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:11:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:11:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:11:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:11:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:11:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:11:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:11:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:11:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:11:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:11:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:11:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:11:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:11:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:11:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:11:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:11:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:11:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:11:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:11:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:11:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:11:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:11:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:11:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:11:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:11:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:11:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:11:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:11:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:11:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:11:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:11:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:11:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:11:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:11:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:11:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:11:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:11:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:11:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:11:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:11:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:11:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:11:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:11:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:11:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:11:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:11:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:11:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:11:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:11:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:11:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:11:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:11:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:11:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:11:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:11:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:11:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:11:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:11:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:11:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:11:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:11:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:11:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:11:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:11:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:11:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:11:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:11:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:11:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:11:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:11:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:11:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:11:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:11:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:11:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:11:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:11:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:11:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:11:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:11:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:11:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:11:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:11:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:11:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:11:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:11:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:11:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:11:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:11:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:11:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:11:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:11:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:11:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:11:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:11:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:11:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:11:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:11:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:11:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:11:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:11:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:11:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:11:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:11:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:11:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:11:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:11:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:11:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:11:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:11:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:11:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:11:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:11:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:11:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:11:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:11:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:11:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:11:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:11:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:11:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:11:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:11:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:11:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:11:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:11:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:11:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:11:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:11:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:11:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:11:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:11:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:11:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:11:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:11:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:11:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:11:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:11:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:11:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:11:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:11:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:11:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:11:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:11:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:11:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:11:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:12:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:12:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:12:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:12:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:12:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:12:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:12:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:12:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:12:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:12:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:12:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:12:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:12:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:12:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:12:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:12:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:12:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:12:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:12:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:12:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:12:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:12:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:12:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:12:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:12:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:12:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:12:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:12:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:12:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:12:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:12:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:12:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:12:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:12:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:12:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:12:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:12:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:12:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:12:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:12:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:12:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:12:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:12:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:12:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:12:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:12:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:12:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:12:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:12:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:12:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:12:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:12:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:12:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:12:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:14:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:14:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:14:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:14:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:14:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:14:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:14:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:14:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:14:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:14:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:14:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:14:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:14:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:14:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:14:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:14:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:14:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:14:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:14:22 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 09:14:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:14:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:14:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:14:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:14:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:14:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:14:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:14:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:14:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:14:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:14:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:14:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:14:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:14:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:14:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:14:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:14:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:14:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:14:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:14:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:14:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:14:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:14:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:14:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:14:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:14:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:14:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:14:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:14:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:14:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:14:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:14:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:14:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:14:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:14:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:14:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:14:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:14:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:14:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:14:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:14:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:14:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:14:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:14:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:14:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:14:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:14:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:14:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:14:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:14:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:14:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:14:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:14:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:14:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:14:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:14:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:14:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:14:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:14:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:14:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:14:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:14:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:14:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:14:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:14:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:14:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:14:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:14:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:14:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:14:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:14:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:14:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:14:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:14:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:14:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:14:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:14:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:14:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:14:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:14:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:14:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:14:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:14:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:14:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:14:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:14:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:14:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:14:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:14:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:14:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:14:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:14:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:14:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:14:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:14:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:14:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:14:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:14:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:14:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:14:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:14:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:14:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:14:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:14:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:14:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:14:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:14:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:14:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:14:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:14:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:14:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:14:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:14:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:14:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:14:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:14:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:14:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:14:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:14:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:14:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:14:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:14:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:14:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:14:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:14:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:14:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:15:00 rdma06 systemd[1]: Starting dnf makecache...
Apr 18 09:15:01 rdma06 dnf[1185842]: Updating Subscription Management repositories.
Apr 18 09:15:01 rdma06 dnf[1185842]: Unable to read consumer identity
Apr 18 09:15:01 rdma06 dnf[1185842]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 18 09:15:01 rdma06 dnf[1185842]: Metadata cache refreshed recently.
Apr 18 09:15:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 18 09:15:01 rdma06 systemd[1]: Finished dnf makecache.
Apr 18 09:15:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:15:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:15:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:15:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:15:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:15:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:15:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:15:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:15:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:15:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:15:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:15:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:15:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:15:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:15:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:15:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:15:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:15:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:15:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:15:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:15:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:15:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:15:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:15:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:15:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:15:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:15:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:15:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:15:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:15:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:15:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:15:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:15:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:15:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:15:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:15:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:15:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:15:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:15:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:15:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:15:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:15:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:15:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:15:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:15:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:15:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:15:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:15:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:15:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:15:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:15:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:15:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:15:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:15:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:15:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:15:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:15:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:15:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:15:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:15:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:15:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:15:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:15:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:15:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:15:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:15:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:15:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:15:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:15:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:15:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:15:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:15:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:15:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:15:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:15:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:15:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:15:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:15:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:15:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:15:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:15:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:15:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:15:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:15:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:15:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:15:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:15:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:15:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:15:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:15:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:15:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:15:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:15:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:15:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:15:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:15:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:15:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:15:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:15:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:15:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:15:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:15:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:15:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:15:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:15:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:15:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:15:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:15:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:15:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:15:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:15:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:15:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:15:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:15:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:15:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:15:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:15:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:15:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:15:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:15:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:15:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:15:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:15:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:15:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:15:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:15:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:15:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:15:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:15:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:15:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:15:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:15:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:15:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:15:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:15:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:15:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:15:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:15:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:15:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:15:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:15:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:15:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:15:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:15:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:15:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:15:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:15:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:15:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:15:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:15:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:15:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:15:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:15:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:15:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:15:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:15:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:15:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:15:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:15:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:15:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:15:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:15:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:15:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:15:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:15:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:15:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:15:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:15:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:15:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:15:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:15:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:15:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:15:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:15:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:15:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:15:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:15:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:15:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:15:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:15:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:15:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:15:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:15:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:15:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:15:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:15:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:15:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:15:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:15:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:15:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:15:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:15:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:15:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:15:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:15:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:15:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:15:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:15:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:15:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:15:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:15:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:15:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:15:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:15:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:15:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:15:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:15:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:15:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:15:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:15:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:15:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:15:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:15:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:15:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:15:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:15:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:16:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:16:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:16:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:16:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:16:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:16:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:16:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:16:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:16:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:16:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:16:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:16:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:16:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:16:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:16:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:16:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:16:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:16:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:16:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:16:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:16:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:16:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:16:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:16:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:16:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:16:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:16:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:16:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:16:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:16:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:16:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:16:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:16:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:16:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:16:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:16:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:16:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:16:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:16:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:16:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:16:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:16:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:16:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:16:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:16:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:16:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:16:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:16:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:16:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:16:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:16:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:16:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:16:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:16:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:16:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:16:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:16:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:16:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:16:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:16:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:16:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:16:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:16:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:16:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:16:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:16:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:16:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:16:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:16:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:16:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:16:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:16:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:16:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:16:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:16:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:16:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:16:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:16:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:16:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:16:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:16:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:16:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:16:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:16:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:16:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:16:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:16:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:16:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:16:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:16:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:16:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:16:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:16:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:16:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:16:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:16:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:16:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:16:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:16:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:16:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:16:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:16:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:16:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:16:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:16:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:16:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:16:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:16:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:16:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:16:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:16:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:16:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:16:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:16:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:16:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:16:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:16:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:16:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:16:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:16:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:16:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:16:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:16:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:16:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:16:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:16:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:16:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:16:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:16:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:16:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:16:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:16:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:16:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:16:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:16:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:16:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:16:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:16:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:16:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:16:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:16:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:16:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:16:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:16:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:16:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:16:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:16:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:16:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:16:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:16:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:16:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:16:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:16:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:16:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:16:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:16:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:16:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:16:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:16:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:16:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:16:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:16:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:16:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:16:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:16:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:16:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:16:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:16:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:16:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:16:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:16:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:16:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:16:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:16:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:16:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:16:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:16:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:16:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:16:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:16:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:16:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:16:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:16:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:16:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:16:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:16:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:16:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:16:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:16:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:16:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:16:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:16:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:16:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:16:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:16:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:16:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:16:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:16:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:16:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:16:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:16:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:16:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:16:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:16:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:16:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:16:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:16:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:16:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:16:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:16:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:16:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:16:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:16:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:16:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:16:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:16:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:17:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:17:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:17:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:17:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:17:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:17:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:17:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:17:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:17:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:17:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:17:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:17:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:17:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:17:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:17:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:17:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:17:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:17:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:17:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:17:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:17:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:17:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:17:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:17:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:17:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:17:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:17:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:17:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:17:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:17:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:17:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:17:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:17:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:17:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:17:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:17:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:17:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:17:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:17:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:17:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:17:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:17:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:17:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:17:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:17:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:17:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:17:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:17:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:17:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:17:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:17:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:17:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:17:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:17:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:17:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:17:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:17:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:17:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:17:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:17:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:17:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:17:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:17:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:17:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:17:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:17:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:17:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:17:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:17:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:17:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:17:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:17:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:17:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:17:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:17:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:17:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:17:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:17:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:17:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:17:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:17:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:17:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:17:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:17:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:17:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:17:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:17:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:17:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:17:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:17:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:17:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:17:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:17:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:17:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:17:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:17:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:17:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:17:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:17:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:17:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:17:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:17:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:17:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:17:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:17:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:17:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:17:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:17:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:17:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:17:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:17:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:17:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:17:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:17:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:17:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:17:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:17:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:17:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:17:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:17:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:17:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:17:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:17:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:17:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:17:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:17:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:17:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:17:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:17:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:17:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:17:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:17:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:17:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:17:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:17:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:17:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:17:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:17:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:17:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:17:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:17:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:17:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:17:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:17:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:17:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:17:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:17:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:17:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:17:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:17:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:17:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:17:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:17:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:17:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:17:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:17:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:17:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:17:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:17:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:17:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:17:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:17:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:17:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:17:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:17:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:17:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:17:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:17:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:17:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:17:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:17:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:17:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:17:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:17:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:17:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:17:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:17:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:17:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:17:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:17:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:17:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:17:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:17:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:17:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:17:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:17:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:17:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:17:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:17:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:17:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:17:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:17:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:17:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:17:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:17:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:17:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:17:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:17:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:17:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:17:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:17:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:17:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:17:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:17:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:17:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:17:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:17:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:17:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:17:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:17:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:17:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:17:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:17:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:17:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:17:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:17:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:18:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:18:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:18:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:18:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:18:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:18:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:18:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:18:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:18:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:18:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:18:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:18:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:18:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:18:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:18:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:18:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:18:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:18:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:18:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:18:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:18:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:18:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:18:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:18:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:18:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:18:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:18:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:18:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:18:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:18:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:18:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:18:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:18:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:18:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:18:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:18:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:18:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:18:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:18:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:18:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:18:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:18:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:18:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:18:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:18:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:18:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:18:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:18:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:18:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:18:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:18:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:18:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:18:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:18:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:18:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:18:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:18:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:18:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:18:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:18:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:18:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:18:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:18:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:18:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:18:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:18:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:18:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:18:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:18:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:18:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:18:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:18:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:18:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:18:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:18:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:18:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:18:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:18:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:18:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:18:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:18:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:18:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:18:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:18:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:18:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:18:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:18:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:18:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:18:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:18:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:18:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:18:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:18:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:18:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:18:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:18:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:18:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:18:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:18:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:18:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:18:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:18:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:18:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:18:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:18:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:18:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:18:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:18:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:18:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:18:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:18:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:18:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:18:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:18:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:18:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:18:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:18:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:18:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:18:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:18:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:18:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:18:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:18:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:18:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:18:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:18:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:18:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:18:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:18:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:18:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:18:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:18:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:18:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:18:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:18:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:18:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:18:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:18:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:18:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:18:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:18:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:18:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:18:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:18:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:18:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:18:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:18:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:18:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:18:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:18:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:18:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:18:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:18:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:18:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:18:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:18:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:18:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:18:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:18:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:18:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:18:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:18:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:18:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:18:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:18:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:18:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:18:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:18:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:18:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:18:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:18:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:18:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:18:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:18:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:18:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:18:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:18:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:18:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:18:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:18:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:18:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:18:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:18:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:18:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:18:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:18:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:18:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:18:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:18:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:18:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:18:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:18:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:18:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:18:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:18:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:18:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:18:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:18:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:18:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:18:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:18:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:18:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:18:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:18:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:18:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:18:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:18:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:18:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:18:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:18:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:18:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:18:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:18:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:18:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:18:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:18:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:19:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:19:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:19:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:19:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:19:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:19:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:19:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:19:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:19:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:19:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:19:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:19:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:19:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:19:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:19:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:19:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:19:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:19:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:19:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:19:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:19:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:19:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:19:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:19:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:19:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:19:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:19:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:19:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:19:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:19:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:19:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:19:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:19:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:19:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:19:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:19:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:19:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:19:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:19:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:19:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:19:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:19:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:19:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:19:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:19:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:19:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:19:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:19:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:19:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:19:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:19:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:19:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:19:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:19:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:19:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:19:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:19:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:19:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:19:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:19:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:19:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:19:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:19:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:19:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:19:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:19:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:19:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:19:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:19:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:19:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:19:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:19:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:19:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:19:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:19:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:19:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:19:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:19:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:19:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:19:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:19:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:19:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:19:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:19:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:19:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:19:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:19:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:19:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:19:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:19:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:19:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:19:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:19:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:19:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:19:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:19:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:19:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:19:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:19:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:19:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:19:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:19:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:19:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:19:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:19:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:19:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:19:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:19:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:19:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:19:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:19:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:19:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:19:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:19:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:19:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:19:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:19:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:19:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:19:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:19:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:19:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:19:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:19:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:19:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:19:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:19:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:19:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:19:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:19:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:19:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:19:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:19:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:19:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:19:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:19:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:19:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:19:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:19:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:19:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:19:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:19:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:19:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:19:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:19:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:19:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:19:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:19:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:19:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:19:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:19:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:19:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:19:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:19:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:19:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:19:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:19:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:19:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:19:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:19:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:19:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:19:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:19:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:19:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:19:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:19:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:19:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:19:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:19:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:19:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:19:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:19:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:19:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:19:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:19:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:19:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:19:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:19:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:19:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:19:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:19:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:19:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:19:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:19:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:19:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:19:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:19:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:19:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:19:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:19:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:19:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:19:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:19:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:19:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:19:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:19:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:19:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:19:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:19:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:19:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:19:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:19:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:19:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:19:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:19:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:19:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:19:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:19:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:19:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:19:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:19:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:19:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:19:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:19:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:19:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:19:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:19:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:20:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:20:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:20:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:20:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:20:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:20:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:20:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:20:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:20:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:20:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:20:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:20:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:20:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:20:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:20:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:20:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:20:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:20:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:20:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:20:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:20:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:20:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:20:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:20:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:20:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:20:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:20:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:20:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:20:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:20:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:20:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:20:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:20:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:20:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:20:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:20:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:20:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:20:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:20:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:20:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:20:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:20:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:20:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:20:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:20:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:20:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:20:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:20:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:20:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:20:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:20:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:20:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:20:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:20:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:20:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:20:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:20:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:20:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:20:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:20:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:20:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:20:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:20:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:20:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:20:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:20:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:20:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:20:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:20:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:20:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:20:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:20:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:20:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:20:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:20:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:20:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:20:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:20:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:20:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:20:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:20:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:20:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:20:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:20:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:20:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:20:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:20:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:20:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:20:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:20:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:20:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:20:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:20:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:20:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:20:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:20:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:20:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:20:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:20:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:20:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:20:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:20:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:20:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:20:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:20:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:20:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:20:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:20:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:20:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:20:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:20:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:20:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:20:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:20:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:20:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:20:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:20:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:20:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:20:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:20:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:20:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:20:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:20:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:20:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:20:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:20:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:20:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:20:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:20:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:20:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:20:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:20:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:20:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:20:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:20:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:20:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:20:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:20:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:20:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:20:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:20:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:20:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:20:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:20:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:20:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:20:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:20:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:20:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:20:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:20:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:20:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:20:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:20:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:20:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:20:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:20:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:20:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:20:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:20:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:20:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:20:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:20:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:20:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:20:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:20:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:20:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:20:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:20:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:20:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:20:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:20:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:20:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:20:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:20:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:20:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:20:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:20:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:20:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:20:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:20:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:20:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:20:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:20:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:20:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:20:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:20:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:20:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:20:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:20:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:20:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:20:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:20:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:20:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:20:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:20:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:20:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:20:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:20:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:20:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:20:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:20:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:20:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:20:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:20:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:20:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:20:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:20:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:20:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:20:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:20:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:20:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:20:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:20:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:20:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:20:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:20:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:21:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:21:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:21:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:21:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:21:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:21:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:21:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:21:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:21:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:21:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:21:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:21:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:21:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:21:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:21:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:21:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:21:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:21:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:21:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:21:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:21:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:21:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:21:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:21:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:21:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:21:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:21:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:21:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:21:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:21:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:21:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:21:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:21:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:21:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:21:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:21:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:21:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:21:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:21:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:21:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:21:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:21:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:21:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:21:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:21:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:21:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:21:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:21:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:21:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:21:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:21:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:21:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:21:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:21:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:21:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:21:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:21:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:21:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:21:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:21:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:21:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:21:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:21:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:21:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:21:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:21:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:21:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:21:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:21:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:21:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:21:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:21:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:21:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:21:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:21:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:21:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:21:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:21:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:21:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:21:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:21:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:21:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:21:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:21:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:21:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:21:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:21:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:21:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:21:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:21:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:21:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:21:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:21:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:21:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:21:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:21:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:21:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:21:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:21:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:21:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:21:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:21:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:21:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:21:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:21:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:21:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:21:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:21:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:21:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:21:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:21:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:21:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:21:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:21:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:21:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:21:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:21:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:21:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:21:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:21:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:21:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:21:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:21:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:21:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:21:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:21:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:21:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:21:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:21:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:21:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:21:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:21:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:21:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:21:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:21:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:21:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:21:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:21:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:21:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:21:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:21:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:21:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:21:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:21:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:21:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:21:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:21:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:21:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:21:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:21:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:21:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:21:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:21:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:21:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:21:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:21:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:21:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:21:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:21:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:21:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:21:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:21:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:21:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:21:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:21:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:21:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:21:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:21:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:21:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:21:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:21:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:21:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:21:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:21:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:21:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:21:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:21:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:21:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:21:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:21:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:21:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:21:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:21:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:21:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:21:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:21:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:21:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:21:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:21:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:21:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:21:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:21:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:21:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:21:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:21:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:21:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:21:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:21:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:21:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:21:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:21:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:21:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:21:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:21:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:21:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:21:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:22:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:22:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:22:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:22:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:22:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:22:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:22:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:22:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:22:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:22:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:22:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:22:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:22:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:22:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:22:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:22:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:22:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:22:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:22:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:22:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:22:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:22:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:22:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:22:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:22:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:22:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:22:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:22:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:22:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:22:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:22:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:22:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:22:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:22:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:22:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:22:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:22:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:22:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:22:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:22:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:22:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:22:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:22:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:22:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:22:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:22:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:22:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:22:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:22:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:22:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:22:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:22:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:24:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:24:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:24:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:24:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:24:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:24:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:24:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:24:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:24:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:24:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:24:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:24:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:24:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:24:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:24:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:24:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:24:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:24:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:24:28 rdma06 rsyslogd[1041]: imjournal: 5298 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 09:24:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:24:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:24:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:24:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:24:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:24:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:24:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:24:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:24:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:24:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:24:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:24:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:24:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:24:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:24:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:24:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:24:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:24:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:24:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:24:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:24:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:24:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:24:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:24:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:24:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:24:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:24:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:24:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:24:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:24:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:24:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:24:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:24:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:24:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:24:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:24:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:24:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:24:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:24:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:24:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:24:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:24:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:24:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:24:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:24:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:24:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:24:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:24:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:24:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:24:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:24:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:24:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:24:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:24:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:24:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:24:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:24:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:24:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:24:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:24:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:24:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:24:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:24:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:24:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:24:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:24:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:24:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:24:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:24:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:24:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:24:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:24:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:24:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:24:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:24:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:24:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:24:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:24:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:24:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:24:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:24:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:24:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:24:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:24:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:24:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:24:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:24:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:24:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:24:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:24:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:24:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:24:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:24:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:24:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:24:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:24:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:24:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:24:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:24:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:24:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:24:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:24:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:24:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:24:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:24:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:24:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:24:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:24:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:25:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:25:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:25:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:25:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:25:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:25:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:25:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:25:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:25:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:25:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:25:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:25:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:25:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:25:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:25:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:25:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:25:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:25:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:25:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:25:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:25:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:25:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:25:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:25:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:25:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:25:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:25:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:25:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:25:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:25:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:25:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:25:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:25:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:25:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:25:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:25:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:25:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:25:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:25:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:25:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:25:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:25:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:25:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:25:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:25:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:25:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:25:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:25:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:25:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:25:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:25:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:25:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:25:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:25:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:25:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:25:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:25:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:25:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:25:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:25:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:25:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:25:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:25:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:25:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:25:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:25:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:25:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:25:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:25:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:25:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:25:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:25:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:25:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:25:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:25:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:25:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:25:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:25:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:25:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:25:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:25:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:25:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:25:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:25:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:25:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:25:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:25:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:25:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:25:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:25:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:25:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:25:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:25:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:25:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:25:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:25:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:25:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:25:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:25:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:25:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:25:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:25:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:25:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:25:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:25:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:25:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:25:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:25:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:25:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:25:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:25:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:25:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:25:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:25:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:25:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:25:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:25:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:25:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:25:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:25:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:25:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:25:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:25:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:25:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:25:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:25:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:25:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:25:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:25:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:25:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:25:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:25:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:25:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:25:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:25:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:25:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:25:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:25:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:25:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:25:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:25:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:25:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:25:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:25:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:25:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:25:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:25:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:25:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:25:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:25:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:25:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:25:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:25:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:25:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:25:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:25:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:25:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:25:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:25:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:25:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:25:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:25:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:25:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:25:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:25:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:25:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:25:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:25:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:25:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:25:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:25:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:25:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:25:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:25:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:25:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:25:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:25:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:25:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:25:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:25:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:25:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:25:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:25:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:25:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:25:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:25:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:25:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:25:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:25:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:25:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:25:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:25:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:25:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:25:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:25:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:25:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:25:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:25:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:25:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:25:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:25:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:25:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:25:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:25:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:25:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:25:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:25:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:25:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:25:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:25:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:25:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:25:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:25:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:25:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:25:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:25:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:26:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:26:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:26:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:26:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:26:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:26:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:26:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:26:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:26:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:26:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:26:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:26:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:26:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:26:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:26:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:26:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:26:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:26:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:26:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:26:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:26:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:26:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:26:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:26:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:26:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:26:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:26:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:26:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:26:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:26:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:26:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:26:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:26:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:26:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:26:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:26:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:26:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:26:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:26:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:26:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:26:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:26:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:26:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:26:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:26:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:26:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:26:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:26:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:26:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:26:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:26:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:26:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:26:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:26:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:26:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:26:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:26:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:26:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:26:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:26:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:26:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:26:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:26:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:26:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:26:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:26:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:26:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:26:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:26:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:26:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:26:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:26:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:26:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:26:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:26:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:26:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:26:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:26:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:26:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:26:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:26:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:26:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:26:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:26:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:26:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:26:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:26:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:26:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:26:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:26:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:26:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:26:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:26:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:26:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:26:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:26:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:26:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:26:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:26:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:26:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:26:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:26:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:26:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:26:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:26:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:26:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:26:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:26:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:26:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:26:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:26:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:26:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:26:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:26:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:26:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:26:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:26:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:26:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:26:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:26:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:26:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:26:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:26:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:26:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:26:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:26:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:26:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:26:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:26:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:26:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:26:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:26:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:26:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:26:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:26:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:26:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:26:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:26:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:26:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:26:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:26:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:26:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:26:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:26:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:26:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:26:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:26:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:26:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:26:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:26:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:26:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:26:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:26:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:26:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:26:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:26:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:26:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:26:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:26:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:26:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:26:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:26:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:26:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:26:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:26:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:26:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:26:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:26:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:26:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:26:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:26:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:26:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:26:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:26:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:26:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:26:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:26:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:26:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:26:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:26:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:26:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:26:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:26:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:26:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:26:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:26:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:26:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:26:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:26:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:26:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:26:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:26:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:26:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:26:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:26:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:26:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:26:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:26:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:26:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:26:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:26:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:26:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:26:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:26:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:26:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:26:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:26:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:26:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:26:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:26:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:26:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:26:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:26:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:26:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:26:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:26:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:27:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:27:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:27:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:27:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:27:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:27:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:27:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:27:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:27:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:27:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:27:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:27:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:27:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:27:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:27:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:27:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:27:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:27:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:27:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:27:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:27:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:27:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:27:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:27:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:27:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:27:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:27:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:27:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:27:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:27:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:27:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:27:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:27:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:27:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:27:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:27:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:27:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:27:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:27:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:27:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:27:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:27:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:27:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:27:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:27:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:27:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:27:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:27:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:27:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:27:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:27:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:27:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:27:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:27:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:27:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:27:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:27:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:27:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:27:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:27:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:27:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:27:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:27:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:27:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:27:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:27:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:27:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:27:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:27:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:27:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:27:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:27:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:27:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:27:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:27:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:27:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:27:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:27:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:27:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:27:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:27:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:27:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:27:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:27:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:27:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:27:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:27:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:27:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:27:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:27:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:27:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:27:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:27:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:27:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:27:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:27:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:27:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:27:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:27:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:27:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:27:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:27:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:27:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:27:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:27:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:27:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:27:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:27:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:27:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:27:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:27:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:27:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:27:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:27:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:27:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:27:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:27:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:27:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:27:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:27:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:27:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:27:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:27:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:27:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:27:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:27:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:27:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:27:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:27:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:27:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:27:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:27:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:27:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:27:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:27:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:27:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:27:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:27:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:27:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:27:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:27:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:27:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:27:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:27:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:27:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:27:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:27:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:27:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:27:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:27:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:27:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:27:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:27:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:27:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:27:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:27:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:27:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:27:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:27:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:27:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:27:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:27:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:27:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:27:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:27:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:27:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:27:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:27:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:27:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:27:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:27:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:27:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:27:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:27:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:27:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:27:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:27:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:27:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:27:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:27:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:27:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:27:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:27:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:27:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:27:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:27:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:27:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:27:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:27:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:27:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:27:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:27:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:27:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:27:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:27:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:27:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:27:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:27:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:27:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:27:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:27:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:27:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:27:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:27:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:27:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:27:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:27:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:27:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:27:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:27:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:27:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:27:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:27:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:27:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:27:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:27:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:28:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:28:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:28:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:28:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:28:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:28:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:28:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:28:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:28:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:28:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:28:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:28:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:28:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:28:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:28:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:28:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:28:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:28:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:28:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:28:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:28:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:28:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:28:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:28:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:28:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:28:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:28:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:28:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:28:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:28:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:28:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:28:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:28:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:28:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:28:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:28:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:28:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:28:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:28:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:28:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:28:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:28:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:28:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:28:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:28:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:28:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:28:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:28:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:28:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:28:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:28:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:28:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:28:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:28:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:28:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:28:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:28:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:28:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:28:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:28:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:28:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:28:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:28:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:28:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:28:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:28:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:28:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:28:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:28:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:28:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:28:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:28:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:28:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:28:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:28:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:28:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:28:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:28:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:28:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:28:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:28:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:28:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:28:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:28:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:28:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:28:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:28:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:28:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:28:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:28:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:28:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:28:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:28:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:28:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:28:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:28:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:28:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:28:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:28:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:28:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:28:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:28:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:28:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:28:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:28:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:28:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:28:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:28:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:28:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:28:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:28:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:28:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:28:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:28:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:28:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:28:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:28:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:28:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:28:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:28:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:28:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:28:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:28:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:28:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:28:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:28:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:28:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:28:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:28:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:28:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:28:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:28:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:28:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:28:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:28:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:28:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:28:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:28:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:28:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:28:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:28:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:28:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:28:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:28:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:28:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:28:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:28:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:28:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:28:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:28:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:28:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:28:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:28:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:28:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:28:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:28:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:28:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:28:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:28:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:28:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:28:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:28:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:28:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:28:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:28:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:28:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:28:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:28:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:28:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:28:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:28:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:28:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:28:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:28:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:28:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:28:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:28:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:28:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:28:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:28:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:28:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:28:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:28:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:28:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:28:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:28:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:28:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:28:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:28:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:28:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:28:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:28:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:28:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:28:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:28:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:28:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:28:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:28:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:28:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:28:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:28:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:28:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:28:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:28:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:28:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:28:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:28:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:28:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:28:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:28:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:28:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:28:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:28:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:28:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:28:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:28:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:29:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:29:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:29:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:29:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:29:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:29:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:29:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:29:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:29:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:29:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:29:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:29:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:29:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:29:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:29:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:29:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:29:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:29:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:29:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:29:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:29:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:29:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:29:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:29:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:29:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:29:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:29:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:29:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:29:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:29:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:29:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:29:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:29:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:29:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:29:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:29:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:29:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:29:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:29:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:29:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:29:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:29:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:29:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:29:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:29:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:29:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:29:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:29:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:29:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:29:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:29:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:29:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:29:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:29:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:29:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:29:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:29:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:29:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:29:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:29:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:29:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:29:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:29:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:29:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:29:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:29:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:29:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:29:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:29:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:29:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:29:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:29:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:29:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:29:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:29:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:29:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:29:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:29:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:29:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:29:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:29:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:29:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:29:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:29:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:29:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:29:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:29:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:29:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:29:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:29:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:29:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:29:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:29:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:29:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:29:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:29:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:29:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:29:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:29:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:29:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:29:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:29:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:29:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:29:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:29:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:29:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:29:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:29:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:29:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:29:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:29:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:29:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:29:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:29:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:29:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:29:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:29:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:29:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:29:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:29:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:29:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:29:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:29:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:29:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:29:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:29:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:29:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:29:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:29:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:29:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:29:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:29:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:29:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:29:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:29:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:29:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:29:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:29:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:29:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:29:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:29:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:29:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:29:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:29:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:29:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:29:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:29:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:29:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:29:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:29:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:29:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:29:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:29:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:29:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:29:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:29:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:29:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:29:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:29:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:29:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:29:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:29:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:29:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:29:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:29:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:29:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:29:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:29:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:29:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:29:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:29:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:29:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:29:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:29:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:29:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:29:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:29:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:29:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:29:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:29:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:29:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:29:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:29:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:29:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:29:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:29:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:29:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:29:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:29:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:29:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:29:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:29:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:29:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:29:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:29:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:29:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:29:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:29:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:29:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:29:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:29:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:29:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:29:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:29:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:29:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:29:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:29:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:29:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:29:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:29:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:29:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:29:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:29:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:29:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:29:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:29:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:30:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:30:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:30:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:30:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:30:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:30:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:30:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:30:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:30:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:30:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:30:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:30:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:30:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:30:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:30:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:30:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:30:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:30:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:30:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:30:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:30:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:30:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:30:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:30:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:30:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:30:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:30:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:30:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:30:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:30:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:30:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:30:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:30:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:30:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:30:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:30:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:30:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:30:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:30:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:30:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:30:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:30:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:30:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:30:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:30:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:30:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:30:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:30:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:30:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:30:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:30:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:30:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:30:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:30:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:30:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:30:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:30:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:30:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:30:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:30:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:30:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:30:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:30:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:30:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:30:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:30:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:30:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:30:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:30:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:30:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:30:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:30:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:30:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:30:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:30:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:30:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:30:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:30:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:30:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:30:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:30:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:30:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:30:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:30:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:30:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:30:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:30:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:30:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:30:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:30:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:30:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:30:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:30:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:30:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:30:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:30:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:30:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:30:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:30:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:30:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:30:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:30:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:30:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:30:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:30:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:30:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:30:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:30:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:30:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:30:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:30:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:30:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:30:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:30:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:30:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:30:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:30:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:30:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:30:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:30:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:30:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:30:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:30:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:31:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:31:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:31:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:31:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:31:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:31:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:31:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:31:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:31:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:31:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:31:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:31:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:31:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:31:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:31:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:31:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:31:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:31:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:31:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:31:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:31:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:31:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:31:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:31:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:31:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:31:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:31:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:31:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:31:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:31:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:31:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:31:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:31:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:31:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:31:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:31:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:31:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:31:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:31:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:31:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:31:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:31:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:31:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:31:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:31:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:31:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:31:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:31:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:31:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:31:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:31:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:31:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:31:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:31:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:31:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:31:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:31:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:31:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:31:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:31:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:31:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:31:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:31:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:31:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:31:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:31:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:31:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:31:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:31:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:31:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:31:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:31:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:31:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:31:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:31:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:31:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:31:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:31:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:31:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:31:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:31:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:31:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:31:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:31:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:31:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:31:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:31:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:31:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:31:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:31:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:31:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:31:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:31:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:31:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:31:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:31:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:31:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:31:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:31:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:31:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:31:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:31:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:31:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:31:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:31:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:31:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:31:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:31:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:31:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:31:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:31:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:31:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:31:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:31:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:31:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:31:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:31:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:31:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:31:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:31:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:31:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:31:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:31:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:31:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:31:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:31:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:31:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:31:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:31:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:31:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:31:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:31:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:31:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:31:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:31:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:31:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:31:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:31:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:31:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:31:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:31:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:31:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:31:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:31:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:31:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:31:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:31:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:31:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:31:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:31:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:31:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:31:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:31:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:31:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:31:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:31:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:31:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:31:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:31:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:31:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:31:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:31:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:31:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:31:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:31:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:31:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:31:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:31:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:31:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:31:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:31:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:31:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:31:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:31:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:31:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:31:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:31:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:31:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:31:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:31:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:31:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:31:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:31:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:31:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:31:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:31:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:31:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:31:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:31:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:31:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:31:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:31:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:31:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:31:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:31:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:31:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:31:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:31:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:31:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:31:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:31:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:31:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:31:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:31:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:31:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:31:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:31:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:31:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:31:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:31:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:31:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:31:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:31:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:31:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:31:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:31:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:32:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:32:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:32:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:32:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:32:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:32:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:32:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:32:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:32:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:32:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:32:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:32:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:32:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:32:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:32:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:32:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:32:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:32:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:32:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:32:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:32:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:32:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:32:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:32:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:32:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:32:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:32:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:32:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:32:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:32:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:32:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:32:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:32:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:32:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:32:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:32:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:32:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:32:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:32:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:32:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:32:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:32:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:32:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:32:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:32:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:32:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:32:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:32:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:32:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:32:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:32:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:32:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:32:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:32:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:32:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:32:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:32:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:32:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:32:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:32:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:32:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:32:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:32:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:32:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:32:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:32:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:32:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:32:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:32:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:32:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:32:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:32:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:32:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:32:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:32:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:32:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:32:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:32:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:32:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:32:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:32:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:32:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:32:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:32:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:32:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:32:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:32:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:32:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:32:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:32:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:34:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:34:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:34:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:34:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:34:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:34:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:34:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:34:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:34:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:34:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:34:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:34:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:34:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:34:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:34:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:34:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:34:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:34:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:34:33 rdma06 rsyslogd[1041]: imjournal: 5295 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 09:34:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:34:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:34:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:34:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:34:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:34:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:34:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:34:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:34:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:34:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:34:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:34:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:34:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:34:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:34:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:34:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:34:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:34:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:34:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:34:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:34:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:34:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:34:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:34:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:34:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:34:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:34:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:34:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:34:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:34:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:34:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:34:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:34:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:34:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:34:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:34:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:34:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:34:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:34:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:34:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:34:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:34:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:34:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:34:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:34:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:34:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:34:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:34:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:34:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:34:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:34:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:34:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:34:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:34:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:34:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:34:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:34:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:34:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:34:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:34:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:34:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:34:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:34:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:34:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:34:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:34:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:34:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:34:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:34:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:34:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:34:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:34:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:34:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:34:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:34:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:34:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:34:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:34:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:34:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:34:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:34:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:34:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:34:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:34:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:34:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:34:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:34:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:34:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:34:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:34:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:35:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:35:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:35:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:35:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:35:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:35:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:35:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:35:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:35:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:35:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:35:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:35:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:35:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:35:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:35:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:35:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:35:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:35:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:35:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:35:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:35:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:35:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:35:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:35:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:35:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:35:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:35:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:35:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:35:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:35:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:35:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:35:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:35:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:35:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:35:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:35:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:35:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:35:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:35:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:35:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:35:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:35:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:35:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:35:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:35:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:35:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:35:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:35:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:35:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:35:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:35:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:35:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:35:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:35:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:35:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:35:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:35:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:35:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:35:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:35:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:35:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:35:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:35:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:35:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:35:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:35:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:35:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:35:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:35:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:35:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:35:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:35:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:35:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:35:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:35:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:35:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:35:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:35:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:35:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:35:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:35:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:35:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:35:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:35:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:35:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:35:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:35:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:35:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:35:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:35:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:35:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:35:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:35:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:35:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:35:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:35:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:35:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:35:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:35:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:35:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:35:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:35:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:35:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:35:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:35:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:35:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:35:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:35:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:35:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:35:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:35:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:35:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:35:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:35:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:35:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:35:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:35:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:35:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:35:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:35:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:35:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:35:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:35:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:35:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:35:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:35:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:35:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:35:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:35:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:35:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:35:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:35:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:35:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:35:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:35:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:35:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:35:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:35:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:35:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:35:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:35:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:35:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:35:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:35:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:35:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:35:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:35:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:35:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:35:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:35:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:35:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:35:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:35:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:35:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:35:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:35:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:35:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:35:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:35:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:35:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:35:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:35:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:35:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:35:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:35:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:35:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:35:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:35:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:35:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:35:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:35:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:35:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:35:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:35:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:35:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:35:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:35:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:35:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:35:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:35:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:35:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:35:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:35:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:35:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:35:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:35:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:35:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:35:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:35:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:35:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:35:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:35:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:35:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:35:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:35:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:35:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:35:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:35:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:35:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:35:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:35:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:35:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:35:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:35:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:35:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:35:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:35:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:35:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:35:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:35:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:35:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:35:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:35:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:35:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:35:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:35:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:36:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:36:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:36:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:36:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:36:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:36:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:36:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:36:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:36:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:36:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:36:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:36:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:36:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:36:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:36:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:36:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:36:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:36:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:36:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:36:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:36:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:36:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:36:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:36:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:36:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:36:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:36:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:36:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:36:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:36:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:36:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:36:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:36:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:36:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:36:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:36:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:36:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:36:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:36:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:36:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:36:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:36:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:36:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:36:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:36:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:36:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:36:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:36:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:36:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:36:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:36:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:36:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:36:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:36:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:36:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:36:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:36:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:36:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:36:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:36:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:36:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:36:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:36:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:36:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:36:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:36:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:36:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:36:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:36:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:36:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:36:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:36:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:36:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:36:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:36:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:36:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:36:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:36:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:36:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:36:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:36:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:36:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:36:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:36:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:36:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:36:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:36:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:36:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:36:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:36:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:36:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:36:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:36:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:36:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:36:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:36:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:36:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:36:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:36:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:36:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:36:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:36:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:36:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:36:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:36:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:36:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:36:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:36:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:36:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:36:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:36:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:36:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:36:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:36:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:36:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:36:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:36:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:36:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:36:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:36:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:36:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:36:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:36:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:36:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:36:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:36:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:36:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:36:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:36:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:36:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:36:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:36:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:36:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:36:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:36:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:36:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:36:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:36:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:36:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:36:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:36:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:36:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:36:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:36:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:36:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:36:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:36:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:36:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:36:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:36:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:36:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:36:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:36:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:36:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:36:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:36:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:36:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:36:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:36:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:36:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:36:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:36:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:36:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:36:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:36:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:36:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:36:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:36:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:36:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:36:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:36:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:36:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:36:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:36:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:36:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:36:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:36:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:36:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:36:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:36:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:36:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:36:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:36:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:36:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:36:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:36:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:36:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:36:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:36:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:36:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:36:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:36:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:36:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:36:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:36:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:36:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:36:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:36:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:36:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:36:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:36:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:36:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:36:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:36:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:36:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:36:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:36:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:36:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:36:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:36:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:36:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:36:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:36:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:36:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:36:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:36:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:37:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:37:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:37:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:37:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:37:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:37:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:37:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:37:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:37:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:37:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:37:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:37:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:37:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:37:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:37:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:37:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:37:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:37:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:37:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:37:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:37:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:37:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:37:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:37:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:37:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:37:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:37:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:37:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:37:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:37:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:37:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:37:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:37:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:37:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:37:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:37:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:37:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:37:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:37:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:37:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:37:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:37:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:37:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:37:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:37:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:37:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:37:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:37:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:37:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:37:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:37:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:37:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:37:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:37:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:37:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:37:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:37:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:37:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:37:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:37:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:37:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:37:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:37:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:37:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:37:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:37:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:37:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:37:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:37:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:37:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:37:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:37:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:37:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:37:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:37:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:37:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:37:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:37:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:37:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:37:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:37:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:37:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:37:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:37:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:37:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:37:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:37:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:37:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:37:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:37:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:37:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:37:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:37:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:37:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:37:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:37:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:37:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:37:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:37:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:37:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:37:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:37:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:37:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:37:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:37:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:37:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:37:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:37:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:37:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:37:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:37:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:37:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:37:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:37:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:37:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:37:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:37:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:37:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:37:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:37:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:37:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:37:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:37:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:37:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:37:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:37:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:37:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:37:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:37:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:37:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:37:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:37:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:37:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:37:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:37:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:37:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:37:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:37:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:37:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:37:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:37:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:37:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:37:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:37:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:37:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:37:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:37:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:37:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:37:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:37:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:37:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:37:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:37:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:37:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:37:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:37:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:37:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:37:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:37:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:37:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:37:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:37:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:37:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:37:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:37:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:37:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:37:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:37:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:37:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:37:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:37:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:37:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:37:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:37:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:37:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:37:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:37:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:37:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:37:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:37:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:37:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:37:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:37:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:37:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:37:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:37:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:37:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:37:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:37:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:37:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:37:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:37:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:37:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:37:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:37:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:37:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:37:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:37:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:37:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:37:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:37:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:37:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:37:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:37:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:37:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:37:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:37:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:37:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:37:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:37:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:37:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:37:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:37:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:37:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:37:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:37:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:38:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:38:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:38:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:38:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:38:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:38:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:38:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:38:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:38:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:38:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:38:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:38:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:38:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:38:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:38:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:38:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:38:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:38:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:38:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:38:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:38:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:38:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:38:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:38:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:38:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:38:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:38:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:38:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:38:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:38:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:38:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:38:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:38:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:38:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:38:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:38:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:38:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:38:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:38:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:38:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:38:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:38:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:38:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:38:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:38:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:38:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:38:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:38:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:38:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:38:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:38:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:38:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:38:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:38:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:38:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:38:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:38:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:38:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:38:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:38:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:38:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:38:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:38:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:38:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:38:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:38:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:38:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:38:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:38:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:38:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:38:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:38:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:38:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:38:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:38:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:38:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:38:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:38:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:38:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:38:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:38:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:38:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:38:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:38:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:38:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:38:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:38:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:38:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:38:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:38:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:38:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:38:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:38:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:38:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:38:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:38:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:38:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:38:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:38:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:38:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:38:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:38:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:38:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:38:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:38:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:38:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:38:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:38:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:38:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:38:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:38:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:38:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:38:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:38:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:38:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:38:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:38:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:38:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:38:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:38:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:38:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:38:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:38:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:38:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:38:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:38:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:38:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:38:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:38:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:38:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:38:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:38:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:38:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:38:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:38:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:38:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:38:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:38:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:38:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:38:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:38:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:38:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:38:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:38:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:38:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:38:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:38:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:38:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:38:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:38:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:38:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:38:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:38:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:38:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:38:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:38:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:38:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:38:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:38:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:38:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:38:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:38:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:38:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:38:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:38:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:38:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:38:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:38:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:38:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:38:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:38:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:38:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:38:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:38:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:38:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:38:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:38:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:38:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:38:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:38:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:38:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:38:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:38:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:38:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:38:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:38:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:38:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:38:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:38:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:38:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:38:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:38:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:38:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:38:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:38:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:38:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:38:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:38:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:38:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:38:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:38:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:38:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:38:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:38:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:38:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:38:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:38:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:38:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:38:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:38:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:38:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:38:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:38:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:38:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:38:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:38:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:39:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:39:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:39:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:39:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:39:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:39:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:39:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:39:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:39:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:39:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:39:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:39:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:39:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:39:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:39:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:39:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:39:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:39:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:39:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:39:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:39:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:39:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:39:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:39:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:39:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:39:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:39:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:39:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:39:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:39:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:39:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:39:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:39:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:39:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:39:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:39:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:39:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:39:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:39:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:39:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:39:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:39:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:39:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:39:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:39:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:39:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:39:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:39:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:39:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:39:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:39:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:39:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:39:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:39:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:39:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:39:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:39:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:39:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:39:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:39:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:39:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:39:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:39:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:39:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:39:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:39:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:39:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:39:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:39:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:39:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:39:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:39:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:39:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:39:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:39:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:39:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:39:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:39:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:39:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:39:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:39:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:39:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:39:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:39:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:39:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:39:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:39:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:39:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:39:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:39:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:39:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:39:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:39:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:39:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:39:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:39:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:39:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:39:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:39:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:39:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:39:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:39:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:39:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:39:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:39:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:39:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:39:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:39:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:39:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:39:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:39:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:39:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:39:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:39:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:39:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:39:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:39:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:39:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:39:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:39:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:39:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:39:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:39:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:39:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:39:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:39:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:39:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:39:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:39:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:39:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:39:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:39:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:39:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:39:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:39:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:39:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:39:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:39:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:39:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:39:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:39:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:39:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:39:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:39:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:39:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:39:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:39:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:39:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:39:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:39:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:39:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:39:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:39:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:39:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:39:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:39:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:39:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:39:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:39:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:39:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:39:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:39:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:39:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:39:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:39:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:39:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:39:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:39:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:39:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:39:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:39:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:39:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:39:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:39:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:39:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:39:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:39:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:39:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:39:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:39:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:39:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:39:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:39:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:39:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:39:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:39:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:39:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:39:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:39:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:39:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:39:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:39:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:39:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:39:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:39:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:39:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:39:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:39:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:39:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:39:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:39:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:39:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:39:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:39:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:39:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:39:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:39:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:39:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:39:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:39:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:39:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:39:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:39:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:39:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:39:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:39:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:40:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:40:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:40:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:40:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:40:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:40:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:40:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:40:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:40:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:40:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:40:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:40:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:40:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:40:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:40:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:40:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:40:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:40:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:40:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:40:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:40:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:40:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:40:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:40:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:40:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:40:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:40:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:40:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:40:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:40:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:40:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:40:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:40:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:40:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:40:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:40:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:40:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:40:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:40:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:40:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:40:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:40:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:40:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:40:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:40:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:40:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:40:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:40:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:40:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:40:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:40:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:40:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:40:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:40:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:40:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:40:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:40:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:40:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:40:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:40:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:40:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:40:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:40:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:40:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:40:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:40:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:40:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:40:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:40:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:40:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:40:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:40:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:40:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:40:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:40:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:40:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:40:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:40:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:40:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:40:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:40:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:40:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:40:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:40:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:40:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:40:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:40:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:40:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:40:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:40:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:40:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:40:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:40:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:40:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:40:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:40:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:40:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:40:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:40:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:40:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:40:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:40:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:40:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:40:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:40:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:40:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:40:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:40:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:40:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:40:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:40:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:40:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:40:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:40:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:40:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:40:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:40:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:40:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:40:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:40:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:40:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:40:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:40:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:40:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:40:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:40:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:40:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:40:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:40:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:40:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:40:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:40:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:40:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:40:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:40:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:40:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:40:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:40:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:40:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:40:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:40:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:40:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:40:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:40:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:40:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:40:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:40:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:40:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:40:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:40:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:40:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:40:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:40:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:40:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:40:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:40:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:40:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:40:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:40:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:40:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:40:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:40:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:40:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:40:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:40:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:40:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:40:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:40:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:40:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:40:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:40:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:40:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:40:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:40:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:40:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:40:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:40:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:40:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:40:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:40:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:40:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:40:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:40:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:40:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:40:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:40:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:40:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:40:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:40:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:40:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:40:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:40:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:40:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:40:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:40:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:40:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:40:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:40:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:40:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:40:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:40:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:40:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:40:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:40:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:40:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:40:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:40:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:40:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:40:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:40:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:40:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:40:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:40:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:40:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:40:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:40:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:41:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:41:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:41:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:41:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:41:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:41:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:41:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:41:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:41:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:41:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:41:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:41:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:41:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:41:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:41:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:41:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:41:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:41:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:41:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:41:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:41:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:41:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:41:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:41:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:41:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:41:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:41:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:41:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:41:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:41:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:41:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:41:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:41:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:41:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:41:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:41:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:41:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:41:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:41:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:41:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:41:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:41:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:41:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:41:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:41:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:41:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:41:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:41:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:41:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:41:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:41:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:41:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:41:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:41:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:41:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:41:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:41:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:41:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:41:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:41:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:41:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:41:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:41:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:41:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:41:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:41:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:41:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:41:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:41:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:41:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:41:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:41:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:41:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:41:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:41:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:41:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:41:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:41:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:41:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:41:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:41:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:41:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:41:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:41:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:41:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:41:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:41:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:41:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:41:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:41:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:41:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:41:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:41:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:41:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:41:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:41:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:41:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:41:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:41:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:41:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:41:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:41:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:41:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:41:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:41:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:41:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:41:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:41:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:41:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:41:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:41:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:41:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:41:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:41:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:41:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:41:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:41:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:41:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:41:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:41:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:41:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:41:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:41:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:41:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:41:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:41:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:41:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:41:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:41:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:41:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:41:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:41:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:41:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:41:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:41:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:41:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:41:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:41:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:41:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:41:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:41:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:41:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:41:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:41:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:41:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:41:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:41:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:41:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:41:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:41:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:41:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:41:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:41:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:41:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:41:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:41:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:41:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:41:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:41:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:41:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:41:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:41:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:41:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:41:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:41:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:41:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:41:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:41:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:41:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:41:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:41:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:41:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:41:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:41:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:41:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:41:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:41:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:41:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:41:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:41:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:41:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:41:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:41:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:41:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:41:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:41:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:41:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:41:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:41:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:41:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:41:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:41:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:41:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:41:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:41:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:41:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:41:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:41:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:41:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:41:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:41:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:41:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:41:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:41:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:41:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:41:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:41:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:41:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:41:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:41:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:41:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:41:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:41:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:41:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:41:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:41:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:42:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:42:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:42:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:42:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:42:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:42:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:42:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:42:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:42:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:42:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:42:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:42:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:42:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:42:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:42:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:42:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:42:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:42:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:42:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:42:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:42:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:42:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:42:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:42:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:42:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:42:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:42:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:42:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:42:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:42:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:42:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:42:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:42:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:42:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:42:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:42:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:42:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:42:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:42:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:42:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:42:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:42:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:42:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:42:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:42:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:42:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:42:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:42:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:42:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:42:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:42:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:42:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:42:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:42:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:42:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:42:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:42:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:42:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:42:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:42:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:42:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:42:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:42:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:42:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:42:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:42:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:42:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:42:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:42:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:42:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:42:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:42:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:42:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:42:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:42:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:42:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:42:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:42:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:42:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:42:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:42:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:42:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:42:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:42:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:42:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:42:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:42:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:42:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:42:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:42:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:42:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:42:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:42:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:42:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:42:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:42:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:42:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:42:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:42:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:42:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:42:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:42:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:42:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:42:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:42:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:42:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:42:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:42:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:44:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:44:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:44:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:44:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:44:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:44:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:44:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:44:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:44:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:44:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:44:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:44:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:44:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:44:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:44:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:44:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:44:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:44:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:44:38 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 09:44:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:44:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:44:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:44:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:44:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:44:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:44:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:44:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:44:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:44:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:44:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:44:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:44:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:44:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:44:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:44:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:44:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:44:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:44:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:44:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:44:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:44:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:44:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:44:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:44:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:44:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:44:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:44:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:44:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:44:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:44:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:44:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:44:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:44:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:44:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:44:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:44:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:44:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:44:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:44:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:44:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:44:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:44:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:44:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:44:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:44:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:44:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:44:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:44:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:44:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:44:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:44:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:44:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:44:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:44:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:44:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:44:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:44:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:44:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:44:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:44:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:44:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:44:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:44:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:44:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:44:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:44:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:44:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:44:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:44:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:44:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:44:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:45:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:45:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:45:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:45:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:45:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:45:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:45:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:45:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:45:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:45:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:45:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:45:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:45:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:45:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:45:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:45:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:45:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:45:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:45:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:45:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:45:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:45:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:45:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:45:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:45:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:45:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:45:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:45:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:45:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:45:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:45:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:45:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:45:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:45:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:45:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:45:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:45:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:45:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:45:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:45:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:45:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:45:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:45:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:45:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:45:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:45:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:45:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:45:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:45:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:45:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:45:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:45:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:45:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:45:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:45:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:45:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:45:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:45:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:45:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:45:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:45:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:45:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:45:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:45:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:45:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:45:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:45:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:45:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:45:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:45:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:45:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:45:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:45:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:45:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:45:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:45:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:45:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:45:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:45:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:45:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:45:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:45:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:45:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:45:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:45:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:45:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:45:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:45:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:45:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:45:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:45:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:45:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:45:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:45:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:45:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:45:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:45:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:45:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:45:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:45:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:45:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:45:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:45:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:45:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:45:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:45:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:45:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:45:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:45:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:45:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:45:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:45:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:45:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:45:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:45:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:45:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:45:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:45:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:45:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:45:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:45:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:45:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:45:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:45:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:45:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:45:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:45:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:45:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:45:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:45:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:45:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:45:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:45:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:45:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:45:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:45:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:45:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:45:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:45:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:45:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:45:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:45:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:45:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:45:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:45:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:45:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:45:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:45:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:45:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:45:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:45:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:45:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:45:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:45:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:45:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:45:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:45:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:45:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:45:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:45:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:45:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:45:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:45:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:45:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:45:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:45:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:45:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:45:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:45:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:45:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:45:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:45:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:45:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:45:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:45:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:45:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:45:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:45:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:45:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:45:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:45:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:45:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:45:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:45:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:45:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:45:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:45:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:45:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:45:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:45:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:45:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:45:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:45:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:45:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:45:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:45:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:45:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:45:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:45:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:45:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:45:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:45:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:45:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:45:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:45:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:45:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:45:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:45:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:45:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:45:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:45:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:45:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:45:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:45:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:45:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:45:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:46:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:46:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:46:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:46:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:46:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:46:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:46:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:46:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:46:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:46:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:46:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:46:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:46:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:46:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:46:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:46:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:46:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:46:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:46:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:46:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:46:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:46:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:46:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:46:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:46:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:46:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:46:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:46:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:46:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:46:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:46:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:46:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:46:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:46:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:46:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:46:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:46:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:46:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:46:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:46:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:46:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:46:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:46:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:46:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:46:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:46:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:46:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:46:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:46:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:46:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:46:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:46:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:46:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:46:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:46:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:46:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:46:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:46:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:46:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:46:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:46:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:46:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:46:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:46:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:46:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:46:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:46:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:46:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:46:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:46:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:46:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:46:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:46:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:46:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:46:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:46:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:46:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:46:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:46:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:46:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:46:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:46:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:46:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:46:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:46:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:46:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:46:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:46:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:46:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:46:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:46:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:46:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:46:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:46:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:46:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:46:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:46:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:46:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:46:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:46:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:46:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:46:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:46:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:46:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:46:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:46:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:46:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:46:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:46:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:46:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:46:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:46:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:46:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:46:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:46:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:46:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:46:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:46:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:46:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:46:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:46:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:46:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:46:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:46:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:46:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:46:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:46:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:46:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:46:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:46:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:46:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:46:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:46:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:46:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:46:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:46:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:46:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:46:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:46:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:46:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:46:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:46:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:46:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:46:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:46:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:46:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:46:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:46:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:46:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:46:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:46:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:46:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:46:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:46:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:46:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:46:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:46:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:46:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:46:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:46:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:46:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:46:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:46:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:46:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:46:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:46:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:46:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:46:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:46:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:46:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:46:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:46:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:46:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:46:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:46:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:46:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:46:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:46:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:46:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:46:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:46:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:46:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:46:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:46:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:46:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:46:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:46:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:46:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:46:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:46:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:46:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:46:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:46:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:46:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:46:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:46:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:46:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:46:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:46:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:46:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:46:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:46:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:46:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:46:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:46:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:46:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:46:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:46:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:46:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:46:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:46:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:46:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:46:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:46:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:46:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:46:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:47:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:47:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:47:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:47:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:47:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:47:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:47:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:47:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:47:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:47:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:47:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:47:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:47:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:47:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:47:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:47:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:47:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:47:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:47:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:47:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:47:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:47:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:47:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:47:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:47:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:47:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:47:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:47:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:47:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:47:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:47:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:47:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:47:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:47:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:47:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:47:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:47:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:47:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:47:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:47:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:47:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:47:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:47:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:47:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:47:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:47:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:47:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:47:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:47:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:47:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:47:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:47:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:47:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:47:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:47:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:47:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:47:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:47:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:47:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:47:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:47:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:47:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:47:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:47:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:47:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:47:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:47:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:47:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:47:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:47:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:47:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:47:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:47:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:47:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:47:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:47:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:47:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:47:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:47:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:47:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:47:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:47:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:47:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:47:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:47:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:47:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:47:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:47:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:47:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:47:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:47:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:47:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:47:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:47:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:47:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:47:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:47:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:47:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:47:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:47:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:47:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:47:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:47:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:47:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:47:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:47:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:47:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:47:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:47:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:47:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:47:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:47:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:47:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:47:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:47:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:47:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:47:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:47:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:47:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:47:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:47:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:47:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:47:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:47:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:47:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:47:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:47:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:47:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:47:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:47:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:47:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:47:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:47:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:47:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:47:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:47:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:47:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:47:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:47:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:47:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:47:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:47:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:47:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:47:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:47:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:47:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:47:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:47:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:47:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:47:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:47:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:47:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:47:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:47:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:47:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:47:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:47:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:47:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:47:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:47:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:47:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:47:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:47:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:47:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:47:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:47:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:47:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:47:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:47:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:47:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:47:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:47:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:47:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:47:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:47:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:47:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:47:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:47:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:47:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:47:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:47:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:47:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:47:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:47:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:47:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:47:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:47:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:47:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:47:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:47:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:47:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:47:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:47:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:47:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:47:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:47:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:47:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:47:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:47:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:47:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:47:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:47:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:47:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:47:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:47:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:47:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:47:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:47:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:47:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:47:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:47:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:47:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:47:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:47:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:47:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:47:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:48:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:48:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:48:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:48:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:48:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:48:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:48:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:48:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:48:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:48:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:48:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:48:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:48:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:48:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:48:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:48:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:48:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:48:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:48:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:48:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:48:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:48:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:48:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:48:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:48:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:48:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:48:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:48:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:48:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:48:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:48:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:48:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:48:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:48:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:48:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:48:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:48:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:48:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:48:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:48:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:48:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:48:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:48:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:48:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:48:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:48:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:48:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:48:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:48:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:48:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:48:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:48:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:48:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:48:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:48:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:48:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:48:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:48:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:48:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:48:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:48:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:48:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:48:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:48:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:48:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:48:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:48:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:48:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:48:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:48:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:48:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:48:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:48:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:48:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:48:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:48:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:48:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:48:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:48:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:48:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:48:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:48:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:48:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:48:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:48:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:48:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:48:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:48:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:48:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:48:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:48:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:48:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:48:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:48:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:48:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:48:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:48:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:48:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:48:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:48:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:48:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:48:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:48:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:48:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:48:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:48:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:48:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:48:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:48:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:48:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:48:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:48:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:48:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:48:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:48:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:48:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:48:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:48:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:48:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:48:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:48:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:48:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:48:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:48:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:48:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:48:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:48:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:48:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:48:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:48:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:48:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:48:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:48:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:48:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:48:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:48:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:48:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:48:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:48:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:48:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:48:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:48:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:48:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:48:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:48:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:48:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:48:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:48:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:48:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:48:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:48:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:48:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:48:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:48:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:48:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:48:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:48:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:48:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:48:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:48:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:48:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:48:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:48:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:48:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:48:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:48:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:48:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:48:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:48:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:48:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:48:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:48:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:48:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:48:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:48:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:48:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:48:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:48:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:48:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:48:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:48:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:48:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:48:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:48:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:48:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:48:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:48:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:48:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:48:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:48:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:48:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:48:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:48:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:48:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:48:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:48:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:48:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:48:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:48:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:48:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:48:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:48:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:48:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:48:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:48:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:48:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:48:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:48:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:48:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:48:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:48:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:48:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:48:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:48:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:48:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:48:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:49:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:49:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:49:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:49:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:49:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:49:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:49:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:49:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:49:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:49:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:49:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:49:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:49:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:49:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:49:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:49:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:49:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:49:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:49:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:49:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:49:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:49:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:49:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:49:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:49:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:49:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:49:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:49:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:49:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:49:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:49:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:49:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:49:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:49:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:49:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:49:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:49:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:49:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:49:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:49:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:49:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:49:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:49:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:49:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:49:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:49:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:49:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:49:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:49:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:49:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:49:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:49:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:49:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:49:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:49:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:49:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:49:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:49:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:49:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:49:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:49:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:49:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:49:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:49:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:49:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:49:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:49:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:49:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:49:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:49:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:49:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:49:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:49:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:49:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:49:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:49:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:49:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:49:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:49:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:49:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:49:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:49:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:49:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:49:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:49:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:49:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:49:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:49:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:49:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:49:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:49:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:49:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:49:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:49:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:49:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:49:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:49:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:49:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:49:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:49:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:49:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:49:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:49:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:49:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:49:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:49:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:49:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:49:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:49:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:49:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:49:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:49:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:49:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:49:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:49:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:49:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:49:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:49:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:49:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:49:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:49:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:49:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:49:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:49:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:49:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:49:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:49:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:49:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:49:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:49:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:49:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:49:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:49:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:49:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:49:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:49:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:49:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:49:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:49:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:49:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:49:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:49:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:49:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:49:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:49:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:49:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:49:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:49:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:49:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:49:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:49:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:49:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:49:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:49:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:49:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:49:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:49:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:49:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:49:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:49:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:49:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:49:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:49:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:49:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:49:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:49:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:49:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:49:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:49:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:49:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:49:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:49:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:49:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:49:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:49:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:49:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:49:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:49:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:49:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:49:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:49:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:49:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:49:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:49:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:49:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:49:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:49:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:49:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:49:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:49:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:49:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:49:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:49:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:49:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:49:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:49:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:49:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:49:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:49:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:49:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:49:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:49:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:49:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:49:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:49:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:49:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:49:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:49:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:49:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:49:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:49:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:49:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:49:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:49:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:49:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:49:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:50:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:50:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:50:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:50:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:50:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:50:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:50:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:50:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:50:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:50:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:50:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:50:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:50:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:50:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:50:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:50:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:50:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:50:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:50:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:50:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:50:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:50:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:50:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:50:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:50:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:50:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:50:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:50:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:50:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:50:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:50:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:50:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:50:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:50:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:50:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:50:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:50:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:50:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:50:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:50:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:50:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:50:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:50:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:50:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:50:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:50:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:50:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:50:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:50:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:50:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:50:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:50:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:50:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:50:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:50:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:50:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:50:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:50:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:50:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:50:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:50:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:50:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:50:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:50:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:50:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:50:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:50:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:50:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:50:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:50:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:50:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:50:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:50:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:50:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:50:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:50:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:50:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:50:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:50:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:50:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:50:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:50:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:50:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:50:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:50:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:50:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:50:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:50:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:50:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:50:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:50:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:50:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:50:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:50:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:50:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:50:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:50:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:50:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:50:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:50:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:50:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:50:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:50:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:50:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:50:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:50:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:50:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:50:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:50:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:50:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:50:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:50:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:50:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:50:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:50:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:50:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:50:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:50:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:50:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:50:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:50:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:50:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:50:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:50:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:50:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:50:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:50:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:50:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:50:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:50:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:50:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:50:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:50:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:50:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:50:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:50:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:50:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:50:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:50:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:50:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:50:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:50:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:50:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:50:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:50:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:50:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:50:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:50:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:50:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:50:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:50:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:50:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:50:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:50:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:50:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:50:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:50:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:50:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:50:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:50:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:50:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:50:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:50:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:50:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:50:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:50:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:50:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:50:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:50:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:50:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:50:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:50:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:50:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:50:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:50:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:50:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:50:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:50:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:50:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:50:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:50:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:50:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:50:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:50:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:50:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:50:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:50:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:50:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:50:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:50:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:50:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:50:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:50:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:50:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:50:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:50:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:50:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:50:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:50:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:50:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:50:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:50:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:50:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:50:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:50:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:50:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:50:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:50:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:50:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:50:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:50:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:50:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:50:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:50:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:50:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:50:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:51:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:51:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:51:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:51:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:51:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:51:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:51:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:51:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:51:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:51:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:51:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:51:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:51:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:51:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:51:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:51:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:51:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:51:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:51:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:51:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:51:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:51:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:51:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:51:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:51:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:51:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:51:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:51:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:51:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:51:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:51:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:51:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:51:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:51:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:51:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:51:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:51:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:51:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:51:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:51:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:51:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:51:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:51:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:51:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:51:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:51:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:51:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:51:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:51:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:51:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:51:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:51:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:51:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:51:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:51:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:51:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:51:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:51:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:51:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:51:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:51:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:51:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:51:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:51:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:51:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:51:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:51:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:51:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:51:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:51:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:51:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:51:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:51:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:51:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:51:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:51:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:51:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:51:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:51:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:51:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:51:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:51:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:51:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:51:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:51:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:51:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:51:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:51:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:51:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:51:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:51:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:51:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:51:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:51:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:51:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:51:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:51:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:51:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:51:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:51:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:51:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:51:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:51:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:51:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:51:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:51:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:51:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:51:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:51:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:51:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:51:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:51:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:51:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:51:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:51:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:51:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:51:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:51:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:51:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:51:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:51:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:51:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:51:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:51:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:51:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:51:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:51:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:51:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:51:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:51:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:51:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:51:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:51:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:51:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:51:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:51:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:51:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:51:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:51:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:51:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:51:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:51:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:51:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:51:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:51:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:51:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:52:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:52:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:52:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:52:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:52:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:52:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:52:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:52:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:52:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:52:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:52:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:52:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:52:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:52:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:52:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:52:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:52:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:52:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:52:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:52:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:52:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:52:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:52:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:52:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:52:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:52:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:52:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:52:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:52:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:52:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:52:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:52:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:52:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:52:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:52:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:52:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:52:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:52:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:52:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:52:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:52:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:52:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:52:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:52:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:52:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:52:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:52:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:52:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:52:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:52:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:52:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:52:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:52:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:52:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:52:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:52:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:52:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:52:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:52:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:52:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:52:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:52:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:52:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:52:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:52:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:52:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:52:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:52:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:52:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:52:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:52:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:52:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:52:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:52:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:52:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:52:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:52:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:52:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:52:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:52:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:52:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:52:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:52:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:52:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:52:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:52:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:54:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:54:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:54:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:54:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:54:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:54:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:54:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:54:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:54:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:54:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:54:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:54:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:54:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:54:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:54:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:54:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:54:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:54:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:54:39 rdma06 rsyslogd[1041]: imjournal: 5082 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 09:54:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:54:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:54:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:54:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:54:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:54:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:54:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:54:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:54:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:54:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:54:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:54:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:54:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:54:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:54:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:54:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:54:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:54:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:54:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:54:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:54:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:54:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:54:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:54:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:54:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:54:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:54:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:54:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:54:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:54:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:54:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:54:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:54:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:54:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:54:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:54:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:54:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:54:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:54:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:54:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:54:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:54:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:54:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:54:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:54:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:54:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:54:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:54:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:54:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:54:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:54:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:54:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:54:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:54:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:54:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:54:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:54:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:54:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:54:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:54:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:54:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:54:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:54:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:54:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:54:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:54:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:54:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:54:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:54:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:54:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:54:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:54:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:55:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:55:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:55:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:55:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:55:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:55:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:55:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:55:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:55:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:55:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:55:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:55:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:55:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:55:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:55:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:55:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:55:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:55:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:55:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:55:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:55:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:55:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:55:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:55:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:55:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:55:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:55:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:55:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:55:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:55:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:55:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:55:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:55:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:55:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:55:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:55:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:55:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:55:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:55:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:55:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:55:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:55:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:55:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:55:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:55:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:55:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:55:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:55:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:55:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:55:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:55:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:55:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:55:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:55:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:55:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:55:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:55:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:55:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:55:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:55:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:55:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:55:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:55:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:55:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:55:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:55:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:55:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:55:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:55:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:55:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:55:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:55:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:55:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:55:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:55:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:55:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:55:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:55:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:55:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:55:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:55:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:55:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:55:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:55:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:55:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:55:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:55:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:55:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:55:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:55:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:55:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:55:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:55:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:55:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:55:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:55:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:55:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:55:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:55:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:55:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:55:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:55:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:55:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:55:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:55:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:55:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:55:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:55:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:55:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:55:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:55:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:55:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:55:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:55:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:55:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:55:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:55:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:55:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:55:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:55:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:55:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:55:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:55:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:55:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:55:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:55:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:55:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:55:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:55:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:55:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:55:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:55:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:55:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:55:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:55:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:55:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:55:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:55:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:55:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:55:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:55:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:55:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:55:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:55:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:55:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:55:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:55:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:55:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:55:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:55:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:55:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:55:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:55:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:55:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:55:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:55:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:55:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:55:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:55:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:55:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:55:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:55:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:55:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:55:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:55:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:55:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:55:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:55:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:55:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:55:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:55:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:55:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:55:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:55:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:55:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:55:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:55:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:55:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:55:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:55:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:55:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:55:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:55:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:55:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:55:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:55:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:55:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:55:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:55:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:55:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:55:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:55:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:55:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:55:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:55:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:55:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:55:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:55:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:55:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:55:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:55:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:55:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:55:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:55:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:55:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:55:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:55:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:55:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:55:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:55:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:55:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:55:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:55:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:55:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:55:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:55:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:56:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:56:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:56:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:56:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:56:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:56:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:56:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:56:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:56:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:56:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:56:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:56:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:56:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:56:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:56:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:56:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:56:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:56:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:56:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:56:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:56:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:56:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:56:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:56:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:56:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:56:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:56:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:56:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:56:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:56:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:56:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:56:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:56:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:56:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:56:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:56:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:56:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:56:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:56:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:56:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:56:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:56:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:56:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:56:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:56:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:56:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:56:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:56:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:56:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:56:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:56:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:56:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:56:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:56:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:56:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:56:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:56:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:56:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:56:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:56:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:56:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:56:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:56:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:56:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:56:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:56:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:56:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:56:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:56:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:56:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:56:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:56:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:56:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:56:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:56:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:56:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:56:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:56:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:56:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:56:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:56:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:56:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:56:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:56:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:56:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:56:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:56:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:56:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:56:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:56:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:56:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:56:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:56:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:56:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:56:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:56:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:56:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:56:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:56:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:56:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:56:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:56:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:56:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:56:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:56:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:56:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:56:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:56:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:56:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:56:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:56:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:56:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:56:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:56:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:56:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:56:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:56:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:56:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:56:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:56:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:56:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:56:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:56:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:56:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:56:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:56:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:56:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:56:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:56:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:56:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:56:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:56:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:56:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:56:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:56:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:56:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:56:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:56:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:56:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:56:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:56:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:56:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:56:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:56:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:56:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:56:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:56:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:56:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:56:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:56:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:56:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:56:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:56:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:56:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:56:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:56:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:56:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:56:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:56:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:56:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:56:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:56:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:56:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:56:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:56:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:56:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:56:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:56:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:56:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:56:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:56:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:56:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:56:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:56:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:56:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:56:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:56:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:56:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:56:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:56:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:56:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:56:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:56:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:56:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:56:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:56:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:56:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:56:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:56:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:56:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:56:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:56:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:56:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:56:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:56:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:56:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:56:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:56:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:56:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:56:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:56:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:56:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:56:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:56:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:56:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:56:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:56:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:56:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:56:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:56:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:56:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:56:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:56:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:56:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:56:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:56:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:57:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:57:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:57:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:57:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:57:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:57:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:57:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:57:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:57:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:57:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:57:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:57:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:57:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:57:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:57:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:57:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:57:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:57:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:57:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:57:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:57:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:57:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:57:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:57:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:57:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:57:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:57:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:57:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:57:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:57:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:57:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:57:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:57:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:57:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:57:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:57:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:57:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:57:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:57:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:57:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:57:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:57:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:57:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:57:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:57:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:57:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:57:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:57:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:57:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:57:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:57:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:57:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:57:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:57:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:57:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:57:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:57:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:57:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:57:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:57:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:57:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:57:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:57:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:57:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:57:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:57:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:57:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:57:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:57:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:57:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:57:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:57:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:57:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:57:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:57:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:57:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:57:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:57:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:57:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:57:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:57:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:57:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:57:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:57:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:57:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:57:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:57:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:57:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:57:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:57:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:57:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:57:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:57:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:57:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:57:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:57:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:57:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:57:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:57:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:57:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:57:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:57:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:57:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:57:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:57:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:57:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:57:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:57:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:57:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:57:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:57:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:57:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:57:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:57:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:57:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:57:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:57:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:57:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:57:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:57:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:57:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:57:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:57:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:57:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:57:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:57:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:57:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:57:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:57:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:57:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:57:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:57:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:57:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:57:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:57:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:57:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:57:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:57:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:57:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:57:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:57:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:57:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:57:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:57:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:57:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:57:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:57:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:57:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:57:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:57:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:57:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:57:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:57:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:57:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:57:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:57:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:57:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:57:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:57:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:57:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:57:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:57:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:57:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:57:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:57:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:57:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:57:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:57:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:57:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:57:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:57:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:57:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:57:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:57:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:57:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:57:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:57:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:57:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:57:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:57:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:57:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:57:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:57:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:57:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:57:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:57:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:57:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:57:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:57:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:57:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:57:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:57:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:57:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:57:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:57:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:57:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:57:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:57:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:57:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:57:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:57:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:57:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:57:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:57:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:57:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:57:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:57:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:57:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:57:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:57:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:57:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:57:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:57:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:57:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:57:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:57:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:58:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:58:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:58:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:58:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:58:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:58:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:58:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:58:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:58:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:58:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:58:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:58:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:58:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:58:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:58:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:58:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:58:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:58:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:58:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:58:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:58:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:58:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:58:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:58:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:58:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:58:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:58:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:58:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:58:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:58:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:58:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:58:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:58:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:58:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:58:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:58:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:58:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:58:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:58:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:58:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:58:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:58:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:58:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:58:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:58:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:58:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:58:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:58:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:58:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:58:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:58:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:58:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:58:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:58:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:58:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:58:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:58:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:58:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:58:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:58:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:58:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:58:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:58:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:58:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:58:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:58:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:58:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:58:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:58:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:58:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:58:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:58:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:58:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:58:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:58:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:58:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:58:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:58:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:58:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:58:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:58:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:58:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:58:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:58:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:58:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:58:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:58:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:58:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:58:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:58:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:58:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:58:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:58:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:58:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:58:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:58:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:58:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:58:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:58:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:58:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:58:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:58:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:58:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:58:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:58:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:58:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:58:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:58:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:58:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:58:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:58:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:58:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:58:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:58:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:58:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:58:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:58:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:58:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:58:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:58:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:58:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:58:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:58:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:58:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:58:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:58:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:58:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:58:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:58:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:58:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:58:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:58:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:58:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:58:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:58:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:58:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:58:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:58:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:58:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:58:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:58:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:58:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:58:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:58:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:58:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:58:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:58:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:58:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:58:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:58:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:58:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:58:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:58:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:58:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:58:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:58:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:58:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:58:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:58:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:58:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:58:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:58:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:58:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:58:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:58:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:58:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:58:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:58:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:58:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:58:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:58:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:58:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:58:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:58:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:58:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:58:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:58:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:58:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:58:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:58:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:58:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:58:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:58:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:58:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:58:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:58:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:58:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:58:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:58:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:58:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:58:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:58:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:58:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:58:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:58:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:58:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:58:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:58:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:58:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:58:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:58:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:58:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:58:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:58:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:58:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:58:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:58:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:58:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:58:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:58:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:58:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:58:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:58:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:58:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:58:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:58:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:59:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:59:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:59:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:59:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:59:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:59:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:59:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:59:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:59:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:59:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:59:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:59:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:59:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:59:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:59:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:59:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:59:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:59:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:59:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:59:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:59:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:59:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:59:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:59:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:59:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:59:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:59:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:59:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:59:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:59:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:59:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:59:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:59:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:59:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:59:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:59:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:59:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:59:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:59:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:59:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:59:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:59:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:59:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:59:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:59:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:59:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:59:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:59:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:59:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:59:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:59:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:59:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:59:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:59:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:59:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:59:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:59:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:59:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:59:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:59:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:59:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:59:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:59:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:59:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:59:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:59:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:59:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:59:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:59:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:59:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:59:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:59:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:59:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:59:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:59:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:59:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:59:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:59:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:59:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:59:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:59:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:59:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:59:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:59:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:59:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:59:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:59:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:59:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:59:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:59:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:59:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:59:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:59:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:59:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:59:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:59:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:59:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:59:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:59:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:59:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:59:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:59:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:59:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:59:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:59:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:59:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:59:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:59:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:59:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:59:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:59:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:59:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:59:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:59:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:59:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:59:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:59:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:59:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:59:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:59:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:59:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:59:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:59:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:59:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:59:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:59:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:59:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:59:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:59:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:59:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:59:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:59:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:59:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:59:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:59:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:59:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:59:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:59:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:59:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:59:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:59:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:59:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:59:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:59:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:59:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 09:59:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 09:59:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:59:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:59:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:59:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 09:59:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 09:59:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 09:59:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 09:59:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 09:59:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 09:59:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 09:59:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 09:59:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 09:59:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 09:59:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 09:59:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 09:59:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 09:59:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:00:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:00:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:00:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:00:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:00:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:00:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:00:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:00:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:00:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:00:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:00:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:00:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:00:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:00:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:00:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:00:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:00:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:00:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:00:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:00:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:00:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:00:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:00:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:00:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:00:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:00:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:00:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:00:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:00:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:00:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:00:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:00:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:00:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:00:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:00:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:00:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:00:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:00:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:00:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:00:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:00:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:00:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:00:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:00:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:00:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:00:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:00:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:00:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:00:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:00:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:00:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:00:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:00:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:00:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:00:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:00:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:00:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:00:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:00:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:00:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:00:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:00:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:00:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:00:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:00:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:00:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:00:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:00:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:00:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:00:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:00:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:00:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:00:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:00:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:00:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:00:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:00:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:00:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:00:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:00:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:00:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:00:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:00:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:00:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:00:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:00:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:00:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:00:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:00:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:00:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:00:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:00:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:00:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:00:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:00:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:00:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:00:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:00:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:00:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:00:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:00:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:00:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:00:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:00:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:00:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:00:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:00:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:00:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:00:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:00:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:00:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:00:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:00:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:00:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:00:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:00:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:00:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:00:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:00:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:00:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:00:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:00:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:00:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:00:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:00:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:00:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:00:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:00:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:00:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:00:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:00:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:00:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:00:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:00:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:00:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:00:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:00:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:00:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:00:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:00:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:00:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:00:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:00:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:00:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:00:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:00:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:00:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:00:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:00:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:00:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:00:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:00:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:00:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:00:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:00:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:00:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:00:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:00:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:00:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:00:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:00:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:00:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:00:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:00:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:00:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:00:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:00:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:00:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:00:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:00:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:00:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:00:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:00:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:00:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:00:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:00:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:00:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:00:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:00:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:00:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:00:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:00:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:00:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:00:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:00:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:00:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:01:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:01:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:01:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:01:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:01:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:01:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:01:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:01:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:01:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:01:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:01:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:01:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:01:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:01:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:01:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:01:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:01:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:01:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:01:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:01:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:01:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:01:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:01:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:01:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:01:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:01:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:01:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:01:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:01:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:01:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:01:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:01:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:01:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:01:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:01:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:01:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:01:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:01:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:01:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:01:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:01:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:01:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:01:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:01:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:01:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:01:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:01:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:01:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:01:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:01:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:01:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:01:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:01:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:01:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:01:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:01:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:01:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:01:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:01:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:01:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:01:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:01:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:01:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:01:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:01:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:01:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:01:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:01:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:01:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:01:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:01:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:01:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:01:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:01:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:01:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:01:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:01:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:01:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:01:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:01:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:01:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:01:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:01:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:01:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:01:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:01:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:01:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:01:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:01:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:01:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:01:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:01:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:01:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:01:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:01:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:01:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:01:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:01:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:01:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:01:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:01:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:01:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:01:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:01:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:01:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:01:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:01:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:01:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:01:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:01:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:01:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:01:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:01:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:01:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:01:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:01:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:01:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:01:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:01:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:01:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:01:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:01:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:01:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:01:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:01:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:01:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:01:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:01:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:01:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:01:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:01:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:01:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:01:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:01:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:01:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:01:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:01:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:01:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:01:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:01:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:01:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:01:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:01:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:01:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:01:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:01:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:01:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:01:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:01:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:01:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:01:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:01:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:01:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:01:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:01:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:01:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:01:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:01:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:01:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:01:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:01:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:01:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:01:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:01:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:01:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:01:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:01:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:01:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:01:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:01:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:01:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:01:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:01:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:01:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:01:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:01:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:01:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:01:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:01:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:01:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:01:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:01:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:01:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:01:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:01:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:01:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:01:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:01:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:01:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:01:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:01:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:01:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:01:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:01:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:01:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:01:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:01:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:01:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:01:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:01:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:01:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:01:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:01:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:01:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:01:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:01:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:01:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:01:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:01:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:01:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:01:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:01:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:01:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:01:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:01:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:01:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:02:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:02:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:02:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:02:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:02:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:02:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:02:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:02:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:02:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:02:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:02:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:02:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:02:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:02:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:02:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:02:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:02:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:02:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:02:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:02:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:02:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:02:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:02:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:02:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:02:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:02:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:02:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:02:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:02:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:02:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:02:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:02:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:02:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:02:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:02:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:02:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:02:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:02:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:02:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:02:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:02:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:02:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:02:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:02:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:02:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:02:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:02:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:02:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:02:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:02:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:02:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:02:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:02:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:02:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:02:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:02:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:02:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:02:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:02:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:02:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:02:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:02:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:02:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:02:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:02:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:02:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:02:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:02:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:02:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:02:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:02:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:02:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:02:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:02:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:02:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:02:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:02:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:02:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:02:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:02:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:02:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:02:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:02:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:02:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:02:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:02:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:02:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:02:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:02:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:02:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:02:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:02:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:02:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:02:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:02:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:02:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:02:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:02:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:02:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:02:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:02:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:02:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:02:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:02:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:02:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:02:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:02:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:02:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:02:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:02:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:02:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:02:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:02:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:02:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:02:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:02:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:02:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:02:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:02:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:02:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:02:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:02:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:02:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:02:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:02:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:02:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:04:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:04:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:04:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:04:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:04:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:04:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:04:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:04:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:04:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:04:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:04:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:04:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:04:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:04:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:04:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:04:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:04:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:04:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:04:44 rdma06 rsyslogd[1041]: imjournal: 5295 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 10:04:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:04:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:04:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:04:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:04:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:04:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:04:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:04:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:04:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:04:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:04:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:04:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:04:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:04:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:04:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:04:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:04:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:04:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:04:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:04:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:04:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:04:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:04:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:04:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:04:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:04:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:04:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:04:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:04:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:04:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:04:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:04:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:04:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:04:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:04:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:04:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:04:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:04:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:04:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:04:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:04:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:04:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:04:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:04:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:04:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:04:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:04:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:04:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:04:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:04:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:04:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:04:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:04:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:04:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:05:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:05:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:05:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:05:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:05:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:05:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:05:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:05:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:05:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:05:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:05:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:05:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:05:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:05:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:05:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:05:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:05:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:05:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:05:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:05:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:05:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:05:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:05:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:05:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:05:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:05:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:05:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:05:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:05:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:05:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:05:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:05:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:05:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:05:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:05:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:05:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:05:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:05:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:05:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:05:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:05:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:05:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:05:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:05:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:05:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:05:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:05:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:05:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:05:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:05:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:05:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:05:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:05:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:05:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:05:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:05:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:05:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:05:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:05:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:05:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:05:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:05:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:05:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:05:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:05:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:05:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:05:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:05:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:05:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:05:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:05:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:05:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:05:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:05:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:05:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:05:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:05:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:05:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:05:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:05:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:05:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:05:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:05:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:05:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:05:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:05:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:05:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:05:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:05:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:05:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:05:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:05:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:05:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:05:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:05:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:05:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:05:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:05:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:05:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:05:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:05:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:05:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:05:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:05:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:05:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:05:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:05:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:05:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:05:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:05:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:05:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:05:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:05:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:05:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:05:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:05:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:05:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:05:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:05:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:05:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:05:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:05:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:05:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:05:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:05:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:05:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:05:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:05:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:05:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:05:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:05:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:05:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:05:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:05:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:05:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:05:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:05:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:05:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:05:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:05:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:05:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:05:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:05:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:05:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:05:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:05:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:05:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:05:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:05:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:05:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:05:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:05:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:05:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:05:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:05:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:05:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:05:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:05:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:05:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:05:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:05:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:05:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:05:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:05:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:05:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:05:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:05:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:05:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:05:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:05:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:05:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:05:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:05:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:05:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:05:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:05:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:05:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:05:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:05:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:05:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:05:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:05:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:05:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:05:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:05:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:05:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:05:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:05:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:05:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:05:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:05:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:05:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:05:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:05:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:05:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:05:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:05:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:05:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:05:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:05:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:05:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:05:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:05:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:05:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:05:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:05:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:05:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:05:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:05:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:05:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:05:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:05:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:05:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:05:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:05:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:05:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:06:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:06:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:06:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:06:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:06:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:06:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:06:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:06:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:06:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:06:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:06:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:06:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:06:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:06:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:06:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:06:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:06:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:06:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:06:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:06:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:06:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:06:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:06:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:06:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:06:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:06:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:06:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:06:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:06:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:06:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:06:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:06:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:06:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:06:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:06:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:06:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:06:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:06:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:06:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:06:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:06:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:06:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:06:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:06:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:06:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:06:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:06:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:06:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:06:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:06:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:06:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:06:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:06:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:06:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:06:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:06:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:06:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:06:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:06:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:06:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:06:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:06:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:06:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:06:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:06:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:06:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:06:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:06:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:06:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:06:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:06:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:06:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:06:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:06:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:06:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:06:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:06:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:06:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:06:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:06:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:06:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:06:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:06:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:06:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:06:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:06:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:06:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:06:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:06:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:06:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:06:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:06:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:06:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:06:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:06:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:06:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:06:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:06:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:06:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:06:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:06:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:06:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:06:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:06:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:06:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:06:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:06:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:06:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:06:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:06:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:06:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:06:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:06:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:06:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:06:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:06:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:06:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:06:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:06:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:06:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:06:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:06:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:06:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:06:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:06:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:06:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:06:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:06:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:06:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:06:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:06:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:06:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:06:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:06:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:06:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:06:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:06:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:06:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:06:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:06:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:06:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:06:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:06:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:06:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:06:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:06:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:06:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:06:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:06:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:06:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:06:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:06:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:06:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:06:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:06:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:06:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:06:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:06:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:06:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:06:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:06:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:06:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:06:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:06:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:06:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:06:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:06:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:06:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:06:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:06:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:06:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:06:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:06:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:06:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:06:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:06:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:06:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:06:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:06:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:06:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:06:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:06:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:06:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:06:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:06:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:06:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:06:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:06:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:06:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:06:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:06:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:06:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:06:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:06:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:06:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:06:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:06:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:06:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:06:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:06:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:06:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:06:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:06:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:06:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:06:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:06:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:06:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:06:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:06:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:06:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:06:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:06:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:06:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:06:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:06:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:06:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:07:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:07:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:07:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:07:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:07:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:07:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:07:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:07:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:07:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:07:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:07:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:07:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:07:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:07:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:07:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:07:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:07:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:07:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:07:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:07:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:07:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:07:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:07:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:07:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:07:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:07:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:07:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:07:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:07:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:07:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:07:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:07:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:07:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:07:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:07:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:07:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:07:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:07:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:07:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:07:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:07:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:07:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:07:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:07:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:07:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:07:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:07:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:07:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:07:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:07:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:07:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:07:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:07:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:07:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:07:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:07:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:07:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:07:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:07:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:07:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:07:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:07:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:07:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:07:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:07:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:07:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:07:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:07:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:07:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:07:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:07:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:07:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:07:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:07:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:07:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:07:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:07:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:07:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:07:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:07:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:07:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:07:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:07:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:07:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:07:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:07:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:07:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:07:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:07:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:07:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:07:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:07:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:07:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:07:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:07:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:07:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:07:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:07:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:07:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:07:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:07:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:07:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:07:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:07:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:07:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:07:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:07:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:07:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:07:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:07:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:07:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:07:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:07:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:07:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:07:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:07:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:07:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:07:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:07:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:07:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:07:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:07:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:07:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:07:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:07:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:07:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:07:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:07:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:07:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:07:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:07:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:07:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:07:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:07:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:07:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:07:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:07:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:07:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:07:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:07:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:07:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:07:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:07:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:07:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:07:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:07:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:07:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:07:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:07:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:07:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:07:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:07:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:07:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:07:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:07:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:07:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:07:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:07:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:07:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:07:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:07:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:07:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:07:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:07:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:07:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:07:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:07:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:07:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:07:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:07:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:07:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:07:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:07:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:07:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:07:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:07:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:07:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:07:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:07:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:07:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:07:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:07:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:07:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:07:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:07:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:07:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:07:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:07:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:07:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:07:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:07:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:07:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:07:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:07:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:07:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:07:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:07:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:07:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:07:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:07:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:07:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:07:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:07:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:07:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:07:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:07:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:07:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:07:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:07:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:07:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:07:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:07:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:07:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:07:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:07:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:07:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:08:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:08:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:08:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:08:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:08:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:08:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:08:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:08:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:08:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:08:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:08:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:08:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:08:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:08:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:08:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:08:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:08:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:08:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:08:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:08:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:08:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:08:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:08:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:08:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:08:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:08:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:08:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:08:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:08:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:08:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:08:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:08:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:08:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:08:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:08:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:08:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:08:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:08:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:08:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:08:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:08:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:08:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:08:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:08:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:08:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:08:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:08:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:08:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:08:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:08:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:08:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:08:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:08:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:08:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:08:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:08:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:08:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:08:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:08:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:08:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:08:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:08:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:08:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:08:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:08:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:08:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:08:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:08:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:08:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:08:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:08:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:08:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:08:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:08:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:08:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:08:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:08:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:08:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:08:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:08:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:08:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:08:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:08:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:08:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:08:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:08:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:08:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:08:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:08:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:08:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:08:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:08:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:08:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:08:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:08:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:08:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:08:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:08:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:08:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:08:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:08:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:08:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:08:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:08:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:08:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:08:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:08:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:08:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:08:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:08:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:08:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:08:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:08:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:08:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:08:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:08:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:08:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:08:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:08:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:08:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:08:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:08:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:08:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:08:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:08:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:08:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:08:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:08:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:08:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:08:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:08:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:08:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:08:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:08:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:08:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:08:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:08:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:08:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:08:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:08:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:08:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:08:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:08:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:08:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:08:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:08:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:08:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:08:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:08:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:08:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:08:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:08:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:08:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:08:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:08:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:08:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:08:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:08:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:08:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:08:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:08:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:08:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:08:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:08:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:08:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:08:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:08:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:08:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:08:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:08:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:08:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:08:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:08:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:08:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:08:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:08:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:08:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:08:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:08:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:08:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:08:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:08:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:08:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:08:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:08:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:08:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:08:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:08:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:08:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:08:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:08:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:08:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:08:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:08:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:08:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:08:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:08:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:08:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:08:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:08:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:08:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:08:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:08:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:08:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:08:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:08:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:08:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:08:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:08:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:08:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:08:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:08:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:08:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:08:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:08:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:08:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:09:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:09:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:09:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:09:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:09:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:09:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:09:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:09:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:09:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:09:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:09:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:09:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:09:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:09:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:09:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:09:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:09:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:09:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:09:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:09:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:09:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:09:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:09:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:09:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:09:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:09:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:09:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:09:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:09:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:09:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:09:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:09:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:09:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:09:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:09:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:09:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:09:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:09:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:09:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:09:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:09:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:09:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:09:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:09:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:09:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:09:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:09:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:09:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:09:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:09:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:09:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:09:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:09:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:09:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:09:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:09:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:09:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:09:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:09:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:09:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:09:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:09:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:09:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:09:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:09:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:09:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:09:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:09:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:09:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:09:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:09:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:09:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:09:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:09:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:09:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:09:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:09:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:09:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:09:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:09:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:09:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:09:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:09:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:09:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:09:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:09:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:09:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:09:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:09:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:09:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:09:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:09:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:09:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:09:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:09:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:09:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:09:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:09:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:09:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:09:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:09:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:09:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:09:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:09:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:09:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:09:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:09:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:09:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:09:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:09:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:09:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:09:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:09:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:09:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:09:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:09:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:09:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:09:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:09:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:09:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:09:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:09:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:09:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:09:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:09:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:09:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:09:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:09:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:09:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:09:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:09:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:09:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:09:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:09:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:09:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:09:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:09:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:09:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:09:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:09:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:09:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:09:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:09:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:09:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:09:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:09:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:09:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:09:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:09:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:09:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:09:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:09:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:09:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:09:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:09:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:09:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:09:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:09:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:09:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:09:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:09:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:09:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:09:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:09:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:09:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:09:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:09:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:09:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:09:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:09:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:09:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:09:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:09:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:09:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:09:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:09:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:09:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:09:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:09:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:09:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:09:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:09:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:09:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:09:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:09:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:09:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:09:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:09:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:09:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:09:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:09:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:09:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:09:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:09:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:09:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:09:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:09:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:09:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:09:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:09:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:09:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:09:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:09:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:09:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:09:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:09:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:09:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:09:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:09:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:09:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:09:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:09:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:09:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:09:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:09:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:09:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:10:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:10:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:10:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:10:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:10:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:10:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:10:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:10:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:10:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:10:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:10:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:10:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:10:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:10:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:10:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:10:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:10:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:10:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:10:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:10:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:10:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:10:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:10:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:10:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:10:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:10:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:10:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:10:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:10:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:10:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:10:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:10:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:10:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:10:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:10:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:10:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:10:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:10:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:10:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:10:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:10:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:10:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:10:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:10:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:10:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:10:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:10:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:10:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:10:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:10:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:10:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:10:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:10:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:10:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:10:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:10:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:10:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:10:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:10:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:10:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:10:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:10:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:10:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:10:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:10:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:10:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:10:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:10:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:10:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:10:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:10:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:10:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:10:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:10:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:10:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:10:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:10:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:10:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:10:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:10:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:10:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:10:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:10:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:10:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:10:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:10:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:10:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:10:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:10:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:10:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:10:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:10:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:10:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:10:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:10:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:10:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:10:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:10:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:10:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:10:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:10:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:10:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:10:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:10:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:10:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:10:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:10:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:10:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:10:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:10:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:10:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:10:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:10:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:10:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:10:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:10:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:10:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:10:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:10:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:10:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:10:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:10:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:10:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:10:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:10:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:10:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:10:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:10:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:10:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:10:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:10:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:10:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:10:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:10:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:10:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:10:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:10:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:10:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:10:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:10:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:10:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:10:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:10:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:10:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:10:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:10:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:10:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:10:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:10:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:10:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:10:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:10:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:10:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:10:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:10:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:10:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:10:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:10:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:10:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:10:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:10:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:10:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:10:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:10:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:10:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:10:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:10:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:10:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:10:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:10:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:10:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:10:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:10:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:10:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:10:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:10:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:10:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:10:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:10:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:10:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:10:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:10:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:10:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:10:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:10:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:10:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:10:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:10:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:10:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:10:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:10:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:10:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:10:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:10:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:10:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:10:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:10:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:10:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:10:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:10:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:10:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:10:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:10:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:10:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:10:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:10:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:10:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:10:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:10:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:10:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:10:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:10:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:10:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:10:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:10:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:10:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:11:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:11:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:11:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:11:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:11:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:11:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:11:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:11:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:11:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:11:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:11:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:11:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:11:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:11:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:11:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:11:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:11:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:11:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:11:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:11:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:11:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:11:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:11:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:11:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:11:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:11:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:11:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:11:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:11:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:11:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:11:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:11:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:11:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:11:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:11:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:11:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:11:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:11:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:11:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:11:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:11:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:11:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:11:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:11:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:11:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:11:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:11:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:11:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:11:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:11:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:11:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:11:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:11:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:11:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:11:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:11:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:11:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:11:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:11:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:11:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:11:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:11:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:11:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:11:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:11:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:11:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:11:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:11:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:11:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:11:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:11:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:11:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:11:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:11:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:11:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:11:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:11:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:11:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:11:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:11:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:11:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:11:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:11:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:11:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:11:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:11:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:11:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:11:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:11:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:11:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:11:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:11:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:11:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:11:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:11:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:11:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:11:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:11:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:11:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:11:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:11:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:11:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:11:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:11:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:11:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:11:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:11:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:11:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:11:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:11:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:11:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:11:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:11:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:11:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:11:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:11:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:11:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:11:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:11:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:11:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:11:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:11:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:11:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:11:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:11:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:11:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:11:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:11:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:11:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:11:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:11:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:11:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:11:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:11:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:11:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:11:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:11:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:11:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:11:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:11:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:11:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:11:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:11:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:11:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:11:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:11:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:11:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:11:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:11:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:11:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:11:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:11:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:11:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:11:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:11:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:11:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:11:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:11:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:11:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:11:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:11:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:11:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:11:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:11:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:11:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:11:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:11:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:11:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:11:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:11:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:11:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:11:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:11:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:11:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:11:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:11:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:11:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:11:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:11:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:11:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:11:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:11:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:11:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:11:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:11:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:11:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:11:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:11:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:11:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:11:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:11:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:11:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:11:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:11:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:11:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:11:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:11:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:11:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:11:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:11:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:11:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:11:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:11:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:11:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:11:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:11:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:11:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:11:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:11:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:11:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:11:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:11:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:11:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:11:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:11:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:11:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:12:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:12:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:12:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:12:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:12:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:12:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:12:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:12:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:12:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:12:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:12:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:12:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:12:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:12:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:12:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:12:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:12:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:12:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:12:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:12:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:12:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:12:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:12:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:12:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:12:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:12:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:12:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:12:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:12:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:12:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:12:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:12:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:12:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:12:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:12:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:12:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:12:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:12:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:12:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:12:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:12:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:12:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:12:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:12:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:12:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:12:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:12:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:12:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:12:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:12:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:12:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:12:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:12:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:12:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:12:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:12:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:12:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:12:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:12:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:12:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:12:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:12:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:12:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:12:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:12:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:12:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:12:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:12:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:12:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:12:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:12:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:12:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:12:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:12:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:12:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:12:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:12:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:12:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:12:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:12:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:12:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:12:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:12:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:12:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:12:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:12:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:12:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:12:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:12:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:12:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:12:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:12:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:12:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:12:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:12:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:12:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:12:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:12:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:12:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:12:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:12:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:12:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:12:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:12:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:12:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:12:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:12:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:12:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:12:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:12:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:12:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:12:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:12:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:12:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:12:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:12:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:12:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:12:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:12:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:12:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:12:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:12:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:12:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:12:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:12:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:12:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:12:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:12:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:12:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:12:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:12:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:12:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:12:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:12:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:12:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:12:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:12:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:12:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:12:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:12:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:12:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:12:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:12:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:12:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:14:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:14:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:14:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:14:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:14:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:14:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:14:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:14:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:14:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:14:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:14:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:14:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:14:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:14:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:14:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:14:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:14:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:14:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:14:49 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 10:14:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:14:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:14:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:14:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:14:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:14:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:14:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:14:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:14:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:14:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:14:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:14:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:14:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:14:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:14:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:14:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:14:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:14:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:14:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:14:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:14:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:14:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:14:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:14:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:14:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:14:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:14:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:14:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:14:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:14:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:14:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:14:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:14:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:14:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:14:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:14:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:15:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:15:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:15:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:15:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:15:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:15:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:15:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:15:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:15:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:15:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:15:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:15:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:15:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:15:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:15:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:15:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:15:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:15:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:15:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:15:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:15:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:15:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:15:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:15:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:15:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:15:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:15:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:15:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:15:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:15:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:15:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:15:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:15:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:15:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:15:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:15:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:15:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:15:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:15:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:15:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:15:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:15:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:15:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:15:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:15:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:15:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:15:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:15:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:15:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:15:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:15:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:15:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:15:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:15:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:15:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:15:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:15:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:15:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:15:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:15:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:15:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:15:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:15:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:15:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:15:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:15:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:15:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:15:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:15:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:15:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:15:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:15:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:15:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:15:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:15:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:15:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:15:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:15:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:15:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:15:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:15:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:15:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:15:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:15:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:15:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:15:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:15:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:15:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:15:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:15:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:15:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:15:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:15:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:15:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:15:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:15:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:15:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:15:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:15:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:15:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:15:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:15:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:15:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:15:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:15:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:15:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:15:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:15:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:15:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:15:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:15:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:15:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:15:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:15:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:15:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:15:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:15:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:15:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:15:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:15:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:15:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:15:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:15:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:15:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:15:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:15:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:15:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:15:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:15:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:15:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:15:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:15:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:15:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:15:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:15:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:15:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:15:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:15:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:15:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:15:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:15:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:15:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:15:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:15:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:15:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:15:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:15:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:15:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:15:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:15:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:15:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:15:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:15:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:15:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:15:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:15:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:15:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:15:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:15:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:15:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:15:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:15:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:15:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:15:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:15:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:15:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:15:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:15:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:15:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:15:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:15:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:15:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:15:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:15:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:15:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:15:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:15:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:15:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:15:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:15:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:15:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:15:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:15:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:15:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:15:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:15:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:15:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:15:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:15:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:15:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:15:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:15:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:15:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:15:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:15:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:15:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:15:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:15:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:15:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:15:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:15:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:15:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:15:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:15:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:15:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:15:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:15:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:15:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:15:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:15:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:15:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:15:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:15:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:15:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:15:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:15:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:16:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:16:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:16:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:16:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:16:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:16:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:16:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:16:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:16:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:16:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:16:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:16:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:16:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:16:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:16:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:16:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:16:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:16:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:16:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:16:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:16:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:16:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:16:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:16:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:16:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:16:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:16:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:16:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:16:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:16:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:16:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:16:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:16:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:16:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:16:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:16:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:16:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:16:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:16:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:16:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:16:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:16:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:16:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:16:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:16:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:16:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:16:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:16:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:16:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:16:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:16:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:16:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:16:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:16:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:16:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:16:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:16:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:16:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:16:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:16:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:16:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:16:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:16:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:16:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:16:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:16:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:16:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:16:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:16:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:16:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:16:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:16:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:16:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:16:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:16:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:16:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:16:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:16:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:16:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:16:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:16:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:16:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:16:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:16:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:16:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:16:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:16:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:16:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:16:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:16:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:16:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:16:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:16:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:16:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:16:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:16:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:16:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:16:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:16:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:16:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:16:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:16:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:16:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:16:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:16:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:16:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:16:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:16:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:16:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:16:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:16:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:16:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:16:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:16:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:16:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:16:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:16:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:16:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:16:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:16:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:16:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:16:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:16:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:16:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:16:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:16:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:16:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:16:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:16:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:16:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:16:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:16:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:16:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:16:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:16:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:16:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:16:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:16:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:16:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:16:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:16:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:16:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:16:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:16:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:16:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:16:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:16:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:16:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:16:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:16:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:16:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:16:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:16:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:16:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:16:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:16:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:16:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:16:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:16:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:16:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:16:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:16:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:16:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:16:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:16:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:16:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:16:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:16:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:16:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:16:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:16:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:16:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:16:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:16:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:16:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:16:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:16:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:16:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:16:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:16:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:16:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:16:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:16:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:16:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:16:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:16:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:16:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:16:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:16:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:16:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:16:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:16:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:16:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:16:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:16:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:16:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:16:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:16:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:16:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:16:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:16:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:16:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:16:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:16:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:16:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:16:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:16:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:16:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:16:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:16:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:16:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:16:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:16:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:16:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:16:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:16:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:17:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:17:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:17:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:17:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:17:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:17:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:17:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:17:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:17:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:17:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:17:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:17:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:17:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:17:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:17:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:17:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:17:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:17:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:17:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:17:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:17:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:17:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:17:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:17:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:17:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:17:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:17:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:17:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:17:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:17:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:17:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:17:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:17:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:17:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:17:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:17:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:17:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:17:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:17:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:17:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:17:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:17:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:17:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:17:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:17:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:17:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:17:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:17:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:17:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:17:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:17:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:17:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:17:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:17:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:17:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:17:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:17:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:17:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:17:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:17:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:17:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:17:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:17:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:17:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:17:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:17:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:17:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:17:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:17:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:17:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:17:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:17:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:17:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:17:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:17:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:17:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:17:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:17:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:17:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:17:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:17:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:17:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:17:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:17:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:17:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:17:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:17:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:17:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:17:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:17:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:17:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:17:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:17:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:17:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:17:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:17:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:17:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:17:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:17:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:17:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:17:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:17:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:17:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:17:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:17:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:17:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:17:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:17:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:17:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:17:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:17:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:17:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:17:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:17:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:17:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:17:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:17:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:17:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:17:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:17:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:17:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:17:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:17:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:17:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:17:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:17:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:17:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:17:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:17:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:17:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:17:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:17:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:17:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:17:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:17:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:17:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:17:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:17:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:17:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:17:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:17:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:17:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:17:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:17:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:17:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:17:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:17:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:17:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:17:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:17:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:17:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:17:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:17:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:17:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:17:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:17:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:17:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:17:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:17:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:17:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:17:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:17:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:17:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:17:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:17:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:17:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:17:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:17:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:17:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:17:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:17:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:17:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:17:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:17:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:17:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:17:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:17:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:17:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:17:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:17:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:17:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:17:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:17:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:17:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:17:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:17:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:17:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:17:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:17:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:17:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:17:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:17:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:17:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:17:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:17:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:17:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:17:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:17:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:17:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:17:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:17:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:17:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:17:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:17:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:17:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:17:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:17:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:17:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:17:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:17:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:17:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:17:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:17:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:17:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:17:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:17:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:18:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:18:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:18:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:18:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:18:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:18:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:18:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:18:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:18:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:18:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:18:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:18:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:18:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:18:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:18:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:18:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:18:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:18:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:18:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:18:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:18:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:18:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:18:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:18:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:18:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:18:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:18:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:18:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:18:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:18:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:18:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:18:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:18:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:18:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:18:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:18:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:18:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:18:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:18:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:18:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:18:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:18:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:18:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:18:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:18:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:18:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:18:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:18:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:18:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:18:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:18:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:18:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:18:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:18:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:18:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:18:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:18:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:18:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:18:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:18:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:18:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:18:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:18:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:18:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:18:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:18:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:18:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:18:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:18:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:18:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:18:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:18:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:18:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:18:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:18:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:18:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:18:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:18:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:18:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:18:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:18:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:18:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:18:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:18:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:18:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:18:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:18:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:18:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:18:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:18:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:18:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:18:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:18:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:18:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:18:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:18:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:18:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:18:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:18:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:18:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:18:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:18:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:18:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:18:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:18:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:18:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:18:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:18:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:18:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:18:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:18:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:18:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:18:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:18:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:18:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:18:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:18:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:18:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:18:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:18:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:18:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:18:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:18:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:18:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:18:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:18:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:18:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:18:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:18:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:18:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:18:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:18:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:18:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:18:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:18:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:18:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:18:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:18:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:18:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:18:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:18:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:18:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:18:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:18:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:18:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:18:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:18:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:18:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:18:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:18:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:18:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:18:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:18:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:18:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:18:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:18:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:18:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:18:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:18:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:18:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:18:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:18:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:18:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:18:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:18:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:18:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:18:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:18:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:18:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:18:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:18:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:18:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:18:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:18:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:18:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:18:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:18:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:18:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:18:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:18:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:18:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:18:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:18:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:18:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:18:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:18:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:18:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:18:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:18:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:18:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:18:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:18:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:18:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:18:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:18:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:18:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:18:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:18:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:18:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:18:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:18:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:18:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:18:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:18:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:18:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:18:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:18:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:18:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:18:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:18:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:18:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:18:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:18:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:18:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:18:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:18:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:19:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:19:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:19:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:19:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:19:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:19:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:19:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:19:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:19:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:19:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:19:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:19:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:19:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:19:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:19:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:19:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:19:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:19:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:19:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:19:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:19:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:19:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:19:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:19:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:19:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:19:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:19:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:19:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:19:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:19:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:19:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:19:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:19:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:19:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:19:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:19:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:19:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:19:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:19:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:19:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:19:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:19:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:19:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:19:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:19:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:19:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:19:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:19:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:19:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:19:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:19:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:19:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:19:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:19:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:19:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:19:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:19:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:19:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:19:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:19:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:19:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:19:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:19:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:19:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:19:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:19:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:19:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:19:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:19:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:19:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:19:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:19:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:19:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:19:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:19:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:19:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:19:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:19:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:19:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:19:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:19:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:19:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:19:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:19:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:19:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:19:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:19:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:19:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:19:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:19:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:19:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:19:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:19:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:19:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:19:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:19:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:19:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:19:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:19:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:19:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:19:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:19:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:19:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:19:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:19:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:19:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:19:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:19:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:19:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:19:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:19:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:19:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:19:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:19:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:19:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:19:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:19:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:19:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:19:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:19:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:19:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:19:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:19:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:19:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:19:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:19:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:19:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:19:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:19:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:19:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:19:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:19:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:19:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:19:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:19:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:19:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:19:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:19:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:19:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:19:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:19:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:19:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:19:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:19:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:19:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:19:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:19:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:19:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:19:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:19:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:19:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:19:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:19:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:19:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:19:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:19:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:19:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:19:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:19:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:19:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:19:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:19:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:19:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:19:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:19:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:19:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:19:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:19:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:19:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:19:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:19:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:19:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:19:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:19:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:19:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:19:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:19:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:19:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:19:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:19:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:19:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:19:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:19:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:19:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:19:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:19:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:19:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:19:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:19:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:19:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:19:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:19:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:19:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:19:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:19:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:19:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:19:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:19:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:19:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:19:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:19:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:19:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:19:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:19:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:19:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:19:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:19:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:19:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:19:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:19:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:19:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:19:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:19:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:19:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:19:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:19:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:20:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:20:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:20:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:20:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:20:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:20:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:20:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:20:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:20:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:20:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:20:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:20:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:20:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:20:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:20:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:20:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:20:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:20:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:20:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:20:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:20:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:20:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:20:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:20:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:20:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:20:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:20:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:20:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:20:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:20:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:20:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:20:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:20:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:20:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:20:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:20:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:20:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:20:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:20:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:20:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:20:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:20:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:20:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:20:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:20:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:20:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:20:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:20:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:20:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:20:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:20:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:20:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:20:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:20:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:20:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:20:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:20:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:20:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:20:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:20:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:20:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:20:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:20:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:20:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:20:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:20:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:20:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:20:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:20:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:20:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:20:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:20:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:20:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:20:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:20:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:20:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:20:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:20:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:20:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:20:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:20:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:20:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:20:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:20:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:20:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:20:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:20:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:20:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:20:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:20:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:20:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:20:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:20:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:20:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:20:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:20:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:20:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:20:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:20:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:20:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:20:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:20:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:20:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:20:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:20:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:20:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:20:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:20:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:20:32 rdma06 NetworkManager[1025]: <info>  [1681827632.8394] dhcp4 (eno1): state changed new lease, address=10.73.131.45
Apr 18 10:20:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:20:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:20:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:20:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:20:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:20:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:20:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:20:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:20:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:20:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:20:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:20:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:20:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:20:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:20:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:20:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:20:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:20:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:20:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:20:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:20:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:20:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:20:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:20:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:20:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:20:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:20:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:20:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:20:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:20:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:20:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:20:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:20:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:20:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:20:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:20:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:20:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:20:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:20:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:20:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:20:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:20:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:20:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:20:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:20:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:20:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:20:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:20:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:20:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:20:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:20:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:20:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:20:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:20:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:20:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:20:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:20:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:20:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:20:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:20:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:20:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:20:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:20:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:20:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:20:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:20:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:20:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:20:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:20:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:20:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:20:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:20:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:20:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:20:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:20:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:20:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:20:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:20:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:20:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:20:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:20:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:20:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:20:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:20:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:20:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:20:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:20:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:20:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:20:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:20:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:20:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:20:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:20:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:20:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:20:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:20:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:20:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:20:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:20:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:20:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:20:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:20:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:20:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:20:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:20:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:20:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:20:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:20:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:21:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:21:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:21:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:21:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:21:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:21:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:21:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:21:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:21:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:21:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:21:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:21:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:21:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:21:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:21:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:21:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:21:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:21:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:21:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:21:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:21:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:21:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:21:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:21:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:21:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:21:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:21:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:21:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:21:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:21:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:21:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:21:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:21:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:21:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:21:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:21:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:21:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:21:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:21:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:21:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:21:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:21:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:21:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:21:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:21:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:21:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:21:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:21:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:21:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:21:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:21:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:21:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:21:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:21:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:21:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:21:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:21:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:21:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:21:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:21:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:21:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:21:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:21:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:21:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:21:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:21:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:21:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:21:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:21:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:21:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:21:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:21:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:21:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:21:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:21:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:21:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:21:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:21:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:21:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:21:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:21:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:21:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:21:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:21:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:21:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:21:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:21:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:21:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:21:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:21:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:21:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:21:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:21:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:21:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:21:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:21:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:21:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:21:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:21:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:21:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:21:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:21:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:21:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:21:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:21:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:21:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:21:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:21:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:21:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:21:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:21:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:21:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:21:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:21:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:21:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:21:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:22:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:22:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:22:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:22:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:22:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:22:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:22:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:22:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:22:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:22:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:22:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:22:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:22:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:22:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:22:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:22:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:22:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:22:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:22:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:22:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:22:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:22:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:22:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:22:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:22:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:22:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:22:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:22:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:22:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:22:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:22:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:22:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:22:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:22:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:22:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:22:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:22:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:22:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:22:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:22:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:22:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:22:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:22:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:22:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:22:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:22:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:22:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:22:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:22:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:22:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:22:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:22:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:22:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:22:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:22:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:22:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:22:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:22:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:22:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:22:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:22:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:22:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:22:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:22:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:22:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:22:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:22:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:22:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:22:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:22:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:22:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:22:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:22:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:22:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:22:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:22:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:22:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:22:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:22:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:22:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:22:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:22:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:22:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:22:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:22:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:22:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:22:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:22:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:22:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:22:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:22:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:22:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:22:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:22:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:22:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:22:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:22:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:22:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:22:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:22:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:22:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:22:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:22:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:22:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:22:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:22:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:22:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:22:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:22:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:22:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:22:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:22:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:22:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:22:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:22:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:22:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:22:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:22:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:22:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:22:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:22:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:22:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:22:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:22:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:22:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:22:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:22:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:22:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:22:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:22:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:22:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:22:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:22:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:22:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:22:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:22:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:22:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:22:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:22:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:22:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:22:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:22:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:22:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:22:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:22:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:22:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:22:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:22:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:22:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:22:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:22:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:22:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:22:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:22:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:22:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:22:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:22:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:22:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:22:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:22:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:22:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:22:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:24:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:24:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:24:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:24:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:24:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:24:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:24:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:24:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:24:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:24:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:24:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:24:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:24:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:24:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:24:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:24:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:24:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:24:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:24:50 rdma06 rsyslogd[1041]: imjournal: 5166 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 10:24:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:24:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:24:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:24:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:24:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:24:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:24:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:24:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:24:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:24:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:24:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:24:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:24:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:24:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:24:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:24:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:24:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:24:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:24:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:24:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:24:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:24:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:24:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:24:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:24:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:24:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:24:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:24:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:24:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:24:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:24:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:24:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:24:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:24:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:24:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:24:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:25:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:25:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:25:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:25:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:25:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:25:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:25:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:25:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:25:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:25:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:25:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:25:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:25:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:25:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:25:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:25:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:25:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:25:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:25:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:25:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:25:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:25:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:25:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:25:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:25:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:25:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:25:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:25:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:25:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:25:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:25:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:25:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:25:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:25:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:25:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:25:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:25:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:25:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:25:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:25:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:25:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:25:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:25:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:25:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:25:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:25:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:25:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:25:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:25:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:25:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:25:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:25:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:25:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:25:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:25:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:25:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:25:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:25:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:25:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:25:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:25:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:25:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:25:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:25:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:25:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:25:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:25:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:25:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:25:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:25:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:25:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:25:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:25:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:25:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:25:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:25:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:25:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:25:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:25:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:25:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:25:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:25:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:25:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:25:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:25:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:25:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:25:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:25:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:25:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:25:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:25:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:25:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:25:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:25:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:25:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:25:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:25:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:25:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:25:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:25:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:25:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:25:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:25:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:25:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:25:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:25:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:25:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:25:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:25:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:25:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:25:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:25:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:25:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:25:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:25:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:25:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:25:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:25:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:25:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:25:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:25:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:25:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:25:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:25:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:25:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:25:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:25:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:25:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:25:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:25:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:25:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:25:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:25:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:25:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:25:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:25:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:25:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:25:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:25:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:25:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:25:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:25:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:25:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:25:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:25:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:25:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:25:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:25:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:25:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:25:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:25:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:25:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:25:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:25:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:25:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:25:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:25:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:25:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:25:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:25:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:25:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:25:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:25:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:25:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:25:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:25:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:25:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:25:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:25:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:25:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:25:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:25:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:25:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:25:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:25:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:25:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:25:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:25:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:25:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:25:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:25:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:25:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:25:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:25:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:25:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:25:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:25:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:25:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:25:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:25:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:25:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:25:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:25:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:25:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:25:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:25:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:25:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:25:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:25:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:25:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:25:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:25:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:25:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:25:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:25:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:25:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:25:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:25:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:25:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:25:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:25:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:25:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:25:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:25:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:25:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:25:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:26:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:26:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:26:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:26:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:26:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:26:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:26:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:26:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:26:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:26:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:26:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:26:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:26:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:26:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:26:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:26:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:26:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:26:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:26:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:26:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:26:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:26:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:26:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:26:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:26:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:26:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:26:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:26:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:26:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:26:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:26:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:26:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:26:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:26:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:26:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:26:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:26:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:26:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:26:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:26:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:26:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:26:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:26:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:26:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:26:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:26:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:26:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:26:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:26:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:26:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:26:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:26:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:26:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:26:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:26:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:26:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:26:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:26:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:26:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:26:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:26:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:26:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:26:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:26:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:26:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:26:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:26:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:26:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:26:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:26:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:26:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:26:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:26:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:26:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:26:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:26:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:26:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:26:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:26:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:26:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:26:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:26:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:26:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:26:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:26:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:26:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:26:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:26:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:26:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:26:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:26:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:26:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:26:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:26:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:26:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:26:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:26:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:26:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:26:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:26:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:26:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:26:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:26:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:26:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:26:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:26:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:26:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:26:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:26:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:26:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:26:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:26:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:26:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:26:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:26:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:26:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:26:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:26:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:26:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:26:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:26:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:26:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:26:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:26:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:26:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:26:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:26:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:26:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:26:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:26:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:26:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:26:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:26:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:26:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:26:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:26:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:26:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:26:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:26:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:26:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:26:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:26:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:26:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:26:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:26:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:26:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:26:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:26:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:26:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:26:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:26:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:26:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:26:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:26:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:26:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:26:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:26:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:26:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:26:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:26:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:26:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:26:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:26:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:26:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:26:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:26:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:26:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:26:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:26:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:26:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:26:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:26:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:26:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:26:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:26:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:26:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:26:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:26:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:26:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:26:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:26:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:26:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:26:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:26:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:26:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:26:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:26:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:26:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:26:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:26:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:26:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:26:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:26:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:26:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:26:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:26:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:26:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:26:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:26:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:26:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:26:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:26:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:26:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:26:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:26:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:26:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:26:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:26:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:26:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:26:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:26:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:26:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:26:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:26:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:26:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:26:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:27:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:27:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:27:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:27:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:27:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:27:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:27:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:27:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:27:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:27:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:27:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:27:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:27:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:27:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:27:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:27:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:27:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:27:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:27:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:27:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:27:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:27:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:27:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:27:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:27:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:27:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:27:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:27:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:27:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:27:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:27:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:27:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:27:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:27:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:27:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:27:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:27:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:27:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:27:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:27:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:27:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:27:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:27:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:27:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:27:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:27:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:27:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:27:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:27:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:27:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:27:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:27:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:27:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:27:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:27:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:27:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:27:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:27:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:27:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:27:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:27:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:27:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:27:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:27:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:27:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:27:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:27:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:27:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:27:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:27:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:27:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:27:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:27:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:27:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:27:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:27:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:27:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:27:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:27:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:27:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:27:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:27:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:27:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:27:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:27:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:27:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:27:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:27:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:27:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:27:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:27:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:27:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:27:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:27:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:27:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:27:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:27:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:27:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:27:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:27:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:27:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:27:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:27:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:27:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:27:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:27:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:27:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:27:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:27:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:27:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:27:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:27:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:27:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:27:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:27:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:27:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:27:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:27:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:27:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:27:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:27:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:27:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:27:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:27:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:27:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:27:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:27:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:27:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:27:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:27:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:27:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:27:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:27:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:27:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:27:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:27:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:27:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:27:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:27:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:27:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:27:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:27:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:27:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:27:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:27:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:27:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:27:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:27:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:27:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:27:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:27:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:27:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:27:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:27:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:27:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:27:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:27:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:27:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:27:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:27:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:27:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:27:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:27:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:27:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:27:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:27:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:27:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:27:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:27:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:27:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:27:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:27:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:27:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:27:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:27:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:27:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:27:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:27:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:27:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:27:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:27:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:27:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:27:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:27:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:27:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:27:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:27:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:27:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:27:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:27:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:27:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:27:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:27:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:27:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:27:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:27:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:27:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:27:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:27:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:27:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:27:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:27:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:27:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:27:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:27:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:27:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:27:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:27:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:27:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:27:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:27:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:27:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:27:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:27:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:27:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:27:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:28:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:28:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:28:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:28:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:28:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:28:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:28:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:28:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:28:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:28:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:28:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:28:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:28:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:28:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:28:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:28:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:28:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:28:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:28:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:28:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:28:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:28:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:28:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:28:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:28:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:28:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:28:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:28:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:28:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:28:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:28:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:28:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:28:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:28:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:28:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:28:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:28:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:28:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:28:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:28:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:28:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:28:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:28:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:28:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:28:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:28:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:28:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:28:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:28:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:28:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:28:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:28:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:28:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:28:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:28:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:28:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:28:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:28:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:28:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:28:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:28:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:28:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:28:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:28:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:28:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:28:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:28:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:28:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:28:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:28:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:28:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:28:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:28:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:28:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:28:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:28:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:28:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:28:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:28:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:28:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:28:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:28:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:28:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:28:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:28:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:28:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:28:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:28:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:28:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:28:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:28:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:28:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:28:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:28:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:28:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:28:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:28:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:28:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:28:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:28:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:28:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:28:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:28:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:28:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:28:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:28:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:28:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:28:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:28:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:28:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:28:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:28:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:28:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:28:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:28:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:28:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:28:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:28:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:28:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:28:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:28:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:28:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:28:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:28:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:28:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:28:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:28:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:28:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:28:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:28:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:28:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:28:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:28:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:28:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:28:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:28:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:28:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:28:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:28:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:28:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:28:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:28:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:28:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:28:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:28:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:28:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:28:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:28:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:28:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:28:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:28:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:28:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:28:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:28:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:28:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:28:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:28:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:28:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:28:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:28:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:28:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:28:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:28:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:28:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:28:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:28:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:28:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:28:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:28:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:28:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:28:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:28:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:28:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:28:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:28:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:28:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:28:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:28:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:28:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:28:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:28:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:28:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:28:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:28:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:28:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:28:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:28:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:28:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:28:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:28:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:28:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:28:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:28:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:28:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:28:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:28:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:28:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:28:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:29:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:29:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:29:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:29:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:29:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:29:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:29:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:29:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:29:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:29:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:29:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:29:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:29:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:29:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:29:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:29:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:29:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:29:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:29:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:29:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:29:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:29:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:29:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:29:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:29:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:29:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:29:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:29:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:29:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:29:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:29:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:29:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:29:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:29:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:29:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:29:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:29:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:29:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:29:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:29:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:29:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:29:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:29:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:29:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:29:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:29:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:29:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:29:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:29:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:29:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:29:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:29:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:29:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:29:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:29:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:29:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:29:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:29:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:29:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:29:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:29:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:29:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:29:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:29:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:29:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:29:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:29:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:29:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:29:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:29:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:29:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:29:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:29:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:29:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:29:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:29:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:29:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:29:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:29:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:29:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:29:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:29:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:29:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:29:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:29:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:29:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:29:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:29:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:29:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:29:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:29:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:29:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:29:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:29:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:29:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:29:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:29:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:29:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:29:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:29:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:29:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:29:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:29:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:29:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:29:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:29:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:29:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:29:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:29:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:29:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:29:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:29:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:29:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:29:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:29:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:29:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:29:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:29:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:29:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:29:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:29:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:29:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:29:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:29:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:29:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:29:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:29:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:29:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:29:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:29:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:29:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:29:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:29:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:29:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:29:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:29:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:29:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:29:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:29:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:29:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:29:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:29:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:29:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:29:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:29:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:29:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:29:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:29:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:29:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:29:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:29:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:29:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:29:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:29:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:29:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:29:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:29:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:29:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:29:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:29:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:29:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:29:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:29:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:29:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:29:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:29:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:29:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:29:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:29:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:29:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:29:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:29:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:29:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:29:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:29:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:29:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:29:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:29:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:29:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:29:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:29:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:29:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:29:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:29:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:29:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:29:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:29:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:29:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:29:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:29:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:29:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:29:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:29:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:29:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:29:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:29:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:29:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:29:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:29:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:29:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:29:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:29:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:29:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:29:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:29:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:29:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:29:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:29:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:29:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:29:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:29:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:29:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:29:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:29:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:29:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:29:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:30:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:30:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:30:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:30:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:30:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:30:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:30:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:30:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:30:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:30:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:30:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:30:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:30:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:30:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:30:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:30:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:30:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:30:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:30:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:30:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:30:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:30:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:30:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:30:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:30:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:30:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:30:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:30:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:30:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:30:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:30:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:30:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:30:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:30:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:30:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:30:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:30:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:30:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:30:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:30:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:30:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:30:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:30:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:30:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:30:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:30:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:30:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:30:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:30:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:30:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:30:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:30:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:30:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:30:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:30:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:30:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:30:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:30:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:30:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:30:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:30:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:30:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:30:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:30:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:30:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:30:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:30:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:30:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:30:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:30:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:30:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:30:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:30:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:30:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:30:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:30:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:30:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:30:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:30:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:30:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:30:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:30:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:30:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:30:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:30:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:30:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:30:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:30:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:30:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:30:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:30:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:30:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:30:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:30:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:30:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:30:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:30:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:30:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:30:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:30:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:30:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:30:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:30:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:30:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:30:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:30:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:30:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:30:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:30:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:30:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:30:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:30:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:30:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:30:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:30:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:30:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:30:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:30:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:30:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:30:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:30:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:30:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:30:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:30:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:30:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:30:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:30:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:30:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:30:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:30:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:30:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:30:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:30:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:30:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:30:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:30:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:30:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:30:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:30:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:30:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:30:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:30:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:30:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:30:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:30:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:30:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:30:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:30:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:30:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:30:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:30:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:30:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:30:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:30:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:30:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:30:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:30:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:30:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:30:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:30:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:30:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:30:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:30:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:30:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:30:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:30:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:30:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:30:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:30:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:30:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:30:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:30:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:30:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:30:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:30:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:30:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:30:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:30:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:30:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:30:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:30:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:30:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:30:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:30:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:30:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:30:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:30:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:30:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:30:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:30:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:30:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:30:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:30:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:30:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:30:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:30:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:30:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:30:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:30:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:30:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:30:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:30:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:30:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:30:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:30:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:30:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:30:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:30:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:30:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:30:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:30:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:30:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:30:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:30:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:30:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:30:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:31:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:31:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:31:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:31:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:31:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:31:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:31:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:31:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:31:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:31:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:31:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:31:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:31:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:31:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:31:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:31:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:31:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:31:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:31:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:31:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:31:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:31:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:31:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:31:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:31:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:31:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:31:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:31:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:31:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:31:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:31:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:31:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:31:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:31:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:31:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:31:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:31:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:31:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:31:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:31:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:31:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:31:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:31:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:31:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:31:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:31:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:31:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:31:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:31:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:31:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:31:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:31:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:31:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:31:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:31:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:31:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:31:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:31:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:31:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:31:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:31:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:31:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:31:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:31:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:31:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:31:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:31:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:31:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:31:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:31:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:31:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:31:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:31:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:31:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:31:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:31:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:31:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:31:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:31:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:31:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:31:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:31:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:31:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:31:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:31:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:31:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:31:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:31:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:31:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:31:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:31:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:31:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:31:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:31:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:31:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:31:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:31:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:31:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:31:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:31:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:31:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:31:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:31:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:31:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:31:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:31:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:31:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:31:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:31:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:31:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:31:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:31:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:31:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:31:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:31:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:31:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:31:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:31:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:31:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:31:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:31:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:31:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:31:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:31:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:31:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:31:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:31:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:31:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:31:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:31:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:31:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:31:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:31:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:31:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:31:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:31:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:31:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:31:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:31:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:31:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:31:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:31:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:31:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:31:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:31:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:31:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:31:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:31:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:31:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:31:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:31:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:31:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:31:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:31:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:31:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:31:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:31:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:31:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:31:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:31:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:31:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:31:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:31:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:31:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:31:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:31:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:31:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:31:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:31:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:31:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:31:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:31:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:31:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:31:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:31:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:31:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:31:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:31:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:31:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:31:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:31:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:31:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:31:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:31:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:31:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:31:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:31:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:31:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:31:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:31:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:31:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:31:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:31:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:31:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:31:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:31:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:31:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:31:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:31:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:31:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:31:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:31:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:31:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:31:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:31:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:31:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:31:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:31:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:31:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:31:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:31:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:31:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:31:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:31:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:31:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:31:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:32:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:32:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:32:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:32:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:32:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:32:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:32:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:32:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:32:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:32:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:32:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:32:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:32:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:32:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:32:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:32:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:32:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:32:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:32:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:32:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:32:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:32:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:32:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:32:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:32:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:32:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:32:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:32:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:32:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:32:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:32:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:32:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:32:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:32:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:32:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:32:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:32:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:32:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:32:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:32:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:32:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:32:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:32:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:32:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:32:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:32:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:32:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:32:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:32:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:32:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:32:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:32:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:32:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:32:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:32:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:32:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:32:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:32:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:32:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:32:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:32:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:32:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:32:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:32:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:32:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:32:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:32:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:32:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:32:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:32:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:32:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:32:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:32:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:32:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:32:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:32:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:32:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:32:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:32:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:32:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:32:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:32:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:32:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:32:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:32:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:32:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:32:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:32:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:32:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:32:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:32:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:32:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:32:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:32:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:32:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:32:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:32:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:32:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:32:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:32:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:32:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:32:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:32:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:32:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:32:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:32:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:32:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:32:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:32:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:32:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:32:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:32:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:32:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:32:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:32:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:32:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:32:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:32:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:32:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:32:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:32:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:32:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:32:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:32:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:32:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:32:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:32:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:32:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:32:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:32:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:32:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:32:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:32:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:32:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:32:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:32:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:32:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:32:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:32:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:32:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:32:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:32:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:32:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:32:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:32:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:32:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:32:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:32:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:32:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:32:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:32:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:32:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:32:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:32:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:32:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:32:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:32:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:32:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:32:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:32:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:32:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:32:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:32:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:32:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:32:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:32:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:32:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:32:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:32:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:32:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:32:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:32:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:32:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:32:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:32:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:32:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:32:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:32:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:32:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:32:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:34:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:34:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:34:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:34:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:34:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:34:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:34:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:34:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:34:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:34:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:34:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:34:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:34:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:34:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:34:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:34:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:34:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:34:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:34:55 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 10:35:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:35:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:35:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:35:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:35:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:35:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:35:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:35:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:35:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:35:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:35:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:35:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:35:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:35:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:35:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:35:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:35:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:35:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:35:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:35:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:35:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:35:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:35:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:35:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:35:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:35:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:35:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:35:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:35:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:35:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:35:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:35:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:35:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:35:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:35:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:35:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:35:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:35:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:35:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:35:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:35:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:35:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:35:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:35:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:35:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:35:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:35:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:35:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:35:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:35:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:35:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:35:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:35:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:35:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:35:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:35:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:35:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:35:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:35:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:35:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:35:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:35:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:35:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:35:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:35:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:35:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:35:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:35:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:35:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:35:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:35:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:35:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:35:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:35:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:35:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:35:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:35:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:35:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:35:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:35:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:35:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:35:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:35:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:35:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:35:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:35:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:35:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:35:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:35:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:35:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:35:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:35:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:35:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:35:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:35:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:35:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:35:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:35:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:35:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:35:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:35:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:35:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:35:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:35:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:35:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:35:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:35:25 rdma06 systemd-journald[790]: Data hash table of /run/log/journal/95379d2f723847b68d9e976dee8967a9/system.journal has a fill level at 75.0 (163751 of 218332 items, 75497472 file size, 461 bytes per hash table item), suggesting rotation.
Apr 18 10:35:25 rdma06 systemd-journald[790]: /run/log/journal/95379d2f723847b68d9e976dee8967a9/system.journal: Journal header limits reached or header out-of-date, rotating.
Apr 18 10:35:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:35:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:35:25 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 18 10:35:25 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 18 10:35:25 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 18 10:35:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:35:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:35:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:35:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:35:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:35:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:35:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:35:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:35:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:35:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:35:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:35:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:35:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:35:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:35:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:35:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:35:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:35:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:35:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:35:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:35:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:35:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:35:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:35:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:35:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:35:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:35:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:35:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:35:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:35:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:35:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:35:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:35:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:35:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:35:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:35:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:35:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:35:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:35:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:35:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:35:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:35:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:35:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:35:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:35:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:35:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:35:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:35:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:35:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:35:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:35:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:35:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:35:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:35:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:35:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:35:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:35:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:35:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:35:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:35:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:35:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:35:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:35:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:35:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:35:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:35:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:35:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:35:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:35:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:35:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:35:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:35:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:35:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:35:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:35:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:35:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:35:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:35:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:35:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:35:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:35:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:35:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:35:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:35:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:35:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:35:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:35:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:35:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:35:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:35:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:35:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:35:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:35:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:35:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:35:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:35:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:35:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:35:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:35:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:35:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:35:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:35:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:35:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:35:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:35:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:35:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:35:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:35:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:36:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:36:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:36:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:36:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:36:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:36:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:36:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:36:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:36:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:36:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:36:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:36:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:36:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:36:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:36:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:36:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:36:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:36:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:36:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:36:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:36:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:36:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:36:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:36:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:36:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:36:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:36:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:36:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:36:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:36:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:36:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:36:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:36:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:36:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:36:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:36:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:36:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:36:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:36:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:36:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:36:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:36:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:36:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:36:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:36:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:36:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:36:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:36:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:36:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:36:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:36:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:36:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:36:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:36:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:36:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:36:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:36:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:36:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:36:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:36:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:36:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:36:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:36:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:36:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:36:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:36:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:36:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:36:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:36:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:36:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:36:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:36:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:36:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:36:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:36:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:36:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:36:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:36:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:36:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:36:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:36:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:36:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:36:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:36:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:36:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:36:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:36:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:36:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:36:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:36:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:36:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:36:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:36:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:36:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:36:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:36:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:36:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:36:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:36:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:36:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:36:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:36:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:36:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:36:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:36:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:36:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:36:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:36:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:36:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:36:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:36:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:36:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:36:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:36:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:36:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:36:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:36:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:36:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:36:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:36:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:36:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:36:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:36:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:36:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:36:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:36:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:36:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:36:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:36:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:36:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:36:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:36:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:36:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:36:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:36:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:36:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:36:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:36:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:36:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:36:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:36:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:36:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:36:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:36:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:36:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:36:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:36:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:36:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:36:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:36:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:36:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:36:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:36:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:36:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:36:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:36:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:36:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:36:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:36:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:36:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:36:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:36:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:36:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:36:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:36:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:36:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:36:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:36:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:36:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:36:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:36:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:36:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:36:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:36:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:36:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:36:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:36:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:36:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:36:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:36:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:36:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:36:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:36:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:36:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:36:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:36:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:36:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:36:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:36:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:36:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:36:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:36:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:36:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:36:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:36:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:36:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:36:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:36:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:36:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:36:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:36:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:36:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:36:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:36:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:36:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:36:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:36:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:36:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:36:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:36:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:36:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:36:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:36:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:36:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:36:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:36:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:37:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:37:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:37:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:37:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:37:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:37:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:37:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:37:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:37:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:37:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:37:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:37:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:37:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:37:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:37:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:37:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:37:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:37:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:37:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:37:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:37:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:37:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:37:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:37:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:37:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:37:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:37:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:37:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:37:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:37:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:37:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:37:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:37:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:37:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:37:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:37:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:37:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:37:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:37:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:37:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:37:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:37:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:37:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:37:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:37:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:37:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:37:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:37:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:37:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:37:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:37:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:37:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:37:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:37:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:37:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:37:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:37:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:37:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:37:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:37:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:37:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:37:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:37:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:37:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:37:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:37:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:37:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:37:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:37:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:37:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:37:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:37:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:37:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:37:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:37:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:37:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:37:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:37:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:37:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:37:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:37:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:37:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:37:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:37:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:37:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:37:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:37:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:37:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:37:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:37:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:37:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:37:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:37:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:37:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:37:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:37:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:37:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:37:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:37:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:37:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:37:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:37:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:37:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:37:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:37:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:37:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:37:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:37:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:37:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:37:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:37:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:37:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:37:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:37:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:37:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:37:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:37:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:37:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:37:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:37:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:37:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:37:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:37:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:37:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:37:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:37:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:37:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:37:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:37:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:37:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:37:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:37:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:37:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:37:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:37:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:37:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:37:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:37:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:37:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:37:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:37:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:37:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:37:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:37:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:37:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:37:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:37:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:37:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:37:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:37:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:37:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:37:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:37:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:37:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:37:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:37:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:37:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:37:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:37:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:37:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:37:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:37:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:37:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:37:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:37:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:37:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:37:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:37:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:37:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:37:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:37:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:37:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:37:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:37:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:37:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:37:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:37:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:37:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:37:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:37:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:37:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:37:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:37:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:37:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:37:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:37:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:37:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:37:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:37:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:37:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:37:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:37:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:37:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:37:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:37:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:37:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:37:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:37:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:37:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:37:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:37:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:37:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:37:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:37:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:37:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:37:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:37:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:37:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:37:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:37:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:37:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:37:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:37:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:37:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:37:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:37:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:38:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:38:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:38:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:38:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:38:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:38:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:38:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:38:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:38:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:38:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:38:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:38:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:38:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:38:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:38:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:38:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:38:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:38:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:38:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:38:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:38:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:38:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:38:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:38:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:38:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:38:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:38:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:38:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:38:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:38:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:38:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:38:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:38:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:38:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:38:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:38:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:38:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:38:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:38:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:38:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:38:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:38:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:38:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:38:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:38:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:38:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:38:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:38:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:38:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:38:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:38:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:38:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:38:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:38:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:38:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:38:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:38:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:38:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:38:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:38:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:38:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:38:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:38:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:38:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:38:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:38:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:38:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:38:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:38:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:38:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:38:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:38:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:38:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:38:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:38:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:38:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:38:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:38:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:38:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:38:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:38:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:38:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:38:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:38:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:38:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:38:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:38:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:38:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:38:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:38:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:38:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:38:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:38:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:38:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:38:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:38:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:38:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:38:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:38:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:38:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:38:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:38:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:38:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:38:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:38:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:38:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:38:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:38:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:38:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:38:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:38:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:38:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:38:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:38:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:38:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:38:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:38:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:38:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:38:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:38:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:38:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:38:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:38:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:38:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:38:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:38:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:38:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:38:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:38:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:38:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:38:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:38:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:38:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:38:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:38:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:38:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:38:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:38:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:38:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:38:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:38:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:38:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:38:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:38:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:38:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:38:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:38:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:38:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:38:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:38:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:38:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:38:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:38:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:38:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:38:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:38:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:38:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:38:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:38:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:38:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:38:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:38:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:38:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:38:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:38:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:38:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:38:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:38:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:38:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:38:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:38:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:38:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:38:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:38:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:38:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:38:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:38:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:38:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:38:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:38:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:38:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:38:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:38:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:38:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:38:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:38:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:38:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:38:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:38:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:38:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:38:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:38:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:38:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:38:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:38:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:38:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:38:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:38:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:38:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:38:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:38:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:38:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:38:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:38:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:38:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:38:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:38:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:38:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:38:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:38:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:38:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:38:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:38:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:38:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:38:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:38:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:39:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:39:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:39:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:39:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:39:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:39:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:39:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:39:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:39:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:39:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:39:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:39:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:39:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:39:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:39:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:39:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:39:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:39:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:39:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:39:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:39:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:39:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:39:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:39:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:39:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:39:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:39:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:39:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:39:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:39:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:39:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:39:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:39:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:39:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:39:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:39:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:39:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:39:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:39:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:39:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:39:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:39:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:39:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:39:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:39:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:39:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:39:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:39:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:39:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:39:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:39:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:39:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:39:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:39:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:39:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:39:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:39:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:39:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:39:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:39:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:39:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:39:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:39:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:39:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:39:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:39:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:39:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:39:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:39:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:39:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:39:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:39:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:39:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:39:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:39:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:39:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:39:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:39:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:39:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:39:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:39:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:39:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:39:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:39:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:39:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:39:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:39:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:39:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:39:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:39:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:39:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:39:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:39:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:39:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:39:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:39:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:39:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:39:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:39:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:39:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:39:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:39:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:39:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:39:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:39:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:39:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:39:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:39:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:39:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:39:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:39:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:39:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:39:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:39:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:39:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:39:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:39:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:39:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:39:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:39:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:39:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:39:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:39:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:39:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:39:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:39:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:39:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:39:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:39:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:39:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:39:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:39:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:39:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:39:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:39:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:39:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:39:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:39:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:39:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:39:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:39:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:39:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:39:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:39:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:39:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:39:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:39:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:39:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:39:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:39:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:39:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:39:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:39:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:39:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:39:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:39:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:39:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:39:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:39:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:39:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:39:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:39:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:39:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:39:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:39:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:39:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:39:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:39:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:39:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:39:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:39:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:39:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:39:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:39:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:39:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:39:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:39:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:39:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:39:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:39:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:39:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:39:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:39:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:39:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:39:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:39:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:39:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:39:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:39:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:39:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:39:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:39:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:39:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:39:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:39:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:39:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:39:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:39:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:39:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:39:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:39:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:39:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:39:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:39:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:39:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:39:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:39:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:39:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:39:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:39:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:39:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:39:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:39:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:39:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:39:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:39:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:40:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:40:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:40:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:40:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:40:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:40:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:40:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:40:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:40:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:40:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:40:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:40:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:40:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:40:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:40:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:40:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:40:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:40:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:40:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:40:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:40:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:40:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:40:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:40:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:40:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:40:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:40:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:40:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:40:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:40:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:40:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:40:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:40:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:40:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:40:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:40:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:40:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:40:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:40:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:40:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:40:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:40:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:40:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:40:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:40:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:40:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:40:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:40:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:40:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:40:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:40:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:40:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:40:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:40:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:40:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:40:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:40:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:40:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:40:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:40:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:40:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:40:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:40:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:40:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:40:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:40:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:40:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:40:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:40:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:40:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:40:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:40:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:40:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:40:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:40:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:40:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:40:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:40:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:40:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:40:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:40:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:40:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:40:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:40:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:40:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:40:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:40:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:40:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:40:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:40:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:40:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:40:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:40:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:40:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:40:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:40:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:40:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:40:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:40:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:40:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:40:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:40:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:40:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:40:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:40:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:40:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:40:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:40:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:40:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:40:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:40:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:40:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:40:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:40:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:40:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:40:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:40:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:40:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:40:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:40:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:40:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:40:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:40:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:40:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:40:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:40:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:40:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:40:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:40:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:40:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:40:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:40:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:40:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:40:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:40:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:40:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:40:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:40:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:40:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:40:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:40:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:40:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:40:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:40:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:40:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:40:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:40:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:40:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:40:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:40:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:40:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:40:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:40:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:40:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:40:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:40:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:40:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:40:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:40:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:40:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:40:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:40:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:40:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:40:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:40:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:40:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:40:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:40:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:40:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:40:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:40:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:40:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:40:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:40:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:40:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:40:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:40:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:40:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:40:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:40:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:40:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:40:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:40:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:40:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:40:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:40:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:40:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:40:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:40:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:40:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:40:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:40:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:40:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:40:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:40:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:40:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:40:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:40:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:40:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:40:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:40:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:40:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:40:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:40:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:40:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:40:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:40:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:40:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:40:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:40:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:40:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:40:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:40:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:40:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:40:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:40:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:41:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:41:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:41:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:41:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:41:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:41:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:41:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:41:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:41:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:41:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:41:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:41:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:41:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:41:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:41:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:41:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:41:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:41:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:41:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:41:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:41:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:41:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:41:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:41:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:41:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:41:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:41:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:41:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:41:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:41:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:41:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:41:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:41:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:41:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:41:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:41:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:41:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:41:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:41:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:41:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:41:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:41:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:41:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:41:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:41:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:41:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:41:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:41:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:41:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:41:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:41:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:41:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:41:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:41:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:41:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:41:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:41:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:41:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:41:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:41:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:41:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:41:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:41:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:41:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:41:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:41:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:41:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:41:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:41:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:41:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:41:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:41:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:41:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:41:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:41:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:41:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:41:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:41:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:41:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:41:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:41:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:41:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:41:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:41:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:41:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:41:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:41:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:41:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:41:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:41:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:41:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:41:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:41:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:41:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:41:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:41:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:41:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:41:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:41:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:41:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:41:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:41:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:41:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:41:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:41:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:41:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:41:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:41:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:41:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:41:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:41:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:41:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:41:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:41:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:41:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:41:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:41:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:41:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:41:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:41:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:41:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:41:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:41:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:41:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:41:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:41:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:41:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:41:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:41:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:41:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:41:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:41:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:41:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:41:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:41:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:41:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:41:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:41:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:41:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:41:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:41:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:41:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:41:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:41:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:41:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:41:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:41:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:41:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:41:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:41:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:41:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:41:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:41:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:41:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:41:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:41:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:41:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:41:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:41:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:41:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:41:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:41:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:41:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:41:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:41:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:41:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:41:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:41:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:41:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:41:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:41:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:41:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:41:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:41:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:41:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:41:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:41:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:41:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:41:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:41:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:41:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:41:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:41:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:41:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:41:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:41:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:41:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:41:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:41:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:41:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:41:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:41:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:41:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:41:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:41:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:41:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:41:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:41:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:41:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:41:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:41:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:41:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:41:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:41:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:41:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:41:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:41:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:41:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:41:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:41:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:41:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:41:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:41:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:41:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:41:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:41:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:42:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:42:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:42:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:42:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:42:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:42:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:42:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:42:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:42:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:42:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:42:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:42:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:42:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:42:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:42:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:42:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:42:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:42:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:42:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:42:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:42:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:42:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:42:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:42:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:42:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:42:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:42:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:42:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:42:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:42:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:42:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:42:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:42:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:42:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:42:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:42:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:42:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:42:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:42:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:42:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:42:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:42:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:42:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:42:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:42:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:42:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:42:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:42:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:42:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:42:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:42:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:42:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:42:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:42:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:42:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:42:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:42:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:42:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:42:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:42:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:42:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:42:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:42:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:42:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:42:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:42:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:42:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:42:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:42:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:42:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:42:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:42:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:42:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:42:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:42:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:42:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:42:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:42:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:42:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:42:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:42:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:42:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:42:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:42:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:42:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:42:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:42:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:42:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:42:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:42:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:42:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:42:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:42:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:42:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:42:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:42:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:42:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:42:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:42:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:42:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:42:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:42:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:42:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:42:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:42:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:42:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:42:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:42:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:42:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:42:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:42:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:42:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:42:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:42:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:42:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:42:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:42:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:42:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:42:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:42:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:42:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:42:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:42:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:42:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:42:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:42:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:42:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:42:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:42:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:42:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:42:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:42:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:42:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:42:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:42:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:42:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:42:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:42:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:42:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:42:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:42:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:42:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:42:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:42:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:42:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:42:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:42:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:42:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:42:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:42:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:42:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:42:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:42:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:42:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:42:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:42:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:42:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:42:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:42:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:42:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:42:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:42:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:42:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:42:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:42:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:42:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:42:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:42:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:42:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:42:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:42:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:42:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:42:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:42:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:42:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:42:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:42:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:42:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:42:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:42:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:42:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:42:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:42:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:42:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:42:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:42:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:42:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:42:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:42:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:42:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:42:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:42:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:42:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:42:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:42:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:42:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:42:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:42:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:44:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:44:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:44:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:44:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:44:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:44:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:44:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:44:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:44:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:44:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:44:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:44:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:44:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:44:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:44:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:44:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:44:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:44:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:44:56 rdma06 rsyslogd[1041]: imjournal: 5087 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 10:45:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:45:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:45:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:45:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:45:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:45:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:45:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:45:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:45:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:45:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:45:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:45:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:45:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:45:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:45:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:45:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:45:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:45:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:45:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:45:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:45:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:45:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:45:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:45:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:45:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:45:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:45:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:45:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:45:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:45:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:45:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:45:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:45:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:45:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:45:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:45:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:45:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:45:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:45:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:45:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:45:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:45:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:45:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:45:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:45:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:45:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:45:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:45:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:45:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:45:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:45:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:45:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:45:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:45:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:45:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:45:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:45:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:45:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:45:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:45:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:45:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:45:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:45:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:45:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:45:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:45:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:45:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:45:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:45:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:45:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:45:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:45:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:45:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:45:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:45:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:45:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:45:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:45:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:45:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:45:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:45:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:45:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:45:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:45:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:45:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:45:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:45:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:45:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:45:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:45:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:45:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:45:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:45:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:45:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:45:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:45:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:45:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:45:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:45:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:45:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:45:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:45:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:45:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:45:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:45:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:45:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:45:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:45:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:45:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:45:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:45:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:45:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:45:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:45:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:45:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:45:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:45:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:45:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:45:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:45:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:45:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:45:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:45:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:45:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:45:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:45:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:45:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:45:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:45:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:45:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:45:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:45:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:45:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:45:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:45:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:45:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:45:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:45:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:45:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:45:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:45:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:45:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:45:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:45:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:45:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:45:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:45:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:45:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:45:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:45:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:45:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:45:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:45:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:45:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:45:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:45:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:45:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:45:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:45:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:45:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:45:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:45:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:45:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:45:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:45:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:45:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:45:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:45:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:45:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:45:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:45:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:45:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:45:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:45:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:45:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:45:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:45:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:45:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:45:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:45:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:45:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:45:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:45:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:45:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:45:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:45:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:45:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:45:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:45:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:45:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:45:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:45:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:45:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:45:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:45:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:45:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:45:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:45:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:45:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:45:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:45:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:45:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:45:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:45:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:45:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:45:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:45:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:45:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:45:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:45:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:45:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:45:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:45:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:45:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:45:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:45:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:46:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:46:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:46:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:46:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:46:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:46:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:46:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:46:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:46:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:46:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:46:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:46:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:46:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:46:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:46:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:46:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:46:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:46:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:46:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:46:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:46:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:46:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:46:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:46:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:46:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:46:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:46:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:46:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:46:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:46:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:46:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:46:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:46:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:46:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:46:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:46:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:46:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:46:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:46:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:46:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:46:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:46:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:46:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:46:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:46:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:46:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:46:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:46:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:46:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:46:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:46:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:46:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:46:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:46:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:46:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:46:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:46:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:46:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:46:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:46:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:46:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:46:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:46:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:46:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:46:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:46:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:46:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:46:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:46:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:46:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:46:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:46:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:46:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:46:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:46:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:46:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:46:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:46:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:46:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:46:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:46:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:46:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:46:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:46:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:46:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:46:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:46:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:46:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:46:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:46:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:46:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:46:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:46:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:46:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:46:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:46:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:46:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:46:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:46:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:46:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:46:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:46:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:46:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:46:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:46:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:46:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:46:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:46:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:46:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:46:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:46:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:46:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:46:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:46:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:46:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:46:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:46:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:46:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:46:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:46:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:46:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:46:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:46:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:46:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:46:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:46:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:46:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:46:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:46:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:46:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:46:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:46:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:46:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:46:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:46:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:46:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:46:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:46:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:46:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:46:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:46:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:46:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:46:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:46:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:46:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:46:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:46:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:46:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:46:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:46:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:46:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:46:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:46:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:46:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:46:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:46:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:46:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:46:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:46:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:46:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:46:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:46:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:46:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:46:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:46:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:46:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:46:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:46:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:46:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:46:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:46:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:46:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:46:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:46:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:46:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:46:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:46:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:46:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:46:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:46:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:46:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:46:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:46:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:46:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:46:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:46:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:46:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:46:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:46:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:46:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:46:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:46:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:46:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:46:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:46:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:46:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:46:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:46:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:46:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:46:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:46:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:46:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:46:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:46:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:46:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:46:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:46:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:46:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:46:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:46:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:46:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:46:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:46:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:46:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:46:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:46:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:47:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:47:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:47:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:47:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:47:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:47:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:47:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:47:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:47:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:47:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:47:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:47:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:47:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:47:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:47:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:47:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:47:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:47:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:47:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:47:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:47:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:47:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:47:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:47:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:47:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:47:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:47:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:47:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:47:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:47:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:47:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:47:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:47:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:47:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:47:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:47:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:47:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:47:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:47:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:47:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:47:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:47:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:47:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:47:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:47:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:47:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:47:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:47:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:47:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:47:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:47:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:47:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:47:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:47:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:47:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:47:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:47:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:47:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:47:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:47:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:47:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:47:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:47:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:47:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:47:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:47:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:47:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:47:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:47:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:47:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:47:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:47:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:47:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:47:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:47:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:47:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:47:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:47:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:47:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:47:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:47:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:47:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:47:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:47:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:47:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:47:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:47:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:47:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:47:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:47:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:47:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:47:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:47:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:47:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:47:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:47:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:47:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:47:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:47:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:47:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:47:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:47:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:47:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:47:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:47:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:47:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:47:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:47:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:47:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:47:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:47:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:47:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:47:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:47:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:47:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:47:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:47:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:47:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:47:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:47:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:47:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:47:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:47:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:47:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:47:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:47:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:47:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:47:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:47:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:47:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:47:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:47:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:47:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:47:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:47:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:47:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:47:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:47:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:47:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:47:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:47:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:47:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:47:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:47:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:47:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:47:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:47:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:47:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:47:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:47:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:47:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:47:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:47:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:47:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:47:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:47:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:47:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:47:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:47:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:47:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:47:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:47:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:47:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:47:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:47:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:47:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:47:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:47:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:47:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:47:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:47:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:47:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:47:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:47:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:47:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:47:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:47:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:47:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:47:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:47:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:47:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:47:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:47:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:47:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:47:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:47:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:47:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:47:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:47:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:47:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:47:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:47:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:47:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:47:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:47:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:47:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:47:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:47:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:47:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:47:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:47:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:47:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:47:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:47:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:47:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:47:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:47:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:47:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:47:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:47:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:47:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:47:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:47:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:47:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:47:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:47:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:48:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:48:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:48:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:48:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:48:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:48:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:48:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:48:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:48:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:48:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:48:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:48:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:48:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:48:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:48:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:48:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:48:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:48:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:48:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:48:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:48:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:48:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:48:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:48:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:48:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:48:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:48:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:48:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:48:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:48:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:48:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:48:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:48:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:48:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:48:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:48:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:48:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:48:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:48:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:48:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:48:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:48:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:48:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:48:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:48:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:48:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:48:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:48:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:48:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:48:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:48:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:48:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:48:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:48:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:48:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:48:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:48:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:48:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:48:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:48:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:48:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:48:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:48:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:48:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:48:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:48:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:48:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:48:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:48:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:48:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:48:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:48:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:48:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:48:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:48:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:48:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:48:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:48:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:48:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:48:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:48:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:48:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:48:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:48:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:48:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:48:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:48:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:48:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:48:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:48:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:48:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:48:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:48:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:48:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:48:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:48:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:48:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:48:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:48:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:48:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:48:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:48:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:48:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:48:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:48:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:48:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:48:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:48:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:48:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:48:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:48:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:48:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:48:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:48:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:48:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:48:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:48:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:48:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:48:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:48:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:48:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:48:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:48:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:48:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:48:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:48:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:48:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:48:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:48:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:48:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:48:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:48:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:48:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:48:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:48:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:48:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:48:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:48:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:48:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:48:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:48:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:48:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:48:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:48:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:48:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:48:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:48:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:48:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:48:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:48:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:48:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:48:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:48:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:48:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:48:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:48:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:48:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:48:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:48:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:48:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:48:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:48:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:48:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:48:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:48:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:48:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:48:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:48:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:48:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:48:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:48:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:48:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:48:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:48:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:48:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:48:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:48:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:48:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:48:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:48:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:48:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:48:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:48:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:48:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:48:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:48:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:48:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:48:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:48:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:48:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:48:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:48:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:48:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:48:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:48:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:48:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:48:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:48:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:48:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:48:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:48:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:48:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:48:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:48:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:48:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:48:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:48:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:48:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:48:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:48:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:48:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:48:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:48:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:48:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:48:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:48:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:49:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:49:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:49:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:49:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:49:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:49:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:49:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:49:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:49:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:49:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:49:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:49:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:49:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:49:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:49:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:49:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:49:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:49:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:49:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:49:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:49:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:49:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:49:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:49:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:49:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:49:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:49:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:49:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:49:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:49:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:49:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:49:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:49:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:49:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:49:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:49:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:49:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:49:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:49:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:49:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:49:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:49:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:49:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:49:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:49:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:49:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:49:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:49:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:49:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:49:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:49:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:49:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:49:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:49:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:49:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:49:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:49:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:49:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:49:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:49:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:49:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:49:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:49:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:49:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:49:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:49:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:49:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:49:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:49:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:49:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:49:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:49:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:49:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:49:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:49:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:49:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:49:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:49:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:49:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:49:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:49:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:49:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:49:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:49:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:49:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:49:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:49:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:49:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:49:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:49:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:49:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:49:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:49:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:49:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:49:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:49:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:49:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:49:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:49:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:49:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:49:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:49:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:49:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:49:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:49:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:49:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:49:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:49:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:49:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:49:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:49:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:49:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:49:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:49:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:49:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:49:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:49:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:49:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:49:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:49:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:49:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:49:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:49:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:49:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:49:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:49:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:49:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:49:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:49:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:49:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:49:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:49:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:49:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:49:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:49:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:49:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:49:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:49:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:49:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:49:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:49:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:49:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:49:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:49:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:49:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:49:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:49:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:49:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:49:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:49:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:49:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:49:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:49:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:49:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:49:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:49:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:49:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:49:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:49:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:49:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:49:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:49:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:49:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:49:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:49:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:49:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:49:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:49:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:49:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:49:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:49:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:49:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:49:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:49:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:49:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:49:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:49:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:49:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:49:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:49:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:49:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:49:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:49:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:49:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:49:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:49:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:49:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:49:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:49:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:49:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:49:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:49:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:49:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:49:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:49:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:49:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:49:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:49:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:49:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:49:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:49:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:49:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:49:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:49:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:49:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:49:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:49:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:49:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:49:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:49:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:49:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:49:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:49:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:49:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:49:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:49:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:50:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:50:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:50:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:50:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:50:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:50:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:50:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:50:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:50:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:50:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:50:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:50:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:50:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:50:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:50:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:50:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:50:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:50:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:50:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:50:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:50:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:50:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:50:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:50:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:50:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:50:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:50:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:50:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:50:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:50:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:50:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:50:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:50:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:50:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:50:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:50:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:50:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:50:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:50:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:50:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:50:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:50:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:50:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:50:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:50:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:50:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:50:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:50:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:50:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:50:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:50:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:50:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:50:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:50:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:50:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:50:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:50:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:50:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:50:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:50:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:50:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:50:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:50:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:50:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:50:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:50:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:50:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:50:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:50:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:50:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:50:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:50:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:50:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:50:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:50:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:50:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:50:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:50:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:50:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:50:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:50:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:50:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:50:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:50:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:50:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:50:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:50:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:50:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:50:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:50:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:50:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:50:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:50:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:50:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:50:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:50:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:50:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:50:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:50:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:50:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:50:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:50:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:50:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:50:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:50:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:50:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:50:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:50:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:50:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:50:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:50:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:50:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:50:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:50:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:50:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:50:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:50:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:50:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:50:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:50:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:50:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:50:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:50:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:50:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:50:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:50:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:50:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:50:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:50:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:50:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:50:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:50:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:50:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:50:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:50:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:50:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:50:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:50:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:50:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:50:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:50:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:50:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:50:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:50:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:50:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:50:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:50:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:50:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:50:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:50:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:50:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:50:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:50:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:50:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:50:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:50:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:50:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:50:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:50:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:50:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:50:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:50:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:50:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:50:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:50:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:50:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:50:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:50:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:50:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:50:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:50:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:50:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:50:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:50:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:50:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:50:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:50:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:50:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:50:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:50:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:50:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:50:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:50:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:50:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:50:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:50:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:50:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:50:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:50:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:50:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:50:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:50:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:50:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:50:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:50:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:50:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:50:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:50:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:50:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:50:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:50:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:50:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:50:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:50:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:50:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:50:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:50:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:50:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:50:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:50:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:50:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:50:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:50:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:50:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:50:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:50:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:51:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:51:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:51:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:51:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:51:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:51:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:51:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:51:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:51:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:51:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:51:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:51:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:51:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:51:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:51:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:51:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:51:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:51:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:51:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:51:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:51:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:51:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:51:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:51:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:51:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:51:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:51:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:51:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:51:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:51:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:51:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:51:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:51:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:51:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:51:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:51:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:51:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:51:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:51:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:51:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:51:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:51:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:51:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:51:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:51:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:51:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:51:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:51:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:51:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:51:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:51:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:51:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:51:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:51:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:51:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:51:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:51:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:51:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:51:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:51:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:51:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:51:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:51:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:51:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:51:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:51:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:51:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:51:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:51:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:51:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:51:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:51:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:51:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:51:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:51:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:51:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:51:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:51:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:51:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:51:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:51:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:51:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:51:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:51:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:51:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:51:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:51:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:51:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:51:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:51:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:51:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:51:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:51:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:51:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:51:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:51:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:51:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:51:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:51:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:51:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:51:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:51:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:51:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:51:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:51:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:51:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:51:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:51:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:51:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:51:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:51:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:51:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:51:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:51:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:51:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:51:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:51:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:51:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:51:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:51:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:51:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:51:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:51:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:51:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:51:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:51:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:51:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:51:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:51:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:51:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:51:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:51:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:51:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:51:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:51:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:51:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:51:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:51:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:51:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:51:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:51:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:51:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:51:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:51:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:51:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:51:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:51:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:51:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:51:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:51:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:51:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:51:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:51:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:51:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:51:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:51:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:51:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:51:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:51:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:51:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:51:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:51:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:51:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:51:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:51:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:51:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:51:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:51:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:51:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:51:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:51:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:51:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:51:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:51:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:51:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:51:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:51:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:51:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:51:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:51:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:51:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:51:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:51:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:51:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:51:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:51:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:52:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:52:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:52:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:52:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:52:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:52:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:52:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:52:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:52:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:52:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:52:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:52:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:52:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:52:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:52:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:52:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:52:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:52:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:52:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:52:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:52:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:52:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:52:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:52:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:52:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:52:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:52:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:52:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:52:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:52:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:52:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:52:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:52:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:52:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:52:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:52:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:52:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:52:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:52:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:52:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:52:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:52:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:52:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:52:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:52:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:52:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:52:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:52:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:52:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:52:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:52:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:52:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:52:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:52:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:52:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:52:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:52:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:52:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:52:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:52:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:52:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:52:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:52:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:52:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:52:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:52:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:52:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:52:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:52:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:52:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:52:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:52:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:52:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:52:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:52:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:52:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:52:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:52:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:52:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:52:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:52:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:52:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:52:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:52:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:52:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:52:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:52:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:52:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:52:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:52:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:52:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:52:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:52:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:52:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:52:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:52:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:52:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:52:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:52:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:52:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:52:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:52:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:52:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:52:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:52:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:52:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:52:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:52:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:52:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:52:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:52:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:52:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:52:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:52:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:52:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:52:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:52:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:52:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:52:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:52:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:52:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:52:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:52:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:52:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:52:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:52:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:52:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:52:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:52:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:52:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:52:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:52:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:52:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:52:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:52:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:52:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:52:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:52:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:52:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:52:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:52:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:52:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:52:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:52:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:52:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:52:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:52:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:52:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:52:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:52:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:52:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:52:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:52:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:52:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:52:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:52:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:52:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:52:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:52:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:52:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:52:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:52:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:52:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:52:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:52:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:52:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:52:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:52:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:52:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:52:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:52:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:52:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:52:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:52:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:52:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:52:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:52:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:52:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:52:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:52:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:52:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:52:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:52:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:52:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:52:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:52:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:52:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:52:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:52:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:52:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:52:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:52:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:52:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:52:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:52:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:52:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:52:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:52:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:55:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:55:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:55:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:55:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:55:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:55:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:55:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:55:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:55:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:55:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:55:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:55:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:55:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:55:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:55:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:55:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:55:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:55:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:55:01 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 10:55:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:55:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:55:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:55:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:55:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:55:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:55:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:55:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:55:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:55:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:55:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:55:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:55:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:55:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:55:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:55:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:55:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:55:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:55:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:55:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:55:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:55:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:55:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:55:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:55:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:55:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:55:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:55:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:55:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:55:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:55:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:55:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:55:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:55:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:55:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:55:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:55:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:55:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:55:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:55:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:55:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:55:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:55:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:55:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:55:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:55:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:55:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:55:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:55:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:55:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:55:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:55:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:55:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:55:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:55:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:55:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:55:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:55:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:55:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:55:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:55:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:55:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:55:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:55:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:55:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:55:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:55:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:55:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:55:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:55:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:55:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:55:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:55:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:55:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:55:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:55:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:55:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:55:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:55:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:55:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:55:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:55:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:55:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:55:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:55:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:55:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:55:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:55:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:55:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:55:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:55:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:55:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:55:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:55:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:55:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:55:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:55:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:55:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:55:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:55:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:55:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:55:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:55:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:55:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:55:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:55:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:55:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:55:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:55:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:55:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:55:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:55:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:55:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:55:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:55:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:55:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:55:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:55:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:55:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:55:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:55:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:55:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:55:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:55:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:55:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:55:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:55:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:55:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:55:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:55:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:55:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:55:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:55:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:55:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:55:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:55:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:55:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:55:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:55:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:55:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:55:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:55:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:55:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:55:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:55:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:55:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:55:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:55:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:55:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:55:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:55:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:55:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:55:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:55:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:55:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:55:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:55:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:55:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:55:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:55:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:55:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:55:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:55:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:55:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:55:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:55:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:55:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:55:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:55:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:55:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:55:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:55:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:55:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:55:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:55:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:55:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:55:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:55:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:55:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:55:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:55:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:55:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:55:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:55:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:55:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:55:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:55:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:55:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:55:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:55:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:55:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:55:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:55:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:55:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:55:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:55:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:55:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:55:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:56:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:56:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:56:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:56:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:56:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:56:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:56:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:56:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:56:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:56:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:56:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:56:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:56:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:56:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:56:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:56:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:56:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:56:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:56:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:56:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:56:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:56:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:56:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:56:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:56:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:56:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:56:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:56:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:56:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:56:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:56:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:56:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:56:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:56:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:56:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:56:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:56:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:56:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:56:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:56:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:56:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:56:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:56:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:56:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:56:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:56:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:56:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:56:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:56:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:56:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:56:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:56:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:56:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:56:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:56:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:56:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:56:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:56:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:56:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:56:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:56:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:56:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:56:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:56:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:56:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:56:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:56:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:56:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:56:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:56:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:56:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:56:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:56:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:56:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:56:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:56:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:56:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:56:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:56:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:56:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:56:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:56:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:56:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:56:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:56:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:56:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:56:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:56:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:56:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:56:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:56:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:56:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:56:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:56:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:56:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:56:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:56:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:56:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:56:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:56:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:56:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:56:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:56:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:56:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:56:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:56:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:56:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:56:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:56:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:56:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:56:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:56:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:56:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:56:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:56:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:56:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:56:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:56:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:56:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:56:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:56:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:56:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:56:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:56:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:56:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:56:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:56:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:56:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:56:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:56:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:56:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:56:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:56:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:56:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:56:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:56:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:56:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:56:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:56:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:56:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:56:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:56:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:56:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:56:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:56:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:56:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:56:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:56:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:56:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:56:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:56:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:56:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:56:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:56:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:56:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:56:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:56:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:56:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:56:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:56:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:56:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:56:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:56:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:56:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:56:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:56:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:56:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:56:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:56:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:56:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:56:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:56:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:56:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:56:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:56:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:56:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:56:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:56:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:56:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:56:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:56:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:56:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:56:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:56:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:56:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:56:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:56:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:56:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:56:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:56:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:56:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:56:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:56:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:56:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:56:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:56:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:56:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:56:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:56:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:56:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:56:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:56:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:56:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:56:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:56:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:56:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:56:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:56:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:56:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:56:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:56:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:56:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:56:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:56:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:56:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:56:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:57:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:57:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:57:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:57:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:57:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:57:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:57:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:57:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:57:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:57:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:57:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:57:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:57:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:57:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:57:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:57:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:57:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:57:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:57:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:57:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:57:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:57:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:57:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:57:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:57:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:57:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:57:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:57:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:57:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:57:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:57:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:57:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:57:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:57:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:57:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:57:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:57:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:57:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:57:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:57:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:57:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:57:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:57:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:57:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:57:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:57:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:57:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:57:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:57:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:57:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:57:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:57:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:57:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:57:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:57:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:57:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:57:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:57:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:57:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:57:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:57:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:57:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:57:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:57:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:57:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:57:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:57:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:57:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:57:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:57:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:57:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:57:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:57:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:57:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:57:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:57:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:57:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:57:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:57:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:57:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:57:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:57:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:57:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:57:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:57:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:57:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:57:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:57:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:57:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:57:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:57:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:57:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:57:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:57:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:57:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:57:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:57:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:57:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:57:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:57:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:57:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:57:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:57:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:57:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:57:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:57:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:57:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:57:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:57:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:57:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:57:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:57:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:57:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:57:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:57:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:57:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:57:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:57:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:57:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:57:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:57:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:57:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:57:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:57:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:57:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:57:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:57:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:57:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:57:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:57:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:57:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:57:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:57:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:57:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:57:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:57:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:57:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:57:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:57:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:57:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:57:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:57:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:57:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:57:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:57:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:57:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:57:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:57:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:57:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:57:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:57:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:57:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:57:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:57:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:57:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:57:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:57:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:57:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:57:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:57:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:57:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:57:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:57:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:57:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:57:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:57:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:57:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:57:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:57:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:57:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:57:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:57:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:57:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:57:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:57:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:57:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:57:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:57:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:57:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:57:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:57:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:57:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:57:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:57:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:57:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:57:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:57:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:57:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:57:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:57:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:57:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:57:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:57:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:57:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:57:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:57:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:57:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:57:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:57:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:57:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:57:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:57:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:57:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:57:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:57:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:57:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:57:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:57:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:57:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:57:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:57:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:57:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:57:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:57:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:57:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:57:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:58:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:58:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:58:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:58:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:58:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:58:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:58:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:58:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:58:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:58:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:58:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:58:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:58:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:58:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:58:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:58:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:58:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:58:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:58:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:58:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:58:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:58:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:58:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:58:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:58:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:58:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:58:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:58:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:58:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:58:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:58:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:58:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:58:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:58:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:58:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:58:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:58:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:58:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:58:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:58:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:58:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:58:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:58:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:58:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:58:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:58:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:58:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:58:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:58:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:58:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:58:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:58:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:58:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:58:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:58:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:58:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:58:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:58:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:58:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:58:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:58:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:58:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:58:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:58:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:58:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:58:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:58:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:58:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:58:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:58:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:58:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:58:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:58:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:58:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:58:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:58:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:58:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:58:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:58:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:58:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:58:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:58:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:58:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:58:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:58:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:58:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:58:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:58:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:58:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:58:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:58:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:58:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:58:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:58:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:58:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:58:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:58:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:58:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:58:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:58:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:58:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:58:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:58:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:58:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:58:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:58:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:58:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:58:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:58:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:58:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:58:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:58:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:58:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:58:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:58:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:58:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:58:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:58:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:58:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:58:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:58:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:58:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:58:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:58:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:58:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:58:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:58:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:58:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:58:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:58:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:58:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:58:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:58:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:58:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:58:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:58:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:58:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:58:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:58:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:58:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:58:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:58:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:58:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:58:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:58:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:58:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:58:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:58:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:58:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:58:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:58:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:58:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:58:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:58:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:58:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:58:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:58:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:58:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:58:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:58:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:58:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:58:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:58:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:58:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:58:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:58:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:58:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:58:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:58:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:58:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:58:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:58:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:58:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:58:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:58:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:58:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:58:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:58:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:58:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:58:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:58:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:58:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:58:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:58:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:58:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:58:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:58:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:58:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:58:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:58:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:58:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:58:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:58:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:58:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:58:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:58:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:58:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:58:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:58:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:58:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:58:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:58:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:58:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:58:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:58:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:58:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:58:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:58:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:58:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:58:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:58:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:58:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:58:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:58:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:58:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:58:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:59:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:59:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:59:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:59:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:59:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:59:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:59:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:59:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:59:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:59:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:59:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:59:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:59:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:59:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:59:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:59:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:59:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:59:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:59:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:59:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:59:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:59:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:59:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:59:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:59:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:59:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:59:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:59:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:59:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:59:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:59:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:59:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:59:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:59:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:59:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:59:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:59:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:59:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:59:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:59:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:59:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:59:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:59:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:59:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:59:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:59:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:59:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:59:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:59:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:59:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:59:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:59:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:59:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:59:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:59:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:59:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:59:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:59:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:59:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:59:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:59:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:59:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:59:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:59:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:59:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:59:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:59:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:59:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:59:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:59:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:59:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:59:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:59:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:59:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:59:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:59:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:59:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:59:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:59:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:59:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:59:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:59:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:59:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:59:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:59:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:59:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:59:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:59:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:59:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:59:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:59:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:59:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:59:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:59:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:59:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:59:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:59:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:59:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:59:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:59:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:59:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:59:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:59:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:59:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:59:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:59:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:59:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:59:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:59:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:59:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:59:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:59:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:59:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:59:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:59:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:59:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:59:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:59:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:59:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:59:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:59:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:59:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:59:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:59:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:59:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:59:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:59:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:59:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:59:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:59:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:59:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:59:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:59:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:59:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:59:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:59:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:59:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:59:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:59:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:59:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:59:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:59:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:59:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:59:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:59:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:59:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:59:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:59:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:59:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:59:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:59:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:59:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:59:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:59:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:59:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:59:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:59:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:59:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:59:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:59:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:59:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:59:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:59:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:59:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:59:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:59:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:59:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:59:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:59:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:59:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:59:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:59:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:59:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:59:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:59:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:59:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:59:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:59:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:59:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:59:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:59:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:59:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:59:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:59:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:59:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:59:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:59:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:59:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:59:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:59:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:59:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:59:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:59:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:59:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:59:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:59:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:59:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:59:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 10:59:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 10:59:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:59:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:59:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:59:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 10:59:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 10:59:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 10:59:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 10:59:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 10:59:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 10:59:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 10:59:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 10:59:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 10:59:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 10:59:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 10:59:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 10:59:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 10:59:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:00:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:00:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:00:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:00:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:00:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:00:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:00:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:00:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:00:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:00:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:00:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:00:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:00:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:00:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:00:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:00:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:00:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:00:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:00:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:00:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:00:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:00:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:00:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:00:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:00:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:00:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:00:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:00:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:00:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:00:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:00:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:00:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:00:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:00:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:00:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:00:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:00:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:00:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:00:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:00:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:00:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:00:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:00:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:00:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:00:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:00:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:00:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:00:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:00:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:00:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:00:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:00:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:00:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:00:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:00:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:00:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:00:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:00:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:00:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:00:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:00:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:00:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:00:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:00:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:00:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:00:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:00:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:00:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:00:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:00:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:00:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:00:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:00:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:00:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:00:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:00:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:00:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:00:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:00:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:00:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:00:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:00:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:00:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:00:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:00:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:00:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:00:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:00:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:00:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:00:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:00:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:00:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:00:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:00:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:00:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:00:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:00:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:00:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:00:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:00:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:00:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:00:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:00:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:00:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:00:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:00:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:00:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:00:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:00:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:00:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:00:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:00:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:00:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:00:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:00:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:00:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:00:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:00:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:00:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:00:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:00:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:00:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:00:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:00:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:00:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:00:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:00:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:00:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:00:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:00:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:00:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:00:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:00:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:00:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:00:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:00:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:00:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:00:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:00:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:00:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:00:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:00:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:00:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:00:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:00:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:00:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:00:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:00:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:00:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:00:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:00:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:00:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:00:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:00:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:00:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:00:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:00:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:00:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:00:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:00:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:00:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:00:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:00:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:00:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:00:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:00:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:00:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:00:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:00:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:00:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:00:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:00:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:00:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:00:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:00:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:00:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:00:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:00:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:00:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:00:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:00:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:00:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:00:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:00:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:00:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:00:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:00:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:00:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:00:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:00:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:00:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:00:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:00:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:00:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:00:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:00:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:00:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:00:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:00:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:00:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:00:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:00:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:00:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:00:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:00:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:00:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:00:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:00:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:00:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:00:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:00:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:00:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:00:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:00:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:00:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:00:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:01:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:01:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:01:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:01:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:01:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:01:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:01:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:01:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:01:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:01:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:01:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:01:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:01:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:01:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:01:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:01:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:01:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:01:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:01:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:01:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:01:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:01:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:01:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:01:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:01:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:01:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:01:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:01:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:01:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:01:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:01:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:01:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:01:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:01:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:01:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:01:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:01:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:01:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:01:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:01:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:01:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:01:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:01:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:01:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:01:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:01:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:01:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:01:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:01:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:01:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:01:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:01:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:01:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:01:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:01:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:01:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:01:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:01:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:01:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:01:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:01:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:01:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:01:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:01:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:01:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:01:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:01:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:01:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:01:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:01:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:01:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:01:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:01:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:01:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:01:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:01:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:01:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:01:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:01:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:01:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:01:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:01:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:01:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:01:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:01:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:01:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:01:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:01:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:01:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:01:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:01:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:01:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:01:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:01:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:01:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:01:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:01:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:01:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:01:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:01:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:01:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:01:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:01:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:01:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:01:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:01:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:01:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:01:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:01:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:01:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:01:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:01:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:01:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:01:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:01:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:01:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:01:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:01:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:01:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:01:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:01:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:01:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:01:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:01:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:01:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:01:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:01:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:01:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:01:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:01:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:01:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:01:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:01:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:01:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:01:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:01:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:01:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:01:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:01:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:01:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:01:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:01:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:01:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:01:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:01:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:01:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:01:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:01:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:01:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:01:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:01:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:01:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:01:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:01:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:01:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:01:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:01:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:01:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:01:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:01:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:01:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:01:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:01:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:01:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:01:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:01:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:01:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:01:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:01:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:01:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:01:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:01:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:01:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:01:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:01:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:01:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:01:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:01:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:01:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:01:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:01:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:01:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:01:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:01:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:01:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:01:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:01:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:01:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:01:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:01:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:01:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:01:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:01:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:01:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:01:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:01:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:01:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:01:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:01:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:01:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:01:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:01:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:01:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:01:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:01:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:01:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:01:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:01:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:01:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:01:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:01:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:01:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:01:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:01:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:01:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:01:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:02:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:02:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:02:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:02:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:02:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:02:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:02:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:02:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:02:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:02:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:02:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:02:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:02:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:02:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:02:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:02:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:02:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:02:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:02:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:02:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:02:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:02:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:02:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:02:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:02:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:02:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:02:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:02:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:02:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:02:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:02:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:02:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:02:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:02:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:02:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:02:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:02:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:02:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:02:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:02:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:02:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:02:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:02:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:02:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:02:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:02:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:02:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:02:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:02:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:02:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:02:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:02:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:02:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:02:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:02:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:02:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:02:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:02:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:02:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:02:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:02:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:02:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:02:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:02:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:02:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:02:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:02:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:02:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:02:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:02:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:02:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:02:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:02:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:02:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:02:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:02:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:02:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:02:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:02:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:02:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:02:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:02:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:02:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:02:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:02:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:02:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:02:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:02:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:02:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:02:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:02:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:02:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:02:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:02:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:02:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:02:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:02:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:02:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:02:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:02:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:02:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:02:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:02:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:02:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:02:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:02:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:02:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:02:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:02:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:02:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:02:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:02:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:02:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:02:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:02:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:02:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:02:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:02:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:02:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:02:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:02:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:02:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:02:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:02:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:02:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:02:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:02:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:02:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:02:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:02:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:02:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:02:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:02:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:02:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:02:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:02:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:02:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:02:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:02:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:02:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:02:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:02:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:02:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:02:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:02:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:02:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:02:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:02:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:02:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:02:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:02:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:02:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:02:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:02:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:02:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:02:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:02:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:02:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:02:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:02:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:02:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:02:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:02:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:02:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:02:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:02:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:02:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:02:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:02:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:02:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:02:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:02:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:02:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:02:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:02:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:02:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:02:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:02:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:02:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:02:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:02:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:02:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:02:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:02:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:02:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:02:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:02:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:02:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:02:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:02:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:02:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:02:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:02:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:02:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:02:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:02:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:02:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:02:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:02:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:02:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:02:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:02:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:02:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:02:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:02:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:02:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:02:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:02:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:02:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:02:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:02:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:02:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:02:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:02:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:02:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:02:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:05:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:05:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:05:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:05:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:05:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:05:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:05:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:05:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:05:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:05:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:05:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:05:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:05:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:05:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:05:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:05:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:05:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:05:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:05:06 rdma06 rsyslogd[1041]: imjournal: 5295 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 11:05:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:05:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:05:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:05:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:05:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:05:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:05:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:05:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:05:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:05:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:05:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:05:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:05:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:05:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:05:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:05:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:05:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:05:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:05:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:05:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:05:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:05:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:05:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:05:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:05:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:05:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:05:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:05:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:05:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:05:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:05:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:05:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:05:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:05:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:05:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:05:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:05:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:05:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:05:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:05:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:05:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:05:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:05:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:05:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:05:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:05:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:05:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:05:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:05:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:05:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:05:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:05:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:05:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:05:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:05:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:05:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:05:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:05:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:05:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:05:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:05:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:05:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:05:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:05:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:05:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:05:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:05:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:05:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:05:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:05:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:05:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:05:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:05:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:05:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:05:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:05:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:05:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:05:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:05:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:05:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:05:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:05:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:05:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:05:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:05:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:05:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:05:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:05:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:05:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:05:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:05:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:05:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:05:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:05:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:05:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:05:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:05:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:05:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:05:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:05:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:05:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:05:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:05:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:05:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:05:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:05:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:05:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:05:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:05:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:05:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:05:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:05:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:05:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:05:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:05:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:05:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:05:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:05:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:05:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:05:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:05:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:05:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:05:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:05:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:05:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:05:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:05:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:05:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:05:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:05:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:05:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:05:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:05:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:05:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:05:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:05:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:05:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:05:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:05:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:05:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:05:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:05:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:05:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:05:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:05:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:05:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:05:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:05:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:05:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:05:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:05:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:05:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:05:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:05:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:05:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:05:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:05:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:05:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:05:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:05:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:05:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:05:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:05:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:05:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:05:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:05:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:05:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:05:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:05:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:05:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:05:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:05:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:05:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:05:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:05:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:05:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:05:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:05:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:05:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:05:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:06:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:06:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:06:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:06:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:06:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:06:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:06:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:06:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:06:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:06:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:06:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:06:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:06:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:06:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:06:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:06:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:06:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:06:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:06:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:06:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:06:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:06:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:06:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:06:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:06:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:06:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:06:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:06:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:06:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:06:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:06:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:06:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:06:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:06:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:06:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:06:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:06:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:06:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:06:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:06:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:06:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:06:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:06:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:06:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:06:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:06:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:06:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:06:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:06:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:06:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:06:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:06:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:06:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:06:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:06:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:06:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:06:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:06:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:06:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:06:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:06:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:06:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:06:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:06:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:06:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:06:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:06:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:06:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:06:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:06:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:06:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:06:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:06:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:06:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:06:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:06:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:06:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:06:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:06:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:06:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:06:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:06:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:06:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:06:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:06:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:06:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:06:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:06:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:06:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:06:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:06:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:06:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:06:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:06:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:06:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:06:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:06:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:06:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:06:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:06:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:06:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:06:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:06:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:06:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:06:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:06:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:06:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:06:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:06:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:06:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:06:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:06:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:06:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:06:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:06:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:06:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:06:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:06:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:06:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:06:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:06:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:06:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:06:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:06:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:06:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:06:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:06:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:06:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:06:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:06:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:06:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:06:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:06:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:06:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:06:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:06:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:06:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:06:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:06:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:06:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:06:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:06:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:06:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:06:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:06:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:06:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:06:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:06:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:06:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:06:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:06:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:06:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:06:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:06:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:06:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:06:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:06:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:06:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:06:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:06:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:06:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:06:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:06:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:06:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:06:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:06:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:06:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:06:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:06:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:06:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:06:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:06:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:06:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:06:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:06:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:06:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:06:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:06:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:06:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:06:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:06:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:06:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:06:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:06:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:06:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:06:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:06:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:06:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:06:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:06:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:06:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:06:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:06:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:06:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:06:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:06:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:06:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:06:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:06:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:06:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:06:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:06:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:06:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:06:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:06:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:06:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:06:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:06:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:06:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:06:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:06:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:06:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:06:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:06:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:06:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:06:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:07:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:07:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:07:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:07:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:07:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:07:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:07:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:07:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:07:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:07:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:07:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:07:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:07:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:07:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:07:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:07:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:07:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:07:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:07:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:07:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:07:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:07:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:07:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:07:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:07:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:07:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:07:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:07:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:07:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:07:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:07:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:07:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:07:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:07:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:07:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:07:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:07:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:07:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:07:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:07:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:07:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:07:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:07:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:07:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:07:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:07:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:07:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:07:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:07:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:07:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:07:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:07:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:07:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:07:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:07:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:07:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:07:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:07:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:07:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:07:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:07:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:07:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:07:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:07:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:07:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:07:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:07:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:07:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:07:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:07:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:07:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:07:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:07:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:07:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:07:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:07:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:07:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:07:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:07:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:07:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:07:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:07:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:07:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:07:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:07:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:07:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:07:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:07:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:07:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:07:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:07:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:07:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:07:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:07:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:07:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:07:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:07:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:07:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:07:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:07:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:07:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:07:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:07:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:07:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:07:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:07:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:07:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:07:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:07:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:07:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:07:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:07:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:07:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:07:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:07:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:07:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:07:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:07:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:07:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:07:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:07:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:07:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:07:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:07:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:07:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:07:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:07:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:07:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:07:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:07:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:07:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:07:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:07:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:07:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:07:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:07:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:07:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:07:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:07:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:07:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:07:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:07:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:07:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:07:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:07:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:07:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:07:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:07:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:07:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:07:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:07:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:07:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:07:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:07:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:07:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:07:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:07:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:07:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:07:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:07:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:07:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:07:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:07:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:07:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:07:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:07:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:07:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:07:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:07:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:07:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:07:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:07:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:07:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:07:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:07:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:07:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:07:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:07:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:07:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:07:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:07:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:07:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:07:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:07:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:07:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:07:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:07:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:07:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:07:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:07:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:07:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:07:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:07:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:07:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:07:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:07:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:07:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:07:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:07:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:07:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:07:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:07:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:07:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:07:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:07:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:07:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:07:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:07:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:07:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:07:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:07:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:07:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:07:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:07:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:07:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:07:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:08:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:08:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:08:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:08:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:08:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:08:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:08:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:08:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:08:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:08:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:08:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:08:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:08:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:08:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:08:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:08:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:08:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:08:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:08:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:08:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:08:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:08:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:08:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:08:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:08:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:08:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:08:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:08:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:08:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:08:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:08:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:08:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:08:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:08:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:08:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:08:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:08:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:08:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:08:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:08:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:08:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:08:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:08:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:08:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:08:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:08:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:08:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:08:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:08:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:08:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:08:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:08:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:08:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:08:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:08:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:08:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:08:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:08:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:08:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:08:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:08:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:08:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:08:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:08:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:08:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:08:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:08:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:08:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:08:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:08:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:08:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:08:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:08:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:08:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:08:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:08:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:08:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:08:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:08:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:08:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:08:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:08:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:08:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:08:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:08:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:08:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:08:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:08:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:08:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:08:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:08:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:08:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:08:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:08:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:08:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:08:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:08:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:08:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:08:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:08:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:08:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:08:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:08:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:08:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:08:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:08:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:08:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:08:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:08:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:08:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:08:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:08:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:08:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:08:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:08:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:08:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:08:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:08:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:08:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:08:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:08:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:08:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:08:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:08:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:08:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:08:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:08:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:08:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:08:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:08:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:08:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:08:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:08:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:08:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:08:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:08:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:08:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:08:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:08:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:08:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:08:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:08:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:08:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:08:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:08:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:08:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:08:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:08:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:08:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:08:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:08:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:08:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:08:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:08:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:08:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:08:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:08:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:08:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:08:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:08:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:08:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:08:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:08:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:08:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:08:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:08:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:08:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:08:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:08:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:08:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:08:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:08:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:08:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:08:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:08:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:08:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:08:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:08:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:08:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:08:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:08:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:08:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:08:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:08:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:08:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:08:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:08:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:08:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:08:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:08:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:08:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:08:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:08:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:08:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:08:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:08:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:08:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:08:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:08:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:08:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:08:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:08:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:08:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:08:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:08:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:08:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:08:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:08:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:08:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:08:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:08:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:08:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:08:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:08:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:08:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:08:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:09:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:09:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:09:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:09:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:09:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:09:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:09:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:09:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:09:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:09:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:09:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:09:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:09:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:09:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:09:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:09:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:09:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:09:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:09:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:09:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:09:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:09:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:09:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:09:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:09:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:09:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:09:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:09:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:09:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:09:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:09:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:09:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:09:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:09:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:09:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:09:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:09:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:09:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:09:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:09:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:09:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:09:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:09:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:09:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:09:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:09:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:09:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:09:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:09:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:09:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:09:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:09:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:09:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:09:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:09:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:09:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:09:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:09:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:09:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:09:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:09:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:09:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:09:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:09:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:09:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:09:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:09:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:09:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:09:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:09:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:09:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:09:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:09:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:09:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:09:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:09:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:09:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:09:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:09:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:09:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:09:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:09:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:09:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:09:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:09:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:09:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:09:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:09:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:09:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:09:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:09:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:09:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:09:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:09:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:09:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:09:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:09:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:09:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:09:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:09:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:09:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:09:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:09:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:09:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:09:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:09:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:09:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:09:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:09:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:09:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:09:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:09:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:09:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:09:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:09:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:09:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:09:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:09:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:09:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:09:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:09:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:09:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:09:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:09:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:09:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:09:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:09:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:09:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:09:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:09:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:09:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:09:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:09:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:09:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:09:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:09:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:09:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:09:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:09:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:09:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:09:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:09:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:09:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:09:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:09:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:09:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:09:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:09:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:09:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:09:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:09:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:09:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:09:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:09:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:09:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:09:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:09:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:09:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:09:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:09:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:09:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:09:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:09:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:09:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:09:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:09:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:09:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:09:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:09:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:09:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:09:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:09:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:09:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:09:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:09:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:09:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:09:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:09:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:09:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:09:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:09:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:09:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:09:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:09:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:09:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:09:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:09:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:09:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:09:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:09:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:09:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:09:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:09:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:09:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:09:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:09:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:09:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:09:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:09:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:09:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:09:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:09:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:09:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:09:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:09:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:09:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:09:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:09:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:09:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:09:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:09:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:09:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:09:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:09:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:09:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:09:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:10:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:10:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:10:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:10:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:10:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:10:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:10:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:10:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:10:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:10:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:10:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:10:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:10:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:10:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:10:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:10:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:10:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:10:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:10:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:10:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:10:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:10:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:10:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:10:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:10:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:10:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:10:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:10:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:10:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:10:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:10:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:10:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:10:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:10:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:10:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:10:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:10:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:10:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:10:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:10:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:10:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:10:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:10:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:10:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:10:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:10:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:10:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:10:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:10:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:10:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:10:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:10:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:10:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:10:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:10:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:10:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:10:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:10:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:10:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:10:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:10:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:10:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:10:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:10:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:10:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:10:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:10:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:10:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:10:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:10:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:10:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:10:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:10:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:10:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:10:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:10:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:10:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:10:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:10:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:10:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:10:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:10:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:10:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:10:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:10:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:10:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:10:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:10:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:10:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:10:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:10:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:10:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:10:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:10:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:10:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:10:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:10:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:10:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:10:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:10:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:10:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:10:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:10:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:10:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:10:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:10:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:10:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:10:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:10:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:10:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:10:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:10:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:10:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:10:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:10:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:10:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:10:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:10:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:10:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:10:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:10:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:10:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:10:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:10:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:10:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:10:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:10:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:10:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:10:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:10:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:10:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:10:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:10:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:10:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:10:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:10:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:10:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:10:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:10:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:10:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:10:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:10:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:10:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:10:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:10:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:10:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:10:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:10:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:10:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:10:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:10:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:10:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:10:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:10:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:10:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:10:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:10:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:10:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:10:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:10:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:10:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:10:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:10:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:10:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:10:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:10:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:10:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:10:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:10:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:10:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:10:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:10:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:10:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:10:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:10:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:10:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:10:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:10:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:10:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:10:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:10:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:10:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:10:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:10:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:10:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:10:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:10:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:10:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:10:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:10:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:10:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:10:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:10:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:10:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:10:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:10:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:10:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:10:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:10:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:10:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:10:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:10:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:10:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:10:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:10:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:10:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:10:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:10:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:10:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:10:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:10:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:10:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:10:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:10:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:10:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:10:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:11:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:11:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:11:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:11:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:11:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:11:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:11:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:11:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:11:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:11:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:11:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:11:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:11:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:11:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:11:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:11:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:11:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:11:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:11:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:11:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:11:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:11:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:11:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:11:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:11:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:11:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:11:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:11:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:11:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:11:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:11:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:11:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:11:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:11:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:11:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:11:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:11:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:11:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:11:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:11:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:11:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:11:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:11:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:11:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:11:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:11:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:11:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:11:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:11:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:11:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:11:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:11:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:11:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:11:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:11:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:11:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:11:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:11:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:11:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:11:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:11:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:11:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:11:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:11:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:11:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:11:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:11:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:11:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:11:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:11:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:11:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:11:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:11:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:11:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:11:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:11:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:11:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:11:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:11:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:11:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:11:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:11:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:11:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:11:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:11:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:11:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:11:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:11:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:11:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:11:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:11:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:11:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:11:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:11:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:11:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:11:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:11:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:11:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:11:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:11:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:11:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:11:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:11:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:11:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:11:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:11:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:11:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:11:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:11:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:11:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:11:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:11:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:11:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:11:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:11:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:11:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:11:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:11:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:11:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:11:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:11:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:11:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:11:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:11:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:11:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:11:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:11:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:11:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:11:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:11:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:11:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:11:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:11:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:11:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:11:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:11:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:11:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:11:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:11:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:11:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:11:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:11:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:11:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:11:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:11:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:11:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:11:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:11:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:11:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:11:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:11:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:11:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:11:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:11:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:11:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:11:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:11:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:11:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:11:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:11:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:11:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:11:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:11:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:11:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:11:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:11:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:11:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:11:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:11:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:11:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:11:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:11:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:11:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:11:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:11:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:11:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:11:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:11:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:11:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:11:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:11:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:11:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:11:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:11:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:11:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:11:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:11:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:11:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:11:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:11:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:11:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:11:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:11:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:11:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:11:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:11:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:11:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:11:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:11:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:11:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:11:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:11:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:11:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:11:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:11:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:11:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:11:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:11:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:11:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:11:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:11:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:11:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:11:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:11:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:11:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:11:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:12:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:12:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:12:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:12:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:12:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:12:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:12:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:12:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:12:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:12:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:12:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:12:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:12:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:12:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:12:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:12:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:12:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:12:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:12:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:12:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:12:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:12:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:12:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:12:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:12:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:12:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:12:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:12:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:12:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:12:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:12:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:12:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:12:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:12:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:12:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:12:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:12:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:12:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:12:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:12:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:12:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:12:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:12:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:12:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:12:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:12:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:12:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:12:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:12:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:12:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:12:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:12:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:12:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:12:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:12:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:12:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:12:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:12:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:12:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:12:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:12:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:12:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:12:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:12:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:12:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:12:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:12:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:12:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:12:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:12:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:12:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:12:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:12:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:12:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:12:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:12:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:12:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:12:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:12:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:12:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:12:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:12:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:12:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:12:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:12:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:12:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:12:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:12:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:12:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:12:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:12:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:12:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:12:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:12:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:12:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:12:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:12:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:12:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:12:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:12:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:12:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:12:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:12:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:12:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:12:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:12:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:12:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:12:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:12:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:12:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:12:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:12:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:12:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:12:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:12:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:12:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:12:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:12:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:12:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:12:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:12:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:12:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:12:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:12:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:12:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:12:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:12:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:12:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:12:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:12:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:12:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:12:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:12:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:12:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:12:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:12:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:12:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:12:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:12:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:12:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:12:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:12:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:12:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:12:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:12:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:12:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:12:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:12:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:12:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:12:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:12:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:12:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:12:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:12:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:12:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:12:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:12:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:12:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:12:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:12:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:12:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:12:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:12:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:12:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:12:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:12:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:12:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:12:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:12:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:12:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:12:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:12:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:12:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:12:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:12:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:12:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:12:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:12:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:12:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:12:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:12:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:12:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:12:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:12:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:12:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:12:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:12:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:12:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:12:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:12:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:12:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:12:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:12:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:12:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:12:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:12:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:12:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:12:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:12:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:12:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:12:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:12:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:12:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:12:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:12:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:12:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:12:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:12:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:12:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:12:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:12:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:12:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:12:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:12:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:12:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:12:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:13:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:13:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:13:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:13:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:13:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:13:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:13:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:13:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:13:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:13:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:13:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:13:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:13:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:13:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:13:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:13:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:13:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:13:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:15:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:15:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:15:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:15:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:15:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:15:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:15:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:15:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:15:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:15:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:15:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:15:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:15:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:15:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:15:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:15:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:15:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:15:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:15:07 rdma06 rsyslogd[1041]: imjournal: 5082 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 11:15:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:15:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:15:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:15:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:15:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:15:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:15:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:15:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:15:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:15:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:15:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:15:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:15:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:15:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:15:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:15:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:15:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:15:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:15:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:15:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:15:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:15:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:15:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:15:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:15:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:15:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:15:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:15:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:15:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:15:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:15:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:15:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:15:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:15:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:15:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:15:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:15:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:15:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:15:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:15:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:15:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:15:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:15:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:15:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:15:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:15:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:15:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:15:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:15:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:15:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:15:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:15:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:15:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:15:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:15:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:15:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:15:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:15:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:15:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:15:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:15:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:15:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:15:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:15:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:15:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:15:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:15:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:15:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:15:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:15:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:15:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:15:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:15:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:15:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:15:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:15:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:15:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:15:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:15:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:15:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:15:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:15:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:15:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:15:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:15:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:15:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:15:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:15:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:15:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:15:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:15:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:15:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:15:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:15:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:15:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:15:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:15:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:15:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:15:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:15:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:15:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:15:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:15:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:15:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:15:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:15:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:15:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:15:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:15:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:15:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:15:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:15:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:15:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:15:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:15:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:15:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:15:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:15:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:15:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:15:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:15:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:15:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:15:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:15:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:15:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:15:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:15:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:15:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:15:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:15:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:15:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:15:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:15:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:15:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:15:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:15:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:15:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:15:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:15:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:15:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:15:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:15:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:15:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:15:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:15:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:15:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:15:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:15:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:15:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:15:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:15:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:15:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:15:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:15:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:15:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:15:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:15:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:15:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:15:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:15:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:15:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:15:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:15:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:15:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:15:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:15:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:15:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:15:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:15:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:15:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:15:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:15:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:15:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:15:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:15:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:15:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:15:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:15:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:15:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:15:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:16:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:16:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:16:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:16:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:16:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:16:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:16:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:16:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:16:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:16:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:16:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:16:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:16:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:16:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:16:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:16:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:16:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:16:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:16:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:16:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:16:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:16:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:16:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:16:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:16:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:16:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:16:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:16:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:16:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:16:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:16:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:16:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:16:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:16:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:16:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:16:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:16:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:16:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:16:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:16:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:16:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:16:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:16:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:16:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:16:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:16:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:16:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:16:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:16:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:16:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:16:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:16:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:16:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:16:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:16:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:16:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:16:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:16:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:16:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:16:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:16:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:16:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:16:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:16:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:16:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:16:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:16:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:16:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:16:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:16:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:16:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:16:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:16:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:16:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:16:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:16:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:16:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:16:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:16:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:16:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:16:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:16:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:16:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:16:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:16:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:16:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:16:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:16:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:16:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:16:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:16:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:16:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:16:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:16:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:16:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:16:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:16:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:16:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:16:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:16:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:16:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:16:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:16:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:16:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:16:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:16:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:16:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:16:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:16:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:16:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:16:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:16:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:16:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:16:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:16:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:16:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:16:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:16:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:16:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:16:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:16:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:16:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:16:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:16:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:16:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:16:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:16:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:16:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:16:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:16:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:16:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:16:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:16:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:16:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:16:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:16:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:16:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:16:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:16:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:16:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:16:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:16:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:16:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:16:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:16:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:16:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:16:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:16:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:16:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:16:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:16:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:16:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:16:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:16:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:16:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:16:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:16:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:16:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:16:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:16:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:16:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:16:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:16:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:16:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:16:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:16:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:16:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:16:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:16:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:16:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:16:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:16:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:16:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:16:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:16:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:16:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:16:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:16:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:16:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:16:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:16:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:16:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:16:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:16:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:16:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:16:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:16:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:16:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:16:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:16:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:16:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:16:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:16:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:16:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:16:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:16:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:16:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:16:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:16:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:16:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:16:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:16:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:16:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:16:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:16:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:16:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:16:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:16:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:16:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:16:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:16:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:16:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:16:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:16:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:16:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:16:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:17:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:17:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:17:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:17:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:17:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:17:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:17:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:17:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:17:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:17:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:17:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:17:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:17:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:17:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:17:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:17:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:17:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:17:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:17:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:17:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:17:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:17:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:17:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:17:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:17:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:17:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:17:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:17:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:17:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:17:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:17:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:17:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:17:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:17:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:17:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:17:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:17:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:17:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:17:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:17:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:17:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:17:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:17:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:17:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:17:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:17:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:17:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:17:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:17:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:17:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:17:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:17:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:17:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:17:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:17:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:17:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:17:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:17:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:17:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:17:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:17:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:17:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:17:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:17:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:17:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:17:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:17:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:17:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:17:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:17:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:17:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:17:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:17:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:17:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:17:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:17:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:17:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:17:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:17:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:17:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:17:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:17:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:17:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:17:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:17:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:17:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:17:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:17:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:17:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:17:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:17:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:17:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:17:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:17:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:17:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:17:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:17:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:17:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:17:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:17:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:17:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:17:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:17:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:17:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:17:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:17:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:17:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:17:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:17:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:17:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:17:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:17:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:17:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:17:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:17:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:17:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:17:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:17:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:17:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:17:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:17:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:17:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:17:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:17:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:17:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:17:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:17:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:17:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:17:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:17:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:17:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:17:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:17:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:17:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:17:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:17:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:17:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:17:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:17:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:17:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:17:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:17:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:17:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:17:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:17:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:17:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:17:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:17:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:17:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:17:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:17:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:17:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:17:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:17:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:17:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:17:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:17:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:17:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:17:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:17:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:17:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:17:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:17:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:17:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:17:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:17:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:17:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:17:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:17:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:17:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:17:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:17:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:17:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:17:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:17:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:17:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:17:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:17:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:17:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:17:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:17:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:17:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:17:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:17:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:17:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:17:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:17:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:17:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:17:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:17:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:17:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:17:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:17:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:17:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:17:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:17:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:17:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:17:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:17:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:17:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:17:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:17:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:17:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:17:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:17:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:17:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:17:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:17:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:17:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:17:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:17:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:17:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:17:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:17:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:17:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:17:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:18:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:18:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:18:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:18:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:18:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:18:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:18:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:18:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:18:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:18:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:18:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:18:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:18:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:18:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:18:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:18:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:18:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:18:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:18:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:18:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:18:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:18:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:18:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:18:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:18:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:18:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:18:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:18:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:18:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:18:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:18:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:18:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:18:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:18:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:18:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:18:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:18:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:18:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:18:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:18:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:18:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:18:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:18:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:18:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:18:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:18:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:18:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:18:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:18:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:18:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:18:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:18:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:18:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:18:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:18:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:18:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:18:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:18:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:18:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:18:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:18:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:18:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:18:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:18:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:18:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:18:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:18:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:18:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:18:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:18:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:18:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:18:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:18:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:18:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:18:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:18:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:18:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:18:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:18:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:18:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:18:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:18:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:18:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:18:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:18:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:18:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:18:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:18:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:18:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:18:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:18:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:18:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:18:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:18:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:18:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:18:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:18:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:18:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:18:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:18:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:18:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:18:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:18:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:18:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:18:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:18:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:18:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:18:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:18:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:18:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:18:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:18:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:18:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:18:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:18:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:18:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:18:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:18:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:18:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:18:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:18:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:18:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:18:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:18:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:18:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:18:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:18:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:18:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:18:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:18:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:18:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:18:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:18:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:18:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:18:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:18:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:18:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:18:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:18:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:18:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:18:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:18:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:18:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:18:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:18:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:18:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:18:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:18:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:18:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:18:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:18:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:18:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:18:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:18:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:18:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:18:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:18:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:18:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:18:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:18:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:18:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:18:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:18:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:18:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:18:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:18:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:18:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:18:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:18:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:18:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:18:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:18:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:18:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:18:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:18:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:18:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:18:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:18:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:18:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:18:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:18:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:18:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:18:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:18:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:18:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:18:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:18:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:18:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:18:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:18:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:18:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:18:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:18:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:18:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:18:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:18:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:18:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:18:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:18:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:18:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:18:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:18:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:18:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:18:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:18:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:18:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:18:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:18:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:18:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:18:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:18:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:18:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:18:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:18:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:18:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:18:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:19:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:19:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:19:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:19:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:19:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:19:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:19:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:19:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:19:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:19:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:19:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:19:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:19:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:19:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:19:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:19:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:19:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:19:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:19:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:19:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:19:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:19:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:19:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:19:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:19:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:19:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:19:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:19:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:19:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:19:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:19:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:19:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:19:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:19:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:19:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:19:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:19:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:19:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:19:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:19:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:19:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:19:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:19:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:19:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:19:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:19:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:19:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:19:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:19:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:19:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:19:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:19:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:19:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:19:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:19:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:19:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:19:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:19:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:19:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:19:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:19:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:19:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:19:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:19:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:19:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:19:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:19:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:19:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:19:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:19:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:19:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:19:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:19:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:19:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:19:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:19:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:19:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:19:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:19:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:19:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:19:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:19:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:19:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:19:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:19:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:19:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:19:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:19:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:19:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:19:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:19:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:19:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:19:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:19:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:19:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:19:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:19:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:19:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:19:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:19:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:19:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:19:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:19:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:19:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:19:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:19:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:19:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:19:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:19:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:19:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:19:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:19:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:19:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:19:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:19:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:19:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:19:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:19:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:19:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:19:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:19:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:19:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:19:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:19:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:19:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:19:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:19:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:19:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:19:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:19:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:19:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:19:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:19:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:19:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:19:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:19:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:19:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:19:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:19:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:19:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:19:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:19:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:19:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:19:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:19:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:19:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:19:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:19:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:19:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:19:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:19:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:19:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:19:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:19:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:19:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:19:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:19:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:19:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:19:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:19:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:19:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:19:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:19:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:19:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:19:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:19:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:19:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:19:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:19:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:19:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:19:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:19:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:19:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:19:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:19:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:19:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:19:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:19:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:19:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:19:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:19:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:19:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:19:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:19:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:19:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:19:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:19:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:19:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:19:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:19:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:19:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:19:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:19:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:19:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:19:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:19:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:19:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:19:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:19:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:19:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:19:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:19:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:19:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:19:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:19:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:19:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:19:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:19:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:19:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:19:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:19:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:19:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:19:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:19:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:19:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:19:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:20:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:20:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:20:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:20:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:20:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:20:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:20:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:20:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:20:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:20:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:20:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:20:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:20:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:20:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:20:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:20:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:20:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:20:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:20:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:20:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:20:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:20:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:20:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:20:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:20:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:20:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:20:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:20:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:20:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:20:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:20:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:20:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:20:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:20:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:20:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:20:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:20:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:20:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:20:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:20:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:20:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:20:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:20:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:20:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:20:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:20:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:20:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:20:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:20:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:20:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:20:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:20:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:20:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:20:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:20:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:20:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:20:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:20:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:20:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:20:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:20:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:20:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:20:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:20:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:20:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:20:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:20:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:20:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:20:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:20:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:20:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:20:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:20:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:20:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:20:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:20:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:20:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:20:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:20:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:20:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:20:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:20:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:20:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:20:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:20:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:20:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:20:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:20:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:20:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:20:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:20:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:20:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:20:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:20:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:20:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:20:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:20:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:20:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:20:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:20:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:20:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:20:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:20:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:20:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:20:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:20:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:20:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:20:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:20:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:20:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:20:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:20:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:20:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:20:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:20:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:20:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:20:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:20:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:20:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:20:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:20:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:20:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:20:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:20:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:20:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:20:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:20:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:20:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:20:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:20:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:20:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:20:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:20:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:20:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:20:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:20:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:20:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:20:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:20:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:20:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:20:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:20:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:20:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:20:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:20:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:20:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:20:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:20:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:20:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:20:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:20:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:20:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:20:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:20:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:20:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:20:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:20:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:20:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:20:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:20:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:20:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:20:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:20:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:20:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:20:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:20:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:20:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:20:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:20:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:20:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:20:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:20:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:20:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:20:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:20:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:20:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:20:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:20:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:20:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:20:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:20:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:20:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:20:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:20:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:20:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:20:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:20:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:20:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:20:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:20:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:20:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:20:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:20:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:20:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:20:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:20:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:20:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:20:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:20:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:20:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:20:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:20:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:20:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:20:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:20:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:20:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:20:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:20:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:20:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:20:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:20:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:20:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:20:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:20:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:20:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:20:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:21:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:21:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:21:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:21:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:21:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:21:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:21:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:21:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:21:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:21:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:21:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:21:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:21:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:21:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:21:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:21:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:21:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:21:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:21:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:21:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:21:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:21:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:21:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:21:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:21:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:21:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:21:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:21:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:21:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:21:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:21:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:21:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:21:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:21:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:21:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:21:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:21:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:21:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:21:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:21:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:21:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:21:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:21:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:21:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:21:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:21:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:21:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:21:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:21:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:21:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:21:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:21:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:21:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:21:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:21:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:21:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:21:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:21:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:21:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:21:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:21:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:21:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:21:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:21:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:21:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:21:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:21:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:21:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:21:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:21:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:21:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:21:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:21:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:21:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:21:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:21:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:21:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:21:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:21:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:21:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:21:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:21:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:21:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:21:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:21:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:21:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:21:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:21:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:21:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:21:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:21:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:21:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:21:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:21:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:21:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:21:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:21:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:21:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:21:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:21:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:21:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:21:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:21:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:21:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:21:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:21:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:21:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:21:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:21:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:21:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:21:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:21:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:21:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:21:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:21:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:21:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:21:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:21:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:21:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:21:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:21:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:21:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:21:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:21:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:21:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:21:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:21:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:21:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:21:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:21:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:21:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:21:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:21:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:21:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:21:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:21:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:21:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:21:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:21:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:21:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:21:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:21:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:21:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:21:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:21:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:21:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:21:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:21:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:21:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:21:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:21:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:21:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:21:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:21:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:21:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:21:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:21:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:21:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:21:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:21:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:21:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:21:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:21:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:21:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:21:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:21:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:21:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:21:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:21:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:21:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:21:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:21:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:21:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:21:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:21:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:21:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:21:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:21:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:21:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:21:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:21:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:21:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:21:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:21:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:21:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:21:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:21:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:21:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:21:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:21:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:21:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:21:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:21:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:21:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:21:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:21:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:21:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:21:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:21:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:21:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:21:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:21:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:21:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:21:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:21:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:21:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:21:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:21:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:21:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:21:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:21:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:21:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:21:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:21:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:21:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:21:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:22:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:22:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:22:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:22:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:22:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:22:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:22:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:22:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:22:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:22:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:22:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:22:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:22:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:22:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:22:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:22:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:22:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:22:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:22:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:22:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:22:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:22:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:22:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:22:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:22:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:22:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:22:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:22:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:22:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:22:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:22:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:22:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:22:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:22:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:22:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:22:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:22:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:22:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:22:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:22:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:22:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:22:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:22:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:22:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:22:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:22:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:22:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:22:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:22:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:22:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:22:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:22:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:22:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:22:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:22:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:22:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:22:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:22:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:22:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:22:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:22:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:22:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:22:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:22:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:22:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:22:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:22:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:22:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:22:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:22:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:22:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:22:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:22:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:22:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:22:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:22:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:22:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:22:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:22:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:22:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:22:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:22:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:22:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:22:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:22:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:22:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:22:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:22:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:22:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:22:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:22:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:22:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:22:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:22:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:22:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:22:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:22:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:22:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:22:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:22:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:22:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:22:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:22:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:22:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:22:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:22:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:22:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:22:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:22:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:22:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:22:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:22:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:22:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:22:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:22:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:22:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:22:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:22:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:22:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:22:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:22:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:22:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:22:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:22:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:22:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:22:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:22:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:22:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:22:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:22:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:22:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:22:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:22:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:22:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:22:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:22:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:22:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:22:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:22:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:22:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:22:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:22:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:22:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:22:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:22:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:22:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:22:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:22:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:22:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:22:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:22:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:22:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:22:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:22:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:22:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:22:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:22:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:22:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:22:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:22:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:22:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:22:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:22:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:22:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:22:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:22:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:22:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:22:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:22:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:22:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:22:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:22:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:22:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:22:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:22:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:22:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:22:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:22:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:22:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:22:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:22:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:22:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:22:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:22:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:22:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:22:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:22:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:22:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:22:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:22:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:22:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:22:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:22:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:22:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:22:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:22:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:22:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:22:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:22:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:22:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:22:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:22:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:22:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:22:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:22:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:22:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:22:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:22:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:22:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:22:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:22:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:22:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:22:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:22:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:22:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:22:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:23:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:23:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:23:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:23:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:23:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:23:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:23:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:23:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:23:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:23:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:23:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:23:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:23:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:23:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:23:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:23:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:23:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:23:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:25:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:25:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:25:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:25:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:25:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:25:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:25:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:25:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:25:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:25:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:25:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:25:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:25:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:25:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:25:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:25:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:25:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:25:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:25:12 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 11:25:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:25:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:25:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:25:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:25:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:25:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:25:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:25:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:25:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:25:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:25:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:25:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:25:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:25:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:25:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:25:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:25:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:25:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:25:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:25:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:25:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:25:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:25:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:25:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:25:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:25:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:25:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:25:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:25:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:25:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:25:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:25:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:25:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:25:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:25:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:25:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:25:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:25:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:25:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:25:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:25:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:25:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:25:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:25:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:25:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:25:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:25:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:25:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:25:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:25:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:25:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:25:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:25:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:25:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:25:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:25:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:25:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:25:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:25:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:25:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:25:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:25:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:25:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:25:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:25:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:25:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:25:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:25:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:25:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:25:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:25:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:25:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:25:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:25:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:25:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:25:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:25:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:25:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:25:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:25:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:25:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:25:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:25:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:25:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:25:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:25:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:25:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:25:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:25:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:25:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:25:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:25:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:25:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:25:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:25:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:25:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:25:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:25:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:25:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:25:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:25:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:25:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:25:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:25:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:25:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:25:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:25:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:25:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:25:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:25:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:25:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:25:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:25:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:25:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:25:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:25:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:25:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:25:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:25:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:25:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:25:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:25:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:25:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:25:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:25:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:25:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:25:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:25:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:25:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:25:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:25:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:25:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:25:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:25:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:25:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:25:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:25:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:25:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:25:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:25:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:25:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:25:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:25:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:25:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:25:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:25:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:25:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:25:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:25:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:25:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:25:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:25:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:25:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:25:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:25:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:25:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:25:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:25:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:25:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:25:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:25:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:25:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:26:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:26:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:26:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:26:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:26:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:26:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:26:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:26:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:26:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:26:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:26:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:26:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:26:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:26:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:26:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:26:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:26:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:26:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:26:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:26:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:26:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:26:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:26:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:26:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:26:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:26:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:26:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:26:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:26:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:26:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:26:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:26:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:26:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:26:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:26:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:26:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:26:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:26:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:26:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:26:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:26:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:26:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:26:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:26:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:26:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:26:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:26:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:26:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:26:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:26:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:26:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:26:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:26:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:26:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:26:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:26:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:26:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:26:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:26:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:26:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:26:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:26:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:26:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:26:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:26:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:26:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:26:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:26:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:26:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:26:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:26:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:26:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:26:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:26:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:26:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:26:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:26:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:26:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:26:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:26:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:26:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:26:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:26:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:26:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:26:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:26:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:26:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:26:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:26:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:26:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:26:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:26:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:26:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:26:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:26:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:26:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:26:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:26:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:26:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:26:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:26:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:26:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:26:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:26:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:26:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:26:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:26:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:26:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:26:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:26:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:26:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:26:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:26:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:26:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:26:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:26:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:26:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:26:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:26:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:26:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:26:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:26:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:26:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:26:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:26:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:26:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:26:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:26:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:26:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:26:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:26:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:26:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:26:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:26:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:26:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:26:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:26:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:26:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:26:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:26:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:26:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:26:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:26:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:26:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:26:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:26:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:26:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:26:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:26:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:26:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:26:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:26:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:26:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:26:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:26:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:26:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:26:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:26:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:26:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:26:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:26:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:26:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:26:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:26:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:26:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:26:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:26:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:26:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:26:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:26:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:26:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:26:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:26:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:26:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:26:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:26:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:26:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:26:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:26:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:26:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:26:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:26:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:26:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:26:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:26:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:26:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:26:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:26:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:26:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:26:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:26:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:26:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:26:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:26:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:26:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:26:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:26:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:26:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:26:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:26:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:26:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:26:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:26:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:26:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:26:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:26:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:26:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:26:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:26:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:26:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:26:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:26:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:26:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:26:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:26:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:26:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:27:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:27:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:27:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:27:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:27:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:27:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:27:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:27:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:27:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:27:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:27:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:27:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:27:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:27:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:27:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:27:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:27:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:27:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:27:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:27:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:27:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:27:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:27:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:27:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:27:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:27:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:27:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:27:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:27:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:27:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:27:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:27:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:27:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:27:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:27:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:27:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:27:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:27:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:27:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:27:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:27:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:27:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:27:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:27:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:27:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:27:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:27:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:27:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:27:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:27:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:27:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:27:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:27:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:27:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:27:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:27:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:27:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:27:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:27:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:27:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:27:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:27:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:27:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:27:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:27:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:27:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:27:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:27:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:27:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:27:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:27:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:27:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:27:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:27:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:27:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:27:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:27:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:27:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:27:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:27:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:27:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:27:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:27:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:27:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:27:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:27:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:27:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:27:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:27:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:27:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:27:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:27:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:27:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:27:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:27:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:27:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:27:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:27:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:27:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:27:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:27:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:27:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:27:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:27:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:27:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:27:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:27:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:27:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:27:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:27:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:27:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:27:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:27:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:27:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:27:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:27:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:27:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:27:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:27:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:27:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:27:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:27:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:27:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:27:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:27:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:27:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:27:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:27:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:27:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:27:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:27:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:27:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:27:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:27:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:27:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:27:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:27:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:27:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:27:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:27:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:27:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:27:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:27:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:27:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:27:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:27:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:27:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:27:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:27:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:27:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:27:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:27:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:27:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:27:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:27:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:27:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:27:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:27:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:27:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:27:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:27:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:27:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:27:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:27:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:27:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:27:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:27:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:27:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:27:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:27:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:27:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:27:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:27:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:27:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:27:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:27:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:27:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:27:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:27:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:27:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:27:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:27:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:27:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:27:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:27:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:27:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:27:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:27:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:27:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:27:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:27:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:27:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:27:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:27:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:27:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:27:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:27:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:27:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:27:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:27:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:27:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:27:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:27:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:27:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:27:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:27:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:27:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:27:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:27:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:27:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:27:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:27:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:27:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:27:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:27:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:27:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:28:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:28:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:28:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:28:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:28:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:28:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:28:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:28:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:28:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:28:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:28:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:28:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:28:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:28:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:28:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:28:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:28:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:28:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:28:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:28:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:28:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:28:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:28:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:28:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:28:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:28:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:28:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:28:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:28:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:28:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:28:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:28:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:28:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:28:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:28:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:28:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:28:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:28:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:28:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:28:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:28:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:28:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:28:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:28:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:28:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:28:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:28:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:28:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:28:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:28:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:28:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:28:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:28:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:28:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:28:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:28:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:28:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:28:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:28:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:28:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:28:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:28:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:28:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:28:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:28:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:28:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:28:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:28:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:28:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:28:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:28:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:28:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:28:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:28:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:28:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:28:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:28:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:28:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:28:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:28:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:28:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:28:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:28:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:28:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:28:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:28:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:28:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:28:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:28:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:28:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:28:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:28:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:28:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:28:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:28:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:28:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:28:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:28:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:28:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:28:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:28:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:28:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:28:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:28:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:28:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:28:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:28:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:28:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:28:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:28:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:28:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:28:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:28:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:28:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:28:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:28:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:28:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:28:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:28:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:28:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:28:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:28:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:28:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:28:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:28:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:28:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:28:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:28:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:28:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:28:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:28:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:28:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:28:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:28:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:28:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:28:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:28:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:28:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:28:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:28:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:28:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:28:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:28:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:28:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:28:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:28:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:28:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:28:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:28:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:28:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:28:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:28:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:28:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:28:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:28:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:28:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:28:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:28:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:28:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:28:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:28:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:28:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:28:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:28:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:28:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:28:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:28:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:28:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:28:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:28:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:28:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:28:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:28:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:28:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:28:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:28:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:28:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:28:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:28:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:28:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:28:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:28:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:28:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:28:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:28:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:28:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:28:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:28:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:28:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:28:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:28:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:28:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:28:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:28:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:28:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:28:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:28:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:28:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:28:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:28:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:28:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:28:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:28:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:28:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:28:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:28:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:28:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:28:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:28:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:28:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:28:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:28:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:28:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:28:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:28:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:28:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:29:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:29:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:29:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:29:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:29:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:29:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:29:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:29:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:29:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:29:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:29:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:29:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:29:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:29:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:29:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:29:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:29:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:29:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:29:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:29:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:29:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:29:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:29:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:29:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:29:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:29:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:29:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:29:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:29:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:29:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:29:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:29:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:29:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:29:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:29:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:29:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:29:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:29:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:29:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:29:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:29:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:29:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:29:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:29:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:29:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:29:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:29:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:29:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:29:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:29:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:29:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:29:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:29:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:29:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:29:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:29:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:29:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:29:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:29:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:29:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:29:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:29:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:29:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:29:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:29:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:29:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:29:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:29:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:29:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:29:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:29:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:29:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:29:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:29:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:29:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:29:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:29:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:29:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:29:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:29:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:29:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:29:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:29:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:29:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:29:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:29:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:29:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:29:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:29:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:29:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:29:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:29:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:29:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:29:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:29:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:29:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:29:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:29:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:29:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:29:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:29:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:29:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:29:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:29:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:29:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:29:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:29:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:29:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:29:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:29:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:29:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:29:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:29:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:29:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:29:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:29:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:29:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:29:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:29:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:29:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:29:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:29:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:29:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:29:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:29:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:29:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:29:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:29:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:29:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:29:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:29:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:29:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:29:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:29:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:29:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:29:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:29:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:29:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:29:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:29:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:29:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:29:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:29:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:29:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:29:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:29:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:29:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:29:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:29:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:29:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:29:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:29:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:29:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:29:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:29:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:29:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:29:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:29:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:29:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:29:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:29:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:29:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:29:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:29:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:29:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:29:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:29:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:29:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:29:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:29:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:29:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:29:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:29:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:29:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:29:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:29:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:29:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:29:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:29:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:29:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:29:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:29:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:29:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:29:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:29:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:29:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:29:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:29:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:29:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:29:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:29:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:29:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:29:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:29:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:29:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:29:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:29:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:29:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:29:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:29:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:29:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:29:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:29:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:29:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:29:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:29:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:29:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:29:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:29:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:29:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:29:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:29:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:29:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:29:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:29:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:29:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:30:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:30:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:30:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:30:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:30:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:30:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:30:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:30:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:30:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:30:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:30:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:30:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:30:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:30:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:30:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:30:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:30:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:30:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:30:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:30:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:30:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:30:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:30:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:30:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:30:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:30:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:30:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:30:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:30:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:30:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:30:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:30:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:30:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:30:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:30:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:30:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:30:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:30:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:30:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:30:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:30:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:30:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:30:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:30:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:30:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:30:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:30:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:30:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:30:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:30:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:30:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:30:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:30:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:30:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:30:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:30:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:30:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:30:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:30:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:30:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:30:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:30:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:30:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:30:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:30:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:30:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:30:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:30:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:30:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:30:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:30:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:30:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:30:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:30:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:30:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:30:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:30:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:30:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:30:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:30:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:30:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:30:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:30:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:30:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:30:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:30:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:30:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:30:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:30:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:30:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:30:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:30:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:30:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:30:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:30:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:30:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:30:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:30:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:30:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:30:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:30:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:30:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:30:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:30:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:30:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:30:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:30:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:30:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:30:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:30:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:30:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:30:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:30:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:30:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:30:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:30:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:30:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:30:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:30:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:30:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:30:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:30:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:30:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:30:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:30:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:30:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:30:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:30:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:30:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:30:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:30:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:30:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:30:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:30:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:30:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:30:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:30:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:30:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:30:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:30:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:30:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:30:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:30:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:30:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:30:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:30:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:30:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:30:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:30:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:30:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:30:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:30:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:30:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:30:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:30:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:30:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:30:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:30:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:30:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:30:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:30:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:30:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:30:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:30:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:30:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:30:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:30:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:30:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:30:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:30:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:30:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:30:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:30:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:30:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:30:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:30:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:30:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:30:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:30:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:30:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:30:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:30:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:30:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:30:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:30:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:30:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:30:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:30:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:30:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:30:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:30:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:30:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:30:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:30:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:30:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:30:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:30:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:30:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:30:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:30:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:30:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:30:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:30:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:30:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:30:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:30:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:30:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:30:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:30:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:30:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:30:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:30:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:30:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:30:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:30:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:30:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:31:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:31:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:31:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:31:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:31:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:31:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:31:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:31:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:31:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:31:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:31:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:31:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:31:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:31:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:31:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:31:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:31:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:31:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:31:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:31:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:31:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:31:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:31:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:31:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:31:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:31:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:31:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:31:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:31:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:31:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:31:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:31:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:31:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:31:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:31:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:31:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:31:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:31:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:31:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:31:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:31:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:31:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:31:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:31:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:31:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:31:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:31:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:31:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:31:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:31:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:31:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:31:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:31:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:31:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:31:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:31:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:31:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:31:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:31:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:31:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:31:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:31:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:31:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:31:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:31:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:31:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:31:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:31:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:31:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:31:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:31:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:31:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:31:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:31:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:31:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:31:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:31:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:31:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:31:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:31:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:31:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:31:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:31:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:31:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:31:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:31:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:31:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:31:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:31:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:31:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:31:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:31:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:31:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:31:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:31:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:31:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:31:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:31:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:31:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:31:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:31:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:31:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:31:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:31:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:31:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:31:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:31:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:31:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:31:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:31:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:31:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:31:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:31:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:31:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:31:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:31:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:31:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:31:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:31:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:31:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:31:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:31:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:31:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:31:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:31:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:31:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:31:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:31:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:31:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:31:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:31:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:31:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:31:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:31:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:31:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:31:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:31:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:31:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:31:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:31:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:31:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:31:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:31:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:31:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:31:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:31:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:31:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:31:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:31:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:31:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:31:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:31:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:31:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:31:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:31:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:31:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:31:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:31:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:31:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:31:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:31:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:31:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:31:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:31:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:31:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:31:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:31:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:31:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:31:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:31:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:31:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:31:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:31:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:31:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:31:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:31:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:31:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:31:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:31:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:31:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:31:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:31:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:31:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:31:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:31:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:31:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:31:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:31:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:31:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:31:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:31:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:31:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:31:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:31:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:31:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:31:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:31:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:31:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:31:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:31:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:31:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:31:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:31:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:31:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:31:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:31:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:31:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:31:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:31:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:31:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:31:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:31:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:31:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:31:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:31:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:31:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:32:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:32:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:32:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:32:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:32:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:32:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:32:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:32:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:32:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:32:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:32:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:32:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:32:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:32:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:32:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:32:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:32:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:32:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:32:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:32:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:32:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:32:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:32:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:32:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:32:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:32:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:32:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:32:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:32:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:32:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:32:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:32:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:32:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:32:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:32:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:32:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:32:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:32:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:32:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:32:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:32:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:32:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:32:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:32:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:32:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:32:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:32:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:32:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:32:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:32:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:32:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:32:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:32:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:32:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:32:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:32:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:32:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:32:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:32:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:32:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:32:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:32:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:32:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:32:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:32:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:32:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:32:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:32:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:32:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:32:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:32:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:32:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:32:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:32:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:32:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:32:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:32:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:32:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:32:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:32:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:32:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:32:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:32:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:32:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:32:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:32:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:32:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:32:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:32:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:32:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:32:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:32:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:32:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:32:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:32:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:32:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:32:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:32:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:32:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:32:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:32:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:32:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:32:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:32:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:32:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:32:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:32:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:32:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:32:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:32:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:32:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:32:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:32:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:32:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:32:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:32:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:32:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:32:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:32:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:32:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:32:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:32:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:32:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:32:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:32:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:32:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:32:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:32:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:32:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:32:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:32:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:32:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:32:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:32:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:32:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:32:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:32:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:32:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:32:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:32:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:32:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:32:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:32:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:32:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:32:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:32:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:32:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:32:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:32:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:32:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:32:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:32:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:32:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:32:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:32:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:32:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:32:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:32:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:32:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:32:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:32:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:32:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:32:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:32:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:32:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:32:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:32:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:32:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:32:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:32:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:32:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:32:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:32:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:32:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:32:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:32:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:32:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:32:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:32:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:32:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:32:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:32:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:32:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:32:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:32:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:32:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:32:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:32:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:32:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:32:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:32:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:32:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:32:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:32:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:32:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:32:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:32:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:32:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:32:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:32:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:32:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:32:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:32:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:32:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:32:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:32:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:32:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:32:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:32:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:32:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:32:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:32:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:32:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:32:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:32:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:32:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:33:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:33:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:33:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:33:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:33:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:33:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:33:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:33:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:33:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:33:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:33:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:33:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:33:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:33:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:33:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:33:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:33:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:33:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:33:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:33:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:33:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:33:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:33:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:33:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:33:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:33:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:33:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:33:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:33:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:33:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:33:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:33:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:33:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:33:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:33:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:33:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:35:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:35:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:35:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:35:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:35:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:35:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:35:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:35:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:35:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:35:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:35:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:35:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:35:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:35:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:35:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:35:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:35:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:35:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:35:17 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 11:35:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:35:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:35:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:35:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:35:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:35:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:35:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:35:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:35:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:35:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:35:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:35:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:35:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:35:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:35:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:35:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:35:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:35:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:35:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:35:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:35:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:35:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:35:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:35:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:35:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:35:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:35:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:35:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:35:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:35:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:35:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:35:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:35:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:35:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:35:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:35:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:35:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:35:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:35:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:35:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:35:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:35:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:35:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:35:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:35:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:35:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:35:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:35:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:35:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:35:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:35:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:35:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:35:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:35:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:35:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:35:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:35:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:35:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:35:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:35:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:35:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:35:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:35:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:35:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:35:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:35:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:35:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:35:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:35:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:35:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:35:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:35:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:35:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:35:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:35:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:35:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:35:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:35:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:35:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:35:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:35:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:35:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:35:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:35:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:35:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:35:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:35:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:35:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:35:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:35:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:35:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:35:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:35:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:35:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:35:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:35:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:35:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:35:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:35:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:35:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:35:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:35:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:35:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:35:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:35:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:35:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:35:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:35:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:35:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:35:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:35:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:35:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:35:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:35:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:35:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:35:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:35:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:35:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:35:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:35:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:35:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:35:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:35:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:35:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:35:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:35:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:35:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:35:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:35:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:35:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:35:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:35:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:35:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:35:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:35:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:35:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:35:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:35:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:35:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:35:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:35:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:35:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:35:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:35:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:36:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:36:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:36:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:36:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:36:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:36:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:36:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:36:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:36:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:36:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:36:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:36:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:36:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:36:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:36:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:36:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:36:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:36:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:36:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:36:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:36:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:36:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:36:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:36:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:36:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:36:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:36:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:36:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:36:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:36:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:36:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:36:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:36:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:36:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:36:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:36:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:36:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:36:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:36:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:36:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:36:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:36:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:36:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:36:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:36:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:36:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:36:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:36:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:36:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:36:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:36:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:36:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:36:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:36:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:36:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:36:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:36:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:36:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:36:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:36:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:36:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:36:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:36:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:36:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:36:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:36:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:36:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:36:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:36:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:36:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:36:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:36:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:36:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:36:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:36:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:36:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:36:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:36:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:36:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:36:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:36:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:36:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:36:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:36:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:36:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:36:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:36:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:36:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:36:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:36:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:36:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:36:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:36:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:36:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:36:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:36:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:36:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:36:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:36:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:36:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:36:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:36:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:36:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:36:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:36:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:36:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:36:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:36:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:36:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:36:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:36:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:36:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:36:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:36:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:36:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:36:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:36:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:36:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:36:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:36:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:36:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:36:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:36:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:36:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:36:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:36:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:36:32 rdma06 systemd-journald[790]: Data hash table of /run/log/journal/95379d2f723847b68d9e976dee8967a9/system.journal has a fill level at 75.0 (163750 of 218332 items, 75497472 file size, 461 bytes per hash table item), suggesting rotation.
Apr 18 11:36:32 rdma06 systemd-journald[790]: /run/log/journal/95379d2f723847b68d9e976dee8967a9/system.journal: Journal header limits reached or header out-of-date, rotating.
Apr 18 11:36:32 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 18 11:36:32 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 18 11:36:32 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 18 11:36:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:36:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:36:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:36:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:36:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:36:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:36:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:36:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:36:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:36:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:36:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:36:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:36:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:36:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:36:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:36:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:36:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:36:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:36:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:36:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:36:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:36:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:36:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:36:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:36:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:36:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:36:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:36:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:36:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:36:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:36:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:36:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:36:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:36:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:36:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:36:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:36:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:36:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:36:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:36:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:36:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:36:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:36:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:36:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:36:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:36:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:36:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:36:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:36:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:36:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:36:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:36:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:36:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:36:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:36:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:36:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:36:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:36:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:36:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:36:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:36:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:36:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:36:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:36:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:36:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:36:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:36:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:36:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:36:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:36:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:36:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:36:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:36:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:36:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:36:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:36:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:36:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:36:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:36:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:36:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:36:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:36:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:36:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:36:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:36:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:36:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:36:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:36:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:36:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:36:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:37:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:37:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:37:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:37:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:37:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:37:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:37:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:37:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:37:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:37:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:37:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:37:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:37:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:37:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:37:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:37:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:37:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:37:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:37:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:37:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:37:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:37:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:37:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:37:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:37:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:37:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:37:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:37:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:37:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:37:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:37:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:37:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:37:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:37:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:37:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:37:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:37:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:37:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:37:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:37:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:37:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:37:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:37:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:37:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:37:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:37:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:37:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:37:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:37:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:37:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:37:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:37:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:37:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:37:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:37:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:37:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:37:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:37:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:37:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:37:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:37:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:37:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:37:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:37:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:37:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:37:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:37:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:37:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:37:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:37:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:37:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:37:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:37:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:37:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:37:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:37:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:37:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:37:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:37:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:37:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:37:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:37:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:37:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:37:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:37:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:37:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:37:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:37:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:37:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:37:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:37:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:37:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:37:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:37:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:37:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:37:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:37:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:37:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:37:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:37:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:37:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:37:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:37:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:37:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:37:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:37:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:37:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:37:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:37:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:37:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:37:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:37:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:37:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:37:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:37:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:37:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:37:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:37:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:37:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:37:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:37:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:37:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:37:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:37:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:37:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:37:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:37:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:37:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:37:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:37:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:37:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:37:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:37:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:37:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:37:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:37:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:37:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:37:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:37:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:37:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:37:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:37:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:37:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:37:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:37:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:37:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:37:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:37:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:37:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:37:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:37:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:37:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:37:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:37:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:37:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:37:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:37:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:37:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:37:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:37:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:37:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:37:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:37:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:37:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:37:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:37:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:37:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:37:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:37:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:37:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:37:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:37:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:37:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:37:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:37:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:37:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:37:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:37:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:37:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:37:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:37:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:37:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:37:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:37:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:37:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:37:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:37:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:37:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:37:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:37:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:37:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:37:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:37:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:37:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:37:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:37:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:37:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:37:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:37:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:37:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:37:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:37:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:37:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:37:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:37:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:37:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:37:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:37:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:37:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:37:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:37:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:37:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:37:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:37:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:37:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:37:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:38:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:38:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:38:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:38:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:38:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:38:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:38:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:38:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:38:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:38:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:38:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:38:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:38:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:38:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:38:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:38:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:38:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:38:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:38:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:38:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:38:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:38:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:38:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:38:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:38:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:38:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:38:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:38:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:38:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:38:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:38:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:38:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:38:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:38:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:38:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:38:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:38:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:38:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:38:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:38:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:38:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:38:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:38:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:38:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:38:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:38:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:38:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:38:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:38:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:38:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:38:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:38:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:38:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:38:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:38:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:38:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:38:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:38:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:38:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:38:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:38:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:38:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:38:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:38:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:38:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:38:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:38:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:38:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:38:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:38:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:38:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:38:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:38:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:38:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:38:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:38:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:38:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:38:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:38:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:38:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:38:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:38:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:38:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:38:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:38:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:38:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:38:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:38:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:38:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:38:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:38:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:38:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:38:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:38:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:38:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:38:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:38:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:38:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:38:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:38:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:38:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:38:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:38:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:38:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:38:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:38:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:38:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:38:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:38:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:38:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:38:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:38:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:38:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:38:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:38:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:38:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:38:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:38:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:38:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:38:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:38:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:38:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:38:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:38:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:38:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:38:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:38:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:38:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:38:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:38:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:38:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:38:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:38:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:38:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:38:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:38:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:38:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:38:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:38:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:38:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:38:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:38:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:38:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:38:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:38:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:38:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:38:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:38:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:38:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:38:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:38:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:38:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:38:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:38:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:38:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:38:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:38:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:38:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:38:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:38:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:38:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:38:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:38:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:38:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:38:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:38:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:38:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:38:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:38:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:38:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:38:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:38:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:38:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:38:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:38:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:38:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:38:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:38:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:38:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:38:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:38:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:38:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:38:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:38:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:38:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:38:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:38:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:38:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:38:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:38:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:38:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:38:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:38:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:38:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:38:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:38:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:38:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:38:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:38:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:38:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:38:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:38:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:38:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:38:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:38:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:38:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:38:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:38:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:38:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:38:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:38:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:38:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:38:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:38:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:38:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:38:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:39:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:39:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:39:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:39:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:39:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:39:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:39:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:39:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:39:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:39:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:39:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:39:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:39:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:39:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:39:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:39:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:39:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:39:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:39:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:39:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:39:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:39:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:39:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:39:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:39:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:39:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:39:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:39:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:39:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:39:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:39:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:39:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:39:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:39:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:39:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:39:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:39:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:39:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:39:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:39:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:39:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:39:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:39:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:39:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:39:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:39:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:39:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:39:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:39:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:39:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:39:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:39:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:39:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:39:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:39:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:39:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:39:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:39:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:39:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:39:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:39:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:39:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:39:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:39:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:39:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:39:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:39:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:39:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:39:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:39:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:39:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:39:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:39:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:39:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:39:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:39:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:39:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:39:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:39:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:39:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:39:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:39:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:39:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:39:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:39:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:39:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:39:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:39:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:39:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:39:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:39:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:39:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:39:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:39:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:39:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:39:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:39:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:39:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:39:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:39:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:39:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:39:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:39:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:39:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:39:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:39:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:39:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:39:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:39:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:39:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:39:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:39:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:39:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:39:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:39:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:39:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:39:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:39:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:39:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:39:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:39:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:39:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:39:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:39:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:39:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:39:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:39:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:39:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:39:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:39:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:39:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:39:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:39:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:39:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:39:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:39:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:39:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:39:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:39:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:39:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:39:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:39:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:39:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:39:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:39:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:39:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:39:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:39:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:39:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:39:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:39:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:39:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:39:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:39:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:39:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:39:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:39:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:39:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:39:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:39:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:39:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:39:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:39:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:39:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:39:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:39:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:39:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:39:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:39:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:39:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:39:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:39:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:39:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:39:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:39:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:39:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:39:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:39:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:39:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:39:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:39:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:39:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:39:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:39:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:39:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:39:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:39:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:39:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:39:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:39:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:39:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:39:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:39:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:39:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:39:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:39:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:39:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:39:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:39:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:39:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:39:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:39:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:39:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:39:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:39:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:39:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:39:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:39:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:39:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:39:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:39:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:39:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:39:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:39:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:39:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:39:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:40:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:40:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:40:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:40:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:40:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:40:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:40:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:40:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:40:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:40:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:40:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:40:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:40:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:40:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:40:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:40:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:40:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:40:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:40:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:40:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:40:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:40:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:40:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:40:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:40:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:40:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:40:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:40:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:40:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:40:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:40:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:40:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:40:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:40:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:40:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:40:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:40:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:40:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:40:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:40:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:40:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:40:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:40:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:40:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:40:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:40:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:40:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:40:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:40:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:40:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:40:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:40:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:40:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:40:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:40:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:40:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:40:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:40:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:40:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:40:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:40:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:40:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:40:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:40:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:40:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:40:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:40:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:40:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:40:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:40:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:40:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:40:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:40:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:40:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:40:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:40:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:40:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:40:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:40:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:40:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:40:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:40:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:40:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:40:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:40:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:40:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:40:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:40:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:40:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:40:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:40:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:40:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:40:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:40:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:40:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:40:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:40:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:40:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:40:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:40:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:40:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:40:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:40:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:40:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:40:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:40:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:40:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:40:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:40:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:40:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:40:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:40:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:40:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:40:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:40:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:40:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:40:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:40:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:40:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:40:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:40:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:40:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:40:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:40:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:40:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:40:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:40:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:40:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:40:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:40:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:40:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:40:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:40:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:40:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:40:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:40:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:40:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:40:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:40:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:40:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:40:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:40:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:40:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:40:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:40:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:40:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:40:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:40:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:40:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:40:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:40:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:40:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:40:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:40:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:40:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:40:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:40:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:40:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:40:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:40:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:40:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:40:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:40:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:40:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:40:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:40:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:40:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:40:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:40:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:40:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:40:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:40:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:40:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:40:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:40:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:40:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:40:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:40:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:40:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:40:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:40:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:40:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:40:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:40:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:40:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:40:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:40:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:40:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:40:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:40:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:40:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:40:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:40:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:40:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:40:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:40:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:40:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:40:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:40:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:40:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:40:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:40:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:40:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:40:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:40:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:40:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:40:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:40:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:40:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:40:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:40:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:40:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:40:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:40:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:40:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:40:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:41:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:41:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:41:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:41:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:41:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:41:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:41:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:41:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:41:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:41:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:41:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:41:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:41:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:41:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:41:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:41:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:41:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:41:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:41:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:41:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:41:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:41:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:41:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:41:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:41:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:41:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:41:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:41:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:41:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:41:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:41:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:41:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:41:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:41:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:41:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:41:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:41:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:41:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:41:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:41:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:41:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:41:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:41:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:41:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:41:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:41:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:41:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:41:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:41:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:41:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:41:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:41:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:41:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:41:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:41:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:41:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:41:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:41:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:41:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:41:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:41:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:41:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:41:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:41:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:41:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:41:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:41:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:41:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:41:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:41:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:41:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:41:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:41:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:41:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:41:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:41:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:41:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:41:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:41:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:41:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:41:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:41:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:41:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:41:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:41:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:41:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:41:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:41:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:41:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:41:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:41:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:41:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:41:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:41:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:41:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:41:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:41:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:41:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:41:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:41:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:41:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:41:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:41:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:41:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:41:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:41:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:41:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:41:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:41:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:41:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:41:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:41:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:41:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:41:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:41:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:41:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:41:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:41:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:41:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:41:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:41:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:41:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:41:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:41:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:41:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:41:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:41:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:41:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:41:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:41:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:41:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:41:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:41:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:41:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:41:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:41:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:41:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:41:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:41:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:41:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:41:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:41:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:41:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:41:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:41:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:41:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:41:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:41:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:41:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:41:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:41:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:41:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:41:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:41:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:41:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:41:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:41:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:41:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:41:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:41:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:41:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:41:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:41:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:41:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:41:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:41:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:41:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:41:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:41:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:41:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:41:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:41:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:41:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:41:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:41:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:41:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:41:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:41:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:41:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:41:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:41:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:41:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:41:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:41:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:41:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:41:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:41:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:41:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:41:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:41:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:41:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:41:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:41:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:41:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:41:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:41:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:41:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:41:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:41:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:41:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:41:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:41:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:41:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:41:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:41:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:41:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:41:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:41:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:41:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:41:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:41:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:41:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:41:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:41:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:41:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:41:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:42:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:42:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:42:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:42:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:42:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:42:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:42:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:42:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:42:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:42:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:42:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:42:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:42:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:42:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:42:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:42:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:42:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:42:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:42:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:42:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:42:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:42:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:42:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:42:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:42:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:42:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:42:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:42:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:42:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:42:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:42:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:42:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:42:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:42:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:42:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:42:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:42:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:42:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:42:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:42:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:42:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:42:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:42:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:42:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:42:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:42:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:42:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:42:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:42:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:42:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:42:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:42:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:42:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:42:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:42:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:42:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:42:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:42:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:42:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:42:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:42:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:42:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:42:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:42:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:42:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:42:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:42:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:42:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:42:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:42:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:42:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:42:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:42:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:42:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:42:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:42:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:42:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:42:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:42:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:42:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:42:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:42:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:42:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:42:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:42:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:42:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:42:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:42:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:42:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:42:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:42:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:42:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:42:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:42:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:42:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:42:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:42:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:42:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:42:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:42:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:42:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:42:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:42:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:42:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:42:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:42:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:42:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:42:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:42:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:42:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:42:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:42:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:42:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:42:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:42:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:42:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:42:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:42:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:42:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:42:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:42:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:42:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:42:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:42:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:42:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:42:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:42:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:42:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:42:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:42:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:42:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:42:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:42:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:42:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:42:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:42:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:42:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:42:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:42:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:42:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:42:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:42:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:42:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:42:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:42:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:42:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:42:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:42:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:42:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:42:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:42:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:42:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:42:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:42:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:42:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:42:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:42:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:42:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:42:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:42:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:42:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:42:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:42:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:42:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:42:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:42:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:42:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:42:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:42:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:42:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:42:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:42:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:42:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:42:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:42:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:42:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:42:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:42:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:42:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:42:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:42:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:42:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:42:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:42:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:42:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:42:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:42:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:42:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:42:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:42:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:42:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:42:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:42:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:42:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:42:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:42:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:42:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:42:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:42:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:42:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:42:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:42:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:42:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:42:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:42:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:42:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:42:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:42:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:42:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:42:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:42:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:42:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:42:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:42:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:42:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:42:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:43:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:43:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:43:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:43:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:43:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:43:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:43:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:43:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:43:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:43:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:43:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:43:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:43:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:43:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:43:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:43:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:43:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:43:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:43:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:43:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:43:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:43:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:43:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:43:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:43:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:43:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:43:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:43:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:43:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:43:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:43:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:43:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:43:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:43:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:43:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:43:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:43:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:43:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:43:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:43:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:43:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:43:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:43:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:43:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:43:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:43:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:43:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:43:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:43:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:43:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:43:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:43:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:43:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:43:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:45:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:45:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:45:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:45:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:45:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:45:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:45:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:45:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:45:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:45:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:45:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:45:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:45:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:45:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:45:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:45:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:45:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:45:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:45:18 rdma06 rsyslogd[1041]: imjournal: 5087 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 11:45:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:45:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:45:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:45:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:45:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:45:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:45:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:45:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:45:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:45:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:45:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:45:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:45:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:45:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:45:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:45:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:45:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:45:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:45:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:45:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:45:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:45:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:45:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:45:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:45:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:45:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:45:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:45:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:45:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:45:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:45:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:45:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:45:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:45:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:45:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:45:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:45:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:45:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:45:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:45:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:45:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:45:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:45:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:45:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:45:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:45:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:45:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:45:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:45:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:45:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:45:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:45:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:45:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:45:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:45:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:45:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:45:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:45:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:45:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:45:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:45:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:45:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:45:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:45:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:45:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:45:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:45:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:45:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:45:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:45:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:45:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:45:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:45:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:45:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:45:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:45:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:45:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:45:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:45:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:45:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:45:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:45:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:45:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:45:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:45:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:45:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:45:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:45:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:45:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:45:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:45:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:45:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:45:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:45:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:45:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:45:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:45:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:45:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:45:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:45:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:45:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:45:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:45:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:45:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:45:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:45:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:45:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:45:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:45:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:45:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:45:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:45:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:45:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:45:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:45:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:45:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:45:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:45:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:45:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:45:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:45:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:45:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:45:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:45:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:45:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:45:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:45:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:45:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:45:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:45:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:45:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:45:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:45:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:45:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:45:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:45:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:45:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:45:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:45:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:45:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:45:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:45:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:45:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:45:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:46:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:46:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:46:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:46:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:46:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:46:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:46:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:46:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:46:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:46:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:46:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:46:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:46:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:46:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:46:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:46:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:46:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:46:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:46:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:46:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:46:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:46:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:46:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:46:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:46:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:46:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:46:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:46:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:46:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:46:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:46:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:46:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:46:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:46:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:46:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:46:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:46:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:46:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:46:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:46:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:46:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:46:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:46:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:46:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:46:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:46:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:46:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:46:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:46:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:46:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:46:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:46:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:46:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:46:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:46:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:46:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:46:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:46:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:46:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:46:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:46:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:46:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:46:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:46:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:46:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:46:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:46:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:46:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:46:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:46:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:46:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:46:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:46:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:46:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:46:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:46:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:46:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:46:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:46:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:46:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:46:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:46:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:46:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:46:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:46:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:46:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:46:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:46:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:46:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:46:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:46:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:46:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:46:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:46:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:46:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:46:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:46:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:46:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:46:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:46:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:46:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:46:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:46:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:46:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:46:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:46:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:46:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:46:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:46:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:46:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:46:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:46:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:46:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:46:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:46:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:46:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:46:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:46:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:46:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:46:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:46:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:46:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:46:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:46:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:46:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:46:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:46:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:46:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:46:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:46:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:46:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:46:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:46:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:46:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:46:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:46:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:46:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:46:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:46:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:46:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:46:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:46:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:46:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:46:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:46:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:46:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:46:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:46:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:46:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:46:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:46:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:46:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:46:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:46:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:46:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:46:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:46:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:46:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:46:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:46:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:46:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:46:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:46:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:46:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:46:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:46:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:46:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:46:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:46:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:46:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:46:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:46:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:46:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:46:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:46:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:46:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:46:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:46:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:46:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:46:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:46:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:46:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:46:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:46:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:46:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:46:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:46:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:46:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:46:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:46:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:46:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:46:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:46:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:46:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:46:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:46:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:46:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:46:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:46:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:46:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:46:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:46:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:46:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:46:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:46:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:46:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:46:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:46:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:46:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:46:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:46:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:46:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:46:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:46:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:46:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:46:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:47:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:47:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:47:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:47:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:47:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:47:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:47:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:47:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:47:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:47:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:47:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:47:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:47:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:47:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:47:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:47:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:47:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:47:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:47:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:47:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:47:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:47:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:47:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:47:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:47:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:47:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:47:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:47:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:47:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:47:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:47:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:47:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:47:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:47:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:47:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:47:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:47:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:47:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:47:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:47:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:47:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:47:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:47:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:47:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:47:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:47:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:47:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:47:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:47:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:47:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:47:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:47:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:47:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:47:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:47:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:47:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:47:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:47:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:47:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:47:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:47:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:47:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:47:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:47:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:47:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:47:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:47:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:47:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:47:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:47:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:47:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:47:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:47:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:47:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:47:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:47:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:47:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:47:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:47:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:47:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:47:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:47:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:47:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:47:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:47:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:47:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:47:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:47:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:47:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:47:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:47:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:47:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:47:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:47:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:47:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:47:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:47:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:47:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:47:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:47:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:47:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:47:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:47:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:47:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:47:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:47:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:47:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:47:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:47:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:47:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:47:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:47:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:47:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:47:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:47:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:47:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:47:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:47:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:47:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:47:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:47:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:47:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:47:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:47:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:47:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:47:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:47:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:47:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:47:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:47:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:47:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:47:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:47:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:47:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:47:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:47:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:47:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:47:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:47:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:47:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:47:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:47:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:47:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:47:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:47:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:47:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:47:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:47:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:47:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:47:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:47:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:47:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:47:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:47:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:47:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:47:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:47:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:47:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:47:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:47:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:47:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:47:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:47:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:47:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:47:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:47:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:47:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:47:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:47:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:47:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:47:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:47:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:47:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:47:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:47:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:47:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:47:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:47:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:47:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:47:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:47:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:47:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:47:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:47:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:47:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:47:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:47:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:47:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:47:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:47:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:47:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:47:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:47:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:47:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:47:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:47:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:47:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:47:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:47:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:47:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:47:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:47:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:47:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:47:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:47:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:47:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:47:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:47:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:47:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:47:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:47:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:47:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:47:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:47:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:47:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:47:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:48:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:48:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:48:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:48:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:48:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:48:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:48:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:48:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:48:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:48:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:48:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:48:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:48:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:48:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:48:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:48:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:48:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:48:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:48:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:48:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:48:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:48:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:48:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:48:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:48:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:48:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:48:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:48:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:48:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:48:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:48:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:48:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:48:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:48:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:48:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:48:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:48:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:48:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:48:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:48:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:48:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:48:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:48:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:48:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:48:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:48:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:48:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:48:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:48:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:48:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:48:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:48:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:48:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:48:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:48:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:48:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:48:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:48:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:48:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:48:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:48:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:48:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:48:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:48:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:48:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:48:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:48:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:48:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:48:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:48:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:48:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:48:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:48:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:48:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:48:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:48:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:48:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:48:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:48:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:48:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:48:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:48:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:48:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:48:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:48:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:48:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:48:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:48:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:48:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:48:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:48:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:48:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:48:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:48:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:48:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:48:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:48:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:48:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:48:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:48:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:48:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:48:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:48:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:48:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:48:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:48:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:48:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:48:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:48:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:48:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:48:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:48:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:48:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:48:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:48:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:48:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:48:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:48:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:48:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:48:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:48:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:48:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:48:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:48:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:48:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:48:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:48:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:48:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:48:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:48:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:48:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:48:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:48:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:48:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:48:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:48:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:48:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:48:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:48:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:48:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:48:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:48:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:48:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:48:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:48:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:48:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:48:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:48:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:48:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:48:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:48:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:48:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:48:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:48:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:48:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:48:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:48:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:48:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:48:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:48:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:48:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:48:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:48:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:48:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:48:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:48:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:48:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:48:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:48:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:48:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:48:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:48:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:48:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:48:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:48:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:48:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:48:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:48:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:48:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:48:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:48:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:48:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:48:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:48:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:48:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:48:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:48:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:48:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:48:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:48:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:48:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:48:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:48:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:48:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:48:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:48:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:48:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:48:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:48:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:48:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:48:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:48:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:48:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:48:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:48:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:48:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:48:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:48:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:48:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:48:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:48:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:48:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:48:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:48:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:48:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:48:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:49:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:49:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:49:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:49:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:49:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:49:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:49:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:49:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:49:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:49:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:49:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:49:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:49:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:49:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:49:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:49:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:49:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:49:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:49:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:49:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:49:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:49:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:49:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:49:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:49:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:49:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:49:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:49:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:49:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:49:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:49:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:49:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:49:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:49:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:49:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:49:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:49:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:49:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:49:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:49:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:49:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:49:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:49:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:49:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:49:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:49:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:49:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:49:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:49:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:49:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:49:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:49:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:49:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:49:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:49:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:49:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:49:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:49:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:49:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:49:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:49:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:49:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:49:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:49:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:49:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:49:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:49:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:49:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:49:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:49:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:49:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:49:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:49:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:49:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:49:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:49:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:49:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:49:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:49:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:49:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:49:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:49:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:49:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:49:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:49:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:49:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:49:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:49:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:49:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:49:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:49:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:49:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:49:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:49:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:49:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:49:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:49:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:49:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:49:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:49:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:49:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:49:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:49:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:49:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:49:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:49:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:49:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:49:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:49:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:49:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:49:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:49:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:49:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:49:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:49:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:49:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:49:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:49:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:49:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:49:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:49:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:49:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:49:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:49:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:49:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:49:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:49:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:49:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:49:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:49:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:49:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:49:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:49:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:49:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:49:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:49:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:49:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:49:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:49:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:49:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:49:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:49:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:49:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:49:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:49:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:49:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:49:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:49:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:49:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:49:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:49:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:49:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:49:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:49:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:49:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:49:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:49:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:49:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:49:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:49:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:49:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:49:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:49:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:49:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:49:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:49:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:49:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:49:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:49:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:49:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:49:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:49:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:49:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:49:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:49:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:49:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:49:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:49:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:49:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:49:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:49:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:49:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:49:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:49:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:49:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:49:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:49:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:49:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:49:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:49:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:49:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:49:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:49:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:49:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:49:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:49:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:49:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:49:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:49:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:49:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:49:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:49:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:49:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:49:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:49:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:49:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:49:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:49:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:49:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:49:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:49:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:49:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:49:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:49:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:49:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:49:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:50:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:50:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:50:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:50:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:50:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:50:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:50:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:50:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:50:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:50:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:50:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:50:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:50:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:50:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:50:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:50:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:50:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:50:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:50:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:50:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:50:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:50:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:50:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:50:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:50:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:50:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:50:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:50:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:50:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:50:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:50:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:50:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:50:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:50:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:50:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:50:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:50:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:50:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:50:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:50:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:50:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:50:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:50:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:50:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:50:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:50:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:50:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:50:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:50:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:50:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:50:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:50:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:50:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:50:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:50:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:50:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:50:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:50:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:50:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:50:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:50:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:50:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:50:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:50:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:50:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:50:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:50:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:50:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:50:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:50:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:50:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:50:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:50:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:50:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:50:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:50:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:50:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:50:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:50:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:50:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:50:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:50:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:50:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:50:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:50:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:50:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:50:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:50:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:50:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:50:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:50:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:50:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:50:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:50:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:50:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:50:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:50:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:50:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:50:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:50:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:50:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:50:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:50:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:50:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:50:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:50:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:50:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:50:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:50:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:50:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:50:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:50:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:50:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:50:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:50:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:50:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:50:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:50:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:50:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:50:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:50:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:50:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:50:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:50:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:50:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:50:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:50:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:50:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:50:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:50:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:50:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:50:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:50:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:50:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:50:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:50:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:50:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:50:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:50:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:50:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:50:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:50:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:50:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:50:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:50:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:50:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:50:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:50:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:50:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:50:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:50:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:50:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:50:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:50:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:50:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:50:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:50:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:50:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:50:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:50:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:50:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:50:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:50:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:50:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:50:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:50:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:50:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:50:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:50:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:50:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:50:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:50:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:50:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:50:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:50:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:50:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:50:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:50:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:50:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:50:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:50:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:50:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:50:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:50:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:50:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:50:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:50:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:50:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:50:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:50:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:50:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:50:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:50:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:50:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:50:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:50:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:50:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:50:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:50:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:50:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:50:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:50:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:50:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:50:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:50:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:50:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:50:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:50:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:50:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:50:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:50:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:50:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:50:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:50:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:50:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:50:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:51:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:51:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:51:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:51:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:51:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:51:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:51:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:51:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:51:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:51:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:51:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:51:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:51:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:51:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:51:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:51:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:51:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:51:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:51:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:51:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:51:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:51:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:51:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:51:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:51:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:51:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:51:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:51:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:51:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:51:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:51:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:51:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:51:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:51:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:51:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:51:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:51:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:51:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:51:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:51:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:51:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:51:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:51:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:51:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:51:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:51:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:51:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:51:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:51:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:51:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:51:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:51:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:51:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:51:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:51:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:51:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:51:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:51:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:51:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:51:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:51:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:51:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:51:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:51:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:51:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:51:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:51:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:51:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:51:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:51:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:51:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:51:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:51:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:51:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:51:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:51:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:51:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:51:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:51:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:51:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:51:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:51:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:51:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:51:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:51:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:51:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:51:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:51:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:51:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:51:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:51:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:51:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:51:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:51:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:51:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:51:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:51:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:51:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:51:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:51:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:51:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:51:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:51:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:51:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:51:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:51:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:51:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:51:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:51:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:51:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:51:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:51:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:51:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:51:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:51:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:51:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:51:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:51:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:51:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:51:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:51:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:51:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:51:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:51:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:51:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:51:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:51:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:51:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:51:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:51:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:51:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:51:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:51:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:51:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:51:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:51:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:51:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:51:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:51:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:51:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:51:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:51:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:51:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:51:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:51:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:51:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:51:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:51:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:51:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:51:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:51:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:51:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:51:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:51:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:51:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:51:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:51:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:51:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:51:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:51:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:51:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:51:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:51:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:51:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:51:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:51:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:51:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:51:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:51:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:51:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:51:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:51:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:51:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:51:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:51:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:51:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:51:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:51:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:51:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:51:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:51:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:51:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:51:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:51:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:51:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:51:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:51:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:51:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:51:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:51:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:51:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:51:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:51:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:51:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:51:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:51:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:51:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:51:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:51:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:51:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:51:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:51:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:51:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:51:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:51:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:51:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:51:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:51:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:51:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:51:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:51:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:51:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:51:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:51:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:51:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:51:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:52:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:52:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:52:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:52:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:52:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:52:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:52:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:52:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:52:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:52:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:52:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:52:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:52:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:52:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:52:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:52:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:52:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:52:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:52:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:52:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:52:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:52:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:52:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:52:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:52:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:52:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:52:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:52:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:52:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:52:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:52:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:52:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:52:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:52:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:52:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:52:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:52:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:52:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:52:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:52:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:52:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:52:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:52:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:52:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:52:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:52:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:52:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:52:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:52:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:52:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:52:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:52:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:52:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:52:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:52:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:52:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:52:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:52:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:52:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:52:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:52:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:52:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:52:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:52:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:52:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:52:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:52:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:52:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:52:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:52:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:52:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:52:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:52:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:52:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:52:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:52:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:52:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:52:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:52:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:52:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:52:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:52:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:52:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:52:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:52:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:52:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:52:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:52:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:52:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:52:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:52:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:52:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:52:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:52:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:52:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:52:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:52:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:52:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:52:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:52:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:52:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:52:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:52:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:52:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:52:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:52:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:52:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:52:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:52:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:52:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:52:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:52:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:52:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:52:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:52:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:52:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:52:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:52:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:52:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:52:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:52:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:52:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:52:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:52:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:52:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:52:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:52:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:52:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:52:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:52:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:52:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:52:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:52:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:52:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:52:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:52:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:52:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:52:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:52:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:52:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:52:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:52:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:52:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:52:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:52:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:52:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:52:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:52:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:52:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:52:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:52:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:52:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:52:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:52:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:52:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:52:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:52:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:52:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:52:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:52:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:52:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:52:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:52:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:52:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:52:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:52:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:52:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:52:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:52:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:52:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:52:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:52:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:52:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:52:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:52:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:52:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:52:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:52:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:52:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:52:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:52:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:52:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:52:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:52:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:52:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:52:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:52:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:52:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:52:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:52:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:52:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:52:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:52:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:52:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:52:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:52:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:52:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:52:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:52:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:52:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:52:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:52:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:52:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:52:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:52:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:52:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:52:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:52:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:52:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:52:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:52:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:52:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:52:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:52:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:52:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:52:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:53:00 rdma06 systemd[1]: Starting dnf makecache...
Apr 18 11:53:01 rdma06 dnf[1189722]: Updating Subscription Management repositories.
Apr 18 11:53:01 rdma06 dnf[1189722]: Unable to read consumer identity
Apr 18 11:53:01 rdma06 dnf[1189722]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 18 11:53:01 rdma06 dnf[1189722]: Metadata cache refreshed recently.
Apr 18 11:53:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 18 11:53:01 rdma06 systemd[1]: Finished dnf makecache.
Apr 18 11:53:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:53:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:53:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:53:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:53:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:53:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:53:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:53:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:53:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:53:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:53:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:53:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:53:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:53:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:53:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:53:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:53:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:53:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:53:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:53:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:53:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:53:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:53:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:53:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:53:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:53:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:53:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:53:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:53:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:53:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:53:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:53:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:53:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:53:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:53:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:53:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:53:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:53:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:53:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:53:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:53:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:53:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:53:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:53:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:53:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:53:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:53:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:53:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:53:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:53:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:53:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:53:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:53:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:53:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:55:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:55:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:55:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:55:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:55:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:55:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:55:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:55:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:55:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:55:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:55:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:55:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:55:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:55:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:55:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:55:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:55:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:55:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:55:23 rdma06 rsyslogd[1041]: imjournal: 5298 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 11:55:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:55:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:55:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:55:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:55:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:55:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:55:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:55:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:55:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:55:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:55:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:55:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:55:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:55:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:55:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:55:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:55:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:55:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:55:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:55:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:55:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:55:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:55:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:55:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:55:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:55:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:55:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:55:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:55:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:55:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:55:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:55:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:55:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:55:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:55:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:55:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:55:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:55:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:55:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:55:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:55:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:55:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:55:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:55:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:55:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:55:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:55:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:55:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:55:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:55:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:55:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:55:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:55:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:55:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:55:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:55:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:55:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:55:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:55:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:55:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:55:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:55:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:55:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:55:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:55:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:55:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:55:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:55:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:55:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:55:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:55:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:55:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:55:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:55:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:55:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:55:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:55:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:55:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:55:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:55:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:55:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:55:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:55:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:55:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:55:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:55:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:55:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:55:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:55:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:55:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:55:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:55:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:55:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:55:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:55:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:55:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:55:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:55:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:55:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:55:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:55:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:55:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:55:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:55:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:55:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:55:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:55:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:55:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:55:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:55:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:55:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:55:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:55:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:55:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:55:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:55:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:55:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:55:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:55:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:55:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:55:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:55:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:55:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:55:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:55:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:55:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:56:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:56:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:56:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:56:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:56:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:56:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:56:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:56:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:56:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:56:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:56:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:56:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:56:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:56:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:56:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:56:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:56:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:56:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:56:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:56:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:56:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:56:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:56:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:56:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:56:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:56:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:56:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:56:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:56:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:56:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:56:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:56:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:56:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:56:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:56:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:56:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:56:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:56:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:56:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:56:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:56:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:56:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:56:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:56:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:56:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:56:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:56:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:56:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:56:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:56:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:56:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:56:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:56:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:56:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:56:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:56:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:56:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:56:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:56:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:56:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:56:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:56:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:56:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:56:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:56:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:56:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:56:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:56:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:56:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:56:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:56:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:56:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:56:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:56:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:56:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:56:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:56:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:56:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:56:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:56:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:56:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:56:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:56:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:56:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:56:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:56:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:56:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:56:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:56:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:56:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:56:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:56:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:56:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:56:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:56:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:56:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:56:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:56:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:56:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:56:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:56:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:56:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:56:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:56:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:56:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:56:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:56:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:56:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:56:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:56:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:56:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:56:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:56:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:56:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:56:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:56:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:56:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:56:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:56:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:56:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:56:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:56:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:56:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:56:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:56:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:56:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:56:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:56:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:56:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:56:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:56:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:56:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:56:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:56:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:56:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:56:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:56:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:56:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:56:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:56:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:56:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:56:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:56:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:56:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:56:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:56:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:56:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:56:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:56:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:56:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:56:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:56:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:56:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:56:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:56:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:56:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:56:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:56:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:56:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:56:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:56:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:56:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:56:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:56:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:56:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:56:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:56:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:56:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:56:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:56:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:56:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:56:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:56:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:56:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:56:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:56:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:56:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:56:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:56:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:56:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:56:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:56:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:56:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:56:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:56:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:56:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:56:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:56:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:56:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:56:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:56:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:56:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:56:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:56:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:56:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:56:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:56:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:56:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:56:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:56:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:56:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:56:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:56:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:56:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:56:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:56:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:56:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:56:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:56:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:56:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:56:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:56:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:56:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:56:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:56:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:56:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:57:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:57:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:57:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:57:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:57:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:57:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:57:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:57:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:57:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:57:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:57:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:57:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:57:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:57:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:57:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:57:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:57:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:57:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:57:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:57:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:57:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:57:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:57:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:57:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:57:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:57:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:57:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:57:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:57:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:57:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:57:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:57:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:57:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:57:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:57:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:57:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:57:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:57:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:57:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:57:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:57:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:57:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:57:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:57:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:57:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:57:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:57:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:57:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:57:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:57:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:57:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:57:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:57:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:57:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:57:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:57:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:57:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:57:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:57:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:57:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:57:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:57:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:57:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:57:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:57:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:57:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:57:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:57:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:57:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:57:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:57:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:57:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:57:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:57:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:57:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:57:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:57:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:57:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:57:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:57:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:57:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:57:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:57:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:57:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:57:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:57:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:57:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:57:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:57:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:57:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:57:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:57:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:57:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:57:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:57:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:57:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:57:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:57:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:57:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:57:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:57:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:57:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:57:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:57:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:57:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:57:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:57:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:57:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:57:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:57:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:57:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:57:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:57:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:57:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:57:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:57:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:57:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:57:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:57:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:57:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:57:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:57:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:57:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:57:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:57:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:57:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:57:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:57:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:57:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:57:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:57:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:57:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:57:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:57:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:57:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:57:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:57:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:57:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:57:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:57:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:57:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:57:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:57:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:57:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:57:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:57:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:57:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:57:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:57:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:57:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:57:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:57:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:57:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:57:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:57:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:57:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:57:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:57:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:57:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:57:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:57:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:57:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:57:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:57:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:57:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:57:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:57:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:57:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:57:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:57:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:57:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:57:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:57:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:57:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:57:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:57:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:57:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:57:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:57:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:57:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:57:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:57:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:57:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:57:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:57:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:57:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:57:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:57:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:57:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:57:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:57:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:57:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:57:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:57:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:57:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:57:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:57:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:57:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:57:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:57:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:57:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:57:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:57:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:57:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:57:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:57:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:57:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:57:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:57:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:57:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:57:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:57:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:57:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:57:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:57:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:57:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:58:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:58:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:58:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:58:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:58:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:58:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:58:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:58:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:58:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:58:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:58:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:58:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:58:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:58:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:58:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:58:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:58:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:58:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:58:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:58:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:58:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:58:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:58:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:58:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:58:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:58:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:58:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:58:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:58:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:58:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:58:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:58:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:58:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:58:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:58:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:58:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:58:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:58:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:58:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:58:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:58:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:58:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:58:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:58:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:58:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:58:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:58:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:58:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:58:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:58:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:58:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:58:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:58:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:58:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:58:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:58:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:58:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:58:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:58:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:58:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:58:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:58:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:58:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:58:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:58:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:58:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:58:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:58:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:58:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:58:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:58:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:58:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:58:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:58:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:58:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:58:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:58:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:58:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:58:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:58:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:58:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:58:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:58:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:58:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:58:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:58:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:58:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:58:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:58:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:58:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:58:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:58:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:58:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:58:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:58:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:58:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:58:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:58:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:58:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:58:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:58:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:58:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:58:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:58:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:58:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:58:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:58:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:58:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:58:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:58:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:58:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:58:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:58:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:58:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:58:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:58:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:58:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:58:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:58:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:58:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:58:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:58:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:58:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:58:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:58:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:58:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:58:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:58:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:58:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:58:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:58:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:58:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:58:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:58:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:58:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:58:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:58:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:58:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:58:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:58:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:58:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:58:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:58:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:58:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:58:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:58:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:58:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:58:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:58:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:58:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:58:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:58:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:58:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:58:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:58:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:58:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:58:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:58:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:58:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:58:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:58:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:58:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:58:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:58:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:58:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:58:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:58:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:58:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:58:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:58:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:58:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:58:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:58:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:58:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:58:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:58:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:58:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:58:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:58:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:58:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:58:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:58:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:58:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:58:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:58:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:58:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:58:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:58:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:58:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:58:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:58:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:58:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:58:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:58:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:58:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:58:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:58:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:58:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:58:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:58:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:58:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:58:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:58:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:58:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:58:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:58:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:58:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:58:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:58:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:58:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:58:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:58:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:58:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:58:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:58:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:58:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:59:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:59:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:59:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:59:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:59:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:59:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:59:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:59:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:59:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:59:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:59:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:59:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:59:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:59:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:59:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:59:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:59:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:59:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:59:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:59:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:59:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:59:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:59:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:59:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:59:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:59:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:59:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:59:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:59:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:59:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:59:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:59:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:59:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:59:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:59:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:59:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:59:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:59:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:59:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:59:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:59:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:59:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:59:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:59:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:59:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:59:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:59:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:59:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:59:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:59:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:59:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:59:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:59:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:59:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:59:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:59:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:59:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:59:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:59:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:59:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:59:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:59:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:59:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:59:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:59:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:59:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:59:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:59:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:59:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:59:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:59:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:59:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:59:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:59:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:59:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:59:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:59:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:59:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:59:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:59:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:59:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:59:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:59:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:59:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:59:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:59:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:59:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:59:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:59:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:59:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:59:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:59:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:59:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:59:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:59:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:59:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:59:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:59:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:59:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:59:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:59:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:59:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:59:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:59:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:59:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:59:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:59:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:59:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:59:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:59:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:59:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:59:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:59:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:59:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:59:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:59:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:59:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:59:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:59:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:59:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:59:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:59:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:59:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:59:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:59:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:59:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:59:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:59:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:59:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:59:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:59:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:59:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:59:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:59:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:59:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:59:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:59:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:59:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:59:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:59:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:59:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:59:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:59:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:59:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:59:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:59:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:59:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:59:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:59:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:59:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:59:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:59:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:59:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:59:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:59:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:59:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:59:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:59:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:59:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:59:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:59:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:59:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:59:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:59:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:59:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:59:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:59:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:59:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:59:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:59:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:59:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:59:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:59:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:59:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:59:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:59:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:59:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:59:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:59:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:59:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:59:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:59:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:59:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:59:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:59:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:59:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:59:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:59:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:59:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:59:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:59:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:59:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:59:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:59:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:59:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:59:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:59:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:59:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 11:59:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 11:59:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:59:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:59:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:59:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 11:59:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 11:59:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 11:59:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 11:59:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 11:59:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 11:59:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 11:59:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 11:59:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 11:59:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 11:59:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 11:59:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 11:59:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 11:59:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:00:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:00:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:00:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:00:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:00:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:00:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:00:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:00:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:00:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:00:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:00:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:00:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:00:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:00:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:00:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:00:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:00:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:00:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:00:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:00:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:00:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:00:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:00:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:00:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:00:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:00:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:00:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:00:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:00:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:00:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:00:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:00:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:00:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:00:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:00:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:00:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:00:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:00:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:00:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:00:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:00:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:00:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:00:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:00:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:00:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:00:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:00:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:00:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:00:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:00:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:00:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:00:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:00:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:00:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:00:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:00:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:00:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:00:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:00:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:00:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:00:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:00:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:00:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:00:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:00:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:00:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:00:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:00:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:00:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:00:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:00:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:00:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:00:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:00:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:00:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:00:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:00:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:00:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:00:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:00:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:00:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:00:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:00:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:00:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:00:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:00:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:00:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:00:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:00:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:00:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:00:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:00:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:00:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:00:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:00:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:00:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:00:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:00:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:00:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:00:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:00:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:00:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:00:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:00:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:00:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:00:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:00:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:00:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:00:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:00:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:00:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:00:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:00:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:00:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:00:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:00:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:00:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:00:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:00:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:00:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:00:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:00:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:00:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:00:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:00:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:00:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:00:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:00:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:00:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:00:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:00:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:00:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:00:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:00:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:00:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:00:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:00:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:00:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:00:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:00:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:00:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:00:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:00:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:00:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:00:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:00:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:00:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:00:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:00:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:00:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:00:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:00:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:00:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:00:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:00:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:00:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:00:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:00:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:00:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:00:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:00:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:00:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:00:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:00:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:00:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:00:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:00:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:00:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:00:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:00:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:00:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:00:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:00:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:00:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:00:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:00:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:00:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:00:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:00:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:00:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:00:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:00:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:00:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:00:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:00:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:00:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:00:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:00:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:00:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:00:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:00:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:00:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:00:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:00:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:00:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:00:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:00:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:00:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:00:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:00:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:00:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:00:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:00:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:00:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:00:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:00:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:00:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:00:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:00:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:00:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:00:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:00:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:00:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:00:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:00:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:00:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:01:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:01:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:01:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:01:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:01:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:01:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:01:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:01:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:01:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:01:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:01:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:01:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:01:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:01:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:01:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:01:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:01:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:01:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:01:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:01:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:01:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:01:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:01:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:01:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:01:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:01:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:01:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:01:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:01:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:01:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:01:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:01:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:01:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:01:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:01:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:01:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:01:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:01:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:01:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:01:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:01:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:01:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:01:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:01:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:01:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:01:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:01:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:01:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:01:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:01:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:01:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:01:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:01:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:01:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:01:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:01:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:01:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:01:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:01:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:01:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:01:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:01:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:01:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:01:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:01:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:01:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:01:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:01:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:01:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:01:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:01:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:01:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:01:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:01:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:01:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:01:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:01:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:01:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:01:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:01:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:01:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:01:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:01:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:01:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:01:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:01:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:01:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:01:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:01:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:01:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:01:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:01:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:01:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:01:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:01:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:01:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:01:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:01:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:01:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:01:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:01:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:01:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:01:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:01:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:01:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:01:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:01:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:01:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:01:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:01:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:01:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:01:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:01:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:01:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:01:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:01:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:01:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:01:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:01:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:01:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:01:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:01:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:01:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:01:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:01:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:01:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:01:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:01:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:01:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:01:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:01:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:01:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:01:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:01:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:01:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:01:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:01:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:01:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:01:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:01:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:01:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:01:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:01:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:01:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:01:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:01:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:01:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:01:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:01:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:01:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:01:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:01:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:01:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:01:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:01:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:01:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:01:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:01:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:01:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:01:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:01:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:01:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:01:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:01:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:01:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:01:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:01:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:01:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:01:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:01:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:01:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:01:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:01:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:01:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:01:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:01:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:01:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:01:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:01:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:01:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:01:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:01:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:01:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:01:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:01:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:01:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:01:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:01:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:01:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:01:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:01:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:01:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:01:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:01:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:01:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:01:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:01:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:01:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:01:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:01:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:01:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:01:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:01:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:01:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:01:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:01:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:01:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:01:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:01:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:01:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:01:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:01:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:01:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:01:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:01:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:01:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:02:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:02:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:02:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:02:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:02:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:02:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:02:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:02:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:02:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:02:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:02:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:02:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:02:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:02:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:02:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:02:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:02:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:02:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:02:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:02:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:02:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:02:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:02:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:02:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:02:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:02:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:02:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:02:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:02:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:02:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:02:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:02:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:02:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:02:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:02:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:02:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:02:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:02:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:02:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:02:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:02:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:02:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:02:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:02:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:02:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:02:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:02:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:02:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:02:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:02:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:02:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:02:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:02:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:02:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:02:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:02:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:02:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:02:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:02:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:02:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:02:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:02:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:02:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:02:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:02:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:02:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:02:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:02:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:02:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:02:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:02:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:02:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:02:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:02:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:02:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:02:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:02:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:02:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:02:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:02:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:02:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:02:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:02:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:02:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:02:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:02:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:02:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:02:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:02:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:02:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:02:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:02:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:02:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:02:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:02:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:02:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:02:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:02:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:02:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:02:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:02:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:02:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:02:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:02:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:02:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:02:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:02:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:02:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:02:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:02:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:02:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:02:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:02:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:02:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:02:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:02:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:02:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:02:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:02:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:02:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:02:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:02:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:02:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:02:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:02:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:02:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:02:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:02:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:02:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:02:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:02:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:02:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:02:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:02:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:02:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:02:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:02:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:02:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:02:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:02:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:02:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:02:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:02:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:02:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:02:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:02:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:02:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:02:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:02:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:02:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:02:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:02:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:02:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:02:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:02:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:02:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:02:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:02:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:02:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:02:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:02:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:02:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:02:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:02:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:02:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:02:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:02:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:02:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:02:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:02:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:02:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:02:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:02:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:02:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:02:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:02:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:02:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:02:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:02:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:02:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:02:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:02:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:02:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:02:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:02:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:02:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:02:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:02:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:02:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:02:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:02:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:02:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:02:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:02:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:02:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:02:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:02:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:02:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:02:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:02:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:02:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:02:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:02:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:02:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:02:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:02:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:02:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:02:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:02:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:02:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:02:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:02:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:02:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:02:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:02:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:02:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:03:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:03:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:03:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:03:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:03:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:03:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:03:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:03:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:03:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:03:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:03:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:03:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:03:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:03:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:03:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:03:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:03:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:03:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:03:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:03:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:03:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:03:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:03:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:03:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:03:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:03:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:03:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:03:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:03:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:03:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:03:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:03:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:03:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:03:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:03:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:03:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:03:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:03:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:03:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:03:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:03:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:03:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:03:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:03:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:03:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:03:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:03:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:03:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:03:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:03:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:03:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:03:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:03:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:03:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:03:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:03:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:03:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:03:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:03:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:03:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:03:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:03:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:03:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:03:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:03:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:03:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:03:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:03:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:03:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:03:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:03:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:03:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:05:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:05:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:05:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:05:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:05:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:05:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:05:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:05:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:05:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:05:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:05:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:05:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:05:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:05:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:05:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:05:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:05:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:05:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:05:28 rdma06 rsyslogd[1041]: imjournal: 5295 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 12:05:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:05:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:05:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:05:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:05:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:05:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:05:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:05:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:05:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:05:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:05:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:05:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:05:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:05:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:05:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:05:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:05:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:05:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:05:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:05:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:05:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:05:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:05:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:05:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:05:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:05:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:05:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:05:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:05:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:05:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:05:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:05:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:05:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:05:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:05:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:05:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:05:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:05:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:05:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:05:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:05:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:05:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:05:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:05:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:05:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:05:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:05:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:05:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:05:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:05:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:05:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:05:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:05:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:05:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:05:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:05:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:05:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:05:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:05:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:05:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:05:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:05:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:05:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:05:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:05:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:05:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:05:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:05:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:05:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:05:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:05:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:05:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:05:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:05:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:05:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:05:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:05:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:05:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:05:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:05:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:05:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:05:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:05:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:05:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:05:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:05:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:05:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:05:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:05:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:05:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:05:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:05:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:05:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:05:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:05:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:05:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:05:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:05:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:05:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:05:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:05:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:05:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:05:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:05:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:05:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:05:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:05:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:05:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:06:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:06:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:06:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:06:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:06:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:06:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:06:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:06:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:06:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:06:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:06:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:06:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:06:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:06:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:06:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:06:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:06:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:06:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:06:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:06:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:06:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:06:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:06:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:06:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:06:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:06:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:06:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:06:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:06:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:06:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:06:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:06:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:06:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:06:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:06:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:06:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:06:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:06:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:06:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:06:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:06:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:06:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:06:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:06:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:06:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:06:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:06:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:06:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:06:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:06:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:06:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:06:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:06:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:06:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:06:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:06:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:06:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:06:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:06:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:06:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:06:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:06:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:06:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:06:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:06:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:06:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:06:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:06:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:06:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:06:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:06:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:06:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:06:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:06:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:06:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:06:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:06:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:06:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:06:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:06:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:06:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:06:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:06:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:06:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:06:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:06:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:06:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:06:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:06:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:06:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:06:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:06:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:06:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:06:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:06:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:06:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:06:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:06:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:06:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:06:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:06:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:06:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:06:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:06:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:06:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:06:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:06:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:06:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:06:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:06:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:06:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:06:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:06:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:06:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:06:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:06:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:06:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:06:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:06:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:06:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:06:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:06:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:06:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:06:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:06:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:06:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:06:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:06:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:06:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:06:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:06:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:06:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:06:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:06:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:06:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:06:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:06:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:06:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:06:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:06:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:06:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:06:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:06:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:06:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:06:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:06:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:06:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:06:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:06:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:06:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:06:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:06:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:06:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:06:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:06:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:06:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:06:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:06:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:06:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:06:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:06:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:06:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:06:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:06:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:06:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:06:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:06:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:06:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:06:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:06:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:06:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:06:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:06:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:06:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:06:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:06:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:06:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:06:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:06:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:06:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:06:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:06:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:06:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:06:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:06:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:06:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:06:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:06:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:06:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:06:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:06:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:06:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:06:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:06:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:06:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:06:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:06:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:06:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:06:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:06:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:06:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:06:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:06:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:06:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:06:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:06:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:06:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:06:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:06:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:06:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:06:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:06:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:06:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:06:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:06:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:06:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:07:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:07:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:07:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:07:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:07:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:07:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:07:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:07:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:07:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:07:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:07:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:07:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:07:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:07:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:07:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:07:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:07:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:07:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:07:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:07:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:07:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:07:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:07:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:07:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:07:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:07:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:07:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:07:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:07:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:07:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:07:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:07:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:07:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:07:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:07:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:07:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:07:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:07:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:07:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:07:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:07:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:07:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:07:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:07:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:07:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:07:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:07:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:07:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:07:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:07:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:07:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:07:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:07:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:07:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:07:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:07:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:07:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:07:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:07:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:07:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:07:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:07:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:07:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:07:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:07:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:07:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:07:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:07:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:07:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:07:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:07:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:07:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:07:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:07:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:07:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:07:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:07:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:07:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:07:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:07:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:07:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:07:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:07:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:07:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:07:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:07:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:07:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:07:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:07:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:07:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:07:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:07:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:07:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:07:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:07:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:07:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:07:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:07:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:07:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:07:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:07:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:07:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:07:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:07:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:07:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:07:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:07:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:07:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:07:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:07:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:07:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:07:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:07:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:07:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:07:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:07:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:07:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:07:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:07:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:07:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:07:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:07:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:07:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:07:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:07:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:07:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:07:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:07:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:07:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:07:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:07:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:07:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:07:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:07:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:07:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:07:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:07:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:07:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:07:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:07:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:07:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:07:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:07:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:07:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:07:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:07:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:07:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:07:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:07:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:07:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:07:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:07:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:07:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:07:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:07:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:07:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:07:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:07:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:07:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:07:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:07:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:07:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:07:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:07:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:07:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:07:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:07:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:07:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:07:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:07:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:07:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:07:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:07:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:07:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:07:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:07:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:07:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:07:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:07:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:07:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:07:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:07:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:07:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:07:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:07:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:07:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:07:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:07:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:07:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:07:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:07:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:07:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:07:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:07:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:07:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:07:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:07:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:07:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:07:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:07:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:07:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:07:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:07:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:07:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:07:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:07:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:07:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:07:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:07:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:07:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:07:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:07:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:07:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:07:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:07:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:07:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:08:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:08:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:08:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:08:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:08:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:08:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:08:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:08:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:08:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:08:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:08:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:08:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:08:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:08:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:08:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:08:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:08:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:08:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:08:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:08:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:08:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:08:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:08:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:08:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:08:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:08:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:08:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:08:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:08:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:08:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:08:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:08:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:08:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:08:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:08:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:08:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:08:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:08:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:08:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:08:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:08:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:08:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:08:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:08:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:08:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:08:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:08:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:08:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:08:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:08:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:08:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:08:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:08:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:08:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:08:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:08:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:08:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:08:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:08:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:08:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:08:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:08:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:08:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:08:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:08:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:08:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:08:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:08:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:08:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:08:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:08:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:08:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:08:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:08:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:08:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:08:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:08:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:08:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:08:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:08:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:08:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:08:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:08:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:08:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:08:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:08:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:08:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:08:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:08:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:08:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:08:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:08:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:08:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:08:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:08:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:08:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:08:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:08:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:08:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:08:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:08:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:08:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:08:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:08:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:08:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:08:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:08:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:08:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:08:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:08:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:08:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:08:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:08:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:08:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:08:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:08:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:08:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:08:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:08:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:08:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:08:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:08:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:08:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:08:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:08:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:08:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:08:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:08:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:08:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:08:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:08:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:08:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:08:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:08:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:08:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:08:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:08:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:08:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:08:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:08:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:08:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:08:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:08:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:08:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:08:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:08:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:08:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:08:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:08:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:08:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:08:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:08:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:08:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:08:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:08:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:08:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:08:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:08:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:08:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:08:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:08:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:08:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:08:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:08:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:08:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:08:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:08:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:08:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:08:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:08:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:08:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:08:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:08:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:08:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:08:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:08:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:08:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:08:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:08:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:08:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:08:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:08:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:08:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:08:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:08:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:08:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:08:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:08:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:08:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:08:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:08:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:08:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:08:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:08:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:08:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:08:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:08:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:08:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:08:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:08:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:08:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:08:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:08:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:08:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:08:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:08:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:08:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:08:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:08:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:08:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:08:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:08:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:08:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:08:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:08:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:08:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:09:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:09:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:09:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:09:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:09:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:09:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:09:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:09:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:09:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:09:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:09:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:09:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:09:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:09:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:09:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:09:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:09:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:09:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:09:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:09:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:09:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:09:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:09:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:09:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:09:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:09:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:09:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:09:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:09:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:09:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:09:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:09:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:09:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:09:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:09:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:09:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:09:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:09:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:09:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:09:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:09:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:09:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:09:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:09:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:09:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:09:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:09:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:09:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:09:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:09:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:09:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:09:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:09:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:09:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:09:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:09:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:09:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:09:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:09:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:09:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:09:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:09:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:09:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:09:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:09:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:09:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:09:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:09:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:09:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:09:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:09:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:09:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:09:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:09:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:09:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:09:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:09:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:09:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:09:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:09:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:09:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:09:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:09:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:09:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:09:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:09:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:09:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:09:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:09:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:09:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:09:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:09:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:09:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:09:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:09:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:09:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:09:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:09:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:09:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:09:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:09:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:09:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:09:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:09:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:09:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:09:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:09:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:09:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:09:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:09:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:09:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:09:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:09:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:09:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:09:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:09:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:09:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:09:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:09:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:09:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:09:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:09:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:09:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:09:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:09:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:09:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:09:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:09:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:09:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:09:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:09:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:09:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:09:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:09:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:09:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:09:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:09:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:09:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:09:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:09:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:09:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:09:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:09:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:09:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:09:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:09:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:09:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:09:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:09:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:09:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:09:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:09:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:09:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:09:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:09:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:09:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:09:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:09:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:09:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:09:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:09:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:09:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:09:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:09:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:09:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:09:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:09:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:09:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:09:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:09:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:09:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:09:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:09:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:09:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:09:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:09:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:09:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:09:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:09:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:09:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:09:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:09:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:09:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:09:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:09:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:09:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:09:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:09:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:09:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:09:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:09:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:09:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:09:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:09:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:09:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:09:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:09:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:09:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:09:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:09:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:09:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:09:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:09:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:09:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:09:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:09:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:09:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:09:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:09:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:09:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:09:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:09:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:09:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:09:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:09:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:09:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:10:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:10:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:10:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:10:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:10:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:10:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:10:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:10:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:10:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:10:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:10:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:10:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:10:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:10:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:10:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:10:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:10:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:10:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:10:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:10:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:10:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:10:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:10:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:10:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:10:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:10:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:10:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:10:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:10:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:10:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:10:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:10:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:10:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:10:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:10:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:10:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:10:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:10:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:10:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:10:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:10:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:10:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:10:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:10:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:10:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:10:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:10:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:10:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:10:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:10:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:10:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:10:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:10:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:10:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:10:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:10:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:10:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:10:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:10:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:10:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:10:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:10:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:10:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:10:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:10:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:10:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:10:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:10:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:10:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:10:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:10:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:10:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:10:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:10:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:10:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:10:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:10:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:10:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:10:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:10:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:10:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:10:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:10:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:10:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:10:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:10:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:10:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:10:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:10:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:10:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:10:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:10:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:10:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:10:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:10:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:10:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:10:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:10:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:10:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:10:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:10:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:10:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:10:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:10:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:10:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:10:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:10:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:10:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:10:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:10:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:10:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:10:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:10:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:10:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:10:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:10:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:10:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:10:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:10:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:10:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:10:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:10:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:10:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:10:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:10:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:10:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:10:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:10:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:10:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:10:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:10:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:10:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:10:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:10:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:10:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:10:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:10:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:10:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:10:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:10:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:10:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:10:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:10:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:10:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:10:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:10:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:10:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:10:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:10:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:10:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:10:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:10:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:10:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:10:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:10:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:10:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:10:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:10:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:10:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:10:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:10:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:10:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:10:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:10:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:10:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:10:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:10:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:10:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:10:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:10:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:10:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:10:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:10:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:10:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:10:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:10:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:10:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:10:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:10:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:10:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:10:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:10:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:10:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:10:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:10:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:10:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:10:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:10:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:10:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:10:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:10:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:10:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:10:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:10:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:10:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:10:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:10:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:10:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:10:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:10:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:10:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:10:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:10:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:10:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:10:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:10:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:10:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:10:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:10:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:10:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:10:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:10:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:10:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:10:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:10:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:10:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:11:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:11:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:11:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:11:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:11:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:11:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:11:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:11:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:11:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:11:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:11:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:11:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:11:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:11:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:11:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:11:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:11:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:11:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:11:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:11:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:11:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:11:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:11:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:11:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:11:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:11:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:11:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:11:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:11:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:11:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:11:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:11:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:11:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:11:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:11:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:11:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:11:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:11:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:11:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:11:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:11:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:11:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:11:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:11:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:11:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:11:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:11:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:11:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:11:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:11:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:11:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:11:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:11:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:11:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:11:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:11:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:11:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:11:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:11:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:11:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:11:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:11:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:11:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:11:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:11:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:11:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:11:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:11:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:11:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:11:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:11:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:11:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:11:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:11:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:11:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:11:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:11:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:11:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:11:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:11:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:11:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:11:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:11:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:11:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:11:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:11:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:11:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:11:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:11:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:11:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:11:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:11:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:11:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:11:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:11:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:11:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:11:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:11:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:11:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:11:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:11:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:11:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:11:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:11:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:11:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:11:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:11:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:11:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:11:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:11:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:11:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:11:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:11:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:11:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:11:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:11:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:11:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:11:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:11:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:11:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:11:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:11:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:11:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:11:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:11:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:11:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:11:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:11:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:11:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:11:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:11:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:11:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:11:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:11:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:11:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:11:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:11:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:11:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:11:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:11:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:11:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:11:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:11:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:11:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:11:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:11:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:11:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:11:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:11:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:11:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:11:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:11:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:11:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:11:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:11:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:11:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:11:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:11:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:11:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:11:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:11:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:11:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:11:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:11:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:11:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:11:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:11:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:11:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:11:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:11:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:11:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:11:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:11:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:11:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:11:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:11:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:11:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:11:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:11:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:11:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:11:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:11:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:11:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:11:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:11:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:11:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:11:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:11:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:11:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:11:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:11:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:11:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:11:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:11:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:11:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:11:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:11:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:11:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:11:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:11:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:11:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:11:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:11:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:11:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:11:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:11:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:11:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:11:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:11:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:11:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:11:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:11:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:11:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:11:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:11:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:11:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:12:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:12:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:12:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:12:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:12:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:12:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:12:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:12:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:12:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:12:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:12:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:12:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:12:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:12:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:12:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:12:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:12:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:12:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:12:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:12:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:12:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:12:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:12:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:12:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:12:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:12:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:12:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:12:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:12:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:12:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:12:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:12:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:12:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:12:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:12:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:12:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:12:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:12:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:12:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:12:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:12:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:12:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:12:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:12:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:12:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:12:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:12:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:12:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:12:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:12:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:12:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:12:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:12:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:12:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:12:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:12:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:12:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:12:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:12:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:12:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:12:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:12:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:12:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:12:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:12:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:12:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:12:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:12:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:12:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:12:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:12:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:12:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:12:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:12:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:12:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:12:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:12:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:12:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:12:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:12:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:12:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:12:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:12:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:12:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:12:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:12:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:12:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:12:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:12:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:12:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:12:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:12:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:12:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:12:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:12:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:12:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:12:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:12:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:12:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:12:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:12:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:12:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:12:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:12:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:12:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:12:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:12:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:12:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:12:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:12:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:12:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:12:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:12:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:12:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:12:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:12:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:12:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:12:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:12:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:12:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:12:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:12:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:12:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:12:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:12:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:12:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:12:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:12:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:12:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:12:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:12:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:12:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:12:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:12:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:12:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:12:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:12:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:12:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:12:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:12:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:12:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:12:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:12:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:12:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:12:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:12:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:12:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:12:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:12:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:12:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:12:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:12:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:12:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:12:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:12:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:12:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:12:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:12:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:12:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:12:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:12:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:12:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:12:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:12:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:12:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:12:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:12:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:12:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:12:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:12:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:12:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:12:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:12:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:12:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:12:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:12:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:12:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:12:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:12:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:12:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:12:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:12:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:12:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:12:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:12:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:12:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:12:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:12:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:12:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:12:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:12:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:12:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:12:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:12:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:12:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:12:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:12:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:12:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:12:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:12:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:12:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:12:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:12:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:12:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:12:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:12:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:12:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:12:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:12:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:12:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:12:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:12:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:12:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:12:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:12:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:12:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:13:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:13:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:13:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:13:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:13:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:13:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:13:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:13:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:13:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:13:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:13:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:13:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:13:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:13:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:13:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:13:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:13:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:13:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:13:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:13:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:13:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:13:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:13:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:13:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:13:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:13:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:13:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:13:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:13:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:13:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:13:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:13:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:13:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:13:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:13:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:13:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:13:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:13:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:13:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:13:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:13:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:13:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:13:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:13:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:13:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:13:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:13:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:13:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:13:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:13:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:13:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:13:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:13:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:13:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:13:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:13:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:13:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:13:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:13:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:13:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:13:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:13:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:13:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:13:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:13:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:13:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:13:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:13:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:13:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:13:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:13:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:13:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:13:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:13:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:13:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:13:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:13:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:13:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:13:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:13:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:13:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:13:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:13:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:13:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:13:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:13:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:13:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:13:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:13:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:13:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:15:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:15:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:15:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:15:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:15:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:15:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:15:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:15:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:15:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:15:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:15:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:15:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:15:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:15:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:15:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:15:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:15:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:15:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:15:29 rdma06 rsyslogd[1041]: imjournal: 5082 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 12:15:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:15:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:15:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:15:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:15:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:15:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:15:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:15:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:15:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:15:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:15:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:15:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:15:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:15:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:15:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:15:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:15:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:15:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:15:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:15:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:15:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:15:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:15:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:15:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:15:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:15:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:15:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:15:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:15:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:15:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:15:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:15:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:15:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:15:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:15:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:15:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:15:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:15:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:15:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:15:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:15:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:15:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:15:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:15:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:15:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:15:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:15:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:15:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:15:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:15:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:15:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:15:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:15:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:15:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:15:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:15:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:15:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:15:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:15:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:15:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:15:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:15:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:15:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:15:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:15:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:15:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:15:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:15:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:15:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:15:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:15:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:15:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:15:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:15:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:15:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:15:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:15:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:15:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:15:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:15:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:15:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:15:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:15:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:15:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:15:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:15:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:15:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:15:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:15:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:15:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:15:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:15:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:15:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:15:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:15:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:15:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:15:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:15:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:15:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:15:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:15:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:15:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:15:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:15:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:15:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:15:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:15:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:15:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:16:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:16:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:16:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:16:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:16:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:16:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:16:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:16:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:16:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:16:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:16:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:16:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:16:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:16:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:16:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:16:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:16:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:16:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:16:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:16:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:16:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:16:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:16:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:16:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:16:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:16:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:16:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:16:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:16:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:16:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:16:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:16:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:16:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:16:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:16:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:16:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:16:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:16:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:16:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:16:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:16:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:16:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:16:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:16:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:16:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:16:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:16:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:16:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:16:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:16:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:16:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:16:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:16:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:16:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:16:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:16:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:16:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:16:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:16:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:16:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:16:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:16:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:16:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:16:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:16:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:16:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:16:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:16:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:16:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:16:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:16:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:16:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:16:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:16:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:16:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:16:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:16:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:16:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:16:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:16:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:16:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:16:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:16:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:16:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:16:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:16:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:16:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:16:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:16:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:16:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:16:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:16:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:16:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:16:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:16:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:16:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:16:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:16:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:16:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:16:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:16:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:16:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:16:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:16:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:16:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:16:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:16:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:16:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:16:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:16:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:16:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:16:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:16:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:16:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:16:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:16:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:16:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:16:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:16:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:16:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:16:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:16:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:16:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:16:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:16:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:16:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:16:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:16:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:16:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:16:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:16:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:16:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:16:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:16:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:16:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:16:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:16:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:16:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:16:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:16:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:16:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:16:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:16:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:16:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:16:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:16:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:16:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:16:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:16:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:16:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:16:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:16:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:16:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:16:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:16:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:16:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:16:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:16:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:16:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:16:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:16:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:16:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:16:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:16:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:16:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:16:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:16:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:16:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:16:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:16:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:16:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:16:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:16:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:16:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:16:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:16:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:16:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:16:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:16:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:16:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:16:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:16:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:16:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:16:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:16:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:16:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:16:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:16:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:16:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:16:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:16:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:16:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:16:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:16:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:16:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:16:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:16:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:16:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:16:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:16:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:16:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:16:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:16:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:16:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:16:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:16:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:16:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:16:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:16:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:16:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:16:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:16:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:16:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:16:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:16:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:16:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:17:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:17:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:17:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:17:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:17:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:17:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:17:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:17:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:17:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:17:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:17:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:17:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:17:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:17:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:17:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:17:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:17:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:17:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:17:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:17:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:17:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:17:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:17:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:17:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:17:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:17:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:17:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:17:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:17:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:17:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:17:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:17:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:17:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:17:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:17:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:17:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:17:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:17:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:17:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:17:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:17:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:17:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:17:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:17:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:17:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:17:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:17:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:17:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:17:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:17:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:17:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:17:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:17:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:17:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:17:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:17:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:17:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:17:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:17:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:17:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:17:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:17:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:17:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:17:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:17:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:17:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:17:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:17:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:17:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:17:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:17:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:17:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:17:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:17:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:17:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:17:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:17:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:17:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:17:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:17:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:17:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:17:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:17:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:17:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:17:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:17:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:17:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:17:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:17:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:17:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:17:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:17:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:17:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:17:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:17:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:17:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:17:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:17:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:17:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:17:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:17:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:17:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:17:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:17:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:17:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:17:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:17:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:17:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:17:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:17:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:17:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:17:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:17:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:17:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:17:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:17:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:17:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:17:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:17:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:17:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:17:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:17:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:17:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:17:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:17:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:17:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:17:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:17:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:17:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:17:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:17:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:17:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:17:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:17:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:17:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:17:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:17:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:17:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:17:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:17:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:17:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:17:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:17:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:17:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:17:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:17:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:17:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:17:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:17:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:17:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:17:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:17:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:17:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:17:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:17:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:17:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:17:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:17:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:17:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:17:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:17:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:17:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:17:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:17:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:17:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:17:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:17:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:17:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:17:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:17:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:17:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:17:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:17:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:17:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:17:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:17:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:17:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:17:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:17:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:17:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:17:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:17:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:17:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:17:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:17:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:17:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:17:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:17:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:17:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:17:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:17:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:17:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:17:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:17:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:17:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:17:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:17:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:17:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:17:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:17:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:17:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:17:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:17:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:17:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:17:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:17:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:17:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:17:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:17:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:17:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:17:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:17:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:17:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:17:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:17:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:17:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:18:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:18:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:18:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:18:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:18:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:18:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:18:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:18:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:18:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:18:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:18:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:18:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:18:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:18:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:18:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:18:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:18:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:18:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:18:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:18:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:18:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:18:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:18:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:18:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:18:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:18:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:18:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:18:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:18:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:18:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:18:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:18:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:18:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:18:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:18:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:18:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:18:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:18:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:18:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:18:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:18:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:18:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:18:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:18:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:18:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:18:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:18:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:18:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:18:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:18:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:18:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:18:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:18:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:18:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:18:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:18:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:18:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:18:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:18:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:18:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:18:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:18:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:18:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:18:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:18:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:18:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:18:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:18:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:18:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:18:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:18:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:18:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:18:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:18:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:18:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:18:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:18:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:18:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:18:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:18:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:18:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:18:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:18:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:18:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:18:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:18:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:18:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:18:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:18:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:18:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:18:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:18:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:18:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:18:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:18:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:18:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:18:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:18:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:18:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:18:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:18:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:18:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:18:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:18:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:18:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:18:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:18:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:18:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:18:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:18:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:18:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:18:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:18:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:18:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:18:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:18:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:18:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:18:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:18:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:18:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:18:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:18:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:18:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:18:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:18:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:18:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:18:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:18:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:18:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:18:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:18:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:18:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:18:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:18:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:18:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:18:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:18:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:18:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:18:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:18:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:18:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:18:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:18:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:18:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:18:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:18:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:18:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:18:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:18:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:18:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:18:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:18:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:18:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:18:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:18:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:18:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:18:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:18:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:18:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:18:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:18:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:18:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:18:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:18:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:18:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:18:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:18:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:18:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:18:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:18:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:18:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:18:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:18:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:18:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:18:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:18:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:18:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:18:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:18:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:18:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:18:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:18:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:18:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:18:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:18:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:18:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:18:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:18:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:18:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:18:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:18:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:18:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:18:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:18:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:18:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:18:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:18:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:18:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:18:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:18:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:18:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:18:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:18:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:18:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:18:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:18:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:18:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:18:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:18:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:18:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:18:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:18:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:18:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:18:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:18:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:18:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:19:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:19:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:19:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:19:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:19:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:19:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:19:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:19:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:19:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:19:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:19:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:19:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:19:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:19:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:19:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:19:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:19:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:19:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:19:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:19:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:19:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:19:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:19:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:19:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:19:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:19:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:19:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:19:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:19:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:19:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:19:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:19:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:19:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:19:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:19:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:19:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:19:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:19:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:19:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:19:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:19:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:19:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:19:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:19:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:19:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:19:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:19:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:19:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:19:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:19:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:19:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:19:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:19:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:19:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:19:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:19:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:19:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:19:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:19:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:19:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:19:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:19:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:19:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:19:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:19:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:19:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:19:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:19:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:19:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:19:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:19:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:19:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:19:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:19:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:19:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:19:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:19:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:19:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:19:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:19:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:19:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:19:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:19:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:19:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:19:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:19:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:19:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:19:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:19:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:19:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:19:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:19:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:19:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:19:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:19:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:19:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:19:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:19:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:19:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:19:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:19:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:19:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:19:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:19:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:19:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:19:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:19:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:19:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:19:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:19:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:19:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:19:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:19:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:19:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:19:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:19:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:19:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:19:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:19:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:19:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:19:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:19:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:19:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:19:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:19:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:19:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:19:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:19:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:19:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:19:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:19:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:19:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:19:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:19:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:19:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:19:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:19:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:19:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:19:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:19:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:19:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:19:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:19:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:19:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:19:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:19:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:19:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:19:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:19:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:19:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:19:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:19:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:19:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:19:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:19:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:19:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:19:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:19:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:19:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:19:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:19:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:19:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:19:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:19:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:19:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:19:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:19:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:19:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:19:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:19:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:19:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:19:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:19:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:19:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:19:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:19:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:19:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:19:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:19:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:19:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:19:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:19:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:19:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:19:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:19:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:19:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:19:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:19:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:19:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:19:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:19:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:19:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:19:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:19:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:19:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:19:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:19:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:19:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:19:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:19:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:19:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:19:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:19:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:19:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:19:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:19:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:19:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:19:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:19:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:19:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:19:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:19:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:19:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:19:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:19:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:19:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:20:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:20:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:20:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:20:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:20:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:20:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:20:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:20:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:20:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:20:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:20:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:20:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:20:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:20:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:20:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:20:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:20:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:20:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:20:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:20:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:20:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:20:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:20:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:20:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:20:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:20:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:20:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:20:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:20:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:20:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:20:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:20:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:20:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:20:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:20:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:20:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:20:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:20:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:20:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:20:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:20:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:20:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:20:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:20:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:20:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:20:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:20:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:20:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:20:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:20:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:20:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:20:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:20:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:20:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:20:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:20:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:20:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:20:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:20:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:20:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:20:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:20:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:20:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:20:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:20:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:20:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:20:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:20:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:20:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:20:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:20:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:20:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:20:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:20:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:20:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:20:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:20:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:20:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:20:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:20:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:20:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:20:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:20:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:20:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:20:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:20:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:20:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:20:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:20:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:20:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:20:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:20:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:20:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:20:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:20:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:20:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:20:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:20:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:20:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:20:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:20:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:20:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:20:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:20:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:20:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:20:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:20:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:20:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:20:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:20:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:20:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:20:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:20:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:20:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:20:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:20:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:20:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:20:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:20:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:20:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:20:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:20:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:20:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:20:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:20:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:20:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:20:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:20:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:20:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:20:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:20:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:20:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:20:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:20:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:20:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:20:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:20:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:20:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:20:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:20:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:20:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:20:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:20:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:20:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:20:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:20:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:20:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:20:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:20:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:20:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:20:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:20:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:20:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:20:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:20:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:20:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:20:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:20:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:20:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:20:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:20:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:20:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:20:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:20:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:20:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:20:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:20:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:20:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:20:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:20:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:20:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:20:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:20:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:20:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:20:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:20:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:20:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:20:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:20:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:20:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:20:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:20:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:20:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:20:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:20:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:20:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:20:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:20:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:20:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:20:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:20:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:20:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:20:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:20:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:20:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:20:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:20:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:20:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:20:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:20:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:20:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:20:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:20:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:20:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:20:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:20:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:20:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:20:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:20:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:20:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:20:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:20:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:20:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:20:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:20:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:20:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:21:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:21:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:21:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:21:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:21:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:21:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:21:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:21:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:21:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:21:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:21:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:21:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:21:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:21:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:21:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:21:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:21:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:21:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:21:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:21:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:21:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:21:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:21:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:21:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:21:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:21:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:21:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:21:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:21:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:21:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:21:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:21:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:21:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:21:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:21:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:21:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:21:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:21:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:21:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:21:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:21:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:21:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:21:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:21:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:21:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:21:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:21:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:21:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:21:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:21:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:21:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:21:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:21:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:21:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:21:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:21:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:21:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:21:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:21:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:21:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:21:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:21:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:21:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:21:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:21:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:21:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:21:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:21:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:21:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:21:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:21:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:21:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:21:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:21:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:21:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:21:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:21:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:21:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:21:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:21:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:21:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:21:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:21:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:21:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:21:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:21:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:21:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:21:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:21:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:21:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:21:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:21:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:21:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:21:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:21:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:21:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:21:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:21:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:21:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:21:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:21:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:21:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:21:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:21:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:21:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:21:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:21:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:21:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:21:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:21:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:21:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:21:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:21:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:21:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:21:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:21:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:21:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:21:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:21:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:21:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:21:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:21:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:21:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:21:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:21:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:21:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:21:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:21:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:21:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:21:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:21:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:21:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:21:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:21:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:21:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:21:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:21:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:21:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:21:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:21:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:21:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:21:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:21:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:21:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:21:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:21:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:21:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:21:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:21:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:21:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:21:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:21:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:21:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:21:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:21:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:21:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:21:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:21:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:21:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:21:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:21:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:21:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:21:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:21:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:21:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:21:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:21:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:21:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:21:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:21:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:21:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:21:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:21:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:21:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:21:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:21:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:21:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:21:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:21:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:21:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:21:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:21:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:21:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:21:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:21:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:21:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:21:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:21:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:21:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:21:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:21:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:21:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:21:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:21:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:21:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:21:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:21:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:21:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:21:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:21:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:21:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:21:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:21:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:21:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:21:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:21:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:21:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:21:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:21:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:21:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:21:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:21:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:21:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:21:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:21:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:21:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:22:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:22:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:22:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:22:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:22:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:22:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:22:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:22:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:22:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:22:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:22:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:22:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:22:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:22:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:22:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:22:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:22:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:22:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:22:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:22:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:22:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:22:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:22:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:22:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:22:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:22:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:22:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:22:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:22:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:22:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:22:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:22:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:22:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:22:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:22:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:22:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:22:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:22:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:22:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:22:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:22:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:22:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:22:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:22:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:22:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:22:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:22:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:22:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:22:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:22:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:22:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:22:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:22:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:22:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:22:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:22:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:22:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:22:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:22:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:22:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:22:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:22:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:22:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:22:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:22:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:22:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:22:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:22:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:22:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:22:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:22:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:22:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:22:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:22:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:22:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:22:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:22:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:22:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:22:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:22:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:22:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:22:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:22:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:22:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:22:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:22:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:22:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:22:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:22:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:22:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:22:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:22:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:22:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:22:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:22:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:22:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:22:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:22:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:22:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:22:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:22:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:22:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:22:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:22:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:22:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:22:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:22:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:22:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:22:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:22:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:22:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:22:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:22:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:22:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:22:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:22:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:22:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:22:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:22:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:22:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:22:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:22:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:22:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:22:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:22:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:22:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:22:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:22:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:22:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:22:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:22:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:22:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:22:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:22:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:22:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:22:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:22:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:22:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:22:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:22:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:22:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:22:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:22:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:22:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:22:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:22:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:22:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:22:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:22:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:22:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:22:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:22:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:22:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:22:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:22:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:22:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:22:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:22:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:22:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:22:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:22:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:22:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:22:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:22:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:22:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:22:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:22:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:22:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:22:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:22:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:22:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:22:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:22:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:22:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:22:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:22:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:22:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:22:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:22:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:22:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:22:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:22:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:22:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:22:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:22:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:22:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:22:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:22:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:22:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:22:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:22:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:22:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:22:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:22:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:22:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:22:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:22:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:22:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:22:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:22:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:22:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:22:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:22:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:22:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:22:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:22:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:22:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:22:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:22:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:22:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:22:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:22:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:22:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:22:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:22:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:22:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:23:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:23:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:23:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:23:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:23:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:23:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:23:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:23:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:23:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:23:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:23:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:23:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:23:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:23:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:23:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:23:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:23:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:23:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:23:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:23:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:23:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:23:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:23:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:23:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:23:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:23:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:23:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:23:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:23:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:23:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:23:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:23:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:23:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:23:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:23:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:23:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:23:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:23:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:23:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:23:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:23:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:23:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:23:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:23:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:23:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:23:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:23:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:23:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:23:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:23:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:23:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:23:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:23:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:23:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:23:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:23:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:23:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:23:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:23:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:23:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:23:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:23:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:23:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:23:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:23:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:23:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:23:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:23:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:23:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:23:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:23:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:23:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:23:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:23:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:23:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:23:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:23:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:23:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:23:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:23:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:23:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:23:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:23:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:23:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:23:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:23:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:23:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:23:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:23:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:23:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:25:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:25:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:25:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:25:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:25:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:25:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:25:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:25:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:25:34 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 12:25:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:25:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:25:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:25:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:25:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:25:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:25:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:25:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:25:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:25:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:25:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:25:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:25:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:25:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:25:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:25:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:25:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:25:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:25:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:25:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:25:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:25:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:25:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:25:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:25:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:25:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:25:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:25:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:25:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:25:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:25:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:25:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:25:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:25:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:25:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:25:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:25:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:25:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:25:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:25:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:25:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:25:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:25:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:25:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:25:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:25:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:25:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:25:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:25:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:25:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:25:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:25:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:25:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:25:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:25:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:25:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:25:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:25:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:25:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:25:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:25:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:26:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:26:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:26:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:26:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:26:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:26:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:26:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:26:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:26:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:26:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:26:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:26:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:26:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:26:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:26:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:26:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:26:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:26:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:26:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:26:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:26:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:26:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:26:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:26:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:26:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:26:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:26:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:26:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:26:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:26:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:26:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:26:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:26:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:26:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:26:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:26:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:26:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:26:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:26:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:26:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:26:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:26:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:26:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:26:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:26:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:26:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:26:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:26:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:26:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:26:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:26:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:26:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:26:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:26:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:26:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:26:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:26:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:26:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:26:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:26:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:26:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:26:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:26:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:26:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:26:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:26:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:26:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:26:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:26:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:26:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:26:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:26:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:26:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:26:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:26:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:26:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:26:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:26:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:26:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:26:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:26:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:26:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:26:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:26:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:26:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:26:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:26:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:26:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:26:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:26:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:26:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:26:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:26:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:26:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:26:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:26:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:26:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:26:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:26:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:26:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:26:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:26:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:26:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:26:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:26:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:26:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:26:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:26:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:26:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:26:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:26:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:26:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:26:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:26:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:26:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:26:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:26:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:26:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:26:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:26:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:26:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:26:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:26:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:26:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:26:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:26:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:26:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:26:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:26:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:26:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:26:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:26:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:26:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:26:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:26:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:26:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:26:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:26:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:26:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:26:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:26:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:26:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:26:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:26:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:26:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:26:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:26:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:26:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:26:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:26:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:26:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:26:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:26:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:26:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:26:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:26:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:26:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:26:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:26:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:26:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:26:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:26:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:26:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:26:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:26:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:26:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:26:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:26:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:26:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:26:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:26:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:26:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:26:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:26:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:26:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:26:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:26:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:26:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:26:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:26:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:26:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:26:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:26:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:26:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:26:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:26:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:26:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:26:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:26:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:26:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:26:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:26:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:26:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:26:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:26:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:26:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:26:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:26:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:26:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:26:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:26:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:26:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:26:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:26:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:26:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:26:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:26:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:26:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:26:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:26:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:26:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:26:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:26:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:26:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:26:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:26:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:27:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:27:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:27:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:27:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:27:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:27:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:27:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:27:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:27:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:27:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:27:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:27:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:27:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:27:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:27:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:27:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:27:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:27:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:27:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:27:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:27:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:27:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:27:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:27:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:27:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:27:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:27:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:27:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:27:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:27:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:27:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:27:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:27:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:27:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:27:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:27:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:27:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:27:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:27:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:27:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:27:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:27:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:27:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:27:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:27:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:27:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:27:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:27:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:27:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:27:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:27:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:27:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:27:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:27:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:27:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:27:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:27:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:27:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:27:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:27:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:27:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:27:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:27:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:27:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:27:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:27:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:27:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:27:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:27:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:27:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:27:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:27:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:27:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:27:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:27:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:27:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:27:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:27:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:27:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:27:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:27:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:27:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:27:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:27:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:27:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:27:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:27:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:27:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:27:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:27:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:27:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:27:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:27:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:27:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:27:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:27:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:27:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:27:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:27:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:27:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:27:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:27:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:27:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:27:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:27:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:27:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:27:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:27:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:27:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:27:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:27:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:27:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:27:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:27:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:27:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:27:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:27:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:27:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:27:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:27:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:27:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:27:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:27:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:27:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:27:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:27:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:27:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:27:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:27:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:27:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:27:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:27:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:27:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:27:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:27:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:27:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:27:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:27:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:27:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:27:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:27:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:27:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:27:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:27:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:27:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:27:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:27:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:27:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:27:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:27:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:27:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:27:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:27:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:27:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:27:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:27:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:27:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:27:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:27:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:27:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:27:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:27:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:27:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:27:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:27:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:27:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:27:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:27:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:27:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:27:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:27:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:27:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:27:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:27:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:27:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:27:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:27:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:27:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:27:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:27:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:27:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:27:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:27:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:27:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:27:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:27:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:27:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:27:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:27:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:27:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:27:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:27:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:27:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:27:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:27:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:27:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:27:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:27:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:27:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:27:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:27:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:27:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:27:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:27:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:27:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:27:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:27:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:27:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:27:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:27:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:27:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:27:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:27:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:27:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:27:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:27:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:28:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:28:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:28:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:28:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:28:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:28:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:28:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:28:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:28:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:28:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:28:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:28:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:28:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:28:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:28:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:28:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:28:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:28:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:28:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:28:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:28:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:28:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:28:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:28:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:28:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:28:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:28:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:28:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:28:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:28:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:28:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:28:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:28:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:28:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:28:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:28:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:28:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:28:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:28:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:28:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:28:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:28:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:28:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:28:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:28:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:28:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:28:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:28:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:28:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:28:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:28:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:28:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:28:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:28:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:28:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:28:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:28:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:28:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:28:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:28:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:28:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:28:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:28:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:28:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:28:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:28:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:28:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:28:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:28:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:28:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:28:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:28:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:28:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:28:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:28:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:28:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:28:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:28:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:28:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:28:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:28:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:28:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:28:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:28:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:28:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:28:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:28:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:28:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:28:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:28:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:28:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:28:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:28:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:28:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:28:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:28:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:28:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:28:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:28:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:28:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:28:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:28:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:28:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:28:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:28:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:28:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:28:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:28:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:28:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:28:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:28:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:28:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:28:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:28:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:28:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:28:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:28:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:28:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:28:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:28:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:28:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:28:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:28:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:28:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:28:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:28:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:28:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:28:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:28:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:28:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:28:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:28:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:28:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:28:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:28:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:28:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:28:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:28:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:28:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:28:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:28:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:28:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:28:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:28:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:28:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:28:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:28:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:28:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:28:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:28:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:28:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:28:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:28:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:28:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:28:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:28:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:28:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:28:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:28:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:28:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:28:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:28:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:28:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:28:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:28:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:28:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:28:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:28:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:28:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:28:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:28:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:28:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:28:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:28:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:28:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:28:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:28:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:28:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:28:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:28:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:28:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:28:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:28:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:28:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:28:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:28:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:28:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:28:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:28:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:28:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:28:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:28:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:28:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:28:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:28:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:28:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:28:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:28:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:28:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:28:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:28:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:28:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:28:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:28:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:28:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:28:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:28:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:28:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:28:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:28:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:28:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:28:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:28:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:28:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:28:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:28:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:29:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:29:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:29:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:29:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:29:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:29:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:29:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:29:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:29:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:29:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:29:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:29:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:29:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:29:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:29:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:29:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:29:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:29:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:29:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:29:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:29:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:29:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:29:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:29:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:29:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:29:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:29:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:29:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:29:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:29:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:29:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:29:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:29:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:29:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:29:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:29:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:29:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:29:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:29:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:29:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:29:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:29:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:29:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:29:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:29:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:29:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:29:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:29:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:29:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:29:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:29:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:29:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:29:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:29:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:29:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:29:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:29:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:29:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:29:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:29:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:29:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:29:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:29:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:29:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:29:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:29:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:29:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:29:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:29:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:29:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:29:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:29:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:29:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:29:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:29:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:29:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:29:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:29:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:29:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:29:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:29:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:29:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:29:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:29:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:29:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:29:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:29:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:29:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:29:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:29:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:29:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:29:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:29:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:29:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:29:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:29:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:29:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:29:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:29:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:29:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:29:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:29:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:29:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:29:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:29:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:29:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:29:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:29:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:29:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:29:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:29:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:29:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:29:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:29:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:29:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:29:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:29:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:29:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:29:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:29:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:29:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:29:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:29:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:29:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:29:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:29:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:29:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:29:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:29:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:29:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:29:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:29:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:29:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:29:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:29:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:29:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:29:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:29:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:29:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:29:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:29:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:29:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:29:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:29:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:29:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:29:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:29:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:29:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:29:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:29:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:29:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:29:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:29:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:29:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:29:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:29:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:29:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:29:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:29:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:29:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:29:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:29:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:29:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:29:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:29:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:29:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:29:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:29:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:29:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:29:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:29:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:29:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:29:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:29:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:29:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:29:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:29:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:29:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:29:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:29:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:29:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:29:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:29:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:29:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:29:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:29:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:29:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:29:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:29:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:29:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:29:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:29:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:29:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:29:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:29:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:29:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:29:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:29:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:29:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:29:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:29:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:29:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:29:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:29:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:29:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:29:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:29:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:29:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:29:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:29:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:29:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:29:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:29:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:29:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:29:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:29:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:30:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:30:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:30:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:30:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:30:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:30:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:30:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:30:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:30:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:30:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:30:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:30:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:30:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:30:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:30:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:30:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:30:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:30:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:30:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:30:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:30:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:30:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:30:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:30:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:30:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:30:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:30:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:30:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:30:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:30:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:30:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:30:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:30:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:30:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:30:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:30:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:30:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:30:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:30:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:30:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:30:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:30:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:30:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:30:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:30:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:30:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:30:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:30:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:30:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:30:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:30:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:30:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:30:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:30:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:30:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:30:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:30:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:30:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:30:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:30:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:30:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:30:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:30:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:30:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:30:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:30:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:30:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:30:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:30:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:30:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:30:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:30:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:30:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:30:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:30:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:30:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:30:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:30:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:30:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:30:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:30:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:30:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:30:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:30:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:30:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:30:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:30:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:30:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:30:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:30:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:30:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:30:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:30:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:30:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:30:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:30:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:30:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:30:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:30:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:30:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:30:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:30:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:30:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:30:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:30:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:30:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:30:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:30:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:30:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:30:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:30:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:30:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:30:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:30:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:30:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:30:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:30:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:30:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:30:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:30:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:30:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:30:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:30:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:30:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:30:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:30:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:30:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:30:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:30:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:30:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:30:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:30:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:30:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:30:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:30:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:30:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:30:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:30:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:30:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:30:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:30:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:30:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:30:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:30:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:30:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:30:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:30:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:30:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:30:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:30:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:30:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:30:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:30:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:30:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:30:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:30:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:30:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:30:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:30:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:30:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:30:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:30:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:30:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:30:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:30:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:30:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:30:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:30:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:30:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:30:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:30:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:30:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:30:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:30:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:30:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:30:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:30:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:30:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:30:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:30:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:30:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:30:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:30:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:30:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:30:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:30:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:30:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:30:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:30:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:30:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:30:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:30:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:30:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:30:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:30:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:30:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:30:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:30:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:30:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:30:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:30:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:30:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:30:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:30:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:30:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:30:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:30:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:30:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:30:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:30:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:30:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:30:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:30:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:30:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:30:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:30:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:31:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:31:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:31:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:31:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:31:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:31:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:31:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:31:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:31:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:31:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:31:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:31:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:31:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:31:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:31:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:31:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:31:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:31:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:31:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:31:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:31:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:31:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:31:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:31:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:31:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:31:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:31:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:31:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:31:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:31:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:31:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:31:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:31:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:31:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:31:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:31:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:31:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:31:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:31:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:31:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:31:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:31:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:31:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:31:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:31:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:31:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:31:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:31:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:31:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:31:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:31:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:31:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:31:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:31:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:31:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:31:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:31:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:31:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:31:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:31:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:31:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:31:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:31:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:31:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:31:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:31:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:31:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:31:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:31:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:31:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:31:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:31:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:31:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:31:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:31:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:31:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:31:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:31:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:31:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:31:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:31:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:31:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:31:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:31:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:31:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:31:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:31:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:31:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:31:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:31:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:31:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:31:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:31:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:31:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:31:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:31:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:31:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:31:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:31:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:31:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:31:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:31:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:31:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:31:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:31:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:31:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:31:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:31:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:31:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:31:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:31:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:31:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:31:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:31:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:31:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:31:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:31:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:31:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:31:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:31:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:31:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:31:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:31:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:31:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:31:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:31:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:31:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:31:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:31:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:31:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:31:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:31:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:31:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:31:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:31:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:31:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:31:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:31:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:31:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:31:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:31:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:31:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:31:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:31:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:31:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:31:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:31:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:31:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:31:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:31:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:31:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:31:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:31:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:31:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:31:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:31:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:31:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:31:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:31:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:31:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:31:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:31:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:31:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:31:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:31:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:31:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:31:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:31:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:31:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:31:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:31:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:31:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:31:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:31:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:31:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:31:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:31:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:31:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:31:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:31:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:31:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:31:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:31:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:31:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:31:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:31:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:31:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:31:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:31:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:31:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:31:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:31:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:31:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:31:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:31:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:31:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:31:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:31:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:31:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:31:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:31:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:31:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:31:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:31:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:31:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:31:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:32:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:32:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:32:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:32:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:32:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:32:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:32:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:32:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:32:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:32:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:32:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:32:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:32:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:32:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:32:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:32:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:32:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:32:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:32:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:32:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:32:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:32:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:32:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:32:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:32:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:32:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:32:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:32:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:32:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:32:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:32:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:32:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:32:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:32:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:32:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:32:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:32:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:32:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:32:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:32:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:32:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:32:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:32:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:32:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:32:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:32:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:32:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:32:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:32:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:32:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:32:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:32:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:32:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:32:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:32:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:32:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:32:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:32:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:32:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:32:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:32:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:32:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:32:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:32:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:32:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:32:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:32:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:32:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:32:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:32:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:32:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:32:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:32:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:32:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:32:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:32:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:32:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:32:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:32:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:32:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:32:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:32:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:32:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:32:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:32:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:32:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:32:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:32:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:32:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:32:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:32:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:32:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:32:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:32:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:32:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:32:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:32:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:32:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:32:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:32:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:32:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:32:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:32:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:32:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:32:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:32:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:32:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:32:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:32:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:32:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:32:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:32:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:32:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:32:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:32:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:32:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:32:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:32:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:32:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:32:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:32:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:32:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:32:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:32:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:32:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:32:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:32:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:32:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:32:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:32:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:32:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:32:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:32:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:32:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:32:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:32:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:32:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:32:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:32:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:32:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:32:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:32:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:32:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:32:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:32:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:32:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:32:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:32:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:32:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:32:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:32:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:32:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:32:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:32:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:32:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:32:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:32:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:32:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:32:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:32:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:32:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:32:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:32:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:32:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:32:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:32:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:32:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:32:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:32:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:32:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:32:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:32:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:32:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:32:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:32:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:32:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:32:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:32:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:32:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:32:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:32:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:32:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:32:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:32:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:32:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:32:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:32:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:32:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:32:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:32:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:32:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:32:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:32:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:32:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:32:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:32:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:32:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:32:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:32:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:32:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:32:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:32:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:32:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:32:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:32:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:32:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:32:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:32:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:32:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:32:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:32:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:32:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:32:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:32:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:32:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:32:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:33:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:33:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:33:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:33:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:33:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:33:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:33:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:33:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:33:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:33:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:33:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:33:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:33:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:33:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:33:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:33:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:33:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:33:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:33:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:33:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:33:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:33:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:33:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:33:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:33:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:33:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:33:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:33:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:33:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:33:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:33:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:33:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:33:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:33:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:33:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:33:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:33:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:33:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:33:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:33:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:33:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:33:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:33:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:33:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:33:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:33:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:33:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:33:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:33:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:33:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:33:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:33:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:33:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:33:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:33:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:33:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:33:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:33:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:33:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:33:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:33:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:33:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:33:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:33:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:33:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:33:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:33:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:33:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:33:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:33:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:33:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:33:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:33:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:33:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:33:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:33:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:33:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:33:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:33:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:33:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:33:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:33:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:33:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:33:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:33:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:33:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:33:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:33:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:33:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:33:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:33:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:33:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:33:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:33:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:33:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:33:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:33:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:33:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:33:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:33:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:33:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:33:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:33:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:33:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:33:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:33:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:33:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:33:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:35:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:35:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:35:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:35:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:35:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:35:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:35:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:35:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:35:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:35:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:35:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:35:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:35:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:35:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:35:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:35:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:35:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:35:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:35:39 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 12:35:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:35:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:35:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:35:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:35:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:35:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:35:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:35:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:35:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:35:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:35:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:35:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:35:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:35:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:35:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:35:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:35:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:35:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:35:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:35:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:35:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:35:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:35:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:35:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:35:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:35:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:35:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:35:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:35:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:35:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:35:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:35:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:35:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:35:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:35:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:35:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:35:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:35:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:35:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:35:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:35:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:35:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:35:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:35:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:35:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:35:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:35:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:35:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:35:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:35:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:35:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:35:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:35:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:35:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:35:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:35:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:35:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:35:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:35:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:35:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:35:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:35:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:35:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:35:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:35:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:35:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:35:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:35:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:35:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:35:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:35:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:35:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:36:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:36:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:36:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:36:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:36:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:36:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:36:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:36:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:36:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:36:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:36:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:36:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:36:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:36:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:36:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:36:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:36:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:36:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:36:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:36:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:36:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:36:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:36:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:36:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:36:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:36:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:36:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:36:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:36:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:36:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:36:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:36:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:36:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:36:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:36:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:36:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:36:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:36:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:36:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:36:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:36:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:36:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:36:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:36:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:36:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:36:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:36:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:36:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:36:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:36:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:36:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:36:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:36:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:36:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:36:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:36:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:36:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:36:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:36:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:36:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:36:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:36:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:36:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:36:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:36:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:36:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:36:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:36:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:36:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:36:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:36:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:36:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:36:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:36:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:36:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:36:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:36:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:36:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:36:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:36:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:36:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:36:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:36:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:36:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:36:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:36:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:36:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:36:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:36:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:36:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:36:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:36:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:36:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:36:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:36:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:36:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:36:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:36:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:36:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:36:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:36:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:36:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:36:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:36:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:36:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:36:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:36:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:36:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:36:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:36:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:36:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:36:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:36:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:36:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:36:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:36:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:36:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:36:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:36:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:36:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:36:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:36:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:36:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:36:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:36:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:36:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:36:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:36:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:36:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:36:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:36:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:36:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:36:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:36:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:36:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:36:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:36:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:36:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:36:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:36:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:36:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:36:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:36:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:36:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:36:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:36:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:36:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:36:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:36:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:36:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:36:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:36:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:36:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:36:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:36:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:36:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:36:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:36:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:36:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:36:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:36:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:36:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:36:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:36:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:36:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:36:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:36:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:36:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:36:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:36:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:36:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:36:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:36:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:36:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:36:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:36:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:36:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:36:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:36:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:36:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:36:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:36:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:36:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:36:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:36:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:36:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:36:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:36:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:36:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:36:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:36:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:36:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:36:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:36:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:36:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:36:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:36:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:36:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:36:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:36:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:36:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:36:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:36:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:36:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:36:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:36:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:36:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:36:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:36:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:36:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:36:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:36:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:36:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:36:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:36:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:36:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:37:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:37:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:37:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:37:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:37:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:37:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:37:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:37:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:37:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:37:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:37:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:37:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:37:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:37:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:37:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:37:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:37:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:37:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:37:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:37:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:37:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:37:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:37:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:37:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:37:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:37:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:37:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:37:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:37:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:37:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:37:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:37:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:37:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:37:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:37:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:37:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:37:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:37:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:37:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:37:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:37:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:37:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:37:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:37:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:37:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:37:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:37:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:37:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:37:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:37:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:37:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:37:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:37:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:37:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:37:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:37:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:37:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:37:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:37:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:37:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:37:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:37:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:37:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:37:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:37:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:37:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:37:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:37:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:37:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:37:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:37:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:37:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:37:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:37:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:37:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:37:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:37:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:37:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:37:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:37:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:37:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:37:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:37:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:37:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:37:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:37:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:37:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:37:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:37:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:37:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:37:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:37:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:37:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:37:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:37:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:37:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:37:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:37:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:37:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:37:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:37:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:37:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:37:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:37:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:37:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:37:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:37:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:37:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:37:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:37:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:37:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:37:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:37:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:37:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:37:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:37:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:37:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:37:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:37:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:37:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:37:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:37:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:37:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:37:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:37:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:37:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:37:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:37:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:37:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:37:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:37:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:37:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:37:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:37:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:37:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:37:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:37:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:37:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:37:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:37:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:37:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:37:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:37:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:37:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:37:39 rdma06 systemd-journald[790]: Data hash table of /run/log/journal/95379d2f723847b68d9e976dee8967a9/system.journal has a fill level at 75.0 (163750 of 218332 items, 75497472 file size, 461 bytes per hash table item), suggesting rotation.
Apr 18 12:37:39 rdma06 systemd-journald[790]: /run/log/journal/95379d2f723847b68d9e976dee8967a9/system.journal: Journal header limits reached or header out-of-date, rotating.
Apr 18 12:37:39 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 18 12:37:39 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 18 12:37:39 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 18 12:37:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:37:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:37:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:37:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:37:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:37:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:37:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:37:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:37:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:37:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:37:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:37:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:37:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:37:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:37:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:37:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:37:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:37:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:37:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:37:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:37:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:37:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:37:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:37:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:37:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:37:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:37:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:37:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:37:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:37:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:37:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:37:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:37:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:37:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:37:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:37:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:37:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:37:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:37:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:37:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:37:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:37:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:37:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:37:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:37:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:37:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:37:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:37:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:37:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:37:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:37:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:37:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:37:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:37:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:37:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:37:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:37:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:37:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:37:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:37:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:37:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:37:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:37:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:37:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:37:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:37:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:37:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:37:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:37:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:37:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:37:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:37:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:38:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:38:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:38:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:38:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:38:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:38:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:38:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:38:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:38:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:38:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:38:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:38:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:38:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:38:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:38:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:38:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:38:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:38:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:38:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:38:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:38:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:38:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:38:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:38:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:38:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:38:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:38:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:38:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:38:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:38:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:38:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:38:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:38:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:38:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:38:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:38:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:38:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:38:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:38:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:38:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:38:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:38:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:38:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:38:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:38:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:38:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:38:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:38:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:38:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:38:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:38:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:38:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:38:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:38:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:38:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:38:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:38:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:38:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:38:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:38:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:38:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:38:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:38:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:38:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:38:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:38:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:38:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:38:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:38:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:38:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:38:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:38:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:38:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:38:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:38:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:38:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:38:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:38:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:38:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:38:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:38:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:38:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:38:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:38:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:38:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:38:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:38:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:38:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:38:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:38:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:38:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:38:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:38:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:38:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:38:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:38:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:38:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:38:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:38:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:38:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:38:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:38:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:38:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:38:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:38:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:38:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:38:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:38:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:38:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:38:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:38:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:38:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:38:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:38:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:38:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:38:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:38:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:38:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:38:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:38:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:38:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:38:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:38:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:38:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:38:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:38:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:38:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:38:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:38:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:38:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:38:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:38:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:38:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:38:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:38:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:38:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:38:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:38:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:38:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:38:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:38:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:38:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:38:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:38:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:38:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:38:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:38:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:38:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:38:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:38:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:38:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:38:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:38:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:38:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:38:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:38:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:38:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:38:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:38:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:38:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:38:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:38:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:38:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:38:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:38:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:38:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:38:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:38:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:38:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:38:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:38:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:38:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:38:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:38:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:38:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:38:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:38:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:38:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:38:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:38:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:38:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:38:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:38:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:38:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:38:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:38:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:38:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:38:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:38:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:38:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:38:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:38:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:38:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:38:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:38:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:38:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:38:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:38:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:38:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:38:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:38:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:38:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:38:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:38:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:38:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:38:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:38:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:38:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:38:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:38:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:38:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:38:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:38:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:38:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:38:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:38:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:39:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:39:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:39:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:39:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:39:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:39:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:39:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:39:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:39:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:39:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:39:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:39:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:39:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:39:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:39:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:39:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:39:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:39:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:39:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:39:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:39:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:39:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:39:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:39:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:39:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:39:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:39:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:39:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:39:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:39:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:39:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:39:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:39:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:39:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:39:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:39:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:39:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:39:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:39:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:39:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:39:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:39:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:39:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:39:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:39:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:39:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:39:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:39:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:39:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:39:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:39:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:39:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:39:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:39:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:39:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:39:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:39:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:39:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:39:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:39:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:39:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:39:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:39:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:39:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:39:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:39:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:39:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:39:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:39:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:39:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:39:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:39:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:39:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:39:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:39:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:39:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:39:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:39:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:39:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:39:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:39:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:39:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:39:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:39:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:39:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:39:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:39:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:39:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:39:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:39:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:39:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:39:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:39:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:39:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:39:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:39:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:39:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:39:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:39:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:39:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:39:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:39:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:39:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:39:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:39:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:39:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:39:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:39:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:39:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:39:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:39:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:39:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:39:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:39:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:39:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:39:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:39:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:39:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:39:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:39:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:39:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:39:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:39:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:39:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:39:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:39:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:39:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:39:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:39:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:39:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:39:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:39:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:39:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:39:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:39:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:39:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:39:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:39:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:39:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:39:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:39:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:39:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:39:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:39:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:39:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:39:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:39:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:39:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:39:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:39:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:39:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:39:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:39:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:39:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:39:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:39:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:40:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:40:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:40:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:40:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:40:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:40:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:40:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:40:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:40:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:40:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:40:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:40:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:40:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:40:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:40:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:40:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:40:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:40:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:40:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:40:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:40:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:40:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:40:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:40:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:40:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:40:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:40:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:40:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:40:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:40:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:40:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:40:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:40:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:40:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:40:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:40:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:40:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:40:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:40:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:40:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:40:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:40:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:40:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:40:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:40:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:40:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:40:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:40:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:40:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:40:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:40:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:40:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:40:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:40:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:40:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:40:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:40:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:40:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:40:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:40:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:40:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:40:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:40:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:40:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:40:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:40:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:40:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:40:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:40:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:40:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:40:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:40:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:40:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:40:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:40:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:40:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:40:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:40:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:40:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:40:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:40:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:40:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:40:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:40:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:40:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:40:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:40:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:40:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:40:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:40:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:40:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:40:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:40:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:40:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:40:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:40:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:40:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:40:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:40:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:40:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:40:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:40:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:40:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:40:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:40:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:40:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:40:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:40:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:40:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:40:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:40:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:40:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:40:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:40:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:40:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:40:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:40:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:40:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:40:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:40:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:40:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:40:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:40:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:40:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:40:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:40:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:40:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:40:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:40:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:40:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:40:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:40:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:40:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:40:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:40:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:40:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:40:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:40:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:40:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:40:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:40:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:40:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:40:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:40:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:40:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:40:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:40:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:40:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:40:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:40:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:40:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:40:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:40:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:40:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:40:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:40:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:40:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:40:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:40:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:40:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:40:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:40:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:40:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:40:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:40:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:40:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:40:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:40:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:40:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:40:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:40:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:40:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:40:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:40:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:40:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:40:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:40:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:40:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:40:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:40:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:40:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:40:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:40:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:40:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:40:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:40:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:40:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:40:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:40:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:40:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:40:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:40:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:40:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:40:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:40:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:40:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:40:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:40:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:40:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:40:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:40:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:40:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:40:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:40:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:40:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:40:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:40:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:40:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:40:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:40:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:40:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:40:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:40:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:40:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:40:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:40:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:41:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:41:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:41:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:41:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:41:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:41:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:41:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:41:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:41:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:41:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:41:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:41:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:41:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:41:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:41:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:41:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:41:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:41:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:41:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:41:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:41:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:41:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:41:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:41:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:41:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:41:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:41:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:41:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:41:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:41:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:41:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:41:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:41:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:41:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:41:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:41:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:41:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:41:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:41:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:41:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:41:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:41:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:41:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:41:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:41:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:41:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:41:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:41:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:41:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:41:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:41:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:41:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:41:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:41:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:41:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:41:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:41:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:41:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:41:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:41:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:41:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:41:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:41:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:41:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:41:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:41:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:41:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:41:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:41:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:41:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:41:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:41:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:41:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:41:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:41:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:41:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:41:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:41:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:41:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:41:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:41:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:41:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:41:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:41:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:41:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:41:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:41:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:41:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:41:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:41:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:41:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:41:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:41:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:41:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:41:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:41:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:41:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:41:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:41:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:41:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:41:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:41:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:41:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:41:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:41:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:41:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:41:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:41:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:41:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:41:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:41:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:41:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:41:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:41:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:41:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:41:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:41:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:41:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:41:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:41:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:41:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:41:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:41:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:41:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:41:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:41:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:41:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:41:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:41:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:41:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:41:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:41:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:41:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:41:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:41:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:41:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:41:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:41:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:41:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:41:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:41:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:41:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:41:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:41:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:41:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:41:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:41:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:41:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:41:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:41:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:41:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:41:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:41:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:41:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:41:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:41:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:41:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:41:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:41:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:41:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:41:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:41:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:41:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:41:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:41:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:41:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:41:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:41:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:41:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:41:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:41:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:41:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:41:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:41:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:41:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:41:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:41:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:41:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:41:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:41:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:41:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:41:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:41:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:41:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:41:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:41:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:41:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:41:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:41:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:41:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:41:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:41:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:41:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:41:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:41:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:41:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:41:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:41:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:41:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:41:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:41:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:41:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:41:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:41:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:41:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:41:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:41:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:41:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:41:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:41:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:41:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:41:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:41:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:41:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:41:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:41:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:42:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:42:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:42:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:42:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:42:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:42:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:42:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:42:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:42:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:42:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:42:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:42:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:42:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:42:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:42:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:42:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:42:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:42:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:42:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:42:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:42:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:42:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:42:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:42:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:42:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:42:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:42:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:42:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:42:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:42:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:42:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:42:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:42:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:42:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:42:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:42:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:42:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:42:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:42:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:42:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:42:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:42:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:42:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:42:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:42:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:42:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:42:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:42:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:42:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:42:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:42:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:42:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:42:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:42:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:42:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:42:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:42:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:42:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:42:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:42:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:42:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:42:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:42:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:42:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:42:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:42:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:42:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:42:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:42:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:42:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:42:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:42:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:42:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:42:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:42:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:42:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:42:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:42:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:42:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:42:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:42:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:42:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:42:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:42:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:42:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:42:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:42:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:42:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:42:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:42:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:42:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:42:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:42:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:42:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:42:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:42:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:42:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:42:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:42:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:42:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:42:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:42:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:42:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:42:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:42:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:42:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:42:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:42:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:42:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:42:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:42:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:42:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:42:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:42:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:42:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:42:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:42:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:42:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:42:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:42:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:42:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:42:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:42:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:42:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:42:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:42:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:42:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:42:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:42:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:42:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:42:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:42:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:42:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:42:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:42:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:42:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:42:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:42:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:42:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:42:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:42:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:42:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:42:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:42:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:42:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:42:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:42:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:42:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:42:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:42:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:42:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:42:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:42:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:42:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:42:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:42:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:42:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:42:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:42:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:42:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:42:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:42:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:42:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:42:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:42:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:42:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:42:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:42:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:42:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:42:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:42:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:42:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:42:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:42:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:42:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:42:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:42:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:42:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:42:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:42:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:42:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:42:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:42:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:42:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:42:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:42:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:42:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:42:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:42:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:42:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:42:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:42:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:42:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:42:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:42:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:42:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:42:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:42:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:42:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:42:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:42:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:42:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:42:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:42:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:42:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:42:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:42:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:42:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:42:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:42:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:42:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:42:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:42:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:42:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:42:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:42:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:43:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:43:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:43:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:43:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:43:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:43:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:43:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:43:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:43:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:43:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:43:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:43:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:43:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:43:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:43:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:43:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:43:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:43:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:43:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:43:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:43:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:43:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:43:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:43:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:43:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:43:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:43:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:43:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:43:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:43:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:43:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:43:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:43:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:43:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:43:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:43:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:43:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:43:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:43:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:43:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:43:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:43:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:43:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:43:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:43:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:43:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:43:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:43:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:43:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:43:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:43:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:43:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:43:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:43:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:43:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:43:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:43:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:43:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:43:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:43:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:43:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:43:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:43:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:43:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:43:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:43:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:43:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:43:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:43:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:43:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:43:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:43:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:43:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:43:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:43:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:43:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:43:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:43:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:43:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:43:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:43:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:43:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:43:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:43:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:43:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:43:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:43:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:43:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:43:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:43:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:43:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:43:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:43:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:43:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:43:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:43:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:43:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:43:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:43:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:43:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:43:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:43:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:43:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:43:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:43:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:43:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:43:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:43:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:43:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:43:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:43:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:43:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:43:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:43:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:43:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:43:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:43:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:43:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:43:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:43:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:43:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:43:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:43:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:43:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:43:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:43:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:45:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:45:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:45:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:45:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:45:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:45:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:45:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:45:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:45:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:45:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:45:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:45:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:45:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:45:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:45:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:45:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:45:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:45:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:45:40 rdma06 rsyslogd[1041]: imjournal: 5087 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 12:45:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:45:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:45:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:45:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:45:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:45:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:45:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:45:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:45:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:45:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:45:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:45:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:45:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:45:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:45:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:45:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:45:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:45:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:45:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:45:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:45:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:45:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:45:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:45:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:45:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:45:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:45:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:45:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:45:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:45:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:45:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:45:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:45:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:45:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:45:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:45:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:45:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:45:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:45:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:45:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:45:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:45:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:45:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:45:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:45:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:45:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:45:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:45:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:45:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:45:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:45:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:45:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:45:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:45:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:45:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:45:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:45:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:45:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:45:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:45:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:45:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:45:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:45:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:45:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:45:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:45:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:45:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:45:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:45:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:45:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:45:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:45:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:46:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:46:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:46:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:46:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:46:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:46:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:46:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:46:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:46:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:46:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:46:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:46:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:46:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:46:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:46:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:46:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:46:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:46:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:46:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:46:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:46:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:46:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:46:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:46:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:46:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:46:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:46:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:46:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:46:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:46:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:46:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:46:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:46:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:46:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:46:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:46:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:46:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:46:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:46:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:46:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:46:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:46:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:46:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:46:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:46:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:46:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:46:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:46:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:46:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:46:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:46:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:46:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:46:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:46:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:46:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:46:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:46:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:46:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:46:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:46:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:46:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:46:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:46:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:46:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:46:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:46:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:46:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:46:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:46:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:46:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:46:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:46:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:46:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:46:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:46:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:46:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:46:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:46:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:46:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:46:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:46:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:46:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:46:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:46:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:46:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:46:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:46:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:46:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:46:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:46:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:46:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:46:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:46:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:46:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:46:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:46:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:46:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:46:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:46:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:46:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:46:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:46:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:46:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:46:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:46:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:46:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:46:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:46:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:46:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:46:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:46:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:46:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:46:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:46:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:46:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:46:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:46:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:46:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:46:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:46:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:46:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:46:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:46:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:46:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:46:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:46:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:46:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:46:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:46:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:46:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:46:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:46:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:46:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:46:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:46:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:46:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:46:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:46:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:46:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:46:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:46:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:46:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:46:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:46:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:46:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:46:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:46:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:46:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:46:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:46:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:46:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:46:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:46:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:46:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:46:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:46:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:46:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:46:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:46:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:46:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:46:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:46:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:46:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:46:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:46:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:46:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:46:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:46:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:46:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:46:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:46:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:46:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:46:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:46:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:46:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:46:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:46:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:46:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:46:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:46:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:46:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:46:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:46:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:46:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:46:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:46:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:46:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:46:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:46:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:46:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:46:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:46:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:46:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:46:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:46:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:46:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:46:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:46:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:46:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:46:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:46:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:46:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:46:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:46:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:46:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:46:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:46:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:46:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:46:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:46:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:46:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:46:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:46:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:46:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:46:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:46:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:47:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:47:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:47:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:47:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:47:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:47:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:47:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:47:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:47:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:47:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:47:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:47:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:47:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:47:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:47:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:47:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:47:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:47:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:47:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:47:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:47:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:47:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:47:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:47:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:47:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:47:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:47:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:47:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:47:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:47:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:47:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:47:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:47:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:47:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:47:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:47:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:47:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:47:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:47:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:47:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:47:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:47:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:47:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:47:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:47:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:47:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:47:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:47:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:47:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:47:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:47:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:47:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:47:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:47:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:47:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:47:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:47:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:47:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:47:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:47:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:47:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:47:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:47:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:47:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:47:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:47:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:47:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:47:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:47:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:47:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:47:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:47:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:47:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:47:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:47:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:47:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:47:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:47:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:47:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:47:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:47:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:47:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:47:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:47:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:47:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:47:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:47:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:47:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:47:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:47:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:47:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:47:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:47:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:47:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:47:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:47:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:47:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:47:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:47:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:47:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:47:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:47:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:47:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:47:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:47:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:47:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:47:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:47:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:47:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:47:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:47:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:47:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:47:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:47:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:47:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:47:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:47:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:47:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:47:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:47:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:47:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:47:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:47:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:47:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:47:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:47:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:47:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:47:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:47:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:47:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:47:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:47:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:47:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:47:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:47:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:47:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:47:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:47:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:47:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:47:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:47:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:47:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:47:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:47:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:47:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:47:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:47:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:47:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:47:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:47:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:47:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:47:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:47:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:47:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:47:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:47:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:47:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:47:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:47:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:47:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:47:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:47:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:47:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:47:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:47:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:47:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:47:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:47:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:47:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:47:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:47:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:47:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:47:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:47:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:47:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:47:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:48:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:48:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:48:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:48:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:48:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:48:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:48:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:48:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:48:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:48:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:48:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:48:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:48:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:48:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:48:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:48:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:48:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:48:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:48:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:48:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:48:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:48:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:48:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:48:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:48:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:48:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:48:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:48:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:48:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:48:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:48:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:48:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:48:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:48:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:48:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:48:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:48:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:48:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:48:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:48:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:48:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:48:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:48:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:48:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:48:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:48:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:48:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:48:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:48:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:48:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:48:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:48:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:48:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:48:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:48:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:48:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:48:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:48:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:48:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:48:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:48:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:48:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:48:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:48:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:48:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:48:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:48:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:48:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:48:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:48:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:48:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:48:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:48:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:48:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:48:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:48:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:48:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:48:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:48:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:48:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:48:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:48:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:48:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:48:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:48:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:48:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:48:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:48:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:48:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:48:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:48:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:48:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:48:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:48:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:48:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:48:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:48:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:48:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:48:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:48:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:48:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:48:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:48:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:48:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:48:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:48:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:48:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:48:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:48:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:48:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:48:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:48:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:48:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:48:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:48:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:48:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:48:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:48:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:48:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:48:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:48:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:48:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:48:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:48:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:48:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:48:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:48:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:48:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:48:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:48:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:48:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:48:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:48:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:48:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:48:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:48:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:48:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:48:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:48:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:48:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:48:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:48:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:48:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:48:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:48:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:48:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:48:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:48:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:48:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:48:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:48:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:48:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:48:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:48:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:48:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:48:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:48:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:48:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:48:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:48:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:48:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:48:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:48:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:48:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:48:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:48:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:48:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:48:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:48:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:48:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:48:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:48:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:48:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:48:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:48:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:48:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:48:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:48:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:48:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:48:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:48:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:48:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:48:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:48:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:48:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:48:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:48:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:48:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:48:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:48:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:48:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:48:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:48:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:48:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:48:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:48:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:48:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:48:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:48:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:48:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:48:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:48:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:48:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:48:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:48:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:48:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:48:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:48:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:48:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:48:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:48:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:48:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:48:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:48:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:48:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:48:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:49:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:49:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:49:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:49:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:49:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:49:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:49:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:49:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:49:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:49:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:49:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:49:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:49:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:49:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:49:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:49:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:49:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:49:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:49:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:49:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:49:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:49:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:49:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:49:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:49:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:49:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:49:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:49:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:49:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:49:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:49:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:49:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:49:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:49:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:49:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:49:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:49:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:49:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:49:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:49:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:49:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:49:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:49:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:49:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:49:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:49:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:49:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:49:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:49:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:49:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:49:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:49:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:49:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:49:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:49:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:49:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:49:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:49:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:49:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:49:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:49:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:49:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:49:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:49:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:49:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:49:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:49:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:49:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:49:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:49:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:49:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:49:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:49:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:49:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:49:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:49:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:49:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:49:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:49:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:49:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:49:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:49:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:49:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:49:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:49:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:49:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:49:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:49:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:49:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:49:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:49:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:49:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:49:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:49:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:49:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:49:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:49:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:49:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:49:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:49:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:49:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:49:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:49:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:49:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:49:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:49:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:49:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:49:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:49:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:49:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:49:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:49:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:49:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:49:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:49:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:49:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:49:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:49:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:49:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:49:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:49:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:49:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:49:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:49:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:49:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:49:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:49:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:49:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:49:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:49:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:49:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:49:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:49:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:49:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:49:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:49:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:49:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:49:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:49:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:49:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:49:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:49:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:49:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:49:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:49:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:49:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:49:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:49:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:49:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:49:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:49:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:49:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:49:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:49:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:49:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:49:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:49:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:49:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:49:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:49:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:49:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:49:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:49:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:49:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:49:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:49:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:49:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:49:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:49:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:49:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:49:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:49:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:49:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:49:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:49:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:49:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:49:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:49:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:49:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:49:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:49:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:49:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:49:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:49:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:49:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:49:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:49:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:49:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:49:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:49:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:49:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:49:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:49:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:49:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:49:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:49:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:49:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:49:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:49:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:49:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:49:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:49:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:49:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:49:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:49:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:49:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:50:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:50:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:50:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:50:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:50:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:50:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:50:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:50:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:50:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:50:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:50:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:50:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:50:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:50:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:50:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:50:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:50:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:50:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:50:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:50:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:50:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:50:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:50:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:50:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:50:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:50:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:50:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:50:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:50:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:50:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:50:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:50:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:50:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:50:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:50:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:50:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:50:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:50:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:50:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:50:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:50:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:50:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:50:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:50:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:50:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:50:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:50:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:50:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:50:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:50:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:50:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:50:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:50:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:50:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:50:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:50:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:50:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:50:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:50:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:50:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:50:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:50:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:50:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:50:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:50:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:50:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:50:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:50:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:50:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:50:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:50:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:50:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:50:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:50:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:50:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:50:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:50:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:50:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:50:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:50:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:50:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:50:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:50:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:50:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:50:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:50:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:50:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:50:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:50:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:50:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:50:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:50:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:50:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:50:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:50:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:50:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:50:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:50:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:50:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:50:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:50:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:50:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:50:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:50:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:50:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:50:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:50:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:50:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:50:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:50:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:50:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:50:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:50:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:50:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:50:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:50:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:50:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:50:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:50:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:50:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:50:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:50:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:50:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:50:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:50:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:50:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:50:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:50:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:50:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:50:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:50:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:50:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:50:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:50:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:50:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:50:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:50:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:50:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:50:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:50:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:50:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:50:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:50:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:50:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:50:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:50:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:50:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:50:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:50:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:50:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:50:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:50:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:50:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:50:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:50:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:50:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:50:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:50:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:50:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:50:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:50:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:50:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:50:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:50:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:50:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:50:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:50:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:50:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:50:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:50:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:50:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:50:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:50:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:50:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:50:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:50:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:50:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:50:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:50:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:50:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:50:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:50:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:50:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:50:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:50:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:50:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:50:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:50:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:50:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:50:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:50:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:50:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:50:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:50:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:50:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:50:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:50:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:50:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:50:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:50:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:50:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:50:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:50:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:50:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:50:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:50:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:50:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:50:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:50:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:50:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:50:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:50:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:50:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:50:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:50:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:50:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:51:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:51:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:51:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:51:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:51:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:51:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:51:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:51:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:51:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:51:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:51:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:51:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:51:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:51:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:51:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:51:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:51:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:51:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:51:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:51:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:51:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:51:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:51:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:51:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:51:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:51:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:51:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:51:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:51:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:51:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:51:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:51:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:51:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:51:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:51:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:51:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:51:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:51:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:51:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:51:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:51:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:51:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:51:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:51:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:51:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:51:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:51:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:51:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:51:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:51:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:51:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:51:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:51:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:51:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:51:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:51:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:51:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:51:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:51:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:51:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:51:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:51:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:51:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:51:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:51:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:51:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:51:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:51:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:51:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:51:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:51:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:51:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:51:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:51:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:51:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:51:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:51:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:51:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:51:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:51:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:51:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:51:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:51:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:51:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:51:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:51:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:51:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:51:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:51:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:51:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:51:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:51:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:51:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:51:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:51:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:51:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:51:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:51:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:51:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:51:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:51:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:51:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:51:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:51:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:51:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:51:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:51:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:51:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:51:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:51:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:51:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:51:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:51:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:51:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:51:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:51:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:51:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:51:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:51:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:51:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:51:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:51:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:51:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:51:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:51:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:51:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:51:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:51:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:51:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:51:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:51:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:51:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:51:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:51:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:51:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:51:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:51:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:51:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:51:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:51:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:51:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:51:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:51:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:51:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:51:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:51:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:51:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:51:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:51:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:51:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:51:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:51:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:51:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:51:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:51:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:51:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:51:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:51:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:51:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:51:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:51:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:51:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:51:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:51:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:51:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:51:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:51:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:51:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:51:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:51:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:51:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:51:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:51:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:51:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:51:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:51:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:51:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:51:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:51:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:51:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:51:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:51:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:51:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:51:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:51:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:51:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:51:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:51:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:51:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:51:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:51:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:51:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:51:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:51:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:51:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:51:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:51:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:51:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:51:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:51:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:51:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:51:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:51:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:51:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:51:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:51:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:51:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:51:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:51:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:51:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:51:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:51:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:51:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:51:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:51:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:51:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:52:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:52:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:52:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:52:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:52:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:52:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:52:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:52:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:52:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:52:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:52:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:52:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:52:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:52:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:52:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:52:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:52:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:52:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:52:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:52:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:52:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:52:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:52:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:52:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:52:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:52:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:52:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:52:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:52:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:52:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:52:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:52:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:52:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:52:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:52:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:52:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:52:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:52:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:52:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:52:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:52:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:52:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:52:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:52:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:52:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:52:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:52:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:52:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:52:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:52:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:52:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:52:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:52:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:52:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:52:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:52:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:52:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:52:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:52:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:52:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:52:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:52:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:52:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:52:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:52:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:52:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:52:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:52:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:52:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:52:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:52:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:52:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:52:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:52:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:52:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:52:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:52:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:52:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:52:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:52:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:52:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:52:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:52:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:52:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:52:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:52:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:52:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:52:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:52:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:52:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:52:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:52:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:52:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:52:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:52:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:52:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:52:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:52:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:52:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:52:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:52:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:52:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:52:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:52:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:52:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:52:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:52:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:52:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:52:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:52:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:52:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:52:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:52:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:52:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:52:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:52:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:52:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:52:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:52:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:52:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:52:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:52:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:52:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:52:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:52:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:52:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:52:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:52:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:52:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:52:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:52:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:52:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:52:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:52:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:52:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:52:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:52:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:52:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:52:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:52:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:52:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:52:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:52:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:52:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:52:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:52:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:52:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:52:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:52:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:52:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:52:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:52:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:52:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:52:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:52:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:52:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:52:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:52:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:52:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:52:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:52:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:52:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:52:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:52:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:52:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:52:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:52:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:52:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:52:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:52:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:52:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:52:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:52:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:52:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:52:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:52:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:52:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:52:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:52:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:52:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:52:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:52:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:52:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:52:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:52:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:52:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:52:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:52:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:52:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:52:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:52:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:52:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:52:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:52:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:52:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:52:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:52:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:52:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:52:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:52:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:52:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:52:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:52:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:52:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:52:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:52:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:52:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:52:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:52:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:52:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:52:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:52:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:52:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:52:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:52:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:52:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:53:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:53:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:53:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:53:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:53:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:53:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:53:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:53:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:53:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:53:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:53:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:53:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:53:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:53:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:53:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:53:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:53:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:53:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:53:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:53:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:53:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:53:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:53:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:53:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:53:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:53:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:53:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:53:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:53:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:53:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:53:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:53:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:53:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:53:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:53:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:53:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:53:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:53:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:53:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:53:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:53:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:53:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:53:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:53:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:53:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:53:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:53:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:53:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:53:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:53:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:53:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:53:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:53:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:53:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:53:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:53:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:53:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:53:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:53:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:53:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:53:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:53:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:53:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:53:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:53:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:53:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:53:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:53:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:53:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:53:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:53:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:53:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:53:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:53:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:53:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:53:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:55:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:55:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:55:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:55:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:55:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:55:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:55:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:55:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:55:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:55:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:55:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:55:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:55:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:55:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:55:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:55:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:55:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:55:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:55:45 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 12:55:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:55:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:55:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:55:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:55:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:55:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:55:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:55:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:55:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:55:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:55:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:55:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:55:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:55:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:55:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:55:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:55:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:55:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:55:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:55:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:55:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:55:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:55:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:55:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:55:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:55:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:55:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:55:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:55:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:55:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:55:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:55:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:55:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:55:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:55:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:55:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:56:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:56:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:56:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:56:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:56:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:56:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:56:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:56:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:56:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:56:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:56:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:56:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:56:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:56:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:56:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:56:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:56:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:56:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:56:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:56:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:56:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:56:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:56:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:56:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:56:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:56:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:56:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:56:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:56:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:56:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:56:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:56:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:56:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:56:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:56:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:56:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:56:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:56:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:56:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:56:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:56:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:56:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:56:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:56:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:56:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:56:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:56:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:56:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:56:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:56:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:56:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:56:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:56:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:56:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:56:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:56:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:56:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:56:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:56:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:56:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:56:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:56:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:56:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:56:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:56:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:56:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:56:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:56:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:56:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:56:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:56:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:56:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:56:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:56:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:56:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:56:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:56:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:56:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:56:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:56:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:56:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:56:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:56:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:56:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:56:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:56:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:56:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:56:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:56:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:56:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:56:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:56:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:56:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:56:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:56:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:56:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:56:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:56:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:56:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:56:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:56:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:56:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:56:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:56:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:56:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:56:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:56:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:56:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:56:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:56:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:56:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:56:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:56:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:56:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:56:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:56:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:56:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:56:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:56:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:56:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:56:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:56:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:56:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:56:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:56:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:56:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:56:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:56:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:56:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:56:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:56:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:56:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:56:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:56:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:56:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:56:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:56:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:56:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:56:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:56:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:56:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:56:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:56:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:56:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:56:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:56:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:56:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:56:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:56:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:56:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:56:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:56:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:56:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:56:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:56:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:56:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:56:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:56:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:56:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:56:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:56:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:56:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:56:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:56:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:56:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:56:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:56:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:56:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:56:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:56:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:56:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:56:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:56:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:56:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:56:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:56:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:56:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:56:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:56:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:56:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:56:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:56:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:56:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:56:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:56:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:56:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:56:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:56:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:56:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:56:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:56:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:56:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:56:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:56:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:56:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:56:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:56:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:56:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:56:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:56:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:56:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:56:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:56:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:56:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:56:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:56:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:56:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:56:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:56:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:56:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:56:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:56:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:56:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:56:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:56:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:56:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:57:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:57:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:57:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:57:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:57:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:57:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:57:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:57:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:57:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:57:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:57:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:57:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:57:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:57:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:57:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:57:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:57:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:57:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:57:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:57:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:57:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:57:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:57:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:57:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:57:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:57:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:57:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:57:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:57:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:57:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:57:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:57:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:57:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:57:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:57:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:57:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:57:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:57:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:57:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:57:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:57:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:57:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:57:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:57:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:57:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:57:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:57:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:57:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:57:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:57:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:57:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:57:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:57:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:57:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:57:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:57:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:57:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:57:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:57:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:57:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:57:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:57:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:57:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:57:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:57:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:57:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:57:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:57:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:57:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:57:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:57:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:57:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:57:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:57:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:57:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:57:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:57:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:57:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:57:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:57:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:57:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:57:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:57:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:57:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:57:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:57:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:57:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:57:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:57:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:57:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:57:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:57:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:57:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:57:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:57:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:57:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:57:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:57:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:57:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:57:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:57:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:57:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:57:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:57:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:57:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:57:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:57:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:57:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:57:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:57:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:57:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:57:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:57:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:57:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:57:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:57:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:57:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:57:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:57:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:57:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:57:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:57:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:57:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:57:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:57:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:57:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:57:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:57:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:57:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:57:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:57:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:57:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:57:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:57:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:57:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:57:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:57:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:57:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:57:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:57:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:57:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:57:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:57:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:57:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:57:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:57:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:57:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:57:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:57:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:57:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:57:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:57:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:57:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:57:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:57:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:57:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:57:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:57:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:57:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:57:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:57:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:57:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:57:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:57:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:57:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:57:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:57:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:57:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:57:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:57:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:57:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:57:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:57:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:57:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:57:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:57:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:57:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:57:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:57:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:57:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:57:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:57:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:57:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:57:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:57:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:57:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:57:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:57:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:57:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:57:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:57:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:57:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:57:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:57:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:57:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:57:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:57:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:57:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:57:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:57:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:57:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:57:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:57:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:57:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:57:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:57:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:57:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:57:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:57:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:57:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:57:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:57:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:57:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:57:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:57:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:57:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:58:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:58:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:58:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:58:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:58:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:58:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:58:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:58:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:58:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:58:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:58:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:58:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:58:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:58:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:58:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:58:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:58:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:58:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:58:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:58:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:58:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:58:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:58:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:58:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:58:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:58:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:58:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:58:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:58:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:58:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:58:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:58:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:58:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:58:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:58:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:58:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:58:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:58:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:58:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:58:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:58:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:58:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:58:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:58:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:58:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:58:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:58:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:58:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:58:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:58:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:58:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:58:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:58:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:58:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:58:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:58:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:58:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:58:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:58:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:58:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:58:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:58:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:58:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:58:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:58:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:58:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:58:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:58:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:58:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:58:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:58:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:58:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:58:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:58:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:58:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:58:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:58:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:58:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:58:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:58:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:58:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:58:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:58:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:58:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:58:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:58:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:58:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:58:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:58:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:58:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:58:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:58:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:58:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:58:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:58:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:58:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:58:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:58:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:58:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:58:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:58:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:58:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:58:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:58:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:58:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:58:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:58:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:58:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:58:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:58:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:58:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:58:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:58:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:58:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:58:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:58:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:58:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:58:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:58:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:58:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:58:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:58:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:58:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:58:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:58:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:58:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:58:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:58:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:58:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:58:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:58:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:58:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:58:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:58:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:58:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:58:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:58:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:58:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:58:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:58:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:58:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:58:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:58:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:58:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:58:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:58:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:58:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:58:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:58:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:58:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:58:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:58:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:58:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:58:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:58:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:58:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:58:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:58:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:58:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:58:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:58:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:58:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:58:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:58:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:58:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:58:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:58:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:58:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:58:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:58:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:58:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:58:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:58:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:58:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:58:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:58:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:58:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:58:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:58:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:58:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:58:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:58:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:58:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:58:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:58:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:58:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:58:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:58:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:58:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:58:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:58:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:58:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:58:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:58:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:58:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:58:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:58:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:58:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:58:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:58:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:58:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:58:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:58:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:58:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:58:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:58:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:58:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:58:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:58:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:58:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:58:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:58:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:58:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:58:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:58:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:58:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:59:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:59:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:59:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:59:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:59:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:59:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:59:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:59:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:59:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:59:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:59:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:59:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:59:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:59:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:59:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:59:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:59:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:59:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:59:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:59:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:59:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:59:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:59:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:59:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:59:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:59:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:59:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:59:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:59:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:59:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:59:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:59:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:59:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:59:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:59:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:59:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:59:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:59:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:59:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:59:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:59:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:59:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:59:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:59:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:59:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:59:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:59:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:59:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:59:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:59:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:59:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:59:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:59:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:59:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:59:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:59:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:59:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:59:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:59:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:59:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:59:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:59:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:59:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:59:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:59:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:59:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:59:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:59:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:59:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:59:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:59:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:59:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:59:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:59:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:59:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:59:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:59:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:59:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:59:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:59:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:59:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:59:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:59:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:59:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:59:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:59:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:59:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:59:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:59:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:59:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:59:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:59:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:59:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:59:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:59:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:59:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:59:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:59:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:59:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:59:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:59:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:59:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:59:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:59:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:59:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:59:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:59:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:59:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:59:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:59:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:59:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:59:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:59:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:59:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:59:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:59:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:59:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:59:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:59:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:59:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:59:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:59:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:59:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:59:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:59:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:59:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:59:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:59:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:59:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:59:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:59:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:59:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:59:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:59:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:59:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:59:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:59:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:59:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:59:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:59:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:59:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:59:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:59:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:59:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:59:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:59:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:59:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:59:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:59:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:59:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:59:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:59:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:59:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:59:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:59:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:59:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:59:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:59:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:59:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:59:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:59:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:59:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:59:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:59:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:59:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:59:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:59:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:59:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:59:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:59:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:59:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:59:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:59:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:59:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:59:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:59:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:59:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:59:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:59:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:59:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:59:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:59:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:59:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:59:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:59:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:59:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:59:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:59:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:59:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:59:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:59:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:59:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:59:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:59:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:59:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:59:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:59:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:59:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 12:59:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 12:59:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:59:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:59:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:59:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 12:59:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 12:59:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 12:59:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 12:59:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 12:59:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 12:59:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 12:59:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 12:59:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 12:59:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 12:59:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 12:59:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 12:59:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 12:59:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:00:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:00:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:00:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:00:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:00:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:00:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:00:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:00:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:00:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:00:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:00:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:00:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:00:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:00:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:00:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:00:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:00:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:00:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:00:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:00:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:00:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:00:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:00:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:00:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:00:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:00:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:00:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:00:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:00:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:00:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:00:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:00:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:00:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:00:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:00:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:00:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:00:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:00:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:00:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:00:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:00:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:00:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:00:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:00:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:00:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:00:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:00:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:00:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:00:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:00:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:00:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:00:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:00:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:00:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:00:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:00:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:00:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:00:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:00:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:00:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:00:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:00:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:00:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:00:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:00:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:00:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:00:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:00:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:00:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:00:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:00:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:00:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:00:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:00:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:00:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:00:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:00:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:00:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:00:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:00:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:00:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:00:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:00:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:00:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:00:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:00:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:00:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:00:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:00:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:00:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:00:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:00:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:00:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:00:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:00:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:00:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:00:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:00:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:00:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:00:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:00:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:00:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:00:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:00:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:00:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:00:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:00:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:00:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:00:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:00:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:00:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:00:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:00:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:00:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:00:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:00:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:00:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:00:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:00:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:00:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:00:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:00:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:00:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:00:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:00:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:00:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:00:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:00:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:00:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:00:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:00:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:00:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:00:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:00:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:00:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:00:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:00:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:00:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:00:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:00:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:00:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:00:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:00:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:00:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:00:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:00:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:00:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:00:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:00:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:00:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:00:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:00:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:00:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:00:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:00:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:00:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:00:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:00:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:00:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:00:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:00:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:00:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:00:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:00:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:00:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:00:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:00:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:00:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:00:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:00:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:00:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:00:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:00:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:00:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:00:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:00:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:00:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:00:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:00:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:00:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:00:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:00:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:00:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:00:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:00:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:00:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:00:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:00:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:00:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:00:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:00:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:00:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:00:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:00:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:00:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:00:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:00:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:00:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:00:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:00:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:00:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:00:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:00:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:00:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:00:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:00:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:00:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:00:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:00:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:00:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:00:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:00:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:00:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:00:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:00:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:00:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:01:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:01:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:01:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:01:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:01:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:01:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:01:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:01:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:01:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:01:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:01:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:01:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:01:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:01:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:01:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:01:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:01:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:01:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:01:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:01:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:01:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:01:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:01:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:01:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:01:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:01:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:01:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:01:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:01:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:01:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:01:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:01:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:01:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:01:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:01:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:01:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:01:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:01:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:01:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:01:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:01:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:01:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:01:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:01:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:01:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:01:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:01:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:01:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:01:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:01:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:01:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:01:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:01:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:01:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:01:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:01:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:01:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:01:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:01:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:01:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:01:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:01:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:01:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:01:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:01:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:01:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:01:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:01:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:01:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:01:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:01:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:01:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:01:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:01:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:01:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:01:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:01:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:01:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:01:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:01:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:01:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:01:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:01:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:01:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:01:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:01:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:01:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:01:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:01:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:01:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:01:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:01:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:01:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:01:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:01:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:01:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:01:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:01:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:01:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:01:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:01:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:01:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:01:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:01:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:01:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:01:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:01:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:01:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:01:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:01:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:01:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:01:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:01:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:01:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:01:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:01:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:01:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:01:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:01:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:01:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:01:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:01:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:01:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:01:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:01:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:01:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:01:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:01:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:01:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:01:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:01:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:01:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:01:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:01:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:01:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:01:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:01:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:01:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:01:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:01:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:01:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:01:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:01:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:01:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:01:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:01:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:01:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:01:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:01:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:01:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:01:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:01:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:01:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:01:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:01:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:01:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:02:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:02:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:02:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:02:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:02:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:02:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:02:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:02:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:02:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:02:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:02:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:02:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:02:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:02:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:02:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:02:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:02:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:02:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:02:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:02:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:02:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:02:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:02:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:02:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:02:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:02:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:02:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:02:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:02:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:02:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:02:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:02:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:02:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:02:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:02:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:02:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:02:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:02:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:02:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:02:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:02:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:02:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:02:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:02:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:02:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:02:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:02:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:02:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:02:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:02:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:02:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:02:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:02:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:02:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:02:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:02:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:02:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:02:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:02:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:02:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:02:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:02:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:02:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:02:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:02:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:02:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:02:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:02:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:02:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:02:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:02:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:02:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:02:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:02:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:02:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:02:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:02:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:02:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:02:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:02:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:02:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:02:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:02:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:02:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:02:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:02:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:02:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:02:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:02:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:02:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:02:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:02:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:02:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:02:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:02:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:02:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:02:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:02:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:02:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:02:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:02:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:02:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:02:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:02:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:02:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:02:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:02:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:02:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:02:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:02:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:02:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:02:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:02:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:02:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:02:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:02:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:02:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:02:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:02:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:02:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:02:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:02:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:02:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:02:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:02:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:02:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:02:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:02:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:02:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:02:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:02:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:02:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:02:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:02:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:02:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:02:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:02:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:02:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:02:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:02:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:02:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:02:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:02:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:02:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:02:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:02:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:02:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:02:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:02:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:02:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:02:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:02:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:02:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:02:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:02:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:02:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:02:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:02:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:02:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:02:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:02:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:02:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:02:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:02:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:02:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:02:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:02:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:02:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:02:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:02:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:02:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:02:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:02:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:02:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:02:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:02:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:02:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:02:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:02:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:02:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:02:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:02:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:02:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:02:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:02:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:02:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:02:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:02:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:02:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:02:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:02:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:02:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:02:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:02:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:02:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:02:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:02:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:02:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:02:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:02:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:02:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:02:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:02:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:02:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:02:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:02:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:02:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:02:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:02:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:02:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:02:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:02:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:02:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:02:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:02:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:02:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:03:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:03:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:03:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:03:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:03:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:03:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:03:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:03:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:03:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:03:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:03:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:03:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:03:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:03:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:03:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:03:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:03:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:03:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:03:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:03:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:03:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:03:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:03:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:03:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:03:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:03:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:03:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:03:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:03:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:03:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:03:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:03:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:03:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:03:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:03:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:03:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:03:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:03:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:03:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:03:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:03:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:03:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:03:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:03:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:03:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:03:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:03:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:03:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:03:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:03:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:03:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:03:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:03:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:03:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:03:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:03:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:03:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:03:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:03:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:03:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:03:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:03:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:03:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:03:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:03:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:03:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:03:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:03:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:03:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:03:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:03:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:03:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:03:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:03:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:03:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:03:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:03:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:03:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:03:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:03:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:03:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:03:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:03:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:03:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:03:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:03:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:03:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:03:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:03:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:03:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:03:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:03:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:03:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:03:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:03:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:03:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:03:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:03:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:03:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:03:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:03:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:03:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:03:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:03:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:03:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:03:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:03:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:03:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:03:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:03:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:03:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:03:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:03:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:03:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:03:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:03:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:03:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:03:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:03:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:03:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:03:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:03:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:03:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:03:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:03:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:03:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:03:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:03:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:03:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:03:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:03:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:03:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:03:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:03:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:03:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:03:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:03:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:03:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:03:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:03:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:03:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:03:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:03:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:03:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:03:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:03:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:03:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:03:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:03:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:03:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:03:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:03:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:03:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:03:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:03:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:03:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:03:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:03:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:03:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:03:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:03:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:03:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:05:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:05:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:05:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:05:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:05:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:05:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:05:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:05:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:05:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:05:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:05:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:05:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:05:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:05:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:05:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:05:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:05:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:05:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:05:50 rdma06 rsyslogd[1041]: imjournal: 5295 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 13:05:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:05:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:05:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:05:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:05:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:05:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:05:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:05:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:05:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:05:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:05:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:05:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:05:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:05:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:05:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:05:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:05:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:05:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:06:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:06:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:06:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:06:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:06:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:06:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:06:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:06:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:06:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:06:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:06:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:06:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:06:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:06:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:06:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:06:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:06:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:06:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:06:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:06:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:06:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:06:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:06:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:06:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:06:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:06:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:06:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:06:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:06:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:06:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:06:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:06:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:06:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:06:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:06:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:06:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:06:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:06:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:06:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:06:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:06:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:06:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:06:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:06:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:06:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:06:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:06:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:06:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:06:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:06:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:06:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:06:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:06:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:06:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:06:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:06:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:06:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:06:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:06:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:06:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:06:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:06:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:06:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:06:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:06:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:06:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:06:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:06:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:06:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:06:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:06:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:06:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:06:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:06:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:06:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:06:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:06:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:06:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:06:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:06:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:06:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:06:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:06:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:06:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:06:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:06:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:06:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:06:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:06:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:06:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:06:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:06:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:06:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:06:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:06:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:06:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:06:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:06:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:06:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:06:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:06:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:06:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:06:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:06:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:06:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:06:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:06:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:06:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:06:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:06:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:06:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:06:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:06:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:06:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:06:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:06:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:06:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:06:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:06:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:06:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:06:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:06:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:06:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:06:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:06:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:06:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:06:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:06:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:06:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:06:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:06:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:06:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:06:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:06:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:06:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:06:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:06:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:06:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:06:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:06:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:06:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:06:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:06:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:06:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:06:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:06:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:06:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:06:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:06:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:06:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:06:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:06:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:06:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:06:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:06:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:06:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:06:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:06:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:06:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:06:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:06:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:06:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:06:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:06:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:06:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:06:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:06:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:06:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:06:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:06:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:06:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:06:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:06:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:06:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:06:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:06:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:06:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:06:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:06:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:06:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:06:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:06:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:06:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:06:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:06:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:06:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:06:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:06:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:06:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:06:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:06:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:06:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:06:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:06:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:06:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:06:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:06:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:06:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:06:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:06:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:06:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:06:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:06:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:06:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:06:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:06:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:06:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:06:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:06:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:06:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:06:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:06:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:06:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:06:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:06:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:06:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:07:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:07:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:07:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:07:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:07:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:07:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:07:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:07:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:07:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:07:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:07:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:07:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:07:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:07:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:07:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:07:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:07:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:07:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:07:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:07:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:07:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:07:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:07:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:07:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:07:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:07:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:07:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:07:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:07:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:07:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:07:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:07:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:07:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:07:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:07:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:07:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:07:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:07:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:07:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:07:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:07:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:07:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:07:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:07:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:07:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:07:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:07:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:07:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:07:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:07:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:07:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:07:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:07:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:07:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:07:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:07:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:07:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:07:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:07:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:07:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:07:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:07:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:07:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:07:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:07:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:07:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:07:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:07:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:07:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:07:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:07:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:07:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:07:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:07:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:07:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:07:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:07:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:07:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:07:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:07:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:07:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:07:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:07:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:07:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:07:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:07:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:07:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:07:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:07:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:07:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:07:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:07:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:07:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:07:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:07:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:07:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:07:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:07:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:07:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:07:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:07:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:07:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:07:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:07:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:07:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:07:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:07:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:07:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:07:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:07:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:07:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:07:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:07:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:07:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:07:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:07:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:07:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:07:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:07:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:07:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:07:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:07:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:07:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:07:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:07:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:07:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:07:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:07:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:07:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:07:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:07:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:07:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:07:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:07:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:07:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:07:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:07:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:07:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:07:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:07:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:07:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:07:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:07:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:07:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:07:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:07:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:07:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:07:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:07:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:07:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:07:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:07:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:07:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:07:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:07:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:07:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:07:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:07:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:07:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:07:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:07:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:07:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:07:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:07:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:07:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:07:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:07:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:07:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:07:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:07:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:07:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:07:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:07:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:07:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:07:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:07:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:07:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:07:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:07:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:07:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:07:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:07:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:07:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:07:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:07:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:07:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:07:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:07:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:07:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:07:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:07:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:07:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:07:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:07:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:07:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:07:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:07:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:07:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:07:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:07:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:07:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:07:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:07:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:07:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:07:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:07:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:07:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:07:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:07:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:07:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:07:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:07:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:07:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:07:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:07:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:07:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:08:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:08:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:08:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:08:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:08:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:08:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:08:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:08:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:08:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:08:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:08:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:08:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:08:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:08:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:08:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:08:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:08:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:08:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:08:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:08:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:08:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:08:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:08:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:08:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:08:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:08:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:08:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:08:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:08:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:08:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:08:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:08:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:08:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:08:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:08:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:08:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:08:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:08:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:08:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:08:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:08:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:08:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:08:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:08:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:08:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:08:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:08:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:08:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:08:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:08:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:08:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:08:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:08:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:08:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:08:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:08:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:08:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:08:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:08:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:08:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:08:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:08:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:08:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:08:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:08:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:08:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:08:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:08:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:08:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:08:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:08:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:08:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:08:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:08:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:08:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:08:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:08:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:08:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:08:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:08:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:08:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:08:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:08:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:08:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:08:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:08:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:08:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:08:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:08:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:08:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:08:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:08:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:08:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:08:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:08:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:08:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:08:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:08:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:08:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:08:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:08:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:08:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:08:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:08:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:08:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:08:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:08:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:08:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:08:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:08:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:08:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:08:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:08:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:08:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:08:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:08:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:08:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:08:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:08:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:08:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:08:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:08:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:08:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:08:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:08:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:08:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:08:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:08:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:08:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:08:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:08:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:08:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:08:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:08:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:08:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:08:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:08:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:08:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:08:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:08:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:08:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:08:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:08:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:08:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:08:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:08:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:08:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:08:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:08:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:08:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:08:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:08:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:08:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:08:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:08:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:08:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:08:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:08:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:08:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:08:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:08:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:08:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:08:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:08:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:08:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:08:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:08:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:08:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:08:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:08:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:08:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:08:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:08:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:08:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:08:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:08:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:08:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:08:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:08:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:08:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:08:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:08:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:08:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:08:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:08:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:08:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:08:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:08:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:08:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:08:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:08:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:08:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:08:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:08:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:08:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:08:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:08:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:08:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:08:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:08:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:08:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:08:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:08:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:08:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:08:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:08:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:08:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:08:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:08:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:08:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:08:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:08:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:08:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:08:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:08:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:08:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:09:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:09:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:09:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:09:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:09:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:09:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:09:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:09:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:09:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:09:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:09:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:09:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:09:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:09:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:09:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:09:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:09:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:09:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:09:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:09:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:09:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:09:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:09:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:09:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:09:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:09:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:09:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:09:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:09:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:09:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:09:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:09:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:09:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:09:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:09:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:09:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:09:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:09:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:09:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:09:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:09:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:09:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:09:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:09:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:09:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:09:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:09:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:09:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:09:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:09:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:09:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:09:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:09:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:09:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:09:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:09:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:09:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:09:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:09:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:09:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:09:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:09:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:09:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:09:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:09:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:09:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:09:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:09:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:09:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:09:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:09:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:09:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:09:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:09:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:09:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:09:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:09:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:09:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:09:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:09:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:09:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:09:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:09:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:09:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:09:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:09:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:09:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:09:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:09:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:09:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:09:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:09:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:09:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:09:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:09:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:09:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:09:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:09:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:09:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:09:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:09:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:09:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:09:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:09:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:09:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:09:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:09:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:09:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:09:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:09:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:09:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:09:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:09:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:09:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:09:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:09:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:09:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:09:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:09:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:09:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:09:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:09:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:09:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:09:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:09:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:09:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:09:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:09:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:09:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:09:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:09:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:09:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:09:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:09:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:09:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:09:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:09:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:09:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:09:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:09:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:09:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:09:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:09:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:09:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:09:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:09:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:09:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:09:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:09:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:09:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:09:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:09:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:09:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:09:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:09:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:09:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:09:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:09:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:09:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:09:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:09:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:09:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:09:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:09:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:09:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:09:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:09:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:09:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:09:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:09:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:09:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:09:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:09:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:09:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:09:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:09:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:09:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:09:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:09:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:09:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:09:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:09:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:09:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:09:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:09:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:09:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:09:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:09:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:09:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:09:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:09:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:09:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:09:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:09:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:09:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:09:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:09:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:09:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:09:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:09:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:09:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:09:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:09:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:09:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:09:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:09:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:09:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:09:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:09:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:09:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:09:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:09:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:09:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:09:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:09:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:09:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:10:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:10:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:10:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:10:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:10:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:10:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:10:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:10:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:10:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:10:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:10:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:10:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:10:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:10:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:10:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:10:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:10:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:10:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:10:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:10:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:10:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:10:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:10:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:10:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:10:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:10:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:10:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:10:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:10:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:10:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:10:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:10:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:10:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:10:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:10:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:10:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:10:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:10:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:10:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:10:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:10:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:10:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:10:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:10:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:10:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:10:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:10:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:10:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:10:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:10:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:10:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:10:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:10:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:10:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:10:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:10:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:10:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:10:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:10:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:10:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:10:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:10:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:10:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:10:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:10:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:10:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:10:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:10:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:10:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:10:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:10:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:10:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:10:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:10:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:10:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:10:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:10:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:10:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:10:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:10:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:10:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:10:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:10:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:10:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:10:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:10:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:10:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:10:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:10:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:10:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:10:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:10:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:10:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:10:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:10:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:10:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:10:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:10:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:10:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:10:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:10:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:10:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:10:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:10:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:10:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:10:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:10:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:10:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:10:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:10:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:10:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:10:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:10:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:10:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:10:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:10:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:10:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:10:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:10:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:10:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:10:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:10:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:10:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:10:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:10:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:10:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:10:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:10:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:10:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:10:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:10:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:10:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:10:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:10:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:10:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:10:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:10:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:10:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:10:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:10:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:10:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:10:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:10:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:10:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:10:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:10:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:10:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:10:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:10:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:10:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:10:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:10:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:10:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:10:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:10:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:10:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:10:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:10:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:10:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:10:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:10:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:10:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:10:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:10:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:10:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:10:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:10:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:10:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:10:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:10:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:10:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:10:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:10:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:10:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:10:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:10:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:10:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:10:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:10:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:10:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:10:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:10:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:10:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:10:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:10:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:10:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:10:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:10:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:10:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:10:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:10:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:10:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:10:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:10:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:10:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:10:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:10:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:10:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:10:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:10:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:10:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:10:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:10:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:10:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:10:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:10:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:10:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:10:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:10:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:10:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:10:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:10:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:10:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:10:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:10:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:10:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:11:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:11:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:11:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:11:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:11:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:11:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:11:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:11:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:11:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:11:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:11:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:11:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:11:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:11:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:11:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:11:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:11:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:11:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:11:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:11:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:11:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:11:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:11:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:11:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:11:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:11:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:11:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:11:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:11:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:11:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:11:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:11:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:11:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:11:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:11:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:11:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:11:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:11:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:11:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:11:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:11:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:11:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:11:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:11:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:11:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:11:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:11:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:11:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:11:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:11:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:11:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:11:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:11:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:11:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:11:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:11:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:11:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:11:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:11:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:11:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:11:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:11:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:11:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:11:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:11:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:11:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:11:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:11:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:11:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:11:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:11:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:11:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:11:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:11:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:11:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:11:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:11:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:11:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:11:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:11:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:11:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:11:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:11:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:11:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:11:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:11:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:11:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:11:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:11:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:11:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:11:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:11:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:11:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:11:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:11:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:11:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:11:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:11:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:11:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:11:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:11:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:11:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:11:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:11:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:11:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:11:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:11:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:11:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:11:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:11:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:11:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:11:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:11:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:11:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:11:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:11:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:11:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:11:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:11:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:11:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:11:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:11:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:11:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:11:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:11:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:11:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:11:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:11:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:11:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:11:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:11:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:11:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:11:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:11:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:11:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:11:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:11:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:11:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:11:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:11:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:11:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:11:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:11:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:11:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:11:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:11:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:11:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:11:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:11:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:11:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:11:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:11:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:11:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:11:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:11:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:11:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:11:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:11:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:11:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:11:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:11:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:11:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:11:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:11:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:11:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:11:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:11:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:11:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:11:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:11:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:11:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:11:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:11:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:11:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:11:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:11:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:11:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:11:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:11:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:11:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:11:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:11:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:11:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:11:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:11:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:11:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:12:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:12:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:12:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:12:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:12:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:12:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:12:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:12:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:12:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:12:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:12:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:12:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:12:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:12:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:12:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:12:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:12:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:12:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:12:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:12:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:12:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:12:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:12:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:12:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:12:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:12:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:12:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:12:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:12:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:12:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:12:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:12:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:12:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:12:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:12:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:12:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:12:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:12:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:12:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:12:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:12:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:12:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:12:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:12:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:12:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:12:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:12:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:12:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:12:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:12:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:12:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:12:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:12:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:12:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:12:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:12:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:12:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:12:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:12:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:12:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:12:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:12:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:12:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:12:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:12:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:12:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:12:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:12:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:12:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:12:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:12:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:12:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:12:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:12:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:12:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:12:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:12:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:12:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:12:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:12:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:12:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:12:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:12:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:12:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:12:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:12:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:12:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:12:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:12:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:12:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:12:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:12:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:12:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:12:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:12:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:12:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:12:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:12:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:12:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:12:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:12:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:12:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:12:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:12:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:12:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:12:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:12:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:12:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:12:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:12:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:12:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:12:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:12:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:12:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:12:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:12:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:12:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:12:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:12:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:12:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:12:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:12:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:12:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:12:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:12:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:12:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:12:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:12:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:12:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:12:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:12:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:12:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:12:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:12:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:12:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:12:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:12:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:12:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:12:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:12:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:12:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:12:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:12:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:12:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:12:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:12:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:12:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:12:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:12:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:12:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:12:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:12:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:12:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:12:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:12:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:12:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:12:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:12:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:12:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:12:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:12:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:12:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:12:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:12:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:12:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:12:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:12:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:12:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:12:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:12:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:12:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:12:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:12:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:12:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:12:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:12:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:12:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:12:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:12:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:12:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:12:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:12:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:12:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:12:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:12:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:12:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:12:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:12:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:12:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:12:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:12:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:12:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:12:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:12:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:12:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:12:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:12:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:12:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:12:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:12:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:12:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:12:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:12:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:12:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:12:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:12:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:12:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:12:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:12:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:12:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:12:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:12:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:12:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:12:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:12:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:12:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:13:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:13:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:13:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:13:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:13:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:13:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:13:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:13:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:13:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:13:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:13:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:13:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:13:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:13:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:13:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:13:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:13:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:13:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:13:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:13:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:13:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:13:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:13:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:13:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:13:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:13:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:13:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:13:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:13:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:13:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:13:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:13:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:13:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:13:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:13:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:13:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:13:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:13:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:13:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:13:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:13:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:13:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:13:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:13:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:13:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:13:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:13:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:13:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:13:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:13:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:13:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:13:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:13:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:13:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:13:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:13:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:13:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:13:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:13:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:13:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:13:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:13:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:13:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:13:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:13:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:13:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:13:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:13:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:13:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:13:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:13:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:13:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:13:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:13:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:13:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:13:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:13:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:13:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:13:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:13:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:13:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:13:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:13:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:13:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:13:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:13:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:13:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:13:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:13:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:13:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:13:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:13:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:13:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:13:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:13:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:13:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:13:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:13:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:13:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:13:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:13:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:13:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:13:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:13:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:13:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:13:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:13:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:13:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:13:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:13:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:13:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:13:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:13:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:13:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:13:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:13:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:13:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:13:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:13:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:13:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:13:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:13:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:13:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:13:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:13:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:13:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:13:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:13:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:13:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:13:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:13:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:13:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:13:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:13:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:13:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:13:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:13:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:13:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:13:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:13:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:13:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:13:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:13:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:13:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:13:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:13:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:13:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:13:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:13:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:13:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:13:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:13:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:13:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:13:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:13:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:13:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:13:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:13:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:13:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:13:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:13:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:13:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:13:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:13:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:13:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:13:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:13:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:13:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:13:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:13:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:13:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:13:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:13:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:13:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:13:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:13:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:13:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:13:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:13:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:13:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:15:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:15:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:15:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:15:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:15:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:15:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:15:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:15:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:15:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:15:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:15:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:15:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:15:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:15:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:15:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:15:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:15:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:15:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:15:51 rdma06 rsyslogd[1041]: imjournal: 5082 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 13:15:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:15:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:15:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:15:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:15:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:15:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:15:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:15:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:15:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:15:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:15:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:15:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:15:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:15:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:15:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:15:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:15:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:15:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:16:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:16:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:16:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:16:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:16:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:16:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:16:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:16:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:16:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:16:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:16:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:16:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:16:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:16:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:16:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:16:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:16:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:16:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:16:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:16:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:16:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:16:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:16:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:16:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:16:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:16:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:16:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:16:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:16:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:16:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:16:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:16:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:16:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:16:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:16:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:16:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:16:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:16:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:16:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:16:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:16:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:16:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:16:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:16:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:16:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:16:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:16:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:16:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:16:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:16:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:16:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:16:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:16:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:16:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:16:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:16:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:16:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:16:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:16:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:16:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:16:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:16:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:16:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:16:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:16:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:16:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:16:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:16:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:16:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:16:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:16:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:16:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:16:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:16:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:16:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:16:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:16:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:16:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:16:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:16:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:16:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:16:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:16:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:16:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:16:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:16:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:16:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:16:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:16:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:16:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:16:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:16:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:16:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:16:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:16:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:16:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:16:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:16:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:16:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:16:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:16:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:16:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:16:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:16:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:16:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:16:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:16:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:16:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:16:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:16:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:16:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:16:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:16:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:16:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:16:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:16:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:16:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:16:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:16:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:16:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:16:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:16:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:16:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:16:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:16:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:16:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:16:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:16:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:16:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:16:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:16:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:16:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:16:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:16:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:16:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:16:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:16:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:16:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:16:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:16:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:16:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:16:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:16:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:16:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:16:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:16:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:16:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:16:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:16:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:16:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:16:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:16:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:16:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:16:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:16:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:16:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:16:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:16:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:16:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:16:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:16:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:16:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:16:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:16:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:16:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:16:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:16:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:16:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:16:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:16:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:16:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:16:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:16:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:16:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:16:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:16:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:16:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:16:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:16:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:16:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:16:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:16:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:16:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:16:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:16:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:16:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:16:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:16:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:16:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:16:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:16:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:16:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:16:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:16:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:16:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:16:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:16:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:16:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:16:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:16:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:16:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:16:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:16:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:16:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:16:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:16:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:16:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:16:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:16:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:16:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:16:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:16:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:16:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:16:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:16:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:16:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:17:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:17:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:17:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:17:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:17:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:17:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:17:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:17:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:17:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:17:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:17:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:17:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:17:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:17:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:17:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:17:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:17:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:17:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:17:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:17:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:17:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:17:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:17:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:17:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:17:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:17:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:17:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:17:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:17:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:17:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:17:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:17:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:17:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:17:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:17:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:17:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:17:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:17:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:17:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:17:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:17:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:17:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:17:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:17:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:17:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:17:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:17:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:17:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:17:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:17:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:17:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:17:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:17:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:17:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:17:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:17:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:17:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:17:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:17:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:17:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:17:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:17:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:17:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:17:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:17:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:17:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:17:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:17:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:17:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:17:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:17:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:17:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:17:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:17:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:17:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:17:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:17:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:17:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:17:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:17:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:17:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:17:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:17:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:17:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:17:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:17:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:17:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:17:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:17:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:17:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:17:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:17:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:17:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:17:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:17:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:17:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:17:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:17:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:17:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:17:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:17:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:17:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:17:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:17:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:17:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:17:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:17:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:17:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:17:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:17:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:17:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:17:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:17:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:17:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:17:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:17:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:17:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:17:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:17:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:17:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:17:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:17:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:17:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:17:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:17:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:17:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:17:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:17:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:17:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:17:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:17:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:17:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:17:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:17:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:17:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:17:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:17:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:17:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:17:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:17:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:17:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:17:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:17:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:17:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:17:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:17:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:17:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:17:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:17:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:17:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:17:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:17:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:17:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:17:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:17:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:17:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:17:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:17:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:17:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:17:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:17:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:17:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:17:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:17:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:17:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:17:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:17:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:17:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:17:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:17:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:17:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:17:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:17:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:17:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:17:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:17:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:17:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:17:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:17:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:17:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:17:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:17:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:17:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:17:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:17:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:17:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:17:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:17:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:17:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:17:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:17:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:17:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:17:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:17:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:17:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:17:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:17:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:17:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:17:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:17:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:17:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:17:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:17:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:17:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:17:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:17:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:17:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:17:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:17:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:17:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:17:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:17:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:17:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:17:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:17:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:17:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:18:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:18:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:18:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:18:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:18:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:18:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:18:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:18:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:18:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:18:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:18:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:18:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:18:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:18:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:18:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:18:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:18:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:18:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:18:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:18:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:18:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:18:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:18:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:18:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:18:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:18:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:18:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:18:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:18:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:18:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:18:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:18:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:18:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:18:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:18:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:18:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:18:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:18:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:18:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:18:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:18:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:18:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:18:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:18:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:18:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:18:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:18:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:18:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:18:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:18:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:18:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:18:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:18:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:18:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:18:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:18:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:18:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:18:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:18:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:18:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:18:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:18:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:18:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:18:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:18:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:18:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:18:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:18:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:18:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:18:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:18:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:18:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:18:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:18:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:18:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:18:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:18:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:18:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:18:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:18:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:18:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:18:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:18:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:18:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:18:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:18:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:18:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:18:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:18:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:18:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:18:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:18:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:18:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:18:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:18:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:18:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:18:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:18:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:18:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:18:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:18:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:18:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:18:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:18:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:18:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:18:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:18:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:18:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:18:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:18:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:18:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:18:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:18:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:18:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:18:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:18:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:18:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:18:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:18:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:18:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:18:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:18:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:18:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:18:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:18:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:18:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:18:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:18:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:18:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:18:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:18:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:18:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:18:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:18:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:18:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:18:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:18:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:18:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:18:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:18:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:18:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:18:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:18:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:18:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:18:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:18:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:18:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:18:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:18:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:18:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:18:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:18:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:18:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:18:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:18:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:18:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:18:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:18:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:18:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:18:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:18:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:18:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:18:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:18:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:18:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:18:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:18:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:18:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:18:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:18:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:18:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:18:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:18:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:18:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:18:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:18:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:18:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:18:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:18:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:18:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:18:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:18:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:18:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:18:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:18:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:18:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:18:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:18:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:18:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:18:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:18:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:18:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:18:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:18:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:18:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:18:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:18:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:18:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:18:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:18:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:18:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:18:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:18:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:18:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:18:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:18:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:18:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:18:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:18:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:18:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:18:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:18:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:18:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:18:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:18:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:18:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:19:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:19:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:19:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:19:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:19:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:19:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:19:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:19:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:19:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:19:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:19:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:19:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:19:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:19:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:19:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:19:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:19:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:19:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:19:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:19:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:19:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:19:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:19:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:19:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:19:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:19:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:19:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:19:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:19:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:19:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:19:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:19:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:19:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:19:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:19:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:19:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:19:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:19:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:19:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:19:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:19:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:19:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:19:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:19:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:19:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:19:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:19:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:19:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:19:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:19:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:19:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:19:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:19:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:19:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:19:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:19:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:19:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:19:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:19:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:19:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:19:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:19:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:19:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:19:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:19:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:19:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:19:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:19:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:19:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:19:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:19:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:19:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:19:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:19:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:19:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:19:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:19:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:19:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:19:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:19:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:19:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:19:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:19:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:19:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:19:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:19:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:19:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:19:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:19:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:19:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:19:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:19:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:19:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:19:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:19:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:19:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:19:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:19:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:19:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:19:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:19:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:19:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:19:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:19:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:19:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:19:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:19:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:19:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:19:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:19:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:19:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:19:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:19:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:19:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:19:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:19:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:19:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:19:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:19:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:19:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:19:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:19:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:19:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:19:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:19:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:19:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:19:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:19:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:19:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:19:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:19:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:19:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:19:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:19:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:19:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:19:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:19:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:19:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:19:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:19:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:19:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:19:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:19:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:19:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:19:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:19:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:19:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:19:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:19:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:19:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:19:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:19:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:19:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:19:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:19:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:19:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:19:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:19:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:19:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:19:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:19:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:19:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:19:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:19:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:19:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:19:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:19:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:19:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:19:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:19:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:19:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:19:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:19:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:19:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:19:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:19:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:19:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:19:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:19:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:19:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:19:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:19:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:19:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:19:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:19:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:19:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:19:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:19:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:19:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:19:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:19:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:19:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:19:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:19:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:19:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:19:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:19:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:19:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:19:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:19:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:19:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:19:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:19:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:19:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:19:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:19:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:19:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:19:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:19:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:19:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:19:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:19:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:19:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:19:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:19:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:19:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:20:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:20:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:20:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:20:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:20:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:20:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:20:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:20:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:20:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:20:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:20:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:20:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:20:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:20:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:20:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:20:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:20:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:20:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:20:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:20:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:20:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:20:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:20:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:20:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:20:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:20:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:20:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:20:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:20:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:20:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:20:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:20:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:20:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:20:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:20:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:20:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:20:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:20:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:20:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:20:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:20:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:20:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:20:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:20:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:20:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:20:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:20:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:20:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:20:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:20:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:20:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:20:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:20:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:20:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:20:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:20:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:20:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:20:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:20:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:20:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:20:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:20:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:20:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:20:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:20:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:20:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:20:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:20:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:20:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:20:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:20:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:20:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:20:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:20:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:20:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:20:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:20:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:20:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:20:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:20:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:20:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:20:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:20:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:20:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:20:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:20:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:20:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:20:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:20:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:20:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:20:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:20:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:20:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:20:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:20:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:20:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:20:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:20:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:20:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:20:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:20:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:20:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:20:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:20:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:20:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:20:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:20:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:20:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:20:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:20:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:20:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:20:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:20:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:20:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:20:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:20:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:20:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:20:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:20:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:20:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:20:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:20:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:20:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:20:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:20:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:20:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:20:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:20:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:20:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:20:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:20:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:20:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:20:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:20:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:20:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:20:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:20:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:20:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:20:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:20:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:20:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:20:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:20:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:20:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:20:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:20:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:20:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:20:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:20:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:20:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:20:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:20:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:20:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:20:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:20:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:20:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:20:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:20:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:20:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:20:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:20:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:20:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:20:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:20:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:20:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:20:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:20:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:20:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:20:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:20:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:20:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:20:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:20:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:20:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:20:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:20:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:20:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:20:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:20:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:20:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:20:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:20:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:20:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:20:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:20:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:20:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:20:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:20:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:20:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:20:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:20:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:20:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:20:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:20:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:20:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:20:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:20:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:20:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:20:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:20:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:20:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:20:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:20:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:20:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:20:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:20:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:20:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:20:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:20:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:20:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:20:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:20:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:20:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:20:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:20:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:20:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:21:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:21:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:21:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:21:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:21:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:21:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:21:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:21:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:21:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:21:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:21:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:21:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:21:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:21:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:21:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:21:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:21:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:21:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:21:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:21:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:21:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:21:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:21:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:21:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:21:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:21:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:21:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:21:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:21:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:21:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:21:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:21:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:21:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:21:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:21:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:21:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:21:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:21:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:21:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:21:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:21:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:21:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:21:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:21:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:21:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:21:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:21:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:21:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:21:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:21:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:21:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:21:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:21:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:21:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:21:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:21:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:21:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:21:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:21:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:21:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:21:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:21:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:21:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:21:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:21:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:21:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:21:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:21:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:21:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:21:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:21:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:21:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:21:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:21:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:21:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:21:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:21:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:21:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:21:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:21:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:21:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:21:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:21:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:21:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:21:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:21:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:21:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:21:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:21:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:21:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:21:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:21:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:21:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:21:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:21:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:21:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:21:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:21:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:21:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:21:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:21:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:21:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:21:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:21:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:21:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:21:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:21:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:21:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:21:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:21:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:21:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:21:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:21:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:21:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:21:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:21:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:21:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:21:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:21:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:21:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:21:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:21:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:21:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:21:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:21:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:21:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:21:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:21:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:21:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:21:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:21:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:21:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:21:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:21:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:21:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:21:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:21:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:21:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:21:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:21:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:21:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:21:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:21:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:21:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:21:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:21:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:21:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:21:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:21:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:21:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:21:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:21:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:21:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:21:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:21:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:21:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:21:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:21:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:21:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:21:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:21:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:21:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:21:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:21:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:21:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:21:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:21:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:21:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:21:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:21:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:21:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:21:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:21:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:21:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:21:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:21:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:21:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:21:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:21:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:21:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:21:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:21:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:21:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:21:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:21:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:21:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:21:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:21:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:21:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:21:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:21:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:21:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:21:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:21:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:21:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:21:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:21:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:21:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:21:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:21:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:21:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:21:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:21:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:21:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:21:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:21:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:21:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:21:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:21:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:21:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:21:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:21:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:21:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:21:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:21:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:21:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:22:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:22:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:22:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:22:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:22:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:22:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:22:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:22:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:22:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:22:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:22:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:22:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:22:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:22:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:22:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:22:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:22:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:22:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:22:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:22:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:22:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:22:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:22:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:22:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:22:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:22:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:22:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:22:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:22:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:22:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:22:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:22:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:22:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:22:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:22:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:22:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:22:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:22:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:22:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:22:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:22:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:22:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:22:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:22:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:22:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:22:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:22:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:22:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:22:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:22:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:22:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:22:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:22:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:22:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:22:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:22:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:22:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:22:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:22:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:22:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:22:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:22:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:22:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:22:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:22:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:22:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:22:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:22:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:22:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:22:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:22:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:22:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:22:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:22:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:22:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:22:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:22:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:22:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:22:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:22:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:22:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:22:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:22:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:22:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:22:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:22:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:22:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:22:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:22:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:22:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:22:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:22:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:22:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:22:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:22:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:22:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:22:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:22:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:22:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:22:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:22:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:22:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:22:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:22:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:22:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:22:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:22:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:22:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:22:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:22:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:22:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:22:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:22:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:22:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:22:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:22:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:22:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:22:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:22:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:22:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:22:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:22:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:22:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:22:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:22:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:22:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:22:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:22:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:22:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:22:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:22:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:22:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:22:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:22:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:22:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:22:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:22:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:22:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:22:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:22:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:22:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:22:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:22:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:22:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:22:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:22:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:22:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:22:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:22:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:22:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:22:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:22:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:22:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:22:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:22:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:22:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:22:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:22:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:22:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:22:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:22:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:22:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:22:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:22:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:22:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:22:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:22:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:22:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:22:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:22:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:22:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:22:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:22:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:22:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:22:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:22:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:22:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:22:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:22:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:22:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:22:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:22:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:22:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:22:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:22:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:22:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:22:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:22:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:22:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:22:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:22:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:22:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:22:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:22:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:22:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:22:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:22:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:22:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:22:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:22:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:22:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:22:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:22:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:22:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:22:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:22:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:22:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:22:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:22:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:22:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:22:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:22:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:22:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:22:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:22:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:22:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:23:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:23:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:23:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:23:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:23:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:23:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:23:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:23:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:23:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:23:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:23:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:23:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:23:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:23:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:23:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:23:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:23:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:23:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:23:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:23:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:23:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:23:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:23:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:23:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:23:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:23:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:23:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:23:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:23:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:23:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:23:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:23:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:23:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:23:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:23:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:23:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:23:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:23:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:23:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:23:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:23:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:23:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:23:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:23:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:23:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:23:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:23:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:23:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:23:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:23:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:23:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:23:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:23:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:23:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:23:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:23:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:23:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:23:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:23:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:23:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:23:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:23:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:23:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:23:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:23:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:23:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:23:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:23:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:23:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:23:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:23:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:23:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:23:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:23:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:23:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:23:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:23:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:23:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:23:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:23:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:23:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:23:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:23:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:23:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:23:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:23:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:23:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:23:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:23:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:23:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:23:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:23:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:23:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:23:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:23:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:23:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:23:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:23:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:23:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:23:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:23:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:23:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:23:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:23:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:23:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:23:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:23:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:23:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:23:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:23:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:23:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:23:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:23:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:23:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:23:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:23:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:23:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:23:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:23:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:23:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:23:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:23:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:23:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:23:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:23:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:23:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:23:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:23:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:23:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:23:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:23:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:23:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:23:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:23:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:23:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:23:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:23:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:23:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:23:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:23:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:23:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:23:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:23:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:23:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:23:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:23:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:23:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:23:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:23:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:23:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:23:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:23:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:23:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:23:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:23:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:23:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:23:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:23:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:23:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:23:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:23:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:23:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:23:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:23:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:23:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:23:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:23:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:23:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:23:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:23:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:23:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:23:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:23:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:23:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:23:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:23:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:23:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:23:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:23:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:23:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:25:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:25:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:25:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:25:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:25:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:25:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:25:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:25:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:25:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:25:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:25:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:25:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:25:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:25:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:25:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:25:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:25:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:25:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:25:56 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 13:26:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:26:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:26:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:26:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:26:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:26:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:26:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:26:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:26:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:26:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:26:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:26:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:26:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:26:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:26:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:26:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:26:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:26:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:26:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:26:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:26:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:26:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:26:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:26:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:26:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:26:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:26:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:26:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:26:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:26:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:26:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:26:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:26:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:26:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:26:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:26:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:26:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:26:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:26:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:26:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:26:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:26:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:26:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:26:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:26:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:26:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:26:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:26:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:26:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:26:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:26:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:26:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:26:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:26:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:26:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:26:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:26:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:26:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:26:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:26:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:26:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:26:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:26:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:26:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:26:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:26:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:26:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:26:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:26:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:26:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:26:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:26:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:26:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:26:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:26:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:26:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:26:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:26:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:26:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:26:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:26:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:26:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:26:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:26:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:26:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:26:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:26:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:26:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:26:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:26:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:26:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:26:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:26:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:26:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:26:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:26:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:26:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:26:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:26:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:26:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:26:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:26:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:26:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:26:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:26:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:26:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:26:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:26:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:26:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:26:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:26:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:26:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:26:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:26:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:26:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:26:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:26:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:26:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:26:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:26:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:26:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:26:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:26:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:26:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:26:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:26:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:26:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:26:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:26:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:26:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:26:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:26:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:26:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:26:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:26:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:26:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:26:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:26:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:26:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:26:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:26:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:26:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:26:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:26:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:26:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:26:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:26:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:26:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:26:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:26:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:26:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:26:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:26:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:26:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:26:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:26:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:26:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:26:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:26:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:26:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:26:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:26:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:26:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:26:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:26:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:26:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:26:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:26:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:26:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:26:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:26:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:26:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:26:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:26:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:26:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:26:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:26:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:26:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:26:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:26:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:26:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:26:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:26:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:26:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:26:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:26:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:26:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:26:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:26:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:26:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:26:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:26:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:26:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:26:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:26:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:26:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:26:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:26:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:26:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:26:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:26:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:26:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:26:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:26:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:26:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:26:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:26:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:26:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:26:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:26:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:26:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:26:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:26:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:26:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:26:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:26:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:27:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:27:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:27:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:27:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:27:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:27:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:27:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:27:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:27:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:27:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:27:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:27:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:27:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:27:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:27:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:27:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:27:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:27:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:27:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:27:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:27:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:27:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:27:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:27:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:27:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:27:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:27:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:27:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:27:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:27:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:27:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:27:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:27:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:27:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:27:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:27:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:27:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:27:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:27:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:27:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:27:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:27:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:27:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:27:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:27:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:27:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:27:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:27:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:27:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:27:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:27:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:27:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:27:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:27:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:27:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:27:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:27:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:27:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:27:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:27:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:27:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:27:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:27:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:27:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:27:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:27:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:27:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:27:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:27:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:27:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:27:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:27:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:27:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:27:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:27:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:27:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:27:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:27:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:27:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:27:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:27:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:27:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:27:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:27:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:27:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:27:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:27:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:27:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:27:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:27:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:27:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:27:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:27:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:27:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:27:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:27:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:27:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:27:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:27:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:27:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:27:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:27:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:27:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:27:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:27:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:27:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:27:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:27:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:27:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:27:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:27:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:27:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:27:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:27:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:27:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:27:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:27:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:27:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:27:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:27:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:27:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:27:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:27:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:27:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:27:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:27:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:27:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:27:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:27:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:27:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:27:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:27:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:27:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:27:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:27:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:27:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:27:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:27:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:27:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:27:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:27:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:27:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:27:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:27:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:27:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:27:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:27:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:27:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:27:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:27:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:27:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:27:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:27:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:27:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:27:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:27:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:27:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:27:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:27:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:27:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:27:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:27:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:27:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:27:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:27:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:27:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:27:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:27:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:27:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:27:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:27:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:27:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:27:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:27:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:27:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:27:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:27:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:27:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:27:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:27:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:27:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:27:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:27:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:27:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:27:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:27:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:27:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:27:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:27:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:27:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:27:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:27:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:27:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:27:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:27:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:27:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:27:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:27:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:27:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:27:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:27:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:27:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:27:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:27:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:27:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:27:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:27:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:27:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:27:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:27:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:27:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:27:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:27:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:27:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:27:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:27:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:28:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:28:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:28:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:28:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:28:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:28:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:28:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:28:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:28:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:28:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:28:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:28:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:28:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:28:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:28:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:28:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:28:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:28:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:28:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:28:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:28:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:28:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:28:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:28:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:28:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:28:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:28:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:28:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:28:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:28:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:28:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:28:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:28:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:28:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:28:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:28:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:28:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:28:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:28:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:28:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:28:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:28:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:28:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:28:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:28:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:28:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:28:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:28:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:28:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:28:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:28:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:28:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:28:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:28:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:28:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:28:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:28:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:28:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:28:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:28:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:28:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:28:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:28:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:28:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:28:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:28:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:28:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:28:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:28:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:28:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:28:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:28:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:28:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:28:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:28:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:28:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:28:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:28:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:28:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:28:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:28:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:28:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:28:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:28:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:28:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:28:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:28:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:28:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:28:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:28:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:28:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:28:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:28:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:28:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:28:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:28:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:28:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:28:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:28:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:28:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:28:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:28:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:28:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:28:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:28:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:28:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:28:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:28:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:28:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:28:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:28:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:28:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:28:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:28:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:28:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:28:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:28:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:28:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:28:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:28:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:28:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:28:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:28:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:28:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:28:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:28:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:28:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:28:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:28:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:28:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:28:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:28:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:28:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:28:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:28:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:28:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:28:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:28:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:28:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:28:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:28:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:28:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:28:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:28:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:28:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:28:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:28:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:28:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:28:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:28:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:28:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:28:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:28:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:28:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:28:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:28:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:28:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:28:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:28:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:28:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:28:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:28:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:28:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:28:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:28:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:28:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:28:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:28:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:28:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:28:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:28:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:28:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:28:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:28:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:28:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:28:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:28:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:28:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:28:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:28:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:28:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:28:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:28:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:28:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:28:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:28:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:28:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:28:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:28:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:28:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:28:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:28:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:28:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:28:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:28:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:28:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:28:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:28:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:28:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:28:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:28:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:28:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:28:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:28:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:28:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:28:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:28:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:28:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:28:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:28:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:28:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:28:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:28:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:28:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:28:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:28:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:29:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:29:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:29:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:29:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:29:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:29:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:29:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:29:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:29:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:29:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:29:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:29:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:29:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:29:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:29:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:29:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:29:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:29:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:29:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:29:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:29:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:29:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:29:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:29:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:29:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:29:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:29:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:29:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:29:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:29:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:29:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:29:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:29:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:29:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:29:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:29:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:29:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:29:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:29:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:29:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:29:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:29:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:29:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:29:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:29:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:29:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:29:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:29:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:29:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:29:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:29:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:29:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:29:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:29:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:29:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:29:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:29:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:29:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:29:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:29:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:29:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:29:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:29:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:29:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:29:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:29:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:29:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:29:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:29:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:29:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:29:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:29:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:29:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:29:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:29:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:29:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:29:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:29:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:29:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:29:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:29:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:29:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:29:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:29:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:29:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:29:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:29:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:29:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:29:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:29:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:29:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:29:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:29:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:29:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:29:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:29:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:29:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:29:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:29:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:29:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:29:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:29:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:29:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:29:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:29:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:29:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:29:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:29:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:29:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:29:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:29:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:29:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:29:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:29:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:29:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:29:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:29:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:29:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:29:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:29:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:29:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:29:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:29:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:29:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:29:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:29:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:29:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:29:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:29:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:29:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:29:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:29:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:29:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:29:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:29:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:29:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:29:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:29:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:29:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:29:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:29:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:29:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:29:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:29:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:29:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:29:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:29:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:29:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:29:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:29:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:29:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:29:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:29:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:29:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:29:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:29:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:29:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:29:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:29:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:29:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:29:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:29:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:29:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:29:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:29:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:29:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:29:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:29:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:29:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:29:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:29:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:29:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:29:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:29:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:29:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:29:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:29:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:29:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:29:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:29:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:29:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:29:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:29:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:29:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:29:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:29:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:29:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:29:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:29:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:29:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:29:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:29:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:29:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:29:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:29:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:29:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:29:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:29:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:29:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:29:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:29:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:29:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:29:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:29:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:29:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:29:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:29:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:29:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:29:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:29:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:29:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:29:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:29:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:29:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:29:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:29:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:30:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:30:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:30:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:30:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:30:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:30:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:30:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:30:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:30:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:30:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:30:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:30:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:30:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:30:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:30:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:30:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:30:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:30:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:30:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:30:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:30:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:30:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:30:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:30:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:30:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:30:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:30:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:30:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:30:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:30:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:30:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:30:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:30:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:30:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:30:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:30:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:30:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:30:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:30:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:30:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:30:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:30:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:30:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:30:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:30:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:30:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:30:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:30:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:30:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:30:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:30:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:30:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:30:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:30:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:30:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:30:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:30:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:30:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:30:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:30:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:30:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:30:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:30:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:30:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:30:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:30:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:30:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:30:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:30:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:30:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:30:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:30:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:30:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:30:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:30:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:30:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:30:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:30:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:30:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:30:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:30:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:30:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:30:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:30:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:30:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:30:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:30:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:30:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:30:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:30:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:30:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:30:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:30:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:30:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:30:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:30:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:30:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:30:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:30:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:30:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:30:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:30:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:30:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:30:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:30:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:30:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:30:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:30:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:30:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:30:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:30:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:30:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:30:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:30:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:30:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:30:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:30:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:30:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:30:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:30:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:30:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:30:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:30:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:30:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:30:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:30:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:30:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:30:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:30:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:30:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:30:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:30:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:30:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:30:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:30:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:30:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:30:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:30:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:30:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:30:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:30:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:30:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:30:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:30:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:30:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:30:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:30:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:30:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:30:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:30:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:30:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:30:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:30:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:30:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:30:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:30:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:30:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:30:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:30:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:30:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:30:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:30:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:30:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:30:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:30:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:30:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:30:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:30:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:30:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:30:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:30:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:30:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:30:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:30:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:30:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:30:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:30:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:30:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:30:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:30:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:30:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:30:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:30:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:30:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:30:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:30:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:30:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:30:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:30:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:30:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:30:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:30:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:30:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:30:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:30:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:30:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:30:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:30:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:30:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:30:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:30:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:30:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:30:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:30:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:30:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:30:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:30:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:30:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:30:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:30:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:30:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:30:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:30:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:30:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:30:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:30:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:31:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:31:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:31:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:31:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:31:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:31:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:31:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:31:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:31:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:31:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:31:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:31:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:31:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:31:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:31:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:31:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:31:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:31:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:31:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:31:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:31:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:31:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:31:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:31:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:31:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:31:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:31:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:31:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:31:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:31:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:31:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:31:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:31:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:31:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:31:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:31:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:31:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:31:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:31:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:31:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:31:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:31:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:31:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:31:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:31:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:31:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:31:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:31:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:31:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:31:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:31:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:31:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:31:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:31:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:31:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:31:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:31:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:31:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:31:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:31:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:31:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:31:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:31:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:31:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:31:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:31:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:31:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:31:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:31:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:31:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:31:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:31:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:31:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:31:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:31:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:31:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:31:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:31:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:31:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:31:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:31:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:31:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:31:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:31:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:31:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:31:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:31:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:31:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:31:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:31:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:31:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:31:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:31:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:31:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:31:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:31:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:31:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:31:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:31:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:31:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:31:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:31:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:31:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:31:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:31:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:31:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:31:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:31:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:31:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:31:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:31:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:31:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:31:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:31:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:31:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:31:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:31:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:31:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:31:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:31:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:31:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:31:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:31:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:31:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:31:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:31:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:31:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:31:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:31:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:31:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:31:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:31:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:31:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:31:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:31:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:31:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:31:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:31:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:31:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:31:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:31:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:31:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:31:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:31:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:31:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:31:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:31:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:31:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:31:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:31:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:31:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:31:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:31:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:31:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:31:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:31:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:31:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:31:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:31:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:31:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:31:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:31:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:31:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:31:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:31:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:31:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:32:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:32:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:32:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:32:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:32:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:32:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:32:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:32:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:32:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:32:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:32:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:32:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:32:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:32:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:32:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:32:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:32:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:32:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:32:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:32:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:32:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:32:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:32:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:32:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:32:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:32:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:32:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:32:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:32:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:32:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:32:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:32:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:32:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:32:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:32:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:32:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:32:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:32:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:32:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:32:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:32:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:32:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:32:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:32:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:32:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:32:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:32:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:32:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:32:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:32:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:32:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:32:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:32:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:32:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:32:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:32:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:32:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:32:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:32:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:32:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:32:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:32:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:32:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:32:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:32:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:32:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:32:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:32:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:32:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:32:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:32:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:32:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:32:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:32:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:32:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:32:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:32:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:32:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:32:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:32:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:32:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:32:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:32:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:32:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:32:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:32:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:32:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:32:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:32:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:32:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:32:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:32:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:32:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:32:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:32:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:32:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:32:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:32:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:32:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:32:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:32:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:32:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:32:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:32:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:32:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:32:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:32:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:32:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:32:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:32:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:32:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:32:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:32:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:32:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:32:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:32:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:32:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:32:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:32:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:32:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:32:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:32:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:32:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:32:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:32:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:32:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:32:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:32:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:32:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:32:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:32:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:32:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:32:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:32:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:32:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:32:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:32:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:32:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:32:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:32:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:32:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:32:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:32:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:32:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:32:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:32:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:32:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:32:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:32:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:32:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:32:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:32:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:32:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:32:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:32:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:32:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:32:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:32:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:32:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:32:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:32:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:32:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:32:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:32:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:32:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:32:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:32:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:32:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:32:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:32:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:32:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:32:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:32:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:32:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:32:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:32:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:32:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:32:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:32:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:32:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:32:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:32:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:32:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:32:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:32:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:32:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:32:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:32:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:32:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:32:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:32:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:32:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:32:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:32:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:32:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:32:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:32:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:32:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:32:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:32:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:32:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:32:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:32:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:32:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:32:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:32:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:32:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:32:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:32:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:32:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:32:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:32:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:32:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:32:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:32:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:32:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:33:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:33:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:33:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:33:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:33:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:33:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:33:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:33:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:33:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:33:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:33:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:33:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:33:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:33:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:33:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:33:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:33:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:33:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:33:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:33:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:33:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:33:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:33:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:33:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:33:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:33:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:33:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:33:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:33:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:33:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:33:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:33:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:33:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:33:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:33:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:33:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:33:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:33:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:33:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:33:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:33:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:33:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:33:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:33:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:33:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:33:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:33:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:33:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:33:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:33:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:33:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:33:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:33:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:33:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:33:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:33:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:33:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:33:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:33:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:33:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:33:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:33:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:33:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:33:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:33:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:33:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:33:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:33:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:33:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:33:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:33:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:33:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:33:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:33:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:33:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:33:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:33:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:33:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:33:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:33:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:33:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:33:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:33:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:33:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:33:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:33:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:33:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:33:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:33:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:33:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:33:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:33:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:33:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:33:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:33:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:33:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:33:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:33:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:33:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:33:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:33:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:33:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:33:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:33:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:33:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:33:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:33:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:33:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:33:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:33:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:33:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:33:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:33:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:33:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:33:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:33:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:33:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:33:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:33:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:33:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:33:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:33:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:33:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:33:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:33:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:33:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:33:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:33:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:33:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:33:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:33:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:33:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:33:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:33:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:33:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:33:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:33:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:33:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:33:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:33:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:33:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:33:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:33:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:33:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:33:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:33:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:33:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:33:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:33:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:33:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:33:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:33:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:33:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:33:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:33:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:33:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:33:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:33:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:33:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:33:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:33:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:33:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:33:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:33:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:33:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:33:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:33:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:33:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:33:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:33:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:33:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:33:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:33:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:33:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:33:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:33:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:33:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:33:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:33:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:33:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:33:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:33:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:33:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:33:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:33:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:33:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:33:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:33:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:33:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:33:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:33:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:33:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:33:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:33:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:33:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:33:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:33:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:33:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:36:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:36:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:36:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:36:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:36:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:36:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:36:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:36:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:36:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:36:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:36:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:36:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:36:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:36:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:36:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:36:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:36:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:36:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:36:01 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 13:36:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:36:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:36:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:36:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:36:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:36:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:36:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:36:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:36:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:36:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:36:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:36:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:36:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:36:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:36:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:36:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:36:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:36:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:36:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:36:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:36:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:36:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:36:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:36:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:36:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:36:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:36:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:36:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:36:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:36:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:36:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:36:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:36:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:36:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:36:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:36:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:36:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:36:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:36:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:36:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:36:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:36:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:36:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:36:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:36:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:36:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:36:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:36:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:36:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:36:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:36:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:36:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:36:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:36:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:36:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:36:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:36:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:36:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:36:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:36:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:36:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:36:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:36:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:36:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:36:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:36:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:36:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:36:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:36:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:36:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:36:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:36:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:36:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:36:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:36:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:36:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:36:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:36:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:36:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:36:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:36:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:36:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:36:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:36:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:36:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:36:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:36:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:36:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:36:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:36:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:36:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:36:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:36:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:36:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:36:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:36:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:36:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:36:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:36:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:36:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:36:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:36:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:36:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:36:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:36:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:36:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:36:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:36:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:36:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:36:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:36:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:36:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:36:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:36:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:36:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:36:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:36:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:36:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:36:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:36:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:36:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:36:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:36:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:36:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:36:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:36:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:36:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:36:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:36:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:36:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:36:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:36:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:36:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:36:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:36:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:36:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:36:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:36:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:36:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:36:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:36:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:36:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:36:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:36:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:36:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:36:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:36:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:36:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:36:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:36:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:36:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:36:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:36:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:36:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:36:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:36:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:36:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:36:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:36:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:36:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:36:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:36:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:36:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:36:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:36:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:36:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:36:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:36:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:36:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:36:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:36:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:36:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:36:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:36:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:36:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:36:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:36:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:36:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:36:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:36:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:36:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:36:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:36:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:36:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:36:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:36:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:36:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:36:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:36:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:36:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:36:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:36:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:36:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:36:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:36:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:36:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:36:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:36:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:37:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:37:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:37:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:37:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:37:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:37:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:37:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:37:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:37:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:37:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:37:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:37:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:37:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:37:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:37:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:37:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:37:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:37:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:37:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:37:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:37:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:37:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:37:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:37:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:37:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:37:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:37:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:37:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:37:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:37:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:37:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:37:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:37:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:37:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:37:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:37:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:37:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:37:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:37:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:37:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:37:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:37:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:37:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:37:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:37:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:37:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:37:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:37:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:37:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:37:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:37:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:37:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:37:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:37:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:37:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:37:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:37:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:37:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:37:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:37:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:37:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:37:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:37:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:37:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:37:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:37:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:37:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:37:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:37:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:37:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:37:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:37:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:37:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:37:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:37:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:37:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:37:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:37:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:37:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:37:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:37:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:37:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:37:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:37:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:37:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:37:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:37:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:37:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:37:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:37:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:37:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:37:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:37:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:37:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:37:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:37:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:37:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:37:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:37:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:37:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:37:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:37:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:37:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:37:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:37:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:37:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:37:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:37:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:37:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:37:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:37:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:37:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:37:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:37:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:37:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:37:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:37:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:37:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:37:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:37:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:37:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:37:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:37:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:37:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:37:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:37:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:37:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:37:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:37:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:37:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:37:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:37:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:37:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:37:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:37:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:37:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:37:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:37:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:37:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:37:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:37:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:37:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:37:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:37:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:37:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:37:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:37:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:37:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:37:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:37:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:37:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:37:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:37:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:37:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:37:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:37:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:37:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:37:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:37:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:37:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:37:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:37:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:37:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:37:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:37:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:37:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:37:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:37:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:37:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:37:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:37:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:37:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:37:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:37:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:37:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:37:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:37:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:37:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:37:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:37:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:37:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:37:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:37:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:37:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:37:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:37:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:37:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:37:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:37:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:37:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:37:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:37:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:37:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:37:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:37:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:37:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:37:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:37:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:37:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:37:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:37:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:37:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:37:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:37:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:37:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:37:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:37:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:37:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:37:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:37:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:37:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:37:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:37:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:37:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:37:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:37:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:38:00 rdma06 systemd[1]: Starting dnf makecache...
Apr 18 13:38:01 rdma06 dnf[1192298]: Updating Subscription Management repositories.
Apr 18 13:38:01 rdma06 dnf[1192298]: Unable to read consumer identity
Apr 18 13:38:01 rdma06 dnf[1192298]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 18 13:38:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:38:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:38:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:38:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:38:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:38:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:38:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:38:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:38:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:38:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:38:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:38:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:38:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:38:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:38:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:38:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:38:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:38:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:38:02 rdma06 dnf[1192298]: Copr repo for fmf owned by psss                 2.4 kB/s | 3.3 kB     00:01
Apr 18 13:38:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:38:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:38:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:38:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:38:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:38:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:38:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:38:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:38:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:38:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:38:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:38:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:38:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:38:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:38:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:38:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:38:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:38:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:38:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:38:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:38:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:38:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:38:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:38:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:38:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:38:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:38:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:38:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:38:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:38:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:38:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:38:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:38:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:38:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:38:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:38:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:38:15 rdma06 dnf[1192298]: beaker-AppStream                                 14  B/s | 168  B     00:12
Apr 18 13:38:15 rdma06 dnf[1192298]: Errors during downloading metadata for repository 'beaker-AppStream':
Apr 18 13:38:15 rdma06 dnf[1192298]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 13:38:15 rdma06 dnf[1192298]: Error: Failed to download metadata for repo 'beaker-AppStream': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 13:38:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:38:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:38:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:38:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:38:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:38:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:38:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:38:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:38:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:38:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:38:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:38:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:38:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:38:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:38:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:38:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:38:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:38:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:38:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:38:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:38:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:38:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:38:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:38:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:38:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:38:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:38:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:38:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:38:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:38:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:38:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:38:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:38:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:38:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:38:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:38:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:38:23 rdma06 dnf[1192298]: beaker-AppStream-debuginfo                       20  B/s | 168  B     00:08
Apr 18 13:38:23 rdma06 dnf[1192298]: Errors during downloading metadata for repository 'beaker-AppStream-debuginfo':
Apr 18 13:38:23 rdma06 dnf[1192298]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 13:38:23 rdma06 dnf[1192298]: Error: Failed to download metadata for repo 'beaker-AppStream-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 13:38:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:38:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:38:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:38:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:38:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:38:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:38:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:38:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:38:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:38:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:38:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:38:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:38:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:38:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:38:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:38:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:38:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:38:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:38:29 rdma06 dnf[1192298]: beaker-BaseOS                                    27  B/s | 168  B     00:06
Apr 18 13:38:29 rdma06 dnf[1192298]: Errors during downloading metadata for repository 'beaker-BaseOS':
Apr 18 13:38:29 rdma06 dnf[1192298]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 13:38:29 rdma06 dnf[1192298]: Error: Failed to download metadata for repo 'beaker-BaseOS': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 13:38:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:38:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:38:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:38:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:38:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:38:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:38:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:38:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:38:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:38:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:38:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:38:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:38:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:38:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:38:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:38:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:38:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:38:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:38:35 rdma06 dnf[1192298]: beaker-BaseOS-debuginfo                          27  B/s | 168  B     00:06
Apr 18 13:38:35 rdma06 dnf[1192298]: Errors during downloading metadata for repository 'beaker-BaseOS-debuginfo':
Apr 18 13:38:35 rdma06 dnf[1192298]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 13:38:35 rdma06 dnf[1192298]: Error: Failed to download metadata for repo 'beaker-BaseOS-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 13:38:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:38:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:38:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:38:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:38:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:38:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:38:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:38:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:38:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:38:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:38:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:38:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:38:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:38:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:38:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:38:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:38:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:38:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:38:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:38:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:38:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:38:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:38:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:38:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:38:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:38:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:38:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:38:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:38:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:38:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:38:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:38:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:38:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:38:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:38:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:38:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:38:42 rdma06 dnf[1192298]: beaker-CRB                                       27  B/s | 168  B     00:06
Apr 18 13:38:42 rdma06 dnf[1192298]: Errors during downloading metadata for repository 'beaker-CRB':
Apr 18 13:38:42 rdma06 dnf[1192298]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 13:38:42 rdma06 dnf[1192298]: Error: Failed to download metadata for repo 'beaker-CRB': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 13:38:46 rdma06 dnf[1192298]: beaker-CRB-debuginfo                             39  B/s | 168  B     00:04
Apr 18 13:38:46 rdma06 dnf[1192298]: Errors during downloading metadata for repository 'beaker-CRB-debuginfo':
Apr 18 13:38:46 rdma06 dnf[1192298]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 13:38:46 rdma06 dnf[1192298]: Error: Failed to download metadata for repo 'beaker-CRB-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 13:38:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:38:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:38:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:38:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:38:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:38:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:38:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:38:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:38:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:38:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:38:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:38:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:38:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:38:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:38:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:38:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:38:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:38:46 rdma06 systemd-journald[790]: Data hash table of /run/log/journal/95379d2f723847b68d9e976dee8967a9/system.journal has a fill level at 75.0 (163751 of 218332 items, 75497472 file size, 461 bytes per hash table item), suggesting rotation.
Apr 18 13:38:46 rdma06 systemd-journald[790]: /run/log/journal/95379d2f723847b68d9e976dee8967a9/system.journal: Journal header limits reached or header out-of-date, rotating.
Apr 18 13:38:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:38:46 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 18 13:38:46 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 18 13:38:46 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 18 13:38:51 rdma06 dnf[1192298]: beaker-HighAvailability                          36  B/s | 168  B     00:04
Apr 18 13:38:51 rdma06 dnf[1192298]: Errors during downloading metadata for repository 'beaker-HighAvailability':
Apr 18 13:38:51 rdma06 dnf[1192298]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 13:38:51 rdma06 dnf[1192298]: Error: Failed to download metadata for repo 'beaker-HighAvailability': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 13:38:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:38:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:38:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:38:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:38:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:38:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:38:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:38:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:38:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:38:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:38:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:38:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:38:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:38:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:38:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:38:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:38:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:38:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:38:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:38:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:38:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:38:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:38:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:38:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:38:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:38:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:38:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:38:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:38:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:38:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:38:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:38:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:38:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:38:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:38:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:38:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:38:57 rdma06 dnf[1192298]: beaker-HighAvailability-debuginfo                27  B/s | 168  B     00:06
Apr 18 13:38:57 rdma06 dnf[1192298]: Errors during downloading metadata for repository 'beaker-HighAvailability-debuginfo':
Apr 18 13:38:57 rdma06 dnf[1192298]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 13:38:57 rdma06 dnf[1192298]: Error: Failed to download metadata for repo 'beaker-HighAvailability-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 13:39:01 rdma06 dnf[1192298]: beaker-NFV                                       40  B/s | 168  B     00:04
Apr 18 13:39:01 rdma06 dnf[1192298]: Errors during downloading metadata for repository 'beaker-NFV':
Apr 18 13:39:01 rdma06 dnf[1192298]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 13:39:01 rdma06 dnf[1192298]: Error: Failed to download metadata for repo 'beaker-NFV': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 13:39:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:39:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:39:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:39:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:39:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:39:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:39:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:39:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:39:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:39:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:39:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:39:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:39:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:39:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:39:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:39:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:39:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:39:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:39:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:39:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:39:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:39:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:39:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:39:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:39:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:39:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:39:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:39:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:39:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:39:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:39:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:39:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:39:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:39:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:39:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:39:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:39:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:39:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:39:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:39:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:39:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:39:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:39:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:39:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:39:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:39:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:39:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:39:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:39:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:39:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:39:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:39:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:39:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:39:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:39:16 rdma06 dnf[1192298]: beaker-NFV-debuginfo                             12  B/s | 168  B     00:14
Apr 18 13:39:16 rdma06 dnf[1192298]: Errors during downloading metadata for repository 'beaker-NFV-debuginfo':
Apr 18 13:39:16 rdma06 dnf[1192298]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 13:39:16 rdma06 dnf[1192298]: Error: Failed to download metadata for repo 'beaker-NFV-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 13:39:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:39:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:39:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:39:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:39:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:39:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:39:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:39:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:39:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:39:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:39:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:39:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:39:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:39:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:39:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:39:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:39:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:39:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:39:19 rdma06 dnf[1192298]: beaker-RT                                        49  B/s | 168  B     00:03
Apr 18 13:39:19 rdma06 dnf[1192298]: Errors during downloading metadata for repository 'beaker-RT':
Apr 18 13:39:19 rdma06 dnf[1192298]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 13:39:19 rdma06 dnf[1192298]: Error: Failed to download metadata for repo 'beaker-RT': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 13:39:21 rdma06 dnf[1192298]: beaker-RT-debuginfo                             105  B/s | 168  B     00:01
Apr 18 13:39:21 rdma06 dnf[1192298]: Errors during downloading metadata for repository 'beaker-RT-debuginfo':
Apr 18 13:39:21 rdma06 dnf[1192298]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 13:39:21 rdma06 dnf[1192298]: Error: Failed to download metadata for repo 'beaker-RT-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 13:39:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:39:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:39:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:39:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:39:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:39:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:39:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:39:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:39:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:39:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:39:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:39:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:39:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:39:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:39:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:39:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:39:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:39:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:39:24 rdma06 dnf[1192298]: beaker-ResilientStorage                          52  B/s | 168  B     00:03
Apr 18 13:39:24 rdma06 dnf[1192298]: Errors during downloading metadata for repository 'beaker-ResilientStorage':
Apr 18 13:39:24 rdma06 dnf[1192298]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 13:39:24 rdma06 dnf[1192298]: Error: Failed to download metadata for repo 'beaker-ResilientStorage': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 13:39:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:39:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:39:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:39:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:39:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:39:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:39:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:39:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:39:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:39:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:39:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:39:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:39:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:39:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:39:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:39:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:39:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:39:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:39:27 rdma06 dnf[1192298]: beaker-ResilientStorage-debuginfo                52  B/s | 168  B     00:03
Apr 18 13:39:27 rdma06 dnf[1192298]: Errors during downloading metadata for repository 'beaker-ResilientStorage-debuginfo':
Apr 18 13:39:27 rdma06 dnf[1192298]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 13:39:27 rdma06 dnf[1192298]: Error: Failed to download metadata for repo 'beaker-ResilientStorage-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 13:39:29 rdma06 dnf[1192298]: beaker-SAP                                      104  B/s | 168  B     00:01
Apr 18 13:39:29 rdma06 dnf[1192298]: Errors during downloading metadata for repository 'beaker-SAP':
Apr 18 13:39:29 rdma06 dnf[1192298]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 13:39:29 rdma06 dnf[1192298]: Error: Failed to download metadata for repo 'beaker-SAP': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 13:39:30 rdma06 dnf[1192298]: beaker-SAP-debuginfo                            140  B/s | 168  B     00:01
Apr 18 13:39:30 rdma06 dnf[1192298]: Errors during downloading metadata for repository 'beaker-SAP-debuginfo':
Apr 18 13:39:30 rdma06 dnf[1192298]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 13:39:30 rdma06 dnf[1192298]: Error: Failed to download metadata for repo 'beaker-SAP-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 13:39:31 rdma06 dnf[1192298]: beaker-SAPHANA                                  139  B/s | 168  B     00:01
Apr 18 13:39:31 rdma06 dnf[1192298]: Errors during downloading metadata for repository 'beaker-SAPHANA':
Apr 18 13:39:31 rdma06 dnf[1192298]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 13:39:31 rdma06 dnf[1192298]: Error: Failed to download metadata for repo 'beaker-SAPHANA': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 13:39:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:39:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:39:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:39:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:39:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:39:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:39:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:39:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:39:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:39:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:39:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:39:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:39:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:39:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:39:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:39:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:39:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:39:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:39:32 rdma06 dnf[1192298]: beaker-SAPHANA-debuginfo                        140  B/s | 168  B     00:01
Apr 18 13:39:32 rdma06 dnf[1192298]: Errors during downloading metadata for repository 'beaker-SAPHANA-debuginfo':
Apr 18 13:39:32 rdma06 dnf[1192298]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 13:39:32 rdma06 dnf[1192298]: Error: Failed to download metadata for repo 'beaker-SAPHANA-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 13:39:33 rdma06 dnf[1192298]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr 18 13:39:33 rdma06 dnf[1192298]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr 18 13:39:34 rdma06 dnf[1192298]: Ignoring repositories: beaker-AppStream, beaker-AppStream-debuginfo, beaker-BaseOS, beaker-BaseOS-debuginfo, beaker-CRB, beaker-CRB-debuginfo, beaker-HighAvailability, beaker-HighAvailability-debuginfo, beaker-NFV, beaker-NFV-debuginfo, beaker-RT, beaker-RT-debuginfo, beaker-ResilientStorage, beaker-ResilientStorage-debuginfo, beaker-SAP, beaker-SAP-debuginfo, beaker-SAPHANA, beaker-SAPHANA-debuginfo
Apr 18 13:39:34 rdma06 dnf[1192298]: Metadata cache created.
Apr 18 13:39:34 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 18 13:39:34 rdma06 systemd[1]: Finished dnf makecache.
Apr 18 13:39:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:39:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:39:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:39:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:39:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:39:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:39:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:39:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:39:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:39:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:39:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:39:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:39:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:39:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:39:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:39:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:39:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:39:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:39:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:39:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:39:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:39:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:39:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:39:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:39:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:39:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:39:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:39:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:39:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:39:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:39:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:39:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:39:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:39:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:39:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:39:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:39:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:39:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:39:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:39:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:39:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:39:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:39:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:39:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:39:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:39:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:39:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:39:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:39:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:39:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:39:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:39:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:39:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:39:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:39:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:39:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:39:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:39:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:39:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:39:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:39:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:39:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:39:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:39:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:39:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:39:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:39:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:39:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:39:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:39:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:39:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:39:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:39:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:39:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:39:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:39:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:39:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:39:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:39:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:39:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:39:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:39:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:39:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:39:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:39:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:39:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:39:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:39:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:39:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:39:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:40:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:40:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:40:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:40:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:40:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:40:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:40:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:40:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:40:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:40:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:40:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:40:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:40:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:40:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:40:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:40:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:40:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:40:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:40:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:40:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:40:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:40:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:40:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:40:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:40:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:40:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:40:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:40:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:40:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:40:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:40:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:40:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:40:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:40:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:40:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:40:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:40:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:40:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:40:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:40:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:40:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:40:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:40:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:40:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:40:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:40:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:40:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:40:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:40:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:40:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:40:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:40:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:40:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:40:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:40:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:40:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:40:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:40:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:40:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:40:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:40:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:40:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:40:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:40:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:40:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:40:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:40:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:40:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:40:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:40:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:40:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:40:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:40:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:40:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:40:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:40:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:40:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:40:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:40:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:40:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:40:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:40:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:40:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:40:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:40:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:40:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:40:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:40:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:40:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:40:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:40:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:40:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:40:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:40:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:40:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:40:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:40:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:40:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:40:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:40:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:40:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:40:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:40:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:40:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:40:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:40:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:40:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:40:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:40:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:40:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:40:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:40:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:40:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:40:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:40:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:40:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:40:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:40:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:40:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:40:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:40:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:40:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:40:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:40:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:40:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:40:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:40:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:40:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:40:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:40:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:40:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:40:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:40:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:40:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:40:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:40:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:40:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:40:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:40:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:40:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:40:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:40:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:40:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:40:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:40:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:40:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:40:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:40:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:40:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:40:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:40:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:40:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:40:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:40:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:40:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:40:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:40:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:40:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:40:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:40:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:40:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:40:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:40:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:40:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:40:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:40:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:40:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:40:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:40:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:40:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:40:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:40:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:40:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:40:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:40:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:40:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:40:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:40:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:40:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:40:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:40:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:40:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:40:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:40:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:40:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:40:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:40:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:40:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:40:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:40:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:40:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:40:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:40:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:40:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:40:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:40:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:40:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:40:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:40:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:40:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:40:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:40:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:40:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:40:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:40:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:40:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:40:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:40:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:40:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:40:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:40:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:40:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:40:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:40:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:40:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:40:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:41:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:41:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:41:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:41:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:41:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:41:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:41:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:41:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:41:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:41:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:41:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:41:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:41:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:41:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:41:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:41:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:41:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:41:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:41:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:41:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:41:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:41:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:41:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:41:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:41:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:41:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:41:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:41:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:41:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:41:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:41:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:41:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:41:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:41:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:41:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:41:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:41:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:41:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:41:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:41:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:41:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:41:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:41:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:41:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:41:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:41:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:41:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:41:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:41:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:41:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:41:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:41:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:41:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:41:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:41:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:41:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:41:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:41:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:41:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:41:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:41:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:41:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:41:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:41:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:41:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:41:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:41:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:41:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:41:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:41:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:41:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:41:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:41:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:41:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:41:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:41:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:41:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:41:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:41:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:41:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:41:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:41:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:41:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:41:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:41:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:41:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:41:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:41:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:41:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:41:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:41:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:41:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:41:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:41:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:41:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:41:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:41:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:41:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:41:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:41:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:41:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:41:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:41:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:41:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:41:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:41:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:41:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:41:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:41:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:41:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:41:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:41:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:41:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:41:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:41:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:41:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:41:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:41:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:41:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:41:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:41:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:41:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:41:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:41:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:41:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:41:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:41:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:41:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:41:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:41:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:41:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:41:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:41:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:41:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:41:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:41:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:41:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:41:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:41:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:41:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:41:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:41:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:41:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:41:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:41:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:41:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:41:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:41:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:41:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:41:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:41:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:41:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:41:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:41:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:41:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:41:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:41:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:41:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:41:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:41:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:41:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:41:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:41:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:41:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:41:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:41:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:41:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:41:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:41:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:41:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:41:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:41:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:41:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:41:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:41:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:41:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:41:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:41:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:41:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:41:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:41:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:41:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:41:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:41:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:41:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:41:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:41:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:41:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:41:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:41:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:41:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:41:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:41:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:41:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:41:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:41:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:41:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:41:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:41:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:41:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:41:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:41:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:41:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:41:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:41:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:41:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:41:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:41:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:41:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:41:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:41:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:41:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:41:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:41:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:41:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:41:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:42:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:42:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:42:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:42:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:42:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:42:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:42:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:42:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:42:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:42:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:42:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:42:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:42:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:42:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:42:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:42:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:42:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:42:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:42:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:42:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:42:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:42:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:42:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:42:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:42:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:42:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:42:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:42:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:42:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:42:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:42:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:42:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:42:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:42:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:42:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:42:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:42:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:42:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:42:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:42:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:42:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:42:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:42:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:42:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:42:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:42:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:42:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:42:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:42:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:42:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:42:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:42:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:42:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:42:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:42:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:42:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:42:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:42:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:42:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:42:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:42:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:42:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:42:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:42:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:42:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:42:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:42:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:42:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:42:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:42:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:42:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:42:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:42:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:42:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:42:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:42:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:42:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:42:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:42:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:42:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:42:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:42:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:42:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:42:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:42:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:42:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:42:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:42:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:42:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:42:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:42:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:42:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:42:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:42:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:42:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:42:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:42:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:42:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:42:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:42:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:42:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:42:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:42:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:42:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:42:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:42:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:42:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:42:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:42:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:42:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:42:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:42:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:42:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:42:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:42:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:42:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:42:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:42:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:42:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:42:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:42:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:42:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:42:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:42:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:42:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:42:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:42:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:42:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:42:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:42:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:42:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:42:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:42:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:42:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:42:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:42:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:42:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:42:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:42:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:42:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:42:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:42:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:42:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:42:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:42:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:42:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:42:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:42:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:42:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:42:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:42:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:42:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:42:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:42:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:42:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:42:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:42:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:42:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:42:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:42:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:42:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:42:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:42:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:42:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:42:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:42:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:42:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:42:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:42:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:42:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:42:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:42:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:42:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:42:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:42:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:42:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:42:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:42:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:42:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:42:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:42:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:42:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:42:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:42:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:42:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:42:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:42:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:42:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:42:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:42:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:42:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:42:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:42:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:42:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:42:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:42:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:42:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:42:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:42:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:42:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:42:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:42:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:42:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:42:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:42:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:42:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:42:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:42:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:42:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:42:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:42:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:42:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:42:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:42:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:42:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:42:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:43:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:43:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:43:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:43:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:43:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:43:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:43:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:43:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:43:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:43:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:43:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:43:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:43:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:43:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:43:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:43:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:43:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:43:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:43:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:43:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:43:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:43:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:43:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:43:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:43:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:43:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:43:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:43:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:43:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:43:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:43:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:43:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:43:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:43:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:43:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:43:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:43:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:43:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:43:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:43:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:43:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:43:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:43:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:43:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:43:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:43:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:43:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:43:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:43:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:43:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:43:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:43:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:43:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:43:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:43:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:43:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:43:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:43:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:43:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:43:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:43:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:43:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:43:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:43:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:43:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:43:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:43:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:43:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:43:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:43:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:43:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:43:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:43:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:43:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:43:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:43:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:43:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:43:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:43:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:43:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:43:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:43:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:43:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:43:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:43:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:43:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:43:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:43:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:43:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:43:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:43:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:43:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:43:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:43:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:43:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:43:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:43:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:43:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:43:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:43:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:43:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:43:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:43:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:43:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:43:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:43:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:43:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:43:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:43:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:43:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:43:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:43:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:43:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:43:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:43:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:43:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:43:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:43:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:43:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:43:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:43:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:43:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:43:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:43:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:43:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:43:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:43:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:43:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:43:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:43:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:43:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:43:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:43:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:43:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:43:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:43:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:43:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:43:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:43:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:43:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:43:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:43:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:43:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:43:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:43:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:43:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:43:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:43:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:43:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:43:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:43:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:43:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:43:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:43:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:43:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:43:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:43:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:43:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:43:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:43:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:43:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:43:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:43:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:43:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:43:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:43:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:43:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:43:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:43:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:43:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:43:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:43:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:43:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:43:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:43:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:43:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:43:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:43:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:43:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:43:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:43:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:43:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:43:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:43:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:43:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:43:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:43:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:43:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:43:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:43:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:43:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:43:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:43:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:43:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:43:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:43:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:43:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:43:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:43:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:43:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:43:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:43:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:43:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:43:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:43:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:43:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:43:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:43:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:43:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:43:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:43:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:43:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:43:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:46:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:46:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:46:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:46:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:46:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:46:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:46:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:46:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:46:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:46:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:46:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:46:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:46:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:46:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:46:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:46:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:46:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:46:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:46:02 rdma06 rsyslogd[1041]: imjournal: 5170 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 13:46:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:46:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:46:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:46:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:46:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:46:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:46:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:46:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:46:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:46:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:46:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:46:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:46:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:46:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:46:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:46:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:46:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:46:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:46:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:46:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:46:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:46:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:46:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:46:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:46:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:46:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:46:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:46:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:46:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:46:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:46:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:46:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:46:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:46:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:46:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:46:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:46:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:46:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:46:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:46:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:46:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:46:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:46:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:46:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:46:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:46:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:46:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:46:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:46:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:46:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:46:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:46:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:46:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:46:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:46:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:46:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:46:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:46:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:46:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:46:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:46:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:46:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:46:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:46:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:46:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:46:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:46:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:46:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:46:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:46:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:46:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:46:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:46:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:46:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:46:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:46:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:46:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:46:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:46:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:46:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:46:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:46:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:46:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:46:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:46:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:46:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:46:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:46:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:46:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:46:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:46:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:46:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:46:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:46:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:46:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:46:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:46:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:46:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:46:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:46:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:46:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:46:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:46:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:46:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:46:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:46:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:46:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:46:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:46:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:46:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:46:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:46:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:46:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:46:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:46:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:46:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:46:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:46:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:46:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:46:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:46:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:46:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:46:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:46:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:46:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:46:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:46:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:46:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:46:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:46:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:46:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:46:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:46:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:46:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:46:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:46:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:46:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:46:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:46:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:46:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:46:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:46:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:46:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:46:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:46:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:46:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:46:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:46:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:46:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:46:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:46:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:46:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:46:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:46:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:46:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:46:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:46:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:46:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:46:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:46:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:46:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:46:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:46:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:46:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:46:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:46:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:46:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:46:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:46:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:46:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:46:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:46:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:46:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:46:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:46:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:46:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:46:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:46:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:46:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:46:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:46:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:46:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:46:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:46:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:46:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:46:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:46:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:46:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:46:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:46:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:46:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:46:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:46:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:46:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:46:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:46:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:46:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:46:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:47:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:47:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:47:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:47:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:47:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:47:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:47:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:47:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:47:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:47:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:47:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:47:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:47:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:47:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:47:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:47:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:47:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:47:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:47:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:47:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:47:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:47:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:47:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:47:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:47:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:47:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:47:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:47:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:47:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:47:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:47:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:47:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:47:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:47:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:47:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:47:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:47:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:47:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:47:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:47:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:47:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:47:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:47:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:47:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:47:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:47:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:47:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:47:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:47:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:47:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:47:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:47:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:47:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:47:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:47:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:47:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:47:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:47:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:47:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:47:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:47:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:47:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:47:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:47:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:47:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:47:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:47:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:47:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:47:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:47:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:47:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:47:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:47:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:47:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:47:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:47:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:47:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:47:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:47:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:47:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:47:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:47:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:47:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:47:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:47:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:47:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:47:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:47:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:47:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:47:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:47:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:47:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:47:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:47:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:47:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:47:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:47:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:47:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:47:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:47:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:47:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:47:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:47:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:47:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:47:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:47:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:47:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:47:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:47:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:47:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:47:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:47:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:47:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:47:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:47:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:47:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:47:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:47:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:47:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:47:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:47:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:47:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:47:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:47:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:47:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:47:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:47:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:47:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:47:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:47:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:47:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:47:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:47:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:47:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:47:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:47:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:47:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:47:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:47:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:47:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:47:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:47:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:47:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:47:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:47:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:47:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:47:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:47:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:47:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:47:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:47:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:47:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:47:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:47:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:47:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:47:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:47:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:47:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:47:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:47:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:47:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:47:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:47:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:47:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:47:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:47:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:47:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:47:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:47:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:47:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:47:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:47:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:47:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:47:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:47:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:47:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:47:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:47:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:47:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:47:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:47:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:47:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:47:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:47:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:47:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:47:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:47:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:47:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:47:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:47:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:47:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:47:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:47:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:47:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:47:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:47:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:47:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:47:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:47:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:47:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:47:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:47:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:47:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:47:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:47:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:47:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:47:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:47:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:47:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:47:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:47:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:47:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:47:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:47:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:47:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:47:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:48:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:48:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:48:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:48:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:48:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:48:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:48:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:48:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:48:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:48:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:48:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:48:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:48:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:48:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:48:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:48:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:48:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:48:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:48:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:48:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:48:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:48:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:48:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:48:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:48:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:48:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:48:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:48:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:48:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:48:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:48:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:48:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:48:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:48:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:48:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:48:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:48:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:48:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:48:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:48:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:48:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:48:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:48:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:48:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:48:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:48:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:48:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:48:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:48:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:48:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:48:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:48:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:48:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:48:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:48:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:48:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:48:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:48:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:48:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:48:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:48:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:48:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:48:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:48:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:48:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:48:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:48:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:48:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:48:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:48:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:48:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:48:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:48:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:48:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:48:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:48:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:48:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:48:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:48:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:48:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:48:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:48:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:48:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:48:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:48:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:48:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:48:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:48:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:48:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:48:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:48:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:48:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:48:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:48:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:48:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:48:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:48:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:48:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:48:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:48:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:48:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:48:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:48:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:48:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:48:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:48:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:48:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:48:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:48:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:48:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:48:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:48:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:48:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:48:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:48:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:48:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:48:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:48:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:48:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:48:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:48:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:48:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:48:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:48:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:48:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:48:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:48:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:48:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:48:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:48:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:48:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:48:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:48:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:48:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:48:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:48:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:48:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:48:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:48:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:48:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:48:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:48:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:48:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:48:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:48:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:48:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:48:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:48:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:48:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:48:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:48:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:48:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:48:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:48:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:48:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:48:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:48:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:48:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:48:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:48:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:48:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:48:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:48:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:48:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:48:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:48:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:48:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:48:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:48:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:48:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:48:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:48:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:48:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:48:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:48:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:48:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:48:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:48:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:48:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:48:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:48:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:48:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:48:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:48:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:48:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:48:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:48:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:48:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:48:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:48:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:48:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:48:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:48:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:48:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:48:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:48:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:48:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:48:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:48:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:48:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:48:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:48:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:48:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:48:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:48:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:48:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:48:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:48:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:48:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:48:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:48:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:48:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:48:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:48:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:48:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:48:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:49:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:49:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:49:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:49:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:49:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:49:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:49:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:49:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:49:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:49:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:49:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:49:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:49:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:49:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:49:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:49:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:49:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:49:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:49:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:49:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:49:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:49:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:49:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:49:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:49:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:49:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:49:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:49:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:49:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:49:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:49:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:49:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:49:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:49:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:49:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:49:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:49:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:49:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:49:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:49:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:49:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:49:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:49:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:49:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:49:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:49:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:49:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:49:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:49:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:49:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:49:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:49:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:49:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:49:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:49:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:49:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:49:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:49:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:49:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:49:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:49:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:49:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:49:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:49:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:49:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:49:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:49:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:49:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:49:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:49:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:49:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:49:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:49:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:49:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:49:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:49:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:49:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:49:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:49:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:49:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:49:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:49:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:49:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:49:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:49:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:49:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:49:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:49:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:49:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:49:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:49:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:49:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:49:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:49:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:49:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:49:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:49:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:49:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:49:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:49:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:49:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:49:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:49:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:49:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:49:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:49:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:49:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:49:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:49:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:49:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:49:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:49:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:49:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:49:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:49:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:49:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:49:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:49:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:49:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:49:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:49:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:49:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:49:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:49:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:49:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:49:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:49:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:49:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:49:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:49:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:49:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:49:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:49:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:49:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:49:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:49:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:49:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:49:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:49:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:49:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:49:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:49:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:49:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:49:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:49:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:49:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:49:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:49:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:49:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:49:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:49:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:49:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:49:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:49:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:49:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:49:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:49:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:49:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:49:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:49:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:49:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:49:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:49:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:49:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:49:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:49:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:49:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:49:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:49:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:49:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:49:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:49:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:49:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:49:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:49:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:49:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:49:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:49:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:49:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:49:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:49:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:49:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:49:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:49:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:49:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:49:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:49:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:49:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:49:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:49:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:49:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:49:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:49:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:49:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:49:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:49:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:49:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:49:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:49:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:49:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:49:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:49:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:49:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:49:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:49:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:49:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:49:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:49:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:49:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:49:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:49:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:49:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:49:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:49:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:49:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:49:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:50:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:50:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:50:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:50:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:50:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:50:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:50:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:50:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:50:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:50:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:50:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:50:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:50:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:50:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:50:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:50:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:50:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:50:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:50:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:50:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:50:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:50:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:50:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:50:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:50:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:50:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:50:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:50:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:50:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:50:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:50:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:50:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:50:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:50:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:50:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:50:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:50:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:50:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:50:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:50:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:50:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:50:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:50:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:50:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:50:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:50:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:50:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:50:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:50:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:50:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:50:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:50:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:50:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:50:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:50:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:50:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:50:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:50:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:50:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:50:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:50:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:50:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:50:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:50:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:50:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:50:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:50:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:50:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:50:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:50:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:50:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:50:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:50:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:50:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:50:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:50:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:50:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:50:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:50:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:50:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:50:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:50:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:50:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:50:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:50:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:50:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:50:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:50:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:50:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:50:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:50:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:50:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:50:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:50:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:50:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:50:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:50:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:50:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:50:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:50:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:50:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:50:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:50:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:50:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:50:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:50:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:50:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:50:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:50:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:50:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:50:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:50:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:50:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:50:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:50:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:50:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:50:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:50:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:50:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:50:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:50:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:50:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:50:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:50:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:50:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:50:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:50:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:50:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:50:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:50:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:50:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:50:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:50:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:50:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:50:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:50:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:50:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:50:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:50:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:50:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:50:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:50:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:50:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:50:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:50:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:50:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:50:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:50:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:50:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:50:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:50:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:50:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:50:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:50:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:50:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:50:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:50:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:50:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:50:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:50:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:50:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:50:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:50:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:50:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:50:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:50:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:50:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:50:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:50:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:50:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:50:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:50:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:50:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:50:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:50:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:50:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:50:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:50:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:50:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:50:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:50:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:50:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:50:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:50:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:50:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:50:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:50:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:50:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:50:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:50:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:50:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:50:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:50:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:50:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:50:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:50:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:50:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:50:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:50:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:50:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:50:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:50:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:50:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:50:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:50:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:50:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:50:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:50:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:50:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:50:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:50:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:50:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:50:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:50:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:50:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:50:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:51:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:51:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:51:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:51:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:51:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:51:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:51:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:51:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:51:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:51:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:51:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:51:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:51:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:51:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:51:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:51:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:51:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:51:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:51:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:51:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:51:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:51:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:51:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:51:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:51:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:51:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:51:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:51:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:51:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:51:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:51:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:51:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:51:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:51:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:51:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:51:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:51:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:51:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:51:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:51:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:51:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:51:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:51:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:51:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:51:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:51:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:51:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:51:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:51:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:51:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:51:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:51:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:51:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:51:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:51:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:51:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:51:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:51:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:51:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:51:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:51:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:51:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:51:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:51:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:51:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:51:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:51:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:51:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:51:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:51:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:51:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:51:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:51:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:51:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:51:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:51:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:51:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:51:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:51:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:51:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:51:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:51:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:51:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:51:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:51:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:51:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:51:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:51:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:51:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:51:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:51:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:51:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:51:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:51:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:51:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:51:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:51:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:51:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:51:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:51:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:51:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:51:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:51:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:51:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:51:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:51:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:51:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:51:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:51:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:51:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:51:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:51:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:51:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:51:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:51:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:51:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:51:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:51:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:51:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:51:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:51:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:51:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:51:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:51:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:51:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:51:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:51:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:51:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:51:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:51:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:51:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:51:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:51:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:51:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:51:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:51:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:51:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:51:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:51:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:51:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:51:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:51:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:51:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:51:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:51:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:51:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:51:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:51:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:51:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:51:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:51:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:51:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:51:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:51:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:51:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:51:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:51:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:51:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:51:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:51:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:51:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:51:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:51:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:51:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:51:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:51:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:51:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:51:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:51:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:51:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:51:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:51:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:51:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:51:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:51:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:51:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:51:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:51:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:51:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:51:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:51:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:51:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:51:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:51:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:51:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:51:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:51:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:51:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:51:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:51:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:51:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:51:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:51:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:51:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:51:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:51:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:51:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:51:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:51:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:51:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:51:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:51:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:51:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:51:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:51:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:51:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:51:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:51:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:51:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:51:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:51:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:51:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:51:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:51:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:51:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:51:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:52:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:52:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:52:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:52:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:52:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:52:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:52:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:52:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:52:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:52:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:52:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:52:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:52:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:52:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:52:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:52:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:52:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:52:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:52:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:52:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:52:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:52:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:52:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:52:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:52:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:52:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:52:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:52:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:52:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:52:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:52:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:52:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:52:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:52:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:52:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:52:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:52:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:52:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:52:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:52:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:52:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:52:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:52:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:52:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:52:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:52:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:52:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:52:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:52:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:52:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:52:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:52:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:52:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:52:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:52:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:52:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:52:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:52:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:52:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:52:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:52:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:52:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:52:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:52:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:52:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:52:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:52:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:52:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:52:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:52:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:52:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:52:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:52:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:52:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:52:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:52:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:52:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:52:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:52:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:52:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:52:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:52:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:52:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:52:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:52:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:52:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:52:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:52:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:52:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:52:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:52:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:52:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:52:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:52:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:52:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:52:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:52:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:52:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:52:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:52:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:52:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:52:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:52:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:52:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:52:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:52:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:52:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:52:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:52:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:52:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:52:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:52:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:52:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:52:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:52:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:52:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:52:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:52:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:52:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:52:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:52:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:52:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:52:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:52:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:52:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:52:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:52:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:52:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:52:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:52:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:52:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:52:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:52:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:52:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:52:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:52:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:52:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:52:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:52:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:52:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:52:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:52:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:52:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:52:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:52:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:52:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:52:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:52:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:52:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:52:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:52:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:52:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:52:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:52:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:52:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:52:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:52:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:52:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:52:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:52:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:52:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:52:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:52:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:52:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:52:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:52:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:52:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:52:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:52:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:52:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:52:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:52:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:52:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:52:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:52:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:52:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:52:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:52:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:52:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:52:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:52:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:52:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:52:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:52:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:52:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:52:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:52:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:52:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:52:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:52:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:52:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:52:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:52:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:52:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:52:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:52:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:52:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:52:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:52:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:52:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:52:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:52:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:52:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:52:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:52:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:52:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:52:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:52:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:52:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:52:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:52:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:52:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:52:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:52:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:52:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:52:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:53:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:53:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:53:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:53:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:53:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:53:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:53:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:53:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:53:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:53:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:53:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:53:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:53:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:53:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:53:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:53:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:53:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:53:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:53:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:53:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:53:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:53:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:53:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:53:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:53:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:53:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:53:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:53:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:53:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:53:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:53:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:53:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:53:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:53:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:53:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:53:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:53:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:53:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:53:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:53:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:53:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:53:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:53:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:53:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:53:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:53:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:53:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:53:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:53:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:53:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:53:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:53:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:53:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:53:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:53:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:53:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:53:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:53:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:53:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:53:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:53:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:53:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:53:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:53:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:53:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:53:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:53:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:53:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:53:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:53:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:53:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:53:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:53:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:53:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:53:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:53:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:53:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:53:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:53:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:53:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:53:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:53:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:53:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:53:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:53:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:53:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:53:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:53:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:53:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:53:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:53:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:53:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:53:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:53:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:53:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:53:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:53:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:53:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:53:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:53:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:53:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:53:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:53:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:53:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:53:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:53:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:53:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:53:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:53:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:53:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:53:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:53:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:53:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:53:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:53:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:53:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:53:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:53:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:53:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:53:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:53:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:53:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:53:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:53:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:53:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:53:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:53:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:53:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:53:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:53:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:53:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:53:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:53:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:53:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:53:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:53:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:53:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:53:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:53:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:53:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:53:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:53:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:53:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:53:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:53:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:53:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:53:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:53:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:53:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:53:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:53:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:53:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:53:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:53:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:53:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:53:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:53:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:53:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:53:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:53:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:53:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:53:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:53:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:53:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:53:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:53:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:53:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:53:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:53:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:53:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:53:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:53:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:53:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:53:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:53:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:53:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:53:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:53:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:53:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:53:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:53:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:53:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:53:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:53:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:53:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:53:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:53:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:53:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:53:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:53:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:53:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:53:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:53:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:53:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:53:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:53:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:53:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:53:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:53:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:53:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:53:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:53:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:53:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:53:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:53:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:53:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:53:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:53:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:53:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:53:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:53:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:53:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:53:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:53:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:53:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:53:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:56:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:56:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:56:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:56:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:56:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:56:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:56:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:56:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:56:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:56:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:56:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:56:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:56:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:56:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:56:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:56:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:56:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:56:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:56:07 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 13:56:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:56:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:56:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:56:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:56:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:56:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:56:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:56:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:56:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:56:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:56:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:56:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:56:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:56:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:56:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:56:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:56:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:56:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:56:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:56:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:56:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:56:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:56:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:56:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:56:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:56:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:56:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:56:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:56:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:56:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:56:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:56:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:56:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:56:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:56:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:56:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:56:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:56:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:56:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:56:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:56:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:56:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:56:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:56:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:56:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:56:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:56:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:56:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:56:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:56:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:56:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:56:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:56:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:56:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:56:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:56:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:56:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:56:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:56:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:56:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:56:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:56:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:56:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:56:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:56:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:56:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:56:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:56:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:56:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:56:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:56:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:56:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:56:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:56:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:56:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:56:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:56:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:56:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:56:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:56:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:56:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:56:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:56:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:56:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:56:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:56:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:56:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:56:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:56:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:56:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:56:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:56:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:56:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:56:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:56:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:56:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:56:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:56:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:56:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:56:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:56:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:56:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:56:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:56:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:56:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:56:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:56:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:56:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:56:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:56:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:56:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:56:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:56:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:56:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:56:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:56:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:56:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:56:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:56:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:56:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:56:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:56:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:56:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:56:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:56:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:56:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:56:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:56:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:56:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:56:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:56:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:56:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:56:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:56:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:56:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:56:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:56:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:56:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:56:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:56:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:56:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:56:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:56:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:56:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:56:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:56:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:56:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:56:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:56:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:56:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:56:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:56:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:56:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:56:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:56:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:56:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:56:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:56:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:56:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:56:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:56:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:56:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:56:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:56:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:56:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:56:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:56:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:56:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:56:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:56:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:56:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:56:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:56:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:56:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:56:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:56:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:56:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:56:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:56:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:56:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:57:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:57:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:57:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:57:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:57:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:57:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:57:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:57:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:57:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:57:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:57:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:57:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:57:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:57:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:57:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:57:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:57:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:57:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:57:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:57:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:57:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:57:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:57:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:57:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:57:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:57:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:57:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:57:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:57:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:57:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:57:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:57:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:57:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:57:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:57:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:57:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:57:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:57:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:57:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:57:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:57:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:57:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:57:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:57:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:57:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:57:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:57:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:57:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:57:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:57:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:57:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:57:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:57:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:57:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:57:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:57:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:57:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:57:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:57:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:57:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:57:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:57:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:57:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:57:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:57:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:57:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:57:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:57:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:57:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:57:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:57:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:57:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:57:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:57:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:57:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:57:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:57:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:57:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:57:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:57:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:57:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:57:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:57:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:57:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:57:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:57:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:57:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:57:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:57:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:57:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:57:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:57:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:57:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:57:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:57:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:57:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:57:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:57:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:57:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:57:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:57:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:57:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:57:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:57:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:57:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:57:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:57:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:57:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:57:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:57:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:57:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:57:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:57:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:57:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:57:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:57:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:57:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:57:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:57:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:57:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:57:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:57:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:57:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:57:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:57:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:57:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:57:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:57:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:57:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:57:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:57:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:57:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:57:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:57:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:57:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:57:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:57:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:57:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:57:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:57:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:57:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:57:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:57:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:57:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:57:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:57:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:57:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:57:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:57:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:57:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:57:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:57:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:57:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:57:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:57:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:57:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:57:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:57:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:57:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:57:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:57:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:57:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:57:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:57:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:57:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:57:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:57:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:57:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:57:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:57:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:57:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:57:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:57:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:57:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:57:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:57:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:57:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:57:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:57:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:57:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:57:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:57:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:57:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:57:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:57:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:57:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:57:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:57:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:57:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:57:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:57:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:57:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:57:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:57:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:57:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:57:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:57:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:57:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:57:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:57:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:57:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:57:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:57:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:57:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:57:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:57:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:57:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:57:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:57:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:57:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:57:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:57:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:57:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:57:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:57:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:57:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:58:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:58:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:58:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:58:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:58:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:58:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:58:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:58:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:58:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:58:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:58:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:58:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:58:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:58:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:58:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:58:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:58:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:58:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:58:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:58:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:58:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:58:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:58:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:58:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:58:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:58:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:58:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:58:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:58:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:58:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:58:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:58:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:58:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:58:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:58:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:58:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:58:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:58:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:58:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:58:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:58:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:58:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:58:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:58:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:58:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:58:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:58:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:58:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:58:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:58:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:58:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:58:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:58:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:58:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:58:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:58:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:58:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:58:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:58:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:58:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:58:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:58:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:58:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:58:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:58:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:58:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:58:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:58:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:58:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:58:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:58:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:58:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:58:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:58:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:58:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:58:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:58:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:58:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:58:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:58:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:58:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:58:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:58:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:58:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:58:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:58:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:58:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:58:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:58:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:58:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:58:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:58:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:58:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:58:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:58:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:58:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:58:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:58:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:58:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:58:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:58:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:58:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:58:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:58:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:58:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:58:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:58:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:58:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:58:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:58:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:58:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:58:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:58:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:58:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:58:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:58:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:58:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:58:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:58:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:58:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:58:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:58:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:58:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:58:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:58:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:58:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:58:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:58:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:58:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:58:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:58:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:58:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:58:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:58:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:58:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:58:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:58:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:58:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:58:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:58:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:58:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:58:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:58:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:58:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:58:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:58:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:58:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:58:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:58:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:58:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:58:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:58:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:58:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:58:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:58:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:58:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:58:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:58:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:58:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:58:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:58:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:58:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:58:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:58:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:58:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:58:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:58:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:58:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:58:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:58:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:58:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:58:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:58:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:58:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:58:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:58:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:58:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:58:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:58:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:58:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:58:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:58:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:58:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:58:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:58:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:58:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:58:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:58:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:58:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:58:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:58:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:58:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:58:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:58:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:58:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:58:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:58:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:58:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:58:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:58:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:58:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:58:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:58:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:58:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:58:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:58:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:58:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:58:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:58:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:58:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:58:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:58:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:58:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:58:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:58:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:58:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:59:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:59:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:59:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:59:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:59:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:59:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:59:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:59:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:59:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:59:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:59:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:59:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:59:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:59:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:59:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:59:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:59:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:59:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:59:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:59:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:59:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:59:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:59:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:59:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:59:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:59:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:59:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:59:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:59:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:59:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:59:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:59:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:59:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:59:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:59:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:59:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:59:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:59:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:59:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:59:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:59:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:59:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:59:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:59:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:59:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:59:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:59:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:59:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:59:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:59:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:59:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:59:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:59:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:59:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:59:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:59:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:59:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:59:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:59:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:59:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:59:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:59:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:59:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:59:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:59:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:59:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:59:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:59:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:59:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:59:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:59:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:59:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:59:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:59:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:59:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:59:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:59:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:59:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:59:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:59:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:59:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:59:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:59:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:59:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:59:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:59:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:59:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:59:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:59:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:59:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:59:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:59:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:59:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:59:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:59:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:59:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:59:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:59:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:59:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:59:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:59:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:59:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:59:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:59:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:59:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:59:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:59:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:59:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:59:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:59:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:59:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:59:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:59:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:59:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:59:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:59:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:59:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:59:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:59:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:59:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:59:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:59:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:59:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:59:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:59:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:59:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:59:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:59:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:59:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:59:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:59:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:59:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:59:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:59:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:59:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:59:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:59:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:59:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:59:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:59:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:59:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:59:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:59:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:59:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:59:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:59:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:59:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:59:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:59:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:59:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:59:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:59:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:59:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:59:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:59:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:59:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:59:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:59:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:59:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:59:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:59:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:59:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:59:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:59:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:59:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:59:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:59:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:59:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:59:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:59:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:59:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:59:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:59:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:59:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:59:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:59:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:59:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:59:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:59:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:59:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:59:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:59:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:59:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:59:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:59:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:59:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:59:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:59:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:59:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:59:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:59:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:59:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:59:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:59:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:59:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:59:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:59:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:59:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 13:59:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 13:59:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:59:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:59:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:59:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 13:59:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 13:59:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 13:59:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 13:59:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 13:59:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 13:59:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 13:59:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 13:59:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 13:59:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 13:59:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 13:59:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 13:59:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 13:59:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:00:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:00:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:00:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:00:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:00:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:00:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:00:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:00:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:00:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:00:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:00:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:00:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:00:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:00:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:00:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:00:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:00:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:00:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:00:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:00:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:00:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:00:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:00:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:00:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:00:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:00:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:00:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:00:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:00:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:00:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:00:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:00:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:00:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:00:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:00:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:00:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:00:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:00:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:00:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:00:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:00:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:00:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:00:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:00:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:00:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:00:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:00:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:00:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:00:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:00:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:00:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:00:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:00:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:00:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:00:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:00:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:00:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:00:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:00:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:00:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:00:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:00:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:00:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:00:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:00:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:00:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:00:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:00:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:00:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:00:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:00:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:00:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:00:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:00:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:00:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:00:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:00:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:00:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:00:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:00:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:00:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:00:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:00:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:00:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:00:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:00:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:00:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:00:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:00:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:00:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:00:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:00:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:00:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:00:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:00:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:00:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:00:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:00:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:00:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:00:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:00:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:00:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:00:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:00:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:00:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:00:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:00:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:00:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:00:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:00:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:00:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:00:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:00:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:00:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:00:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:00:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:00:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:00:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:00:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:00:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:00:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:00:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:00:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:00:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:00:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:00:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:00:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:00:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:00:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:00:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:00:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:00:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:00:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:00:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:00:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:00:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:00:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:00:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:00:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:00:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:00:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:00:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:00:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:00:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:00:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:00:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:00:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:00:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:00:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:00:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:00:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:00:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:00:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:00:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:00:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:00:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:00:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:00:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:00:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:00:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:00:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:00:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:00:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:00:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:00:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:00:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:00:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:00:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:00:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:00:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:00:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:00:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:00:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:00:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:00:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:00:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:00:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:00:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:00:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:00:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:00:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:00:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:00:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:00:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:00:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:00:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:00:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:00:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:00:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:00:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:00:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:00:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:00:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:00:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:00:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:00:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:00:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:00:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:00:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:00:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:00:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:00:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:00:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:00:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:00:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:00:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:00:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:00:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:00:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:00:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:00:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:00:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:00:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:00:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:00:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:00:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:01:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:01:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:01:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:01:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:01:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:01:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:01:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:01:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:01:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:01:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:01:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:01:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:01:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:01:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:01:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:01:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:01:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:01:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:01:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:01:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:01:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:01:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:01:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:01:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:01:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:01:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:01:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:01:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:01:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:01:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:01:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:01:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:01:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:01:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:01:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:01:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:01:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:01:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:01:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:01:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:01:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:01:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:01:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:01:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:01:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:01:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:01:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:01:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:01:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:01:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:01:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:01:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:01:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:01:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:01:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:01:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:01:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:01:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:01:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:01:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:01:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:01:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:01:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:01:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:01:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:01:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:01:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:01:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:01:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:01:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:01:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:01:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:01:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:01:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:01:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:01:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:01:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:01:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:01:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:01:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:01:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:01:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:01:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:01:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:01:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:01:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:01:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:01:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:01:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:01:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:01:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:01:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:01:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:01:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:01:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:01:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:01:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:01:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:01:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:01:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:01:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:01:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:01:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:01:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:01:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:01:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:01:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:01:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:01:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:01:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:01:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:01:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:01:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:01:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:01:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:01:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:01:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:01:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:01:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:01:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:01:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:01:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:01:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:01:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:01:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:01:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:01:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:01:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:01:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:01:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:01:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:01:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:01:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:01:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:01:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:01:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:01:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:01:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:01:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:01:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:01:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:01:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:01:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:01:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:01:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:01:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:01:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:01:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:01:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:01:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:01:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:01:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:01:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:01:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:01:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:01:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:01:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:01:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:01:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:01:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:01:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:01:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:01:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:01:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:01:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:01:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:01:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:01:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:01:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:01:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:01:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:01:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:01:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:01:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:01:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:01:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:01:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:01:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:01:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:01:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:01:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:01:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:01:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:01:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:01:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:01:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:01:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:01:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:01:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:01:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:01:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:01:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:01:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:01:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:01:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:01:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:01:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:01:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:01:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:01:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:01:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:01:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:01:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:01:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:01:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:01:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:01:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:01:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:01:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:01:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:01:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:01:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:01:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:01:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:01:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:01:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:02:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:02:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:02:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:02:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:02:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:02:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:02:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:02:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:02:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:02:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:02:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:02:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:02:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:02:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:02:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:02:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:02:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:02:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:02:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:02:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:02:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:02:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:02:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:02:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:02:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:02:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:02:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:02:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:02:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:02:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:02:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:02:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:02:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:02:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:02:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:02:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:02:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:02:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:02:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:02:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:02:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:02:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:02:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:02:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:02:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:02:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:02:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:02:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:02:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:02:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:02:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:02:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:02:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:02:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:02:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:02:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:02:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:02:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:02:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:02:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:02:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:02:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:02:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:02:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:02:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:02:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:02:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:02:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:02:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:02:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:02:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:02:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:02:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:02:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:02:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:02:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:02:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:02:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:02:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:02:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:02:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:02:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:02:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:02:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:02:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:02:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:02:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:02:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:02:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:02:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:02:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:02:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:02:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:02:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:02:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:02:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:02:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:02:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:02:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:02:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:02:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:02:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:02:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:02:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:02:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:02:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:02:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:02:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:02:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:02:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:02:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:02:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:02:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:02:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:02:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:02:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:02:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:02:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:02:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:02:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:02:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:02:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:02:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:02:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:02:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:02:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:02:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:02:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:02:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:02:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:02:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:02:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:02:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:02:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:02:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:02:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:02:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:02:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:02:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:02:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:02:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:02:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:02:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:02:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:02:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:02:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:02:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:02:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:02:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:02:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:02:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:02:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:02:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:02:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:02:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:02:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:02:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:02:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:02:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:02:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:02:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:02:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:02:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:02:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:02:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:02:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:02:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:02:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:02:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:02:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:02:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:02:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:02:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:02:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:02:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:02:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:02:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:02:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:02:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:02:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:02:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:02:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:02:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:02:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:02:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:02:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:02:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:02:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:02:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:02:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:02:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:02:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:02:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:02:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:02:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:02:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:02:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:02:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:02:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:02:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:02:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:02:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:02:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:02:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:02:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:02:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:02:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:02:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:02:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:02:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:02:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:02:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:02:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:02:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:02:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:02:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:03:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:03:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:03:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:03:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:03:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:03:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:03:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:03:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:03:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:03:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:03:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:03:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:03:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:03:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:03:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:03:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:03:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:03:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:03:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:03:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:03:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:03:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:03:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:03:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:03:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:03:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:03:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:03:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:03:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:03:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:03:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:03:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:03:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:03:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:03:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:03:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:03:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:03:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:03:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:03:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:03:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:03:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:03:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:03:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:03:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:03:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:03:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:03:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:03:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:03:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:03:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:03:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:03:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:03:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:03:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:03:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:03:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:03:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:03:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:03:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:03:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:03:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:03:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:03:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:03:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:03:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:03:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:03:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:03:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:03:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:03:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:03:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:03:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:03:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:03:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:03:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:03:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:03:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:03:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:03:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:03:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:03:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:03:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:03:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:03:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:03:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:03:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:03:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:03:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:03:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:03:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:03:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:03:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:03:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:03:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:03:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:03:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:03:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:03:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:03:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:03:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:03:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:03:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:03:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:03:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:03:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:03:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:03:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:03:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:03:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:03:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:03:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:03:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:03:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:03:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:03:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:03:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:03:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:03:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:03:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:03:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:03:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:03:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:03:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:03:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:03:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:03:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:03:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:03:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:03:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:03:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:03:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:03:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:03:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:03:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:03:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:03:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:03:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:03:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:03:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:03:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:03:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:03:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:03:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:03:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:03:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:03:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:03:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:03:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:03:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:03:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:03:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:03:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:03:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:03:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:03:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:03:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:03:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:03:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:03:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:03:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:03:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:03:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:03:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:03:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:03:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:03:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:03:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:03:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:03:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:03:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:03:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:03:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:03:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:03:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:03:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:03:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:03:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:03:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:03:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:03:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:03:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:03:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:03:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:03:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:03:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:03:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:03:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:03:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:03:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:03:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:03:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:03:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:03:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:03:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:03:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:03:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:03:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:03:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:03:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:03:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:03:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:03:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:03:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:03:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:03:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:03:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:03:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:03:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:03:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:03:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:03:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:03:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:03:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:03:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:03:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:04:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:04:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:04:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:04:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:04:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:04:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:04:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:04:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:04:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:04:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:04:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:04:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:04:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:04:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:04:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:04:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:04:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:04:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:06:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:06:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:06:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:06:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:06:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:06:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:06:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:06:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:06:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:06:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:06:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:06:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:06:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:06:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:06:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:06:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:06:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:06:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:06:12 rdma06 rsyslogd[1041]: imjournal: 5295 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 14:06:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:06:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:06:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:06:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:06:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:06:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:06:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:06:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:06:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:06:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:06:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:06:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:06:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:06:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:06:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:06:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:06:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:06:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:06:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:06:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:06:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:06:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:06:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:06:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:06:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:06:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:06:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:06:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:06:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:06:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:06:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:06:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:06:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:06:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:06:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:06:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:06:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:06:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:06:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:06:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:06:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:06:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:06:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:06:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:06:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:06:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:06:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:06:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:06:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:06:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:06:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:06:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:06:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:06:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:06:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:06:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:06:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:06:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:06:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:06:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:06:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:06:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:06:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:06:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:06:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:06:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:06:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:06:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:06:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:06:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:06:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:06:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:06:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:06:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:06:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:06:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:06:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:06:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:06:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:06:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:06:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:06:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:06:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:06:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:06:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:06:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:06:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:06:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:06:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:06:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:06:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:06:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:06:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:06:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:06:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:06:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:06:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:06:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:06:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:06:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:06:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:06:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:06:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:06:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:06:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:06:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:06:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:06:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:06:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:06:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:06:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:06:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:06:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:06:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:06:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:06:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:06:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:06:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:06:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:06:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:06:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:06:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:06:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:06:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:06:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:06:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:06:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:06:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:06:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:06:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:06:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:06:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:06:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:06:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:06:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:06:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:06:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:06:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:06:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:06:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:06:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:06:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:06:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:06:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:06:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:06:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:06:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:06:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:06:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:06:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:06:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:06:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:06:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:06:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:06:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:06:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:06:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:06:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:06:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:06:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:06:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:06:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:07:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:07:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:07:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:07:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:07:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:07:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:07:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:07:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:07:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:07:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:07:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:07:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:07:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:07:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:07:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:07:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:07:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:07:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:07:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:07:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:07:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:07:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:07:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:07:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:07:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:07:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:07:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:07:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:07:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:07:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:07:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:07:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:07:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:07:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:07:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:07:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:07:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:07:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:07:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:07:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:07:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:07:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:07:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:07:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:07:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:07:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:07:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:07:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:07:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:07:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:07:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:07:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:07:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:07:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:07:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:07:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:07:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:07:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:07:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:07:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:07:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:07:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:07:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:07:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:07:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:07:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:07:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:07:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:07:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:07:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:07:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:07:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:07:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:07:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:07:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:07:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:07:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:07:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:07:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:07:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:07:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:07:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:07:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:07:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:07:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:07:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:07:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:07:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:07:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:07:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:07:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:07:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:07:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:07:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:07:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:07:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:07:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:07:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:07:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:07:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:07:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:07:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:07:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:07:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:07:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:07:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:07:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:07:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:07:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:07:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:07:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:07:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:07:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:07:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:07:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:07:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:07:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:07:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:07:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:07:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:07:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:07:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:07:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:07:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:07:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:07:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:07:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:07:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:07:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:07:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:07:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:07:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:07:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:07:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:07:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:07:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:07:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:07:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:07:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:07:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:07:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:07:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:07:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:07:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:07:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:07:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:07:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:07:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:07:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:07:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:07:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:07:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:07:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:07:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:07:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:07:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:07:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:07:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:07:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:07:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:07:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:07:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:07:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:07:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:07:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:07:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:07:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:07:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:07:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:07:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:07:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:07:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:07:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:07:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:07:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:07:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:07:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:07:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:07:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:07:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:07:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:07:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:07:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:07:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:07:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:07:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:07:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:07:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:07:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:07:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:07:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:07:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:07:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:07:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:07:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:07:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:07:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:07:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:07:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:07:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:07:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:07:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:07:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:07:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:07:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:07:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:07:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:07:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:07:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:07:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:07:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:07:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:07:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:07:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:07:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:07:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:08:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:08:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:08:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:08:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:08:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:08:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:08:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:08:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:08:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:08:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:08:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:08:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:08:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:08:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:08:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:08:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:08:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:08:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:08:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:08:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:08:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:08:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:08:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:08:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:08:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:08:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:08:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:08:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:08:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:08:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:08:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:08:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:08:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:08:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:08:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:08:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:08:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:08:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:08:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:08:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:08:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:08:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:08:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:08:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:08:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:08:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:08:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:08:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:08:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:08:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:08:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:08:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:08:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:08:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:08:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:08:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:08:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:08:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:08:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:08:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:08:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:08:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:08:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:08:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:08:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:08:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:08:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:08:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:08:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:08:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:08:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:08:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:08:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:08:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:08:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:08:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:08:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:08:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:08:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:08:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:08:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:08:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:08:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:08:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:08:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:08:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:08:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:08:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:08:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:08:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:08:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:08:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:08:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:08:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:08:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:08:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:08:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:08:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:08:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:08:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:08:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:08:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:08:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:08:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:08:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:08:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:08:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:08:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:08:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:08:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:08:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:08:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:08:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:08:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:08:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:08:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:08:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:08:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:08:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:08:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:08:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:08:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:08:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:08:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:08:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:08:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:08:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:08:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:08:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:08:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:08:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:08:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:08:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:08:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:08:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:08:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:08:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:08:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:08:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:08:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:08:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:08:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:08:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:08:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:08:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:08:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:08:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:08:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:08:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:08:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:08:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:08:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:08:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:08:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:08:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:08:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:08:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:08:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:08:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:08:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:08:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:08:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:08:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:08:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:08:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:08:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:08:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:08:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:08:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:08:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:08:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:08:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:08:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:08:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:08:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:08:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:08:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:08:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:08:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:08:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:08:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:08:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:08:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:08:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:08:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:08:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:08:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:08:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:08:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:08:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:08:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:08:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:08:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:08:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:08:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:08:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:08:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:08:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:08:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:08:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:08:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:08:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:08:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:08:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:08:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:08:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:08:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:08:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:08:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:08:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:08:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:08:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:08:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:08:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:08:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:08:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:09:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:09:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:09:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:09:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:09:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:09:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:09:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:09:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:09:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:09:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:09:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:09:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:09:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:09:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:09:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:09:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:09:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:09:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:09:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:09:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:09:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:09:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:09:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:09:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:09:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:09:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:09:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:09:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:09:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:09:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:09:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:09:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:09:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:09:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:09:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:09:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:09:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:09:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:09:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:09:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:09:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:09:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:09:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:09:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:09:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:09:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:09:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:09:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:09:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:09:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:09:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:09:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:09:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:09:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:09:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:09:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:09:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:09:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:09:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:09:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:09:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:09:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:09:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:09:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:09:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:09:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:09:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:09:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:09:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:09:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:09:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:09:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:09:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:09:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:09:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:09:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:09:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:09:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:09:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:09:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:09:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:09:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:09:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:09:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:09:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:09:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:09:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:09:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:09:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:09:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:09:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:09:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:09:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:09:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:09:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:09:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:09:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:09:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:09:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:09:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:09:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:09:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:09:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:09:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:09:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:09:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:09:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:09:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:09:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:09:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:09:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:09:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:09:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:09:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:09:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:09:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:09:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:09:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:09:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:09:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:09:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:09:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:09:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:09:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:09:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:09:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:09:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:09:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:09:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:09:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:09:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:09:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:09:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:09:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:09:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:09:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:09:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:09:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:09:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:09:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:09:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:09:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:09:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:09:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:09:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:09:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:09:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:09:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:09:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:09:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:09:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:09:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:09:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:09:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:09:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:09:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:09:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:09:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:09:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:09:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:09:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:09:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:09:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:09:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:09:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:09:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:09:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:09:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:09:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:09:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:09:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:09:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:09:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:09:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:09:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:09:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:09:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:09:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:09:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:09:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:09:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:09:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:09:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:09:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:09:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:09:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:09:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:09:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:09:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:09:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:09:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:09:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:09:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:09:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:09:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:09:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:09:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:09:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:09:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:09:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:09:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:09:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:09:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:09:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:09:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:09:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:09:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:09:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:09:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:09:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:09:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:09:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:09:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:09:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:09:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:09:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:10:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:10:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:10:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:10:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:10:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:10:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:10:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:10:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:10:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:10:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:10:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:10:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:10:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:10:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:10:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:10:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:10:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:10:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:10:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:10:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:10:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:10:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:10:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:10:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:10:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:10:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:10:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:10:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:10:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:10:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:10:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:10:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:10:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:10:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:10:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:10:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:10:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:10:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:10:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:10:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:10:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:10:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:10:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:10:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:10:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:10:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:10:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:10:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:10:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:10:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:10:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:10:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:10:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:10:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:10:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:10:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:10:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:10:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:10:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:10:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:10:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:10:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:10:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:10:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:10:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:10:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:10:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:10:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:10:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:10:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:10:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:10:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:10:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:10:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:10:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:10:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:10:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:10:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:10:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:10:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:10:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:10:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:10:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:10:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:10:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:10:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:10:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:10:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:10:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:10:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:10:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:10:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:10:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:10:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:10:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:10:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:10:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:10:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:10:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:10:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:10:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:10:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:10:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:10:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:10:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:10:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:10:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:10:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:10:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:10:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:10:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:10:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:10:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:10:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:10:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:10:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:10:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:10:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:10:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:10:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:10:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:10:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:10:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:10:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:10:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:10:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:10:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:10:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:10:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:10:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:10:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:10:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:10:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:10:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:10:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:10:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:10:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:10:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:10:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:10:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:10:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:10:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:10:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:10:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:10:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:10:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:10:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:10:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:10:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:10:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:10:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:10:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:10:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:10:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:10:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:10:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:10:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:10:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:10:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:10:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:10:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:10:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:10:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:10:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:10:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:10:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:10:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:10:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:10:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:10:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:10:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:10:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:10:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:10:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:10:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:10:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:10:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:10:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:10:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:10:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:10:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:10:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:10:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:10:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:10:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:10:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:10:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:10:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:10:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:10:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:10:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:10:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:10:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:10:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:10:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:10:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:10:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:10:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:10:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:10:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:10:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:10:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:10:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:10:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:10:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:10:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:10:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:10:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:10:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:10:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:10:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:10:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:10:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:10:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:10:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:10:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:11:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:11:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:11:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:11:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:11:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:11:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:11:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:11:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:11:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:11:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:11:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:11:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:11:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:11:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:11:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:11:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:11:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:11:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:11:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:11:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:11:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:11:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:11:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:11:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:11:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:11:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:11:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:11:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:11:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:11:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:11:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:11:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:11:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:11:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:11:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:11:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:11:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:11:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:11:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:11:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:11:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:11:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:11:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:11:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:11:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:11:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:11:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:11:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:11:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:11:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:11:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:11:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:11:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:11:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:11:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:11:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:11:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:11:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:11:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:11:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:11:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:11:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:11:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:11:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:11:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:11:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:11:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:11:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:11:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:11:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:11:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:11:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:11:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:11:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:11:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:11:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:11:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:11:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:11:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:11:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:11:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:11:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:11:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:11:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:11:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:11:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:11:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:11:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:11:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:11:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:11:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:11:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:11:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:11:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:11:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:11:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:11:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:11:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:11:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:11:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:11:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:11:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:11:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:11:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:11:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:11:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:11:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:11:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:11:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:11:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:11:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:11:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:11:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:11:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:11:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:11:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:11:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:11:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:11:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:11:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:11:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:11:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:11:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:11:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:11:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:11:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:11:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:11:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:11:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:11:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:11:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:11:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:11:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:11:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:11:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:11:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:11:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:11:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:11:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:11:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:11:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:11:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:11:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:11:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:11:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:11:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:11:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:11:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:11:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:11:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:11:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:11:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:11:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:11:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:11:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:11:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:11:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:11:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:11:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:11:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:11:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:11:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:11:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:11:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:11:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:11:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:11:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:11:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:11:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:11:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:11:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:11:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:11:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:11:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:11:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:11:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:11:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:11:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:11:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:11:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:11:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:11:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:11:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:11:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:11:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:11:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:11:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:11:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:11:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:11:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:11:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:11:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:11:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:11:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:11:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:11:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:11:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:11:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:11:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:11:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:11:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:11:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:11:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:11:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:11:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:11:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:11:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:11:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:11:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:11:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:11:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:11:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:11:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:11:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:11:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:11:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:12:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:12:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:12:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:12:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:12:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:12:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:12:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:12:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:12:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:12:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:12:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:12:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:12:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:12:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:12:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:12:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:12:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:12:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:12:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:12:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:12:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:12:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:12:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:12:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:12:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:12:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:12:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:12:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:12:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:12:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:12:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:12:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:12:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:12:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:12:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:12:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:12:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:12:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:12:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:12:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:12:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:12:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:12:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:12:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:12:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:12:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:12:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:12:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:12:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:12:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:12:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:12:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:12:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:12:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:12:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:12:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:12:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:12:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:12:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:12:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:12:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:12:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:12:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:12:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:12:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:12:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:12:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:12:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:12:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:12:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:12:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:12:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:12:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:12:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:12:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:12:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:12:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:12:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:12:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:12:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:12:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:12:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:12:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:12:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:12:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:12:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:12:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:12:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:12:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:12:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:12:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:12:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:12:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:12:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:12:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:12:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:12:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:12:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:12:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:12:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:12:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:12:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:12:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:12:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:12:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:12:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:12:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:12:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:12:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:12:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:12:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:12:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:12:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:12:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:12:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:12:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:12:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:12:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:12:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:12:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:12:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:12:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:12:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:12:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:12:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:12:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:12:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:12:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:12:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:12:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:12:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:12:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:12:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:12:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:12:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:12:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:12:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:12:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:12:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:12:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:12:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:12:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:12:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:12:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:12:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:12:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:12:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:12:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:12:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:12:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:12:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:12:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:12:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:12:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:12:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:12:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:12:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:12:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:12:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:12:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:12:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:12:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:12:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:12:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:12:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:12:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:12:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:12:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:12:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:12:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:12:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:12:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:12:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:12:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:12:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:12:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:12:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:12:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:12:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:12:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:12:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:12:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:12:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:12:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:12:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:12:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:12:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:12:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:12:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:12:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:12:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:12:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:12:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:12:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:12:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:12:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:12:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:12:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:12:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:12:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:12:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:12:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:12:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:12:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:12:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:12:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:12:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:12:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:12:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:12:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:12:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:12:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:12:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:12:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:12:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:12:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:13:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:13:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:13:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:13:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:13:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:13:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:13:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:13:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:13:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:13:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:13:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:13:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:13:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:13:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:13:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:13:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:13:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:13:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:13:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:13:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:13:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:13:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:13:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:13:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:13:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:13:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:13:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:13:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:13:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:13:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:13:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:13:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:13:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:13:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:13:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:13:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:13:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:13:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:13:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:13:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:13:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:13:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:13:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:13:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:13:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:13:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:13:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:13:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:13:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:13:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:13:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:13:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:13:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:13:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:13:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:13:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:13:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:13:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:13:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:13:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:13:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:13:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:13:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:13:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:13:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:13:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:13:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:13:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:13:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:13:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:13:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:13:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:13:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:13:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:13:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:13:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:13:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:13:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:13:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:13:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:13:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:13:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:13:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:13:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:13:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:13:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:13:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:13:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:13:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:13:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:13:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:13:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:13:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:13:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:13:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:13:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:13:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:13:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:13:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:13:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:13:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:13:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:13:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:13:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:13:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:13:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:13:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:13:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:13:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:13:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:13:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:13:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:13:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:13:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:13:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:13:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:13:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:13:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:13:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:13:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:13:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:13:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:13:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:13:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:13:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:13:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:13:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:13:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:13:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:13:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:13:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:13:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:13:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:13:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:13:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:13:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:13:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:13:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:13:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:13:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:13:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:13:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:13:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:13:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:13:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:13:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:13:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:13:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:13:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:13:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:13:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:13:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:13:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:13:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:13:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:13:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:13:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:13:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:13:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:13:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:13:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:13:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:13:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:13:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:13:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:13:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:13:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:13:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:13:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:13:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:13:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:13:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:13:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:13:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:13:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:13:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:13:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:13:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:13:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:13:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:13:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:13:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:13:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:13:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:13:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:13:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:13:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:13:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:13:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:13:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:13:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:13:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:13:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:13:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:13:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:13:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:13:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:13:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:13:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:13:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:13:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:13:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:13:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:13:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:13:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:13:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:13:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:13:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:13:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:13:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:13:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:13:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:13:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:13:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:13:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:13:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:14:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:14:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:14:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:14:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:14:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:14:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:14:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:14:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:14:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:14:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:14:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:14:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:14:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:14:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:14:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:14:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:14:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:14:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:14:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:14:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:14:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:14:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:14:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:14:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:14:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:14:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:14:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:14:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:14:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:14:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:14:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:14:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:14:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:14:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:14:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:14:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:16:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:16:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:16:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:16:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:16:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:16:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:16:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:16:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:16:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:16:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:16:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:16:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:16:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:16:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:16:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:16:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:16:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:16:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:16:13 rdma06 rsyslogd[1041]: imjournal: 5082 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 14:16:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:16:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:16:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:16:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:16:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:16:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:16:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:16:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:16:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:16:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:16:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:16:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:16:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:16:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:16:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:16:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:16:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:16:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:16:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:16:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:16:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:16:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:16:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:16:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:16:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:16:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:16:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:16:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:16:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:16:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:16:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:16:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:16:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:16:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:16:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:16:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:16:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:16:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:16:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:16:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:16:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:16:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:16:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:16:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:16:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:16:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:16:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:16:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:16:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:16:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:16:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:16:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:16:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:16:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:16:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:16:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:16:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:16:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:16:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:16:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:16:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:16:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:16:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:16:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:16:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:16:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:16:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:16:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:16:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:16:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:16:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:16:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:16:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:16:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:16:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:16:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:16:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:16:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:16:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:16:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:16:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:16:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:16:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:16:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:16:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:16:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:16:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:16:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:16:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:16:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:16:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:16:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:16:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:16:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:16:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:16:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:16:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:16:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:16:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:16:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:16:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:16:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:16:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:16:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:16:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:16:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:16:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:16:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:16:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:16:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:16:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:16:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:16:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:16:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:16:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:16:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:16:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:16:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:16:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:16:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:16:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:16:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:16:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:16:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:16:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:16:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:16:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:16:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:16:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:16:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:16:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:16:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:16:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:16:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:16:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:16:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:16:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:16:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:16:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:16:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:16:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:16:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:16:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:16:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:16:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:16:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:16:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:16:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:16:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:16:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:16:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:16:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:16:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:16:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:16:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:16:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:16:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:16:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:16:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:16:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:16:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:16:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:17:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:17:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:17:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:17:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:17:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:17:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:17:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:17:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:17:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:17:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:17:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:17:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:17:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:17:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:17:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:17:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:17:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:17:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:17:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:17:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:17:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:17:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:17:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:17:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:17:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:17:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:17:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:17:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:17:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:17:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:17:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:17:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:17:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:17:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:17:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:17:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:17:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:17:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:17:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:17:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:17:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:17:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:17:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:17:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:17:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:17:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:17:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:17:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:17:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:17:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:17:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:17:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:17:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:17:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:17:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:17:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:17:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:17:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:17:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:17:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:17:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:17:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:17:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:17:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:17:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:17:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:17:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:17:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:17:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:17:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:17:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:17:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:17:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:17:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:17:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:17:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:17:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:17:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:17:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:17:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:17:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:17:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:17:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:17:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:17:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:17:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:17:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:17:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:17:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:17:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:17:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:17:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:17:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:17:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:17:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:17:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:17:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:17:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:17:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:17:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:17:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:17:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:17:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:17:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:17:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:17:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:17:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:17:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:17:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:17:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:17:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:17:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:17:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:17:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:17:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:17:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:17:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:17:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:17:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:17:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:17:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:17:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:17:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:17:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:17:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:17:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:17:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:17:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:17:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:17:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:17:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:17:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:17:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:17:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:17:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:17:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:17:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:17:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:17:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:17:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:17:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:17:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:17:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:17:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:17:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:17:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:17:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:17:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:17:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:17:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:17:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:17:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:17:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:17:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:17:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:17:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:17:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:17:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:17:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:17:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:17:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:17:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:17:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:17:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:17:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:17:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:17:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:17:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:17:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:17:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:17:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:17:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:17:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:17:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:17:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:17:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:17:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:17:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:17:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:17:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:17:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:17:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:17:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:17:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:17:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:17:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:17:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:17:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:17:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:17:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:17:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:17:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:17:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:17:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:17:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:17:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:17:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:17:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:17:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:17:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:17:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:17:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:17:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:17:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:17:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:17:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:17:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:17:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:17:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:17:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:17:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:17:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:17:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:17:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:17:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:17:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:18:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:18:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:18:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:18:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:18:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:18:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:18:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:18:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:18:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:18:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:18:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:18:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:18:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:18:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:18:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:18:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:18:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:18:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:18:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:18:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:18:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:18:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:18:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:18:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:18:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:18:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:18:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:18:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:18:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:18:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:18:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:18:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:18:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:18:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:18:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:18:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:18:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:18:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:18:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:18:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:18:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:18:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:18:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:18:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:18:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:18:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:18:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:18:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:18:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:18:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:18:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:18:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:18:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:18:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:18:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:18:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:18:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:18:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:18:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:18:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:18:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:18:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:18:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:18:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:18:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:18:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:18:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:18:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:18:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:18:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:18:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:18:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:18:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:18:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:18:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:18:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:18:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:18:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:18:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:18:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:18:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:18:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:18:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:18:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:18:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:18:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:18:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:18:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:18:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:18:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:18:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:18:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:18:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:18:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:18:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:18:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:18:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:18:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:18:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:18:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:18:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:18:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:18:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:18:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:18:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:18:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:18:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:18:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:18:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:18:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:18:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:18:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:18:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:18:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:18:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:18:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:18:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:18:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:18:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:18:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:18:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:18:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:18:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:18:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:18:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:18:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:18:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:18:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:18:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:18:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:18:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:18:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:18:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:18:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:18:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:18:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:18:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:18:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:18:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:18:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:18:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:18:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:18:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:18:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:18:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:18:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:18:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:18:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:18:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:18:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:18:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:18:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:18:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:18:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:18:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:18:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:18:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:18:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:18:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:18:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:18:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:18:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:18:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:18:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:18:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:18:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:18:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:18:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:18:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:18:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:18:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:18:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:18:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:18:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:18:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:18:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:18:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:18:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:18:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:18:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:18:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:18:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:18:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:18:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:18:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:18:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:18:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:18:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:18:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:18:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:18:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:18:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:18:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:18:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:18:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:18:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:18:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:18:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:18:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:18:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:18:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:18:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:18:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:18:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:18:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:18:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:18:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:18:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:18:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:18:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:18:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:18:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:18:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:18:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:18:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:18:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:19:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:19:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:19:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:19:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:19:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:19:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:19:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:19:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:19:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:19:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:19:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:19:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:19:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:19:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:19:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:19:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:19:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:19:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:19:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:19:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:19:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:19:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:19:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:19:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:19:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:19:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:19:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:19:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:19:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:19:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:19:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:19:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:19:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:19:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:19:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:19:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:19:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:19:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:19:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:19:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:19:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:19:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:19:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:19:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:19:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:19:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:19:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:19:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:19:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:19:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:19:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:19:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:19:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:19:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:19:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:19:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:19:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:19:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:19:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:19:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:19:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:19:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:19:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:19:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:19:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:19:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:19:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:19:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:19:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:19:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:19:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:19:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:19:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:19:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:19:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:19:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:19:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:19:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:19:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:19:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:19:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:19:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:19:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:19:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:19:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:19:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:19:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:19:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:19:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:19:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:19:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:19:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:19:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:19:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:19:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:19:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:19:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:19:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:19:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:19:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:19:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:19:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:19:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:19:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:19:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:19:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:19:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:19:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:19:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:19:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:19:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:19:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:19:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:19:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:19:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:19:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:19:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:19:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:19:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:19:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:19:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:19:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:19:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:19:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:19:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:19:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:19:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:19:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:19:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:19:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:19:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:19:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:19:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:19:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:19:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:19:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:19:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:19:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:19:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:19:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:19:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:19:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:19:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:19:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:19:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:19:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:19:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:19:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:19:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:19:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:19:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:19:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:19:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:19:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:19:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:19:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:19:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:19:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:19:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:19:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:19:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:19:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:19:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:19:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:19:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:19:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:19:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:19:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:19:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:19:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:19:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:19:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:19:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:19:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:19:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:19:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:19:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:19:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:19:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:19:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:19:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:19:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:19:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:19:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:19:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:19:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:19:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:19:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:19:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:19:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:19:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:19:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:19:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:19:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:19:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:19:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:19:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:19:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:19:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:19:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:19:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:19:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:19:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:19:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:19:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:19:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:19:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:19:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:19:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:19:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:19:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:19:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:19:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:19:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:19:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:19:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:20:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:20:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:20:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:20:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:20:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:20:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:20:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:20:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:20:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:20:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:20:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:20:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:20:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:20:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:20:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:20:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:20:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:20:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:20:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:20:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:20:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:20:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:20:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:20:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:20:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:20:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:20:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:20:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:20:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:20:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:20:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:20:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:20:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:20:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:20:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:20:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:20:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:20:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:20:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:20:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:20:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:20:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:20:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:20:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:20:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:20:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:20:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:20:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:20:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:20:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:20:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:20:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:20:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:20:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:20:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:20:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:20:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:20:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:20:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:20:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:20:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:20:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:20:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:20:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:20:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:20:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:20:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:20:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:20:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:20:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:20:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:20:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:20:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:20:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:20:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:20:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:20:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:20:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:20:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:20:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:20:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:20:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:20:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:20:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:20:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:20:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:20:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:20:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:20:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:20:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:20:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:20:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:20:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:20:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:20:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:20:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:20:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:20:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:20:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:20:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:20:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:20:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:20:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:20:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:20:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:20:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:20:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:20:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:20:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:20:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:20:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:20:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:20:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:20:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:20:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:20:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:20:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:20:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:20:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:20:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:20:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:20:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:20:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:20:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:20:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:20:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:20:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:20:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:20:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:20:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:20:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:20:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:20:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:20:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:20:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:20:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:20:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:20:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:20:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:20:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:20:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:20:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:20:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:20:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:20:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:20:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:20:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:20:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:20:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:20:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:20:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:20:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:20:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:20:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:20:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:20:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:20:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:20:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:20:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:20:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:20:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:20:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:20:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:20:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:20:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:20:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:20:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:20:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:20:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:20:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:20:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:20:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:20:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:20:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:20:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:20:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:20:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:20:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:20:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:20:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:20:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:20:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:20:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:20:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:20:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:20:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:20:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:20:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:20:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:20:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:20:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:20:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:20:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:20:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:20:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:20:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:20:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:20:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:20:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:20:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:20:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:20:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:20:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:20:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:20:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:20:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:20:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:20:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:20:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:20:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:20:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:20:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:20:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:20:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:20:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:20:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:21:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:21:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:21:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:21:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:21:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:21:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:21:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:21:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:21:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:21:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:21:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:21:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:21:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:21:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:21:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:21:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:21:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:21:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:21:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:21:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:21:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:21:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:21:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:21:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:21:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:21:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:21:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:21:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:21:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:21:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:21:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:21:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:21:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:21:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:21:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:21:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:21:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:21:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:21:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:21:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:21:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:21:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:21:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:21:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:21:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:21:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:21:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:21:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:21:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:21:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:21:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:21:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:21:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:21:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:21:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:21:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:21:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:21:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:21:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:21:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:21:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:21:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:21:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:21:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:21:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:21:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:21:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:21:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:21:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:21:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:21:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:21:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:21:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:21:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:21:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:21:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:21:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:21:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:21:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:21:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:21:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:21:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:21:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:21:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:21:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:21:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:21:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:21:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:21:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:21:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:21:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:21:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:21:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:21:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:21:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:21:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:21:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:21:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:21:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:21:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:21:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:21:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:21:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:21:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:21:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:21:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:21:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:21:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:21:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:21:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:21:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:21:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:21:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:21:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:21:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:21:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:21:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:21:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:21:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:21:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:21:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:21:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:21:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:21:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:21:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:21:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:21:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:21:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:21:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:21:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:21:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:21:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:21:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:21:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:21:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:21:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:21:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:21:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:21:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:21:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:21:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:21:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:21:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:21:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:21:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:21:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:21:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:21:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:21:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:21:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:21:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:21:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:21:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:21:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:21:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:21:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:21:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:21:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:21:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:21:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:21:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:21:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:21:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:21:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:21:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:21:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:21:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:21:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:21:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:21:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:21:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:21:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:21:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:21:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:21:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:21:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:21:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:21:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:21:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:21:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:21:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:21:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:21:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:21:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:21:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:21:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:21:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:21:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:21:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:21:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:21:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:21:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:21:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:21:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:21:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:21:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:21:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:21:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:21:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:21:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:21:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:21:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:21:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:21:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:21:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:21:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:22:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:22:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:22:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:22:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:22:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:22:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:22:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:22:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:22:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:22:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:22:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:22:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:22:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:22:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:22:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:22:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:22:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:22:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:22:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:22:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:22:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:22:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:22:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:22:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:22:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:22:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:22:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:22:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:22:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:22:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:22:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:22:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:22:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:22:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:22:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:22:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:22:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:22:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:22:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:22:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:22:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:22:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:22:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:22:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:22:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:22:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:22:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:22:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:22:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:22:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:22:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:22:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:22:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:22:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:22:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:22:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:22:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:22:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:22:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:22:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:22:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:22:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:22:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:22:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:22:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:22:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:22:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:22:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:22:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:22:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:22:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:22:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:22:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:22:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:22:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:22:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:22:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:22:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:22:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:22:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:22:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:22:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:22:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:22:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:22:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:22:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:22:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:22:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:22:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:22:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:22:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:22:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:22:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:22:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:22:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:22:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:22:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:22:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:22:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:22:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:22:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:22:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:22:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:22:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:22:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:22:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:22:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:22:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:22:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:22:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:22:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:22:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:22:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:22:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:22:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:22:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:22:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:22:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:22:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:22:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:22:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:22:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:22:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:22:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:22:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:22:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:22:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:22:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:22:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:22:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:22:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:22:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:22:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:22:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:22:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:22:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:22:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:22:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:22:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:22:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:22:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:22:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:22:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:22:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:22:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:22:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:22:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:22:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:22:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:22:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:22:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:22:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:22:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:22:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:22:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:22:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:22:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:22:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:22:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:22:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:22:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:22:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:22:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:22:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:22:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:22:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:22:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:22:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:22:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:22:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:22:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:22:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:22:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:22:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:22:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:22:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:22:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:22:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:22:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:22:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:22:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:22:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:22:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:22:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:22:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:22:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:22:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:22:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:22:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:22:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:22:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:22:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:22:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:22:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:22:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:22:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:23:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:23:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:23:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:23:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:23:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:23:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:23:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:23:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:23:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:23:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:23:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:23:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:23:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:23:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:23:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:23:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:23:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:23:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:23:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:23:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:23:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:23:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:23:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:23:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:23:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:23:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:23:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:23:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:23:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:23:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:23:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:23:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:23:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:23:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:23:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:23:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:23:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:23:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:23:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:23:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:23:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:23:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:23:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:23:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:23:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:23:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:23:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:23:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:23:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:23:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:23:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:23:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:23:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:23:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:23:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:23:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:23:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:23:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:23:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:23:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:23:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:23:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:23:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:23:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:23:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:23:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:23:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:23:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:23:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:23:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:23:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:23:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:23:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:23:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:23:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:23:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:23:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:23:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:23:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:23:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:23:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:23:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:23:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:23:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:23:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:23:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:23:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:23:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:23:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:23:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:23:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:23:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:23:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:23:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:23:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:23:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:23:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:23:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:23:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:23:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:23:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:23:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:23:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:23:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:23:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:23:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:23:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:23:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:23:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:23:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:23:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:23:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:23:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:23:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:23:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:23:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:23:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:23:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:23:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:23:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:23:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:23:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:23:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:23:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:23:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:23:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:23:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:23:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:23:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:23:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:23:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:23:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:23:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:23:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:23:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:23:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:23:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:23:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:23:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:23:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:23:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:23:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:23:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:23:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:23:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:23:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:23:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:23:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:23:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:23:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:23:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:23:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:23:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:23:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:23:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:23:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:23:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:23:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:23:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:23:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:23:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:23:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:23:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:23:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:23:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:23:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:24:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:24:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:24:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:24:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:24:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:24:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:24:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:24:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:24:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:24:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:24:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:24:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:24:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:24:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:24:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:24:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:26:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:26:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:26:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:26:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:26:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:26:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:26:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:26:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:26:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:26:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:26:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:26:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:26:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:26:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:26:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:26:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:26:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:26:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:26:18 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 14:26:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:26:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:26:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:26:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:26:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:26:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:26:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:26:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:26:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:26:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:26:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:26:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:26:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:26:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:26:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:26:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:26:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:26:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:26:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:26:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:26:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:26:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:26:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:26:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:26:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:26:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:26:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:26:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:26:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:26:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:26:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:26:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:26:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:26:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:26:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:26:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:26:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:26:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:26:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:26:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:26:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:26:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:26:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:26:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:26:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:26:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:26:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:26:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:26:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:26:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:26:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:26:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:26:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:26:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:26:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:26:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:26:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:26:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:26:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:26:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:26:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:26:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:26:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:26:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:26:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:26:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:26:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:26:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:26:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:26:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:26:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:26:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:26:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:26:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:26:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:26:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:26:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:26:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:26:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:26:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:26:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:26:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:26:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:26:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:26:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:26:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:26:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:26:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:26:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:26:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:26:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:26:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:26:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:26:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:26:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:26:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:26:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:26:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:26:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:26:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:26:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:26:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:26:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:26:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:26:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:26:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:26:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:26:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:26:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:26:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:26:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:26:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:26:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:26:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:26:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:26:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:26:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:26:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:26:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:26:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:26:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:26:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:26:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:26:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:26:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:26:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:26:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:26:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:26:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:26:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:26:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:26:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:26:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:26:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:26:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:26:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:26:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:26:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:26:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:26:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:26:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:26:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:26:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:26:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:27:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:27:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:27:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:27:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:27:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:27:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:27:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:27:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:27:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:27:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:27:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:27:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:27:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:27:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:27:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:27:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:27:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:27:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:27:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:27:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:27:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:27:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:27:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:27:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:27:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:27:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:27:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:27:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:27:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:27:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:27:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:27:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:27:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:27:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:27:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:27:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:27:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:27:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:27:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:27:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:27:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:27:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:27:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:27:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:27:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:27:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:27:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:27:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:27:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:27:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:27:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:27:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:27:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:27:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:27:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:27:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:27:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:27:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:27:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:27:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:27:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:27:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:27:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:27:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:27:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:27:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:27:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:27:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:27:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:27:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:27:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:27:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:27:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:27:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:27:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:27:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:27:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:27:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:27:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:27:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:27:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:27:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:27:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:27:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:27:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:27:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:27:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:27:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:27:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:27:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:27:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:27:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:27:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:27:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:27:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:27:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:27:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:27:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:27:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:27:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:27:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:27:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:27:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:27:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:27:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:27:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:27:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:27:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:27:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:27:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:27:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:27:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:27:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:27:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:27:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:27:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:27:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:27:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:27:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:27:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:27:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:27:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:27:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:27:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:27:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:27:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:27:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:27:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:27:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:27:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:27:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:27:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:27:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:27:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:27:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:27:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:27:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:27:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:27:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:27:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:27:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:27:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:27:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:27:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:27:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:27:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:27:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:27:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:27:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:27:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:27:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:27:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:27:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:27:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:27:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:27:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:27:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:27:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:27:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:27:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:27:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:27:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:27:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:27:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:27:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:27:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:27:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:27:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:27:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:27:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:27:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:27:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:27:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:27:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:27:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:27:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:27:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:27:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:27:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:27:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:27:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:27:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:27:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:27:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:27:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:27:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:27:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:27:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:27:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:27:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:27:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:27:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:27:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:27:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:27:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:27:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:27:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:27:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:27:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:27:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:27:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:27:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:27:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:27:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:27:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:27:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:27:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:27:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:27:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:27:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:27:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:27:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:27:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:27:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:27:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:27:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:28:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:28:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:28:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:28:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:28:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:28:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:28:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:28:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:28:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:28:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:28:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:28:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:28:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:28:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:28:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:28:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:28:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:28:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:28:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:28:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:28:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:28:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:28:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:28:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:28:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:28:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:28:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:28:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:28:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:28:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:28:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:28:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:28:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:28:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:28:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:28:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:28:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:28:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:28:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:28:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:28:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:28:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:28:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:28:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:28:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:28:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:28:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:28:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:28:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:28:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:28:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:28:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:28:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:28:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:28:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:28:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:28:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:28:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:28:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:28:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:28:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:28:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:28:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:28:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:28:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:28:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:28:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:28:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:28:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:28:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:28:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:28:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:28:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:28:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:28:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:28:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:28:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:28:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:28:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:28:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:28:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:28:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:28:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:28:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:28:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:28:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:28:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:28:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:28:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:28:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:28:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:28:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:28:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:28:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:28:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:28:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:28:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:28:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:28:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:28:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:28:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:28:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:28:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:28:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:28:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:28:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:28:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:28:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:28:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:28:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:28:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:28:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:28:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:28:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:28:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:28:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:28:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:28:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:28:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:28:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:28:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:28:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:28:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:28:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:28:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:28:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:28:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:28:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:28:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:28:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:28:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:28:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:28:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:28:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:28:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:28:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:28:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:28:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:28:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:28:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:28:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:28:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:28:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:28:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:28:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:28:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:28:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:28:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:28:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:28:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:28:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:28:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:28:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:28:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:28:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:28:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:28:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:28:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:28:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:28:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:28:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:28:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:28:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:28:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:28:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:28:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:28:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:28:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:28:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:28:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:28:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:28:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:28:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:28:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:28:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:28:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:28:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:28:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:28:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:28:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:28:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:28:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:28:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:28:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:28:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:28:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:28:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:28:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:28:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:28:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:28:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:28:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:28:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:28:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:28:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:28:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:28:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:28:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:28:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:28:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:28:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:28:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:28:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:28:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:28:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:28:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:28:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:28:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:28:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:28:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:28:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:28:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:28:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:28:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:28:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:28:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:29:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:29:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:29:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:29:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:29:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:29:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:29:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:29:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:29:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:29:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:29:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:29:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:29:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:29:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:29:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:29:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:29:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:29:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:29:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:29:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:29:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:29:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:29:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:29:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:29:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:29:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:29:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:29:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:29:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:29:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:29:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:29:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:29:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:29:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:29:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:29:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:29:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:29:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:29:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:29:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:29:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:29:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:29:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:29:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:29:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:29:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:29:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:29:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:29:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:29:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:29:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:29:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:29:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:29:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:29:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:29:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:29:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:29:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:29:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:29:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:29:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:29:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:29:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:29:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:29:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:29:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:29:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:29:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:29:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:29:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:29:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:29:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:29:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:29:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:29:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:29:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:29:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:29:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:29:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:29:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:29:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:29:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:29:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:29:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:29:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:29:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:29:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:29:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:29:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:29:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:29:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:29:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:29:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:29:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:29:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:29:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:29:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:29:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:29:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:29:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:29:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:29:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:29:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:29:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:29:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:29:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:29:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:29:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:29:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:29:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:29:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:29:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:29:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:29:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:29:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:29:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:29:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:29:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:29:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:29:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:29:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:29:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:29:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:29:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:29:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:29:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:29:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:29:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:29:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:29:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:29:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:29:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:29:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:29:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:29:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:29:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:29:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:29:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:29:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:29:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:29:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:29:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:29:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:29:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:29:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:29:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:29:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:29:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:29:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:29:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:29:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:29:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:29:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:29:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:29:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:29:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:29:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:29:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:29:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:29:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:29:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:29:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:29:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:29:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:29:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:29:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:29:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:29:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:29:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:29:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:29:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:29:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:29:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:29:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:29:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:29:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:29:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:29:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:29:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:29:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:29:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:29:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:29:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:29:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:29:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:29:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:29:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:29:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:29:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:29:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:29:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:29:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:29:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:29:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:29:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:29:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:29:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:29:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:29:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:29:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:29:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:29:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:29:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:29:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:29:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:29:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:29:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:29:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:29:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:29:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:29:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:29:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:29:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:29:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:29:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:29:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:30:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:30:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:30:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:30:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:30:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:30:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:30:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:30:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:30:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:30:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:30:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:30:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:30:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:30:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:30:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:30:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:30:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:30:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:30:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:30:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:30:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:30:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:30:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:30:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:30:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:30:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:30:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:30:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:30:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:30:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:30:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:30:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:30:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:30:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:30:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:30:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:30:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:30:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:30:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:30:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:30:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:30:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:30:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:30:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:30:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:30:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:30:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:30:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:30:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:30:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:30:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:30:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:30:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:30:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:30:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:30:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:30:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:30:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:30:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:30:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:30:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:30:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:30:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:30:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:30:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:30:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:30:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:30:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:30:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:30:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:30:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:30:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:30:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:30:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:30:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:30:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:30:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:30:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:30:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:30:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:30:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:30:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:30:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:30:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:30:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:30:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:30:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:30:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:30:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:30:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:30:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:30:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:30:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:30:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:30:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:30:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:30:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:30:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:30:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:30:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:30:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:30:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:30:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:30:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:30:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:30:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:30:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:30:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:30:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:30:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:30:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:30:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:30:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:30:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:30:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:30:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:31:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:31:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:31:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:31:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:31:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:31:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:31:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:31:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:31:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:31:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:31:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:31:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:31:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:31:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:31:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:31:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:31:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:31:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:31:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:31:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:31:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:31:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:31:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:31:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:31:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:31:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:31:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:31:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:31:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:31:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:31:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:31:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:31:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:31:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:31:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:31:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:31:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:31:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:31:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:31:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:31:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:31:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:31:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:31:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:31:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:31:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:31:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:31:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:31:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:31:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:31:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:31:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:31:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:31:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:31:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:31:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:31:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:31:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:31:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:31:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:31:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:31:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:31:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:31:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:31:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:31:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:31:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:31:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:31:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:31:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:31:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:31:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:31:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:31:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:31:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:31:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:31:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:31:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:31:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:31:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:31:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:31:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:31:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:31:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:31:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:31:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:31:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:31:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:31:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:31:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:31:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:31:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:31:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:31:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:31:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:31:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:31:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:31:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:31:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:31:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:31:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:31:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:31:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:31:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:31:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:31:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:31:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:31:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:31:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:31:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:31:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:31:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:31:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:31:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:31:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:31:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:31:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:31:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:31:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:31:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:31:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:31:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:31:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:31:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:31:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:31:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:31:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:31:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:31:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:31:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:31:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:31:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:31:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:31:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:31:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:31:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:31:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:31:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:31:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:31:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:31:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:31:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:31:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:31:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:31:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:31:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:31:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:31:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:31:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:31:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:31:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:31:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:31:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:31:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:31:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:31:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:31:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:31:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:31:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:31:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:31:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:31:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:31:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:31:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:31:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:31:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:31:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:31:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:31:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:31:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:31:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:31:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:31:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:31:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:31:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:31:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:31:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:31:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:31:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:31:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:31:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:31:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:31:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:31:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:31:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:31:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:31:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:31:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:31:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:31:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:31:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:31:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:31:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:31:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:31:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:31:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:31:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:31:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:31:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:31:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:31:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:31:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:31:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:31:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:31:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:31:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:31:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:31:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:31:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:31:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:31:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:31:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:31:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:31:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:31:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:31:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:32:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:32:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:32:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:32:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:32:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:32:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:32:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:32:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:32:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:32:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:32:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:32:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:32:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:32:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:32:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:32:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:32:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:32:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:32:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:32:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:32:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:32:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:32:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:32:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:32:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:32:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:32:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:32:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:32:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:32:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:32:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:32:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:32:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:32:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:32:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:32:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:32:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:32:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:32:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:32:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:32:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:32:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:32:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:32:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:32:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:32:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:32:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:32:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:32:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:32:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:32:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:32:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:32:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:32:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:32:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:32:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:32:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:32:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:32:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:32:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:32:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:32:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:32:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:32:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:32:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:32:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:32:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:32:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:32:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:32:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:32:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:32:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:32:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:32:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:32:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:32:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:32:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:32:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:32:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:32:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:32:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:32:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:32:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:32:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:32:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:32:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:32:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:32:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:32:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:32:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:32:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:32:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:32:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:32:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:32:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:32:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:32:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:32:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:32:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:32:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:32:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:32:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:32:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:32:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:32:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:32:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:32:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:32:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:32:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:32:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:32:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:32:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:32:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:32:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:32:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:32:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:32:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:32:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:32:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:32:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:32:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:32:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:32:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:32:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:32:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:32:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:32:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:32:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:32:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:32:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:32:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:32:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:32:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:32:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:32:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:32:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:32:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:32:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:32:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:32:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:32:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:32:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:32:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:32:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:32:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:32:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:32:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:32:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:32:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:32:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:32:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:32:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:32:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:32:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:32:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:32:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:32:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:32:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:32:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:32:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:32:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:32:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:32:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:32:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:32:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:32:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:32:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:32:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:32:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:32:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:32:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:32:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:32:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:32:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:32:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:32:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:32:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:32:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:32:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:32:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:32:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:32:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:32:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:32:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:32:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:32:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:32:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:32:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:32:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:32:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:32:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:32:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:32:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:32:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:32:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:32:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:32:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:32:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:32:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:32:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:32:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:32:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:32:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:32:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:32:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:32:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:32:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:32:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:32:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:32:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:32:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:32:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:32:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:32:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:32:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:32:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:33:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:33:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:33:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:33:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:33:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:33:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:33:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:33:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:33:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:33:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:33:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:33:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:33:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:33:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:33:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:33:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:33:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:33:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:33:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:33:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:33:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:33:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:33:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:33:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:33:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:33:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:33:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:33:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:33:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:33:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:33:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:33:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:33:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:33:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:33:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:33:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:33:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:33:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:33:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:33:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:33:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:33:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:33:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:33:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:33:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:33:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:33:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:33:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:33:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:33:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:33:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:33:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:33:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:33:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:33:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:33:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:33:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:33:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:33:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:33:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:33:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:33:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:33:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:33:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:33:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:33:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:33:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:33:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:33:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:33:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:33:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:33:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:33:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:33:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:33:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:33:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:33:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:33:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:33:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:33:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:33:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:33:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:33:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:33:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:33:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:33:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:33:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:33:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:33:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:33:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:33:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:33:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:33:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:33:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:33:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:33:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:33:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:33:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:33:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:33:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:33:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:33:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:33:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:33:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:33:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:33:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:33:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:33:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:33:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:33:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:33:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:33:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:33:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:33:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:33:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:33:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:33:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:33:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:33:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:33:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:33:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:33:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:33:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:33:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:33:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:33:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:33:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:33:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:33:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:33:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:33:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:33:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:33:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:33:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:33:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:33:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:33:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:33:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:33:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:33:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:33:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:33:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:33:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:33:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:33:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:33:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:33:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:33:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:33:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:33:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:33:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:33:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:33:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:33:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:33:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:33:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:33:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:33:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:33:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:33:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:33:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:33:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:33:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:33:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:33:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:33:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:33:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:33:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:33:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:33:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:33:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:33:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:33:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:33:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:33:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:33:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:33:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:33:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:33:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:33:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:33:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:33:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:33:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:33:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:33:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:33:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:33:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:33:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:33:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:33:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:33:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:33:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:33:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:33:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:33:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:33:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:33:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:33:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:33:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:33:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:33:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:33:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:33:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:33:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:33:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:33:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:33:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:33:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:33:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:33:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:33:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:33:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:33:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:33:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:33:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:33:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:34:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:34:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:34:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:34:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:34:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:34:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:34:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:34:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:34:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:34:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:34:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:34:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:34:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:34:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:34:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:34:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:34:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:34:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:34:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:34:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:34:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:34:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:34:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:34:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:34:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:34:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:34:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:34:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:34:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:34:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:34:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:34:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:34:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:34:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:34:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:34:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:34:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:34:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:34:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:34:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:34:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:34:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:34:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:34:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:34:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:34:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:34:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:34:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:34:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:34:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:34:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:34:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:34:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:34:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:36:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:36:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:36:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:36:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:36:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:36:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:36:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:36:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:36:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:36:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:36:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:36:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:36:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:36:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:36:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:36:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:36:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:36:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:36:23 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 14:36:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:36:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:36:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:36:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:36:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:36:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:36:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:36:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:36:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:36:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:36:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:36:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:36:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:36:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:36:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:36:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:36:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:36:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:36:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:36:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:36:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:36:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:36:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:36:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:36:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:36:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:36:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:36:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:36:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:36:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:36:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:36:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:36:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:36:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:36:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:36:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:36:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:36:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:36:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:36:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:36:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:36:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:36:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:36:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:36:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:36:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:36:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:36:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:36:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:36:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:36:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:36:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:36:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:36:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:36:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:36:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:36:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:36:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:36:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:36:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:36:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:36:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:36:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:36:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:36:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:36:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:36:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:36:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:36:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:36:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:36:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:36:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:36:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:36:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:36:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:36:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:36:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:36:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:36:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:36:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:36:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:36:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:36:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:36:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:36:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:36:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:36:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:36:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:36:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:36:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:36:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:36:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:36:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:36:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:36:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:36:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:36:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:36:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:36:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:36:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:36:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:36:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:36:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:36:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:36:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:36:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:36:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:36:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:36:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:36:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:36:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:36:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:36:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:36:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:36:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:36:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:36:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:36:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:36:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:36:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:36:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:36:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:36:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:36:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:36:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:36:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:37:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:37:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:37:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:37:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:37:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:37:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:37:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:37:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:37:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:37:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:37:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:37:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:37:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:37:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:37:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:37:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:37:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:37:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:37:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:37:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:37:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:37:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:37:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:37:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:37:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:37:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:37:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:37:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:37:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:37:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:37:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:37:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:37:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:37:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:37:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:37:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:37:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:37:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:37:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:37:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:37:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:37:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:37:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:37:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:37:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:37:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:37:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:37:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:37:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:37:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:37:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:37:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:37:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:37:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:37:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:37:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:37:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:37:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:37:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:37:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:37:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:37:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:37:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:37:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:37:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:37:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:37:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:37:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:37:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:37:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:37:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:37:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:37:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:37:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:37:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:37:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:37:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:37:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:37:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:37:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:37:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:37:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:37:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:37:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:37:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:37:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:37:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:37:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:37:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:37:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:37:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:37:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:37:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:37:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:37:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:37:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:37:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:37:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:37:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:37:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:37:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:37:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:37:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:37:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:37:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:37:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:37:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:37:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:37:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:37:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:37:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:37:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:37:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:37:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:37:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:37:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:37:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:37:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:37:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:37:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:37:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:37:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:37:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:37:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:37:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:37:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:37:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:37:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:37:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:37:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:37:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:37:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:37:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:37:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:37:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:37:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:37:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:37:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:37:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:37:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:37:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:37:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:37:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:37:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:37:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:37:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:37:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:37:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:37:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:37:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:37:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:37:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:37:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:37:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:37:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:37:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:37:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:37:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:37:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:37:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:37:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:37:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:37:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:37:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:37:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:37:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:37:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:37:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:37:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:37:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:37:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:37:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:37:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:37:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:37:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:37:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:37:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:37:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:37:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:37:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:37:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:37:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:37:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:37:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:37:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:37:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:37:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:37:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:37:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:37:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:37:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:37:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:37:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:37:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:37:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:37:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:37:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:37:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:37:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:37:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:37:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:37:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:37:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:37:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:37:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:37:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:37:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:37:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:37:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:37:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:37:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:37:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:37:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:37:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:37:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:37:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:38:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:38:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:38:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:38:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:38:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:38:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:38:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:38:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:38:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:38:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:38:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:38:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:38:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:38:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:38:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:38:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:38:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:38:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:38:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:38:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:38:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:38:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:38:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:38:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:38:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:38:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:38:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:38:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:38:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:38:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:38:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:38:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:38:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:38:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:38:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:38:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:38:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:38:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:38:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:38:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:38:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:38:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:38:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:38:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:38:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:38:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:38:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:38:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:38:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:38:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:38:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:38:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:38:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:38:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:38:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:38:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:38:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:38:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:38:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:38:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:38:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:38:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:38:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:38:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:38:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:38:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:38:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:38:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:38:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:38:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:38:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:38:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:38:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:38:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:38:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:38:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:38:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:38:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:38:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:38:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:38:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:38:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:38:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:38:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:38:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:38:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:38:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:38:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:38:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:38:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:38:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:38:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:38:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:38:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:38:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:38:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:38:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:38:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:38:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:38:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:38:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:38:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:38:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:38:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:38:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:38:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:38:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:38:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:38:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:38:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:38:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:38:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:38:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:38:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:38:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:38:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:38:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:38:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:38:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:38:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:38:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:38:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:38:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:38:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:38:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:38:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:38:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:38:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:38:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:38:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:38:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:38:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:38:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:38:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:38:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:38:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:38:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:38:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:38:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:38:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:38:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:38:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:38:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:38:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:38:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:38:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:38:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:38:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:38:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:38:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:38:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:38:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:38:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:38:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:38:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:38:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:38:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:38:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:38:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:38:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:38:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:38:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:38:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:38:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:38:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:38:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:38:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:38:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:38:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:38:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:38:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:38:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:38:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:38:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:38:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:38:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:38:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:38:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:38:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:38:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:38:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:38:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:38:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:38:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:38:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:38:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:38:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:38:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:38:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:38:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:38:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:38:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:38:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:38:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:38:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:38:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:38:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:38:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:38:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:38:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:38:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:38:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:38:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:38:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:38:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:38:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:38:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:38:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:38:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:38:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:38:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:38:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:38:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:38:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:38:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:38:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:39:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:39:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:39:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:39:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:39:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:39:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:39:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:39:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:39:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:39:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:39:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:39:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:39:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:39:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:39:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:39:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:39:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:39:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:39:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:39:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:39:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:39:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:39:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:39:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:39:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:39:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:39:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:39:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:39:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:39:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:39:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:39:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:39:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:39:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:39:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:39:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:39:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:39:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:39:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:39:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:39:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:39:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:39:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:39:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:39:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:39:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:39:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:39:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:39:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:39:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:39:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:39:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:39:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:39:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:39:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:39:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:39:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:39:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:39:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:39:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:39:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:39:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:39:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:39:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:39:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:39:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:39:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:39:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:39:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:39:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:39:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:39:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:39:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:39:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:39:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:39:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:39:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:39:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:39:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:39:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:39:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:39:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:39:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:39:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:39:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:39:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:39:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:39:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:39:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:39:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:39:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:39:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:39:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:39:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:39:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:39:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:39:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:39:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:39:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:39:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:39:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:39:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:39:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:39:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:39:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:39:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:39:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:39:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:39:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:39:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:39:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:39:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:39:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:39:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:39:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:39:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:39:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:39:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:39:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:39:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:39:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:39:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:39:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:39:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:39:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:39:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:39:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:39:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:39:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:39:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:39:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:39:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:39:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:39:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:39:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:39:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:39:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:39:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:39:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:39:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:39:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:39:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:39:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:39:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:39:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:39:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:39:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:39:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:39:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:39:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:39:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:39:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:39:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:39:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:39:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:39:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:39:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:39:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:39:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:39:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:39:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:39:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:39:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:39:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:39:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:39:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:39:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:39:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:39:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:39:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:39:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:39:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:39:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:39:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:39:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:39:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:39:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:39:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:39:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:39:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:39:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:39:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:39:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:39:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:39:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:39:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:39:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:39:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:39:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:39:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:39:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:39:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:39:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:39:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:39:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:39:53 rdma06 systemd-journald[790]: Data hash table of /run/log/journal/95379d2f723847b68d9e976dee8967a9/system.journal has a fill level at 75.0 (163751 of 218332 items, 75497472 file size, 461 bytes per hash table item), suggesting rotation.
Apr 18 14:39:53 rdma06 systemd-journald[790]: /run/log/journal/95379d2f723847b68d9e976dee8967a9/system.journal: Journal header limits reached or header out-of-date, rotating.
Apr 18 14:39:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:39:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:39:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:39:53 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 18 14:39:53 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 18 14:39:54 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 18 14:39:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:39:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:39:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:39:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:39:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:39:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:39:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:39:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:39:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:39:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:39:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:39:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:39:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:39:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:39:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:39:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:39:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:39:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:40:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:40:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:40:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:40:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:40:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:40:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:40:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:40:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:40:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:40:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:40:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:40:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:40:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:40:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:40:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:40:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:40:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:40:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:40:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:40:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:40:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:40:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:40:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:40:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:40:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:40:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:40:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:40:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:40:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:40:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:40:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:40:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:40:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:40:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:40:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:40:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:40:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:40:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:40:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:40:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:40:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:40:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:40:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:40:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:40:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:40:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:40:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:40:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:40:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:40:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:40:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:40:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:40:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:40:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:40:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:40:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:40:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:40:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:40:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:40:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:40:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:40:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:40:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:40:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:40:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:40:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:40:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:40:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:40:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:40:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:40:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:40:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:40:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:40:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:40:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:40:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:40:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:40:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:40:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:40:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:40:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:40:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:40:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:40:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:40:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:40:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:40:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:40:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:40:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:40:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:40:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:40:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:40:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:40:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:40:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:40:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:40:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:40:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:40:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:40:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:40:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:40:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:40:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:40:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:40:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:40:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:40:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:40:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:40:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:40:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:40:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:40:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:40:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:40:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:40:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:40:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:40:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:40:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:40:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:40:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:40:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:40:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:40:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:40:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:40:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:40:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:40:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:40:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:40:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:40:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:40:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:40:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:40:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:40:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:40:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:40:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:40:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:40:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:40:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:40:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:40:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:40:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:40:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:40:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:40:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:40:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:40:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:40:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:40:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:40:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:40:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:40:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:40:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:40:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:40:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:40:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:40:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:40:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:40:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:40:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:40:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:40:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:40:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:40:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:40:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:40:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:40:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:40:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:40:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:40:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:40:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:40:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:40:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:40:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:40:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:40:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:40:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:40:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:40:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:40:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:40:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:40:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:40:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:40:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:40:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:40:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:40:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:40:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:40:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:40:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:40:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:40:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:40:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:40:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:40:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:40:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:40:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:40:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:40:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:40:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:40:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:40:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:40:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:40:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:40:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:40:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:40:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:40:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:40:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:40:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:40:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:40:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:40:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:40:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:40:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:40:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:41:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:41:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:41:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:41:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:41:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:41:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:41:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:41:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:41:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:41:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:41:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:41:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:41:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:41:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:41:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:41:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:41:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:41:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:41:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:41:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:41:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:41:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:41:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:41:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:41:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:41:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:41:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:41:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:41:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:41:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:41:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:41:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:41:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:41:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:41:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:41:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:41:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:41:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:41:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:41:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:41:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:41:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:41:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:41:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:41:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:41:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:41:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:41:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:41:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:41:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:41:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:41:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:41:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:41:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:41:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:41:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:41:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:41:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:41:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:41:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:41:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:41:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:41:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:41:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:41:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:41:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:41:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:41:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:41:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:41:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:41:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:41:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:41:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:41:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:41:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:41:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:41:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:41:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:41:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:41:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:41:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:41:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:41:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:41:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:41:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:41:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:41:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:41:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:41:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:41:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:41:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:41:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:41:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:41:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:41:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:41:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:41:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:41:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:41:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:41:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:41:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:41:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:41:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:41:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:41:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:41:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:41:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:41:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:41:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:41:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:41:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:41:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:41:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:41:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:41:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:41:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:41:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:41:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:41:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:41:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:41:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:41:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:41:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:41:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:41:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:41:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:41:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:41:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:41:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:41:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:41:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:41:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:41:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:41:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:41:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:41:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:41:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:41:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:41:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:41:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:41:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:41:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:41:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:41:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:41:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:41:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:41:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:41:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:41:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:41:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:41:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:41:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:41:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:41:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:41:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:41:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:41:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:41:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:41:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:41:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:41:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:41:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:41:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:41:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:41:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:41:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:41:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:41:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:41:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:41:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:41:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:41:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:41:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:41:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:41:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:41:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:41:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:41:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:41:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:41:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:41:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:41:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:41:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:41:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:41:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:41:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:41:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:41:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:41:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:41:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:41:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:41:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:41:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:41:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:41:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:41:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:41:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:41:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:41:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:41:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:41:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:41:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:41:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:41:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:41:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:41:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:41:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:41:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:41:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:41:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:41:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:41:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:41:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:41:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:41:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:41:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:42:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:42:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:42:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:42:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:42:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:42:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:42:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:42:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:42:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:42:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:42:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:42:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:42:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:42:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:42:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:42:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:42:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:42:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:42:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:42:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:42:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:42:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:42:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:42:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:42:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:42:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:42:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:42:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:42:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:42:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:42:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:42:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:42:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:42:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:42:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:42:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:42:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:42:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:42:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:42:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:42:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:42:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:42:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:42:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:42:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:42:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:42:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:42:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:42:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:42:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:42:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:42:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:42:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:42:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:42:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:42:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:42:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:42:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:42:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:42:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:42:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:42:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:42:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:42:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:42:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:42:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:42:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:42:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:42:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:42:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:42:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:42:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:42:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:42:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:42:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:42:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:42:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:42:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:42:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:42:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:42:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:42:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:42:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:42:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:42:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:42:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:42:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:42:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:42:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:42:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:42:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:42:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:42:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:42:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:42:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:42:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:42:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:42:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:42:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:42:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:42:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:42:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:42:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:42:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:42:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:42:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:42:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:42:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:42:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:42:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:42:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:42:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:42:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:42:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:42:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:42:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:42:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:42:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:42:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:42:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:42:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:42:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:42:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:42:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:42:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:42:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:42:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:42:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:42:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:42:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:42:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:42:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:42:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:42:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:42:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:42:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:42:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:42:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:42:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:42:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:42:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:42:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:42:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:42:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:42:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:42:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:42:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:42:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:42:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:42:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:42:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:42:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:42:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:42:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:42:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:42:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:42:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:42:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:42:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:42:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:42:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:42:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:42:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:42:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:42:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:42:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:42:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:42:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:42:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:42:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:42:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:42:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:42:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:42:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:42:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:42:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:42:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:42:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:42:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:42:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:42:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:42:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:42:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:42:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:42:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:42:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:42:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:42:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:42:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:42:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:42:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:42:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:42:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:42:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:42:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:42:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:42:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:42:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:42:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:42:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:42:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:42:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:42:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:42:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:42:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:42:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:42:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:42:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:42:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:42:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:42:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:42:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:42:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:42:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:42:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:42:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:43:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:43:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:43:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:43:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:43:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:43:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:43:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:43:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:43:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:43:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:43:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:43:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:43:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:43:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:43:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:43:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:43:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:43:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:43:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:43:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:43:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:43:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:43:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:43:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:43:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:43:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:43:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:43:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:43:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:43:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:43:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:43:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:43:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:43:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:43:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:43:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:43:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:43:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:43:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:43:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:43:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:43:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:43:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:43:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:43:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:43:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:43:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:43:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:43:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:43:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:43:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:43:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:43:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:43:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:43:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:43:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:43:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:43:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:43:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:43:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:43:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:43:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:43:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:43:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:43:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:43:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:43:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:43:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:43:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:43:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:43:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:43:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:43:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:43:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:43:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:43:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:43:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:43:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:43:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:43:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:43:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:43:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:43:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:43:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:43:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:43:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:43:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:43:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:43:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:43:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:43:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:43:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:43:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:43:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:43:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:43:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:43:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:43:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:43:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:43:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:43:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:43:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:43:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:43:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:43:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:43:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:43:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:43:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:43:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:43:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:43:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:43:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:43:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:43:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:43:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:43:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:43:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:43:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:43:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:43:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:43:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:43:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:43:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:43:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:43:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:43:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:43:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:43:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:43:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:43:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:43:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:43:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:43:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:43:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:43:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:43:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:43:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:43:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:43:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:43:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:43:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:43:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:43:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:43:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:43:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:43:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:43:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:43:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:43:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:43:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:43:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:43:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:43:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:43:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:43:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:43:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:43:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:43:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:43:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:43:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:43:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:43:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:43:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:43:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:43:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:43:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:43:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:43:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:43:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:43:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:43:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:43:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:43:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:43:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:43:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:43:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:43:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:43:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:43:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:43:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:43:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:43:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:43:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:43:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:43:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:43:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:43:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:43:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:43:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:43:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:43:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:43:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:43:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:43:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:43:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:43:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:43:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:43:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:43:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:43:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:43:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:43:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:43:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:43:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:43:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:43:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:43:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:43:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:43:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:43:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:43:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:43:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:43:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:43:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:43:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:43:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:44:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:44:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:44:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:44:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:44:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:44:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:44:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:44:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:44:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:44:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:44:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:44:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:44:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:44:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:44:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:44:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:44:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:44:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:44:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:44:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:44:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:44:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:44:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:44:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:44:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:44:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:44:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:44:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:44:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:44:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:44:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:44:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:44:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:44:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:44:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:44:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:44:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:44:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:44:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:44:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:44:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:44:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:44:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:44:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:44:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:44:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:44:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:44:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:44:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:44:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:44:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:44:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:44:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:44:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:44:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:44:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:44:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:44:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:44:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:44:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:44:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:44:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:44:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:44:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:44:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:44:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:44:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:44:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:44:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:44:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:44:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:44:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:46:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:46:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:46:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:46:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:46:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:46:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:46:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:46:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:46:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:46:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:46:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:46:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:46:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:46:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:46:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:46:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:46:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:46:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:46:24 rdma06 rsyslogd[1041]: imjournal: 5087 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 14:46:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:46:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:46:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:46:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:46:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:46:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:46:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:46:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:46:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:46:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:46:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:46:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:46:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:46:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:46:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:46:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:46:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:46:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:46:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:46:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:46:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:46:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:46:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:46:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:46:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:46:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:46:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:46:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:46:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:46:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:46:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:46:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:46:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:46:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:46:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:46:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:46:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:46:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:46:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:46:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:46:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:46:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:46:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:46:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:46:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:46:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:46:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:46:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:46:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:46:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:46:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:46:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:46:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:46:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:46:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:46:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:46:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:46:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:46:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:46:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:46:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:46:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:46:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:46:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:46:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:46:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:46:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:46:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:46:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:46:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:46:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:46:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:46:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:46:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:46:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:46:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:46:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:46:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:46:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:46:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:46:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:46:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:46:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:46:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:46:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:46:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:46:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:46:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:46:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:46:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:46:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:46:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:46:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:46:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:46:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:46:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:46:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:46:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:46:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:46:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:46:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:46:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:46:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:46:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:46:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:46:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:46:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:46:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:46:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:46:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:46:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:46:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:46:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:46:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:46:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:46:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:46:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:46:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:46:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:46:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:46:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:46:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:46:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:46:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:46:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:46:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:47:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:47:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:47:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:47:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:47:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:47:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:47:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:47:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:47:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:47:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:47:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:47:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:47:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:47:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:47:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:47:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:47:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:47:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:47:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:47:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:47:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:47:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:47:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:47:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:47:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:47:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:47:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:47:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:47:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:47:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:47:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:47:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:47:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:47:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:47:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:47:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:47:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:47:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:47:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:47:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:47:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:47:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:47:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:47:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:47:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:47:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:47:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:47:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:47:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:47:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:47:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:47:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:47:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:47:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:47:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:47:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:47:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:47:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:47:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:47:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:47:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:47:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:47:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:47:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:47:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:47:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:47:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:47:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:47:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:47:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:47:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:47:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:47:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:47:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:47:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:47:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:47:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:47:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:47:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:47:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:47:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:47:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:47:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:47:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:47:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:47:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:47:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:47:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:47:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:47:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:47:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:47:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:47:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:47:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:47:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:47:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:47:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:47:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:47:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:47:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:47:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:47:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:47:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:47:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:47:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:47:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:47:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:47:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:47:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:47:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:47:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:47:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:47:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:47:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:47:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:47:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:47:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:47:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:47:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:47:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:47:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:47:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:47:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:47:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:47:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:47:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:47:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:47:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:47:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:47:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:47:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:47:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:47:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:47:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:47:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:47:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:47:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:47:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:47:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:47:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:47:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:47:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:47:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:47:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:47:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:47:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:47:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:47:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:47:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:47:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:47:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:47:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:47:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:47:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:47:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:47:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:47:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:47:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:47:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:47:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:47:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:47:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:47:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:47:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:47:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:47:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:47:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:47:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:47:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:47:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:47:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:47:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:47:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:47:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:47:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:47:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:47:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:47:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:47:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:47:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:47:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:47:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:47:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:47:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:47:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:47:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:47:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:47:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:47:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:47:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:47:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:47:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:47:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:47:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:47:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:47:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:47:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:47:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:47:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:47:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:47:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:47:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:47:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:47:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:47:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:47:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:47:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:47:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:47:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:47:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:47:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:47:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:47:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:47:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:47:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:47:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:48:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:48:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:48:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:48:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:48:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:48:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:48:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:48:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:48:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:48:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:48:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:48:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:48:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:48:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:48:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:48:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:48:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:48:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:48:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:48:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:48:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:48:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:48:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:48:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:48:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:48:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:48:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:48:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:48:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:48:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:48:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:48:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:48:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:48:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:48:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:48:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:48:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:48:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:48:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:48:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:48:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:48:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:48:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:48:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:48:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:48:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:48:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:48:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:48:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:48:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:48:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:48:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:48:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:48:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:48:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:48:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:48:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:48:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:48:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:48:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:48:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:48:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:48:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:48:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:48:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:48:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:48:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:48:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:48:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:48:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:48:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:48:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:48:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:48:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:48:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:48:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:48:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:48:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:48:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:48:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:48:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:48:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:48:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:48:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:48:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:48:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:48:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:48:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:48:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:48:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:48:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:48:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:48:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:48:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:48:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:48:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:48:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:48:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:48:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:48:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:48:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:48:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:48:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:48:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:48:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:48:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:48:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:48:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:48:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:48:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:48:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:48:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:48:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:48:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:48:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:48:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:48:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:48:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:48:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:48:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:48:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:48:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:48:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:48:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:48:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:48:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:48:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:48:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:48:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:48:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:48:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:48:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:48:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:48:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:48:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:48:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:48:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:48:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:48:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:48:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:48:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:48:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:48:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:48:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:48:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:48:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:48:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:48:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:48:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:48:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:48:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:48:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:48:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:48:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:48:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:48:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:48:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:48:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:48:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:48:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:48:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:48:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:48:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:48:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:48:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:48:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:48:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:48:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:48:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:48:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:48:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:48:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:48:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:48:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:48:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:48:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:48:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:48:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:48:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:48:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:48:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:48:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:48:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:48:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:48:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:48:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:48:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:48:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:48:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:48:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:48:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:48:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:48:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:48:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:48:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:48:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:48:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:48:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:48:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:48:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:48:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:48:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:48:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:48:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:48:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:48:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:48:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:48:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:48:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:48:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:48:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:48:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:48:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:48:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:48:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:48:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:49:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:49:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:49:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:49:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:49:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:49:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:49:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:49:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:49:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:49:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:49:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:49:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:49:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:49:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:49:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:49:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:49:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:49:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:49:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:49:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:49:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:49:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:49:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:49:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:49:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:49:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:49:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:49:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:49:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:49:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:49:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:49:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:49:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:49:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:49:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:49:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:49:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:49:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:49:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:49:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:49:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:49:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:49:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:49:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:49:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:49:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:49:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:49:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:49:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:49:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:49:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:49:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:49:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:49:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:49:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:49:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:49:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:49:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:49:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:49:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:49:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:49:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:49:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:49:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:49:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:49:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:49:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:49:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:49:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:49:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:49:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:49:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:49:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:49:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:49:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:49:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:49:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:49:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:49:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:49:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:49:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:49:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:49:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:49:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:49:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:49:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:49:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:49:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:49:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:49:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:49:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:49:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:49:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:49:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:49:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:49:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:49:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:49:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:49:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:49:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:49:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:49:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:49:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:49:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:49:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:49:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:49:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:49:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:49:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:49:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:49:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:49:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:49:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:49:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:49:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:49:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:49:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:49:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:49:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:49:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:49:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:49:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:49:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:49:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:49:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:49:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:49:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:49:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:49:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:49:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:49:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:49:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:49:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:49:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:49:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:49:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:49:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:49:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:49:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:49:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:49:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:49:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:49:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:49:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:49:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:49:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:49:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:49:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:49:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:49:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:49:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:49:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:49:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:49:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:49:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:49:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:49:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:49:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:49:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:49:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:49:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:49:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:49:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:49:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:49:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:49:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:49:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:49:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:49:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:49:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:49:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:49:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:49:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:49:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:49:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:49:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:49:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:49:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:49:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:49:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:49:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:49:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:49:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:49:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:49:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:49:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:49:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:49:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:49:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:49:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:49:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:49:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:49:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:49:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:49:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:49:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:49:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:49:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:49:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:49:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:49:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:49:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:49:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:49:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:49:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:49:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:49:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:49:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:49:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:49:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:49:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:49:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:49:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:49:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:49:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:49:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:50:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:50:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:50:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:50:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:50:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:50:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:50:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:50:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:50:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:50:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:50:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:50:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:50:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:50:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:50:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:50:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:50:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:50:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:50:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:50:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:50:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:50:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:50:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:50:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:50:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:50:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:50:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:50:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:50:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:50:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:50:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:50:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:50:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:50:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:50:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:50:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:50:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:50:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:50:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:50:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:50:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:50:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:50:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:50:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:50:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:50:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:50:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:50:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:50:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:50:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:50:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:50:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:50:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:50:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:50:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:50:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:50:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:50:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:50:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:50:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:50:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:50:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:50:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:50:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:50:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:50:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:50:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:50:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:50:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:50:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:50:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:50:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:50:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:50:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:50:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:50:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:50:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:50:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:50:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:50:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:50:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:50:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:50:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:50:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:50:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:50:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:50:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:50:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:50:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:50:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:50:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:50:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:50:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:50:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:50:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:50:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:50:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:50:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:50:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:50:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:50:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:50:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:50:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:50:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:50:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:50:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:50:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:50:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:50:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:50:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:50:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:50:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:50:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:50:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:50:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:50:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:50:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:50:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:50:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:50:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:50:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:50:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:50:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:50:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:50:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:50:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:50:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:50:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:50:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:50:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:50:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:50:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:50:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:50:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:50:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:50:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:50:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:50:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:50:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:50:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:50:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:50:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:50:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:50:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:50:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:50:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:50:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:50:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:50:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:50:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:50:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:50:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:50:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:50:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:50:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:50:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:50:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:50:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:50:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:50:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:50:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:50:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:50:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:50:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:50:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:50:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:50:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:50:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:50:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:50:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:50:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:50:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:50:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:50:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:50:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:50:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:50:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:50:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:50:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:50:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:50:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:50:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:50:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:50:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:50:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:50:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:50:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:50:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:50:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:50:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:50:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:50:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:50:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:50:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:50:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:50:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:50:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:50:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:50:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:50:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:50:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:50:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:50:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:50:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:50:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:50:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:50:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:50:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:50:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:50:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:50:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:50:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:50:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:50:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:50:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:50:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:51:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:51:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:51:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:51:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:51:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:51:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:51:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:51:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:51:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:51:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:51:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:51:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:51:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:51:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:51:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:51:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:51:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:51:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:51:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:51:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:51:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:51:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:51:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:51:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:51:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:51:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:51:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:51:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:51:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:51:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:51:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:51:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:51:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:51:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:51:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:51:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:51:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:51:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:51:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:51:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:51:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:51:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:51:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:51:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:51:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:51:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:51:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:51:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:51:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:51:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:51:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:51:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:51:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:51:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:51:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:51:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:51:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:51:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:51:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:51:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:51:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:51:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:51:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:51:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:51:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:51:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:51:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:51:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:51:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:51:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:51:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:51:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:51:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:51:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:51:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:51:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:51:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:51:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:51:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:51:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:51:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:51:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:51:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:51:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:51:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:51:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:51:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:51:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:51:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:51:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:51:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:51:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:51:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:51:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:51:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:51:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:51:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:51:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:51:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:51:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:51:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:51:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:51:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:51:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:51:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:51:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:51:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:51:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:51:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:51:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:51:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:51:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:51:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:51:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:51:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:51:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:51:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:51:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:51:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:51:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:51:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:51:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:51:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:51:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:51:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:51:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:51:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:51:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:51:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:51:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:51:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:51:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:51:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:51:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:51:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:51:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:51:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:51:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:51:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:51:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:51:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:51:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:51:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:51:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:51:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:51:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:52:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:52:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:52:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:52:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:52:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:52:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:52:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:52:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:52:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:52:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:52:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:52:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:52:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:52:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:52:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:52:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:52:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:52:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:52:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:52:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:52:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:52:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:52:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:52:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:52:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:52:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:52:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:52:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:52:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:52:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:52:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:52:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:52:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:52:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:52:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:52:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:52:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:52:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:52:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:52:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:52:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:52:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:52:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:52:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:52:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:52:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:52:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:52:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:52:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:52:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:52:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:52:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:52:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:52:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:52:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:52:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:52:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:52:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:52:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:52:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:52:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:52:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:52:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:52:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:52:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:52:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:52:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:52:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:52:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:52:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:52:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:52:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:52:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:52:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:52:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:52:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:52:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:52:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:52:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:52:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:52:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:52:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:52:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:52:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:52:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:52:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:52:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:52:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:52:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:52:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:52:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:52:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:52:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:52:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:52:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:52:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:52:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:52:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:52:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:52:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:52:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:52:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:52:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:52:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:52:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:52:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:52:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:52:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:52:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:52:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:52:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:52:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:52:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:52:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:52:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:52:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:52:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:52:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:52:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:52:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:52:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:52:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:52:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:52:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:52:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:52:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:52:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:52:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:52:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:52:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:52:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:52:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:52:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:52:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:52:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:52:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:52:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:52:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:52:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:52:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:52:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:52:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:52:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:52:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:52:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:52:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:52:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:52:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:52:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:52:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:52:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:52:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:52:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:52:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:52:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:52:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:52:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:52:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:52:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:52:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:52:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:52:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:52:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:52:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:52:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:52:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:52:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:52:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:52:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:52:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:52:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:52:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:52:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:52:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:52:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:52:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:53:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:53:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:53:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:53:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:53:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:53:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:53:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:53:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:53:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:53:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:53:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:53:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:53:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:53:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:53:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:53:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:53:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:53:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:53:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:53:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:53:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:53:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:53:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:53:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:53:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:53:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:53:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:53:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:53:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:53:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:53:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:53:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:53:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:53:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:53:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:53:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:53:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:53:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:53:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:53:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:53:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:53:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:53:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:53:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:53:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:53:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:53:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:53:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:53:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:53:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:53:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:53:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:53:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:53:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:53:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:53:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:53:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:53:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:53:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:53:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:53:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:53:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:53:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:53:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:53:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:53:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:53:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:53:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:53:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:53:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:53:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:53:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:53:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:53:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:53:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:53:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:53:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:53:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:53:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:53:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:53:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:53:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:53:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:53:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:53:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:53:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:53:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:53:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:53:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:53:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:53:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:53:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:53:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:53:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:53:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:53:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:53:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:53:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:53:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:53:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:53:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:53:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:53:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:53:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:53:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:53:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:53:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:53:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:53:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:53:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:53:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:53:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:53:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:53:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:53:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:53:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:53:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:53:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:53:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:53:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:53:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:53:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:53:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:53:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:53:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:53:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:53:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:53:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:53:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:53:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:53:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:53:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:53:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:53:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:53:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:53:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:53:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:53:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:53:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:53:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:53:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:53:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:53:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:53:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:53:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:53:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:53:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:54:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:54:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:54:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:54:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:54:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:54:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:54:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:54:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:54:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:54:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:54:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:54:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:54:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:54:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:54:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:54:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:54:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:54:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:54:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:54:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:54:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:54:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:54:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:54:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:54:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:54:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:54:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:54:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:54:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:54:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:54:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:54:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:56:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:56:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:56:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:56:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:56:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:56:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:56:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:56:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:56:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:56:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:56:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:56:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:56:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:56:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:56:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:56:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:56:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:56:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:56:29 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 14:56:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:56:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:56:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:56:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:56:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:56:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:56:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:56:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:56:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:56:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:56:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:56:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:56:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:56:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:56:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:56:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:56:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:56:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:56:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:56:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:56:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:56:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:56:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:56:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:56:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:56:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:56:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:56:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:56:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:56:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:56:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:56:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:56:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:56:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:56:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:56:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:56:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:56:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:56:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:56:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:56:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:56:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:56:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:56:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:56:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:56:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:56:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:56:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:56:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:56:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:56:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:56:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:56:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:56:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:56:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:56:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:56:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:56:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:56:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:56:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:56:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:56:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:56:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:56:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:56:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:56:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:56:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:56:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:56:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:56:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:56:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:56:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:56:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:56:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:56:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:56:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:56:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:56:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:56:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:56:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:56:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:56:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:56:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:56:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:56:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:56:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:56:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:56:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:56:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:56:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:56:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:56:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:56:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:56:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:56:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:56:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:56:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:56:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:56:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:56:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:56:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:56:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:56:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:56:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:56:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:56:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:56:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:56:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:57:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:57:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:57:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:57:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:57:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:57:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:57:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:57:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:57:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:57:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:57:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:57:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:57:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:57:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:57:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:57:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:57:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:57:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:57:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:57:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:57:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:57:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:57:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:57:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:57:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:57:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:57:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:57:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:57:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:57:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:57:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:57:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:57:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:57:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:57:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:57:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:57:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:57:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:57:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:57:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:57:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:57:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:57:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:57:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:57:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:57:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:57:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:57:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:57:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:57:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:57:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:57:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:57:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:57:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:57:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:57:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:57:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:57:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:57:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:57:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:57:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:57:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:57:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:57:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:57:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:57:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:57:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:57:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:57:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:57:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:57:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:57:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:57:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:57:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:57:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:57:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:57:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:57:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:57:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:57:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:57:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:57:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:57:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:57:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:57:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:57:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:57:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:57:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:57:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:57:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:57:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:57:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:57:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:57:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:57:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:57:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:57:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:57:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:57:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:57:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:57:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:57:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:57:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:57:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:57:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:57:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:57:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:57:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:57:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:57:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:57:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:57:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:57:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:57:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:57:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:57:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:57:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:57:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:57:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:57:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:57:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:57:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:57:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:57:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:57:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:57:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:57:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:57:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:57:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:57:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:57:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:57:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:57:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:57:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:57:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:57:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:57:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:57:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:57:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:57:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:57:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:57:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:57:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:57:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:57:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:57:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:57:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:57:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:57:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:57:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:57:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:57:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:57:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:57:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:57:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:57:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:57:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:57:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:57:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:57:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:57:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:57:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:57:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:57:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:57:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:57:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:57:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:57:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:57:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:57:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:57:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:57:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:57:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:57:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:57:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:57:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:57:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:57:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:57:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:57:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:57:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:57:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:57:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:57:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:57:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:57:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:57:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:57:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:57:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:57:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:57:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:57:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:57:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:57:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:57:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:57:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:57:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:57:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:57:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:57:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:57:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:57:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:57:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:57:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:57:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:57:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:57:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:57:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:57:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:57:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:57:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:57:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:57:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:57:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:57:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:57:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:58:00 rdma06 systemd[1]: Starting dnf makecache...
Apr 18 14:58:01 rdma06 dnf[1194281]: Updating Subscription Management repositories.
Apr 18 14:58:01 rdma06 dnf[1194281]: Unable to read consumer identity
Apr 18 14:58:01 rdma06 dnf[1194281]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 18 14:58:01 rdma06 dnf[1194281]: Metadata cache refreshed recently.
Apr 18 14:58:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 18 14:58:01 rdma06 systemd[1]: Finished dnf makecache.
Apr 18 14:58:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:58:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:58:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:58:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:58:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:58:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:58:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:58:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:58:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:58:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:58:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:58:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:58:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:58:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:58:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:58:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:58:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:58:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:58:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:58:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:58:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:58:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:58:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:58:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:58:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:58:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:58:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:58:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:58:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:58:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:58:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:58:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:58:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:58:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:58:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:58:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:58:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:58:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:58:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:58:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:58:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:58:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:58:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:58:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:58:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:58:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:58:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:58:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:58:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:58:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:58:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:58:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:58:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:58:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:58:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:58:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:58:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:58:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:58:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:58:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:58:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:58:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:58:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:58:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:58:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:58:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:58:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:58:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:58:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:58:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:58:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:58:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:58:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:58:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:58:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:58:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:58:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:58:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:58:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:58:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:58:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:58:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:58:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:58:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:58:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:58:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:58:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:58:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:58:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:58:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:58:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:58:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:58:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:58:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:58:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:58:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:58:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:58:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:58:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:58:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:58:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:58:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:58:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:58:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:58:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:58:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:58:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:58:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:58:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:58:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:58:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:58:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:58:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:58:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:58:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:58:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:58:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:58:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:58:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:58:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:58:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:58:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:58:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:58:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:58:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:58:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:58:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:58:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:58:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:58:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:58:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:58:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:58:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:58:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:58:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:58:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:58:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:58:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:58:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:58:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:58:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:58:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:58:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:58:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:58:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:58:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:58:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:58:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:58:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:58:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:58:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:58:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:58:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:58:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:58:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:58:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:58:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:58:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:58:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:58:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:58:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:58:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:58:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:58:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:58:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:58:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:58:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:58:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:58:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:58:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:58:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:58:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:58:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:58:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:58:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:58:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:58:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:58:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:58:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:58:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:58:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:58:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:58:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:58:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:58:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:58:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:58:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:58:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:58:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:58:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:58:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:58:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:58:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:58:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:58:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:58:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:58:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:58:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:58:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:58:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:58:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:58:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:58:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:58:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:58:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:58:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:58:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:58:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:58:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:58:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:58:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:58:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:58:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:58:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:58:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:58:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:59:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:59:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:59:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:59:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:59:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:59:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:59:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:59:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:59:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:59:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:59:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:59:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:59:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:59:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:59:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:59:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:59:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:59:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:59:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:59:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:59:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:59:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:59:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:59:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:59:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:59:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:59:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:59:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:59:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:59:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:59:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:59:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:59:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:59:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:59:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:59:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:59:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:59:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:59:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:59:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:59:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:59:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:59:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:59:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:59:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:59:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:59:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:59:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:59:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:59:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:59:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:59:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:59:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:59:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:59:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:59:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:59:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:59:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:59:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:59:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:59:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:59:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:59:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:59:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:59:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:59:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:59:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:59:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:59:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:59:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:59:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:59:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:59:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:59:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:59:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:59:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:59:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:59:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:59:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:59:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:59:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:59:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:59:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:59:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:59:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:59:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:59:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:59:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:59:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:59:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:59:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:59:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:59:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:59:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:59:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:59:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:59:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:59:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:59:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:59:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:59:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:59:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:59:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:59:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:59:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:59:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:59:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:59:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:59:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:59:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:59:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:59:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:59:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:59:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:59:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:59:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:59:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:59:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:59:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:59:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:59:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:59:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:59:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:59:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:59:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:59:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:59:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:59:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:59:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:59:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:59:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:59:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:59:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:59:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:59:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:59:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:59:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:59:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 14:59:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 14:59:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:59:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:59:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:59:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 14:59:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 14:59:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 14:59:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 14:59:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 14:59:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 14:59:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 14:59:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 14:59:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 14:59:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 14:59:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 14:59:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 14:59:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 14:59:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:00:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:00:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:00:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:00:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:00:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:00:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:00:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:00:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:00:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:00:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:00:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:00:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:00:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:00:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:00:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:00:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:00:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:00:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:00:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:00:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:00:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:00:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:00:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:00:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:00:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:00:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:00:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:00:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:00:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:00:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:00:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:00:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:00:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:00:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:00:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:00:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:00:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:00:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:00:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:00:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:00:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:00:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:00:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:00:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:00:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:00:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:00:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:00:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:00:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:00:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:00:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:00:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:00:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:00:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:00:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:00:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:00:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:00:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:00:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:00:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:00:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:00:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:00:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:00:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:00:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:00:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:00:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:00:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:00:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:00:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:00:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:00:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:00:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:00:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:00:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:00:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:00:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:00:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:00:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:00:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:00:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:00:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:00:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:00:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:00:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:00:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:00:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:00:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:00:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:00:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:00:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:00:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:00:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:00:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:00:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:00:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:00:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:00:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:00:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:00:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:00:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:00:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:00:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:00:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:00:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:00:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:00:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:00:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:00:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:00:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:00:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:00:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:00:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:00:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:00:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:00:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:00:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:00:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:00:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:00:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:00:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:00:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:00:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:00:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:00:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:00:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:00:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:00:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:00:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:00:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:00:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:00:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:00:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:00:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:00:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:00:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:00:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:00:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:00:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:00:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:00:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:00:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:00:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:00:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:00:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:00:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:00:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:00:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:00:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:00:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:00:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:00:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:00:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:00:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:00:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:00:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:00:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:00:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:00:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:00:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:00:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:00:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:00:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:00:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:00:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:00:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:00:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:00:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:00:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:00:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:00:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:00:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:00:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:00:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:00:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:00:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:00:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:00:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:00:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:00:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:00:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:00:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:00:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:00:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:00:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:00:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:01:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:01:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:01:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:01:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:01:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:01:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:01:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:01:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:01:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:01:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:01:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:01:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:01:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:01:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:01:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:01:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:01:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:01:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:01:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:01:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:01:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:01:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:01:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:01:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:01:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:01:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:01:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:01:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:01:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:01:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:01:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:01:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:01:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:01:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:01:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:01:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:01:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:01:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:01:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:01:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:01:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:01:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:01:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:01:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:01:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:01:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:01:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:01:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:01:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:01:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:01:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:01:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:01:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:01:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:01:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:01:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:01:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:01:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:01:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:01:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:01:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:01:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:01:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:01:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:01:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:01:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:01:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:01:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:01:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:01:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:01:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:01:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:01:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:01:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:01:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:01:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:01:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:01:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:01:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:01:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:01:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:01:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:01:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:01:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:01:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:01:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:01:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:01:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:01:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:01:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:01:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:01:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:01:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:01:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:01:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:01:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:01:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:01:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:01:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:01:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:01:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:01:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:01:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:01:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:01:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:01:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:01:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:01:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:01:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:01:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:01:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:01:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:01:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:01:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:01:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:01:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:01:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:01:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:01:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:01:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:01:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:01:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:01:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:01:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:01:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:01:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:01:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:01:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:01:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:01:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:01:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:01:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:01:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:01:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:01:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:01:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:01:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:01:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:01:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:01:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:01:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:01:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:01:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:01:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:01:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:01:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:01:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:01:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:01:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:01:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:01:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:01:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:01:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:01:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:01:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:01:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:01:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:01:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:01:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:01:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:01:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:01:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:01:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:01:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:01:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:01:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:01:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:01:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:01:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:01:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:01:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:01:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:01:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:01:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:01:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:01:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:01:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:01:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:01:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:01:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:01:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:01:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:01:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:01:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:01:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:01:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:01:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:01:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:01:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:01:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:01:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:01:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:01:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:01:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:01:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:01:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:01:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:01:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:01:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:01:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:01:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:01:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:01:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:01:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:01:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:01:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:01:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:01:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:01:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:01:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:01:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:01:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:01:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:01:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:01:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:01:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:02:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:02:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:02:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:02:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:02:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:02:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:02:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:02:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:02:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:02:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:02:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:02:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:02:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:02:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:02:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:02:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:02:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:02:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:02:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:02:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:02:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:02:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:02:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:02:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:02:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:02:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:02:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:02:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:02:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:02:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:02:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:02:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:02:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:02:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:02:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:02:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:02:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:02:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:02:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:02:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:02:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:02:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:02:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:02:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:02:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:02:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:02:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:02:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:02:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:02:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:02:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:02:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:02:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:02:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:02:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:02:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:02:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:02:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:02:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:02:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:02:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:02:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:02:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:02:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:02:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:02:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:02:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:02:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:02:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:02:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:02:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:02:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:02:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:02:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:02:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:02:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:02:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:02:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:02:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:02:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:02:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:02:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:02:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:02:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:02:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:02:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:02:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:02:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:02:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:02:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:02:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:02:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:02:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:02:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:02:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:02:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:02:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:02:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:02:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:02:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:02:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:02:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:02:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:02:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:02:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:02:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:02:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:02:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:02:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:02:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:02:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:02:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:02:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:02:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:02:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:02:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:02:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:02:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:02:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:02:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:02:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:02:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:02:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:02:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:02:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:02:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:02:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:02:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:02:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:02:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:02:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:02:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:02:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:02:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:02:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:02:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:02:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:02:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:02:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:02:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:02:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:02:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:02:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:02:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:02:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:02:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:02:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:02:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:02:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:02:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:02:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:02:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:02:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:02:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:02:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:02:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:02:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:02:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:02:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:02:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:02:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:02:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:02:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:02:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:02:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:02:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:02:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:02:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:02:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:02:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:02:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:02:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:02:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:02:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:02:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:02:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:02:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:02:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:02:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:02:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:02:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:02:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:02:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:02:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:02:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:02:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:02:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:02:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:02:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:02:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:02:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:02:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:02:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:02:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:02:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:02:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:02:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:02:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:02:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:02:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:02:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:02:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:02:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:02:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:02:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:02:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:02:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:02:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:02:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:02:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:02:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:02:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:02:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:02:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:02:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:02:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:03:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:03:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:03:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:03:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:03:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:03:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:03:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:03:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:03:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:03:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:03:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:03:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:03:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:03:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:03:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:03:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:03:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:03:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:03:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:03:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:03:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:03:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:03:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:03:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:03:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:03:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:03:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:03:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:03:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:03:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:03:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:03:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:03:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:03:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:03:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:03:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:03:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:03:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:03:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:03:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:03:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:03:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:03:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:03:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:03:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:03:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:03:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:03:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:03:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:03:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:03:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:03:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:03:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:03:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:03:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:03:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:03:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:03:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:03:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:03:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:03:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:03:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:03:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:03:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:03:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:03:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:03:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:03:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:03:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:03:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:03:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:03:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:03:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:03:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:03:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:03:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:03:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:03:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:03:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:03:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:03:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:03:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:03:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:03:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:03:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:03:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:03:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:03:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:03:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:03:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:03:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:03:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:03:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:03:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:03:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:03:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:03:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:03:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:03:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:03:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:03:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:03:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:03:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:03:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:03:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:03:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:03:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:03:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:03:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:03:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:03:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:03:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:03:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:03:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:03:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:03:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:03:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:03:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:03:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:03:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:03:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:03:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:03:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:03:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:03:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:03:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:03:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:03:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:03:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:03:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:03:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:03:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:03:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:03:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:03:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:03:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:03:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:03:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:03:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:03:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:03:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:03:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:03:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:03:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:03:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:03:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:03:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:03:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:03:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:03:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:03:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:03:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:03:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:03:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:03:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:03:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:03:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:03:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:03:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:03:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:03:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:03:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:03:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:03:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:03:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:03:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:03:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:03:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:03:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:03:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:03:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:03:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:03:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:03:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:03:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:03:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:03:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:03:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:03:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:03:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:03:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:03:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:03:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:03:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:03:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:03:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:03:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:03:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:03:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:03:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:03:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:03:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:03:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:03:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:03:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:03:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:03:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:03:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:03:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:03:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:03:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:03:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:03:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:03:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:03:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:03:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:03:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:03:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:03:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:03:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:03:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:03:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:03:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:03:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:03:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:03:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:04:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:04:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:04:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:04:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:04:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:04:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:04:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:04:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:04:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:04:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:04:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:04:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:04:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:04:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:04:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:04:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:04:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:04:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:04:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:04:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:04:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:04:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:04:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:04:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:04:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:04:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:04:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:04:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:04:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:04:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:04:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:04:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:04:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:04:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:04:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:04:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:04:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:04:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:04:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:04:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:04:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:04:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:04:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:04:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:04:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:04:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:04:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:04:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:04:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:04:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:04:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:04:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:04:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:04:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:04:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:04:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:04:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:04:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:04:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:04:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:04:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:04:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:04:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:04:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:04:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:04:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:04:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:04:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:04:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:04:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:04:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:04:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:04:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:04:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:04:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:04:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:04:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:04:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:04:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:04:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:04:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:04:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:04:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:04:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:04:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:04:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:04:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:04:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:04:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:04:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:06:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:06:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:06:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:06:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:06:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:06:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:06:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:06:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:06:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:06:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:06:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:06:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:06:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:06:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:06:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:06:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:06:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:06:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:06:34 rdma06 rsyslogd[1041]: imjournal: 5302 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 15:06:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:06:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:06:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:06:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:06:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:06:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:06:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:06:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:06:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:06:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:06:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:06:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:06:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:06:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:06:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:06:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:06:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:06:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:06:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:06:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:06:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:06:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:06:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:06:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:06:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:06:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:06:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:06:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:06:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:06:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:06:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:06:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:06:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:06:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:06:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:06:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:06:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:06:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:06:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:06:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:06:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:06:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:06:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:06:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:06:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:06:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:06:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:06:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:06:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:06:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:06:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:06:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:06:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:06:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:06:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:06:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:06:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:06:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:06:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:06:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:06:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:06:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:06:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:06:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:06:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:06:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:06:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:06:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:06:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:06:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:06:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:06:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:06:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:06:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:06:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:06:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:06:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:06:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:06:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:06:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:06:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:06:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:06:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:06:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:06:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:06:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:06:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:06:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:06:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:06:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:07:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:07:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:07:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:07:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:07:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:07:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:07:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:07:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:07:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:07:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:07:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:07:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:07:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:07:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:07:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:07:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:07:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:07:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:07:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:07:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:07:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:07:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:07:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:07:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:07:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:07:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:07:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:07:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:07:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:07:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:07:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:07:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:07:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:07:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:07:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:07:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:07:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:07:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:07:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:07:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:07:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:07:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:07:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:07:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:07:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:07:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:07:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:07:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:07:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:07:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:07:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:07:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:07:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:07:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:07:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:07:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:07:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:07:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:07:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:07:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:07:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:07:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:07:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:07:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:07:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:07:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:07:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:07:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:07:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:07:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:07:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:07:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:07:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:07:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:07:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:07:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:07:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:07:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:07:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:07:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:07:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:07:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:07:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:07:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:07:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:07:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:07:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:07:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:07:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:07:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:07:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:07:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:07:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:07:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:07:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:07:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:07:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:07:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:07:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:07:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:07:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:07:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:07:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:07:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:07:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:07:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:07:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:07:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:07:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:07:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:07:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:07:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:07:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:07:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:07:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:07:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:07:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:07:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:07:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:07:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:07:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:07:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:07:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:07:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:07:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:07:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:07:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:07:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:07:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:07:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:07:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:07:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:07:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:07:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:07:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:07:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:07:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:07:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:07:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:07:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:07:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:07:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:07:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:07:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:07:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:07:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:07:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:07:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:07:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:07:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:07:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:07:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:07:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:07:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:07:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:07:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:07:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:07:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:07:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:07:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:07:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:07:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:07:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:07:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:07:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:07:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:07:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:07:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:07:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:07:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:07:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:07:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:07:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:07:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:07:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:07:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:07:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:07:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:07:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:07:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:07:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:07:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:07:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:07:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:07:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:07:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:07:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:07:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:07:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:07:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:07:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:07:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:07:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:07:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:07:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:07:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:07:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:07:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:07:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:07:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:07:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:07:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:07:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:07:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:07:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:07:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:07:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:07:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:07:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:07:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:07:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:07:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:07:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:07:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:07:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:07:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:08:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:08:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:08:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:08:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:08:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:08:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:08:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:08:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:08:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:08:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:08:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:08:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:08:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:08:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:08:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:08:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:08:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:08:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:08:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:08:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:08:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:08:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:08:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:08:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:08:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:08:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:08:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:08:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:08:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:08:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:08:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:08:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:08:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:08:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:08:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:08:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:08:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:08:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:08:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:08:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:08:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:08:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:08:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:08:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:08:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:08:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:08:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:08:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:08:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:08:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:08:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:08:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:08:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:08:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:08:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:08:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:08:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:08:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:08:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:08:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:08:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:08:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:08:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:08:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:08:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:08:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:08:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:08:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:08:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:08:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:08:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:08:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:08:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:08:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:08:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:08:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:08:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:08:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:08:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:08:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:08:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:08:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:08:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:08:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:08:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:08:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:08:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:08:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:08:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:08:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:08:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:08:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:08:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:08:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:08:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:08:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:08:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:08:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:08:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:08:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:08:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:08:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:08:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:08:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:08:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:08:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:08:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:08:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:08:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:08:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:08:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:08:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:08:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:08:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:08:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:08:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:08:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:08:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:08:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:08:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:08:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:08:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:08:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:08:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:08:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:08:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:08:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:08:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:08:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:08:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:08:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:08:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:08:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:08:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:08:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:08:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:08:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:08:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:08:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:08:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:08:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:08:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:08:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:08:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:08:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:08:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:08:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:08:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:08:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:08:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:08:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:08:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:08:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:08:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:08:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:08:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:08:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:08:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:08:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:08:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:08:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:08:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:08:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:08:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:08:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:08:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:08:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:08:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:08:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:08:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:08:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:08:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:08:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:08:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:08:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:08:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:08:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:08:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:08:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:08:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:08:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:08:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:08:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:08:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:08:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:08:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:08:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:08:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:08:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:08:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:08:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:08:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:08:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:08:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:08:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:08:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:08:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:08:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:08:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:08:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:08:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:08:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:08:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:08:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:08:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:08:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:08:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:08:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:08:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:08:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:08:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:08:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:08:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:08:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:08:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:08:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:09:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:09:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:09:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:09:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:09:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:09:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:09:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:09:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:09:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:09:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:09:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:09:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:09:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:09:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:09:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:09:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:09:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:09:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:09:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:09:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:09:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:09:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:09:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:09:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:09:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:09:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:09:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:09:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:09:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:09:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:09:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:09:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:09:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:09:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:09:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:09:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:09:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:09:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:09:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:09:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:09:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:09:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:09:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:09:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:09:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:09:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:09:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:09:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:09:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:09:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:09:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:09:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:09:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:09:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:09:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:09:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:09:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:09:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:09:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:09:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:09:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:09:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:09:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:09:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:09:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:09:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:09:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:09:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:09:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:09:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:09:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:09:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:09:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:09:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:09:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:09:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:09:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:09:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:09:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:09:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:09:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:09:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:09:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:09:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:09:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:09:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:09:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:09:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:09:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:09:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:09:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:09:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:09:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:09:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:09:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:09:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:09:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:09:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:09:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:09:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:09:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:09:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:09:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:09:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:09:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:09:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:09:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:09:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:09:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:09:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:09:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:09:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:09:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:09:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:09:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:09:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:09:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:09:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:09:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:09:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:09:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:09:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:09:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:09:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:09:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:09:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:09:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:09:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:09:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:09:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:09:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:09:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:09:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:09:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:09:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:09:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:09:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:09:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:09:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:09:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:09:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:09:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:09:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:09:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:09:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:09:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:09:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:09:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:09:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:09:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:09:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:09:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:09:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:09:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:09:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:09:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:09:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:09:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:09:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:09:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:09:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:09:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:09:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:09:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:09:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:09:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:09:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:09:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:09:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:09:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:09:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:09:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:09:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:09:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:09:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:09:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:09:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:09:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:09:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:09:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:09:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:09:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:09:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:09:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:09:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:09:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:09:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:09:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:09:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:09:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:09:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:09:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:09:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:09:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:09:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:09:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:09:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:09:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:09:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:09:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:09:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:09:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:09:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:09:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:09:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:09:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:09:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:09:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:09:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:09:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:09:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:09:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:09:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:09:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:09:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:09:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:10:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:10:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:10:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:10:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:10:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:10:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:10:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:10:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:10:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:10:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:10:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:10:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:10:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:10:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:10:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:10:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:10:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:10:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:10:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:10:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:10:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:10:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:10:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:10:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:10:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:10:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:10:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:10:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:10:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:10:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:10:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:10:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:10:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:10:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:10:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:10:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:10:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:10:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:10:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:10:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:10:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:10:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:10:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:10:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:10:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:10:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:10:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:10:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:10:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:10:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:10:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:10:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:10:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:10:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:10:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:10:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:10:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:10:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:10:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:10:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:10:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:10:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:10:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:10:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:10:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:10:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:10:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:10:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:10:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:10:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:10:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:10:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:10:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:10:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:10:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:10:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:10:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:10:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:10:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:10:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:10:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:10:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:10:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:10:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:10:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:10:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:10:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:10:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:10:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:10:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:10:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:10:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:10:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:10:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:10:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:10:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:10:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:10:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:10:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:10:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:10:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:10:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:10:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:10:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:10:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:10:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:10:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:10:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:10:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:10:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:10:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:10:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:10:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:10:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:10:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:10:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:10:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:10:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:10:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:10:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:10:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:10:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:10:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:10:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:10:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:10:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:10:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:10:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:10:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:10:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:10:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:10:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:10:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:10:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:10:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:10:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:10:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:10:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:10:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:10:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:10:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:10:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:10:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:10:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:10:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:10:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:10:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:10:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:10:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:10:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:10:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:10:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:10:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:10:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:10:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:10:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:10:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:10:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:10:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:10:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:10:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:10:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:10:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:10:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:10:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:10:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:10:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:10:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:10:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:10:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:10:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:10:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:10:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:10:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:10:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:10:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:10:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:10:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:10:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:10:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:10:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:10:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:10:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:10:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:10:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:10:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:10:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:10:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:10:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:10:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:10:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:10:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:10:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:10:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:10:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:10:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:10:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:10:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:10:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:10:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:10:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:10:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:10:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:10:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:10:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:10:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:10:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:10:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:10:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:10:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:10:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:10:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:10:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:10:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:10:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:10:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:11:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:11:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:11:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:11:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:11:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:11:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:11:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:11:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:11:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:11:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:11:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:11:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:11:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:11:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:11:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:11:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:11:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:11:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:11:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:11:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:11:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:11:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:11:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:11:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:11:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:11:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:11:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:11:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:11:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:11:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:11:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:11:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:11:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:11:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:11:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:11:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:11:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:11:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:11:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:11:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:11:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:11:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:11:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:11:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:11:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:11:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:11:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:11:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:11:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:11:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:11:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:11:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:11:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:11:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:11:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:11:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:11:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:11:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:11:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:11:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:11:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:11:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:11:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:11:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:11:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:11:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:11:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:11:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:11:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:11:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:11:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:11:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:11:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:11:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:11:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:11:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:11:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:11:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:11:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:11:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:11:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:11:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:11:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:11:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:11:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:11:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:11:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:11:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:11:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:11:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:11:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:11:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:11:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:11:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:11:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:11:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:11:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:11:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:11:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:11:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:11:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:11:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:11:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:11:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:11:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:11:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:11:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:11:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:11:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:11:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:11:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:11:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:11:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:11:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:11:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:11:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:11:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:11:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:11:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:11:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:11:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:11:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:11:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:11:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:11:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:11:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:11:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:11:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:11:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:11:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:11:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:11:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:11:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:11:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:11:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:11:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:11:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:11:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:11:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:11:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:11:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:11:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:11:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:11:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:11:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:11:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:11:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:11:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:11:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:11:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:11:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:11:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:11:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:11:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:11:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:11:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:11:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:11:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:11:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:11:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:11:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:11:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:11:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:11:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:11:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:11:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:11:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:11:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:11:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:11:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:11:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:11:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:11:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:11:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:11:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:11:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:11:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:11:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:11:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:11:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:11:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:11:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:11:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:11:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:11:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:11:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:11:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:11:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:11:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:11:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:11:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:11:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:11:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:11:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:11:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:11:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:11:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:11:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:11:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:11:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:11:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:11:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:11:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:11:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:11:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:11:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:11:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:11:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:11:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:11:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:11:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:11:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:11:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:11:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:11:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:11:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:12:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:12:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:12:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:12:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:12:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:12:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:12:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:12:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:12:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:12:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:12:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:12:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:12:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:12:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:12:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:12:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:12:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:12:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:12:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:12:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:12:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:12:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:12:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:12:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:12:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:12:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:12:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:12:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:12:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:12:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:12:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:12:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:12:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:12:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:12:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:12:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:12:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:12:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:12:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:12:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:12:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:12:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:12:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:12:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:12:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:12:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:12:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:12:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:12:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:12:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:12:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:12:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:12:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:12:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:12:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:12:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:12:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:12:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:12:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:12:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:12:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:12:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:12:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:12:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:12:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:12:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:12:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:12:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:12:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:12:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:12:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:12:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:12:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:12:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:12:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:12:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:12:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:12:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:12:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:12:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:12:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:12:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:12:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:12:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:12:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:12:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:12:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:12:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:12:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:12:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:12:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:12:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:12:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:12:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:12:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:12:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:12:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:12:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:12:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:12:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:12:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:12:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:12:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:12:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:12:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:12:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:12:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:12:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:12:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:12:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:12:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:12:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:12:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:12:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:12:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:12:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:12:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:12:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:12:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:12:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:12:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:12:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:12:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:12:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:12:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:12:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:12:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:12:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:12:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:12:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:12:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:12:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:12:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:12:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:12:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:12:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:12:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:12:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:12:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:12:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:12:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:12:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:12:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:12:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:12:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:12:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:12:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:12:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:12:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:12:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:12:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:12:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:12:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:12:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:12:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:12:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:12:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:12:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:12:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:12:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:12:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:12:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:12:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:12:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:12:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:12:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:12:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:12:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:12:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:12:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:12:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:12:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:12:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:12:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:12:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:12:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:12:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:12:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:12:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:12:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:12:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:12:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:12:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:12:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:12:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:12:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:12:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:12:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:12:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:12:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:12:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:12:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:12:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:12:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:12:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:12:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:12:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:12:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:12:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:12:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:12:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:12:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:12:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:12:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:12:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:12:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:12:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:12:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:12:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:12:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:12:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:12:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:12:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:12:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:12:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:12:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:13:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:13:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:13:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:13:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:13:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:13:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:13:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:13:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:13:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:13:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:13:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:13:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:13:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:13:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:13:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:13:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:13:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:13:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:13:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:13:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:13:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:13:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:13:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:13:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:13:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:13:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:13:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:13:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:13:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:13:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:13:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:13:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:13:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:13:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:13:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:13:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:13:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:13:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:13:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:13:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:13:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:13:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:13:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:13:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:13:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:13:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:13:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:13:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:13:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:13:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:13:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:13:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:13:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:13:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:13:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:13:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:13:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:13:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:13:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:13:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:13:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:13:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:13:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:13:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:13:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:13:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:13:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:13:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:13:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:13:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:13:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:13:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:13:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:13:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:13:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:13:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:13:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:13:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:13:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:13:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:13:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:13:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:13:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:13:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:13:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:13:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:13:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:13:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:13:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:13:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:13:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:13:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:13:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:13:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:13:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:13:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:13:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:13:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:13:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:13:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:13:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:13:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:13:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:13:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:13:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:13:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:13:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:13:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:13:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:13:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:13:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:13:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:13:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:13:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:13:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:13:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:13:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:13:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:13:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:13:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:13:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:13:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:13:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:13:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:13:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:13:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:13:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:13:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:13:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:13:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:13:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:13:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:13:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:13:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:13:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:13:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:13:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:13:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:13:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:13:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:13:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:13:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:13:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:13:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:13:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:13:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:13:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:13:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:13:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:13:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:13:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:13:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:13:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:13:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:13:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:13:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:13:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:13:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:13:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:13:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:13:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:13:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:13:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:13:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:13:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:13:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:13:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:13:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:13:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:13:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:13:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:13:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:13:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:13:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:13:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:13:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:13:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:13:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:13:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:13:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:13:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:13:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:13:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:13:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:13:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:13:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:13:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:13:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:13:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:13:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:13:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:13:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:13:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:13:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:13:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:13:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:13:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:13:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:13:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:13:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:13:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:13:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:13:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:13:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:13:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:13:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:13:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:13:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:13:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:13:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:13:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:13:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:13:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:13:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:13:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:13:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:14:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:14:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:14:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:14:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:14:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:14:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:14:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:14:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:14:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:14:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:14:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:14:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:14:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:14:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:14:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:14:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:14:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:14:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:14:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:14:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:14:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:14:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:14:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:14:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:14:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:14:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:14:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:14:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:14:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:14:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:14:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:14:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:14:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:14:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:14:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:14:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:14:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:14:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:14:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:14:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:14:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:14:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:14:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:14:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:14:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:14:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:14:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:14:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:14:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:14:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:14:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:14:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:14:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:14:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:14:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:14:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:14:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:14:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:14:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:14:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:14:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:14:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:14:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:14:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:14:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:14:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:14:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:14:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:14:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:14:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:14:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:14:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:14:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:14:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:14:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:14:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:14:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:14:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:14:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:14:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:14:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:14:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:14:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:14:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:14:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:14:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:14:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:14:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:14:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:14:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:14:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:14:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:14:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:14:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:14:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:14:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:14:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:14:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:14:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:14:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:14:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:14:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:14:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:14:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:14:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:14:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:14:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:14:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:16:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:16:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:16:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:16:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:16:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:16:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:16:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:16:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:16:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:16:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:16:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:16:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:16:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:16:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:16:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:16:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:16:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:16:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:16:35 rdma06 rsyslogd[1041]: imjournal: 5082 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 15:16:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:16:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:16:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:16:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:16:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:16:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:16:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:16:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:16:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:16:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:16:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:16:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:16:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:16:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:16:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:16:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:16:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:16:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:16:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:16:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:16:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:16:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:16:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:16:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:16:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:16:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:16:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:16:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:16:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:16:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:16:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:16:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:16:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:16:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:16:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:16:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:16:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:16:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:16:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:16:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:16:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:16:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:16:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:16:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:16:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:16:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:16:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:16:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:16:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:16:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:16:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:16:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:16:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:16:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:16:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:16:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:16:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:16:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:16:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:16:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:16:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:16:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:16:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:16:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:16:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:16:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:16:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:16:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:16:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:16:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:16:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:16:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:16:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:16:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:16:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:16:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:16:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:16:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:16:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:16:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:16:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:16:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:16:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:16:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:16:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:16:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:16:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:16:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:16:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:16:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:17:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:17:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:17:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:17:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:17:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:17:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:17:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:17:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:17:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:17:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:17:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:17:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:17:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:17:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:17:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:17:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:17:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:17:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:17:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:17:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:17:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:17:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:17:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:17:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:17:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:17:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:17:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:17:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:17:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:17:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:17:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:17:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:17:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:17:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:17:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:17:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:17:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:17:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:17:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:17:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:17:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:17:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:17:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:17:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:17:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:17:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:17:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:17:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:17:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:17:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:17:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:17:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:17:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:17:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:17:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:17:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:17:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:17:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:17:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:17:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:17:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:17:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:17:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:17:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:17:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:17:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:17:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:17:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:17:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:17:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:17:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:17:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:17:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:17:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:17:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:17:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:17:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:17:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:17:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:17:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:17:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:17:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:17:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:17:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:17:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:17:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:17:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:17:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:17:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:17:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:17:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:17:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:17:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:17:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:17:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:17:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:17:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:17:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:17:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:17:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:17:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:17:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:17:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:17:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:17:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:17:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:17:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:17:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:17:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:17:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:17:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:17:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:17:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:17:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:17:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:17:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:17:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:17:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:17:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:17:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:17:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:17:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:17:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:17:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:17:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:17:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:17:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:17:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:17:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:17:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:17:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:17:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:17:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:17:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:17:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:17:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:17:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:17:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:17:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:17:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:17:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:17:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:17:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:17:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:17:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:17:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:17:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:17:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:17:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:17:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:17:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:17:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:17:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:17:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:17:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:17:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:17:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:17:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:17:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:17:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:17:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:17:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:17:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:17:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:17:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:17:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:17:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:17:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:17:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:17:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:17:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:17:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:17:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:17:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:17:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:17:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:17:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:17:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:17:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:17:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:17:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:17:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:17:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:17:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:17:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:17:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:17:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:17:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:17:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:17:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:17:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:17:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:17:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:17:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:17:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:17:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:17:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:17:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:17:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:17:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:17:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:17:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:17:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:17:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:17:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:17:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:17:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:17:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:17:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:17:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:17:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:17:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:17:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:17:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:17:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:17:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:18:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:18:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:18:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:18:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:18:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:18:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:18:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:18:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:18:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:18:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:18:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:18:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:18:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:18:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:18:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:18:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:18:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:18:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:18:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:18:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:18:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:18:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:18:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:18:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:18:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:18:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:18:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:18:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:18:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:18:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:18:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:18:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:18:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:18:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:18:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:18:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:18:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:18:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:18:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:18:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:18:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:18:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:18:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:18:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:18:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:18:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:18:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:18:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:18:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:18:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:18:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:18:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:18:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:18:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:18:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:18:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:18:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:18:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:18:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:18:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:18:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:18:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:18:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:18:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:18:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:18:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:18:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:18:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:18:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:18:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:18:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:18:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:18:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:18:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:18:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:18:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:18:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:18:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:18:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:18:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:18:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:18:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:18:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:18:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:18:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:18:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:18:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:18:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:18:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:18:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:18:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:18:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:18:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:18:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:18:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:18:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:18:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:18:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:18:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:18:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:18:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:18:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:18:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:18:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:18:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:18:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:18:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:18:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:18:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:18:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:18:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:18:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:18:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:18:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:18:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:18:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:18:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:18:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:18:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:18:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:18:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:18:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:18:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:18:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:18:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:18:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:18:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:18:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:18:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:18:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:18:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:18:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:18:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:18:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:18:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:18:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:18:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:18:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:18:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:18:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:18:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:18:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:18:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:18:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:18:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:18:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:18:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:18:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:18:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:18:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:18:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:18:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:18:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:18:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:18:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:18:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:18:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:18:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:18:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:18:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:18:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:18:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:18:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:18:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:18:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:18:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:18:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:18:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:18:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:18:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:18:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:18:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:18:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:18:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:18:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:18:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:18:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:18:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:18:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:18:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:18:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:18:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:18:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:18:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:18:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:18:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:18:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:18:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:18:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:18:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:18:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:18:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:18:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:18:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:18:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:18:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:18:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:18:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:18:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:18:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:18:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:18:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:18:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:18:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:18:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:18:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:18:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:18:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:18:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:18:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:18:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:18:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:18:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:18:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:18:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:18:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:19:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:19:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:19:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:19:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:19:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:19:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:19:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:19:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:19:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:19:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:19:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:19:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:19:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:19:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:19:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:19:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:19:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:19:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:19:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:19:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:19:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:19:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:19:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:19:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:19:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:19:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:19:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:19:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:19:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:19:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:19:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:19:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:19:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:19:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:19:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:19:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:19:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:19:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:19:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:19:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:19:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:19:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:19:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:19:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:19:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:19:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:19:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:19:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:19:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:19:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:19:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:19:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:19:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:19:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:19:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:19:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:19:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:19:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:19:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:19:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:19:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:19:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:19:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:19:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:19:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:19:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:19:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:19:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:19:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:19:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:19:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:19:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:19:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:19:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:19:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:19:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:19:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:19:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:19:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:19:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:19:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:19:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:19:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:19:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:19:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:19:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:19:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:19:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:19:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:19:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:19:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:19:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:19:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:19:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:19:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:19:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:19:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:19:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:19:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:19:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:19:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:19:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:19:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:19:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:19:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:19:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:19:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:19:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:19:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:19:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:19:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:19:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:19:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:19:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:19:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:19:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:19:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:19:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:19:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:19:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:19:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:19:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:19:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:19:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:19:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:19:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:19:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:19:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:19:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:19:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:19:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:19:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:19:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:19:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:19:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:19:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:19:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:19:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:19:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:19:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:19:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:19:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:19:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:19:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:19:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:19:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:19:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:19:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:19:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:19:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:19:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:19:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:19:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:19:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:19:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:19:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:19:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:19:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:19:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:19:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:19:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:19:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:19:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:19:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:19:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:19:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:19:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:19:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:19:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:19:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:19:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:19:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:19:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:19:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:19:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:19:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:19:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:19:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:19:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:19:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:19:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:19:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:19:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:19:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:19:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:19:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:19:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:19:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:19:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:19:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:19:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:19:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:19:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:19:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:19:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:19:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:19:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:19:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:19:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:19:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:19:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:19:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:19:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:19:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:19:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:19:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:19:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:19:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:19:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:19:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:19:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:19:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:19:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:19:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:19:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:19:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:20:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:20:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:20:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:20:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:20:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:20:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:20:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:20:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:20:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:20:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:20:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:20:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:20:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:20:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:20:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:20:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:20:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:20:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:20:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:20:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:20:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:20:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:20:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:20:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:20:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:20:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:20:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:20:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:20:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:20:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:20:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:20:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:20:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:20:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:20:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:20:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:20:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:20:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:20:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:20:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:20:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:20:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:20:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:20:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:20:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:20:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:20:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:20:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:20:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:20:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:20:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:20:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:20:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:20:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:20:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:20:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:20:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:20:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:20:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:20:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:20:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:20:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:20:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:20:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:20:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:20:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:20:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:20:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:20:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:20:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:20:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:20:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:20:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:20:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:20:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:20:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:20:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:20:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:20:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:20:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:20:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:20:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:20:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:20:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:20:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:20:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:20:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:20:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:20:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:20:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:20:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:20:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:20:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:20:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:20:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:20:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:20:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:20:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:20:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:20:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:20:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:20:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:20:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:20:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:20:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:20:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:20:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:20:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:20:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:20:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:20:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:20:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:20:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:20:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:20:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:20:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:20:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:20:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:20:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:20:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:20:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:20:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:20:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:20:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:20:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:20:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:20:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:20:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:20:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:20:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:20:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:20:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:20:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:20:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:20:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:20:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:20:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:20:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:20:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:20:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:20:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:20:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:20:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:20:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:20:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:20:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:20:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:20:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:20:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:20:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:20:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:20:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:20:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:20:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:20:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:20:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:20:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:20:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:20:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:20:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:20:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:20:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:20:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:20:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:20:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:20:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:20:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:20:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:20:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:20:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:20:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:20:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:20:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:20:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:20:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:20:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:20:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:20:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:20:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:20:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:20:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:20:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:20:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:20:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:20:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:20:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:20:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:20:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:20:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:20:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:20:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:20:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:20:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:20:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:20:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:20:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:20:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:20:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:20:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:20:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:20:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:20:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:20:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:20:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:20:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:20:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:20:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:20:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:20:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:20:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:20:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:20:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:20:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:20:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:20:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:20:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:21:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:21:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:21:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:21:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:21:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:21:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:21:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:21:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:21:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:21:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:21:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:21:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:21:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:21:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:21:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:21:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:21:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:21:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:21:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:21:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:21:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:21:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:21:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:21:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:21:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:21:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:21:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:21:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:21:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:21:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:21:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:21:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:21:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:21:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:21:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:21:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:21:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:21:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:21:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:21:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:21:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:21:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:21:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:21:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:21:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:21:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:21:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:21:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:21:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:21:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:21:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:21:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:21:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:21:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:21:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:21:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:21:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:21:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:21:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:21:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:21:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:21:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:21:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:21:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:21:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:21:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:21:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:21:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:21:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:21:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:21:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:21:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:21:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:21:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:21:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:21:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:21:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:21:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:21:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:21:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:21:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:21:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:21:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:21:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:21:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:21:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:21:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:21:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:21:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:21:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:21:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:21:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:21:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:21:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:21:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:21:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:21:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:21:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:21:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:21:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:21:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:21:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:21:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:21:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:21:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:21:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:21:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:21:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:21:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:21:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:21:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:21:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:21:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:21:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:21:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:21:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:22:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:22:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:22:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:22:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:22:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:22:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:22:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:22:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:22:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:22:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:22:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:22:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:22:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:22:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:22:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:22:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:22:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:22:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:22:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:22:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:22:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:22:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:22:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:22:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:22:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:22:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:22:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:22:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:22:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:22:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:22:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:22:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:22:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:22:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:22:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:22:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:22:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:22:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:22:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:22:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:22:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:22:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:22:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:22:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:22:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:22:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:22:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:22:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:22:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:22:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:22:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:22:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:22:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:22:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:22:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:22:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:22:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:22:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:22:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:22:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:22:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:22:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:22:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:22:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:22:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:22:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:22:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:22:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:22:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:22:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:22:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:22:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:22:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:22:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:22:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:22:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:22:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:22:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:22:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:22:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:22:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:22:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:22:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:22:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:22:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:22:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:22:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:22:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:22:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:22:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:22:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:22:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:22:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:22:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:22:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:22:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:22:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:22:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:22:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:22:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:22:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:22:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:22:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:22:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:22:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:22:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:22:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:22:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:22:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:22:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:22:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:22:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:22:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:22:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:22:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:22:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:22:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:22:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:22:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:22:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:22:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:22:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:22:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:22:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:22:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:22:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:22:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:22:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:22:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:22:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:22:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:22:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:22:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:22:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:22:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:22:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:22:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:22:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:22:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:22:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:22:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:22:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:22:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:22:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:22:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:22:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:22:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:22:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:22:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:22:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:22:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:22:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:22:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:22:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:22:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:22:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:22:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:22:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:22:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:22:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:22:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:22:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:22:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:22:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:22:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:22:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:22:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:22:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:22:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:22:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:22:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:22:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:22:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:22:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:22:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:22:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:22:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:22:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:22:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:22:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:22:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:22:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:22:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:22:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:22:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:22:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:22:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:22:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:22:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:22:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:22:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:22:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:22:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:22:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:22:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:22:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:22:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:22:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:23:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:23:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:23:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:23:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:23:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:23:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:23:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:23:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:23:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:23:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:23:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:23:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:23:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:23:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:23:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:23:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:23:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:23:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:23:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:23:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:23:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:23:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:23:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:23:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:23:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:23:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:23:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:23:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:23:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:23:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:23:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:23:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:23:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:23:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:23:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:23:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:23:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:23:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:23:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:23:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:23:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:23:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:23:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:23:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:23:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:23:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:23:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:23:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:23:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:23:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:23:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:23:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:23:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:23:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:23:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:23:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:23:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:23:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:23:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:23:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:23:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:23:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:23:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:23:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:23:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:23:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:23:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:23:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:23:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:23:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:23:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:23:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:23:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:23:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:23:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:23:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:23:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:23:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:23:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:23:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:23:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:23:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:23:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:23:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:23:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:23:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:23:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:23:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:23:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:23:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:23:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:23:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:23:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:23:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:23:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:23:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:23:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:23:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:23:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:23:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:23:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:23:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:23:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:23:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:23:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:23:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:23:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:23:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:23:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:23:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:23:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:23:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:23:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:23:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:23:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:23:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:23:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:23:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:23:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:23:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:23:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:23:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:23:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:23:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:23:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:23:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:23:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:23:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:23:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:23:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:23:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:23:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:23:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:23:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:23:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:23:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:23:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:23:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:23:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:23:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:23:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:23:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:23:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:23:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:23:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:23:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:23:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:23:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:23:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:23:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:23:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:23:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:23:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:23:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:23:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:23:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:23:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:23:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:23:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:23:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:23:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:23:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:23:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:23:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:23:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:23:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:23:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:23:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:23:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:23:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:23:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:23:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:23:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:23:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:23:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:23:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:23:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:23:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:23:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:23:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:23:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:23:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:23:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:23:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:23:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:23:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:23:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:23:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:23:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:23:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:23:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:23:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:23:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:23:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:23:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:23:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:23:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:23:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:23:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:23:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:23:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:23:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:23:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:23:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:23:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:23:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:23:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:23:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:23:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:23:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:23:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:23:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:23:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:23:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:23:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:23:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:24:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:24:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:24:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:24:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:24:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:24:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:24:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:24:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:24:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:24:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:24:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:24:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:24:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:24:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:24:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:24:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:24:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:24:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:24:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:24:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:24:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:24:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:24:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:24:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:24:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:24:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:24:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:24:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:24:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:24:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:24:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:24:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:24:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:24:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:24:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:24:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:24:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:24:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:24:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:24:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:24:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:24:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:24:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:24:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:24:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:24:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:24:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:24:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:24:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:24:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:24:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:24:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:24:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:24:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:24:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:24:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:24:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:24:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:24:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:24:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:24:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:24:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:24:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:24:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:24:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:24:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:24:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:24:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:24:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:24:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:24:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:24:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:24:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:24:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:24:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:24:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:24:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:24:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:24:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:24:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:24:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:24:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:24:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:24:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:24:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:24:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:24:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:24:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:24:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:24:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:24:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:24:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:24:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:24:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:24:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:24:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:24:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:24:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:24:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:24:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:24:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:24:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:24:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:24:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:24:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:24:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:24:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:24:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:24:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:24:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:24:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:24:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:24:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:24:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:24:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:24:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:24:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:24:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:24:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:24:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:24:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:24:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:24:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:24:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:24:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:24:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:26:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:26:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:26:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:26:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:26:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:26:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:26:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:26:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:26:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:26:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:26:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:26:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:26:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:26:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:26:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:26:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:26:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:26:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:26:40 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 15:26:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:26:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:26:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:26:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:26:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:26:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:26:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:26:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:26:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:26:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:26:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:26:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:26:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:26:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:26:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:26:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:26:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:26:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:26:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:26:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:26:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:26:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:26:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:26:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:26:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:26:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:26:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:26:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:26:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:26:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:26:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:26:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:26:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:26:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:26:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:26:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:26:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:26:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:26:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:26:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:26:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:26:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:26:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:26:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:26:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:26:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:26:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:26:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:26:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:26:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:26:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:26:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:26:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:26:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:27:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:27:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:27:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:27:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:27:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:27:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:27:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:27:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:27:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:27:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:27:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:27:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:27:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:27:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:27:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:27:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:27:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:27:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:27:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:27:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:27:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:27:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:27:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:27:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:27:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:27:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:27:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:27:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:27:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:27:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:27:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:27:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:27:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:27:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:27:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:27:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:27:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:27:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:27:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:27:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:27:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:27:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:27:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:27:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:27:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:27:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:27:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:27:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:27:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:27:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:27:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:27:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:27:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:27:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:27:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:27:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:27:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:27:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:27:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:27:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:27:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:27:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:27:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:27:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:27:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:27:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:27:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:27:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:27:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:27:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:27:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:27:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:27:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:27:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:27:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:27:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:27:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:27:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:27:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:27:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:27:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:27:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:27:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:27:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:27:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:27:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:27:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:27:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:27:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:27:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:27:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:27:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:27:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:27:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:27:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:27:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:27:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:27:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:27:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:27:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:27:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:27:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:27:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:27:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:27:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:27:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:27:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:27:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:27:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:27:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:27:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:27:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:27:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:27:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:27:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:27:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:27:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:27:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:27:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:27:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:27:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:27:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:27:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:27:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:27:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:27:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:27:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:27:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:27:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:27:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:27:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:27:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:27:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:27:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:27:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:27:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:27:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:27:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:27:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:27:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:27:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:27:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:27:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:27:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:27:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:27:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:27:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:27:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:27:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:27:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:27:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:27:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:27:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:27:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:27:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:27:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:27:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:27:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:27:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:27:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:27:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:27:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:27:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:27:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:27:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:27:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:27:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:27:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:27:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:27:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:27:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:27:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:27:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:27:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:27:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:27:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:27:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:27:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:27:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:27:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:27:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:27:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:27:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:27:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:27:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:27:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:27:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:27:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:27:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:27:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:27:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:27:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:27:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:27:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:27:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:27:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:27:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:27:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:27:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:27:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:27:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:27:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:27:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:27:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:27:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:27:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:27:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:27:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:27:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:27:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:27:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:27:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:27:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:27:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:27:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:27:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:28:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:28:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:28:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:28:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:28:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:28:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:28:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:28:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:28:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:28:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:28:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:28:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:28:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:28:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:28:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:28:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:28:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:28:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:28:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:28:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:28:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:28:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:28:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:28:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:28:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:28:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:28:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:28:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:28:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:28:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:28:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:28:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:28:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:28:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:28:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:28:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:28:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:28:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:28:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:28:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:28:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:28:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:28:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:28:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:28:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:28:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:28:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:28:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:28:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:28:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:28:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:28:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:28:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:28:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:28:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:28:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:28:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:28:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:28:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:28:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:28:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:28:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:28:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:28:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:28:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:28:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:28:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:28:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:28:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:28:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:28:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:28:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:28:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:28:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:28:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:28:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:28:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:28:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:28:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:28:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:28:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:28:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:28:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:28:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:28:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:28:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:28:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:28:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:28:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:28:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:28:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:28:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:28:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:28:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:28:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:28:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:28:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:28:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:28:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:28:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:28:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:28:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:28:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:28:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:28:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:28:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:28:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:28:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:28:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:28:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:28:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:28:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:28:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:28:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:28:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:28:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:28:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:28:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:28:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:28:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:28:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:28:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:28:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:28:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:28:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:28:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:28:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:28:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:28:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:28:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:28:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:28:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:28:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:28:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:28:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:28:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:28:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:28:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:28:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:28:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:28:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:28:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:28:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:28:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:28:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:28:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:28:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:28:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:28:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:28:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:28:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:28:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:28:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:28:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:28:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:28:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:28:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:28:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:28:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:28:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:28:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:28:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:28:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:28:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:28:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:28:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:28:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:28:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:28:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:28:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:28:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:28:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:28:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:28:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:28:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:28:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:28:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:28:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:28:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:28:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:28:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:28:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:28:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:28:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:28:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:28:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:28:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:28:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:28:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:28:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:28:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:28:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:28:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:28:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:28:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:28:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:28:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:28:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:28:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:28:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:28:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:28:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:28:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:28:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:28:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:28:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:28:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:28:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:28:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:28:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:28:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:28:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:28:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:28:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:28:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:28:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:29:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:29:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:29:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:29:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:29:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:29:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:29:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:29:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:29:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:29:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:29:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:29:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:29:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:29:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:29:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:29:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:29:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:29:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:29:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:29:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:29:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:29:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:29:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:29:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:29:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:29:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:29:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:29:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:29:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:29:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:29:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:29:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:29:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:29:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:29:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:29:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:29:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:29:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:29:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:29:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:29:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:29:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:29:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:29:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:29:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:29:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:29:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:29:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:29:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:29:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:29:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:29:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:29:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:29:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:29:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:29:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:29:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:29:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:29:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:29:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:29:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:29:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:29:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:29:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:29:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:29:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:29:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:29:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:29:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:29:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:29:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:29:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:29:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:29:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:29:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:29:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:29:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:29:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:29:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:29:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:29:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:29:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:29:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:29:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:29:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:29:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:29:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:29:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:29:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:29:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:29:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:29:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:29:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:29:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:29:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:29:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:29:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:29:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:29:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:29:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:29:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:29:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:29:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:29:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:29:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:29:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:29:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:29:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:29:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:29:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:29:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:29:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:29:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:29:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:29:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:29:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:29:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:29:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:29:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:29:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:29:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:29:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:29:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:29:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:29:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:29:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:29:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:29:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:29:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:29:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:29:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:29:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:29:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:29:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:29:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:29:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:29:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:29:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:29:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:29:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:29:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:29:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:29:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:29:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:29:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:29:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:29:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:29:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:29:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:29:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:29:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:29:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:29:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:29:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:29:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:29:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:29:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:29:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:29:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:29:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:29:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:29:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:29:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:29:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:29:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:29:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:29:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:29:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:29:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:29:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:29:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:29:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:29:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:29:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:29:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:29:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:29:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:29:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:29:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:29:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:29:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:29:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:29:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:29:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:29:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:29:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:29:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:29:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:29:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:29:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:29:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:29:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:29:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:29:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:29:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:29:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:29:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:29:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:29:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:29:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:29:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:29:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:29:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:29:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:29:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:29:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:29:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:29:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:29:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:29:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:29:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:29:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:29:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:29:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:29:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:29:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:30:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:30:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:30:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:30:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:30:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:30:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:30:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:30:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:30:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:30:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:30:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:30:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:30:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:30:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:30:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:30:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:30:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:30:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:30:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:30:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:30:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:30:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:30:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:30:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:30:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:30:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:30:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:30:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:30:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:30:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:30:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:30:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:30:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:30:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:30:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:30:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:30:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:30:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:30:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:30:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:30:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:30:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:30:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:30:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:30:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:30:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:30:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:30:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:30:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:30:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:30:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:30:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:30:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:30:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:30:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:30:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:30:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:30:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:30:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:30:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:30:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:30:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:30:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:30:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:30:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:30:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:30:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:30:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:30:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:30:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:30:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:30:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:30:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:30:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:30:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:30:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:30:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:30:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:30:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:30:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:30:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:30:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:30:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:30:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:30:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:30:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:30:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:30:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:30:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:30:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:30:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:30:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:30:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:30:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:30:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:30:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:30:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:30:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:30:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:30:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:30:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:30:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:30:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:30:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:30:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:30:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:30:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:30:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:30:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:30:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:30:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:30:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:30:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:30:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:30:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:30:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:30:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:30:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:30:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:30:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:30:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:30:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:30:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:30:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:30:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:30:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:30:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:30:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:30:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:30:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:30:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:30:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:30:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:30:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:30:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:30:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:30:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:30:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:30:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:30:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:30:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:30:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:30:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:30:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:30:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:30:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:30:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:30:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:30:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:30:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:30:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:30:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:30:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:30:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:30:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:30:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:30:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:30:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:30:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:30:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:30:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:30:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:30:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:30:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:30:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:30:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:30:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:30:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:30:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:30:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:30:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:30:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:30:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:30:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:30:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:30:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:30:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:30:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:30:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:30:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:30:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:30:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:30:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:30:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:30:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:30:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:30:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:30:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:30:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:30:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:30:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:30:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:30:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:30:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:30:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:30:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:30:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:30:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:30:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:30:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:30:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:30:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:30:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:30:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:30:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:30:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:30:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:30:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:30:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:30:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:30:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:30:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:30:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:30:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:30:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:30:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:31:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:31:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:31:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:31:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:31:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:31:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:31:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:31:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:31:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:31:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:31:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:31:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:31:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:31:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:31:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:31:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:31:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:31:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:31:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:31:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:31:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:31:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:31:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:31:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:31:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:31:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:31:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:31:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:31:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:31:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:31:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:31:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:31:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:31:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:31:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:31:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:31:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:31:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:31:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:31:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:31:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:31:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:31:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:31:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:31:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:31:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:31:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:31:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:31:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:31:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:31:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:31:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:31:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:31:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:31:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:31:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:31:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:31:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:31:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:31:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:31:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:31:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:31:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:31:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:31:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:31:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:31:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:31:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:31:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:31:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:31:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:31:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:31:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:31:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:31:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:31:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:31:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:31:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:31:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:31:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:31:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:31:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:31:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:31:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:31:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:31:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:31:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:31:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:31:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:31:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:31:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:31:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:31:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:31:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:31:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:31:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:31:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:31:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:31:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:31:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:31:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:31:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:31:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:31:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:31:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:31:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:31:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:31:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:31:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:31:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:31:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:31:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:31:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:31:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:31:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:31:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:31:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:31:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:31:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:31:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:31:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:31:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:31:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:31:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:31:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:31:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:31:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:31:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:31:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:31:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:31:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:31:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:31:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:31:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:31:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:31:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:31:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:31:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:31:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:31:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:31:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:31:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:31:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:31:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:31:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:31:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:31:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:31:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:31:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:31:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:31:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:31:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:31:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:31:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:31:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:31:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:31:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:31:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:31:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:31:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:31:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:31:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:31:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:31:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:31:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:31:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:31:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:31:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:31:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:31:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:31:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:31:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:31:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:31:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:31:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:31:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:31:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:31:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:31:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:31:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:31:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:31:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:31:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:31:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:31:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:31:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:31:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:31:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:31:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:31:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:31:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:31:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:31:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:31:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:31:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:31:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:31:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:31:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:31:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:31:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:31:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:31:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:31:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:31:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:31:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:31:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:31:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:31:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:31:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:31:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:31:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:31:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:31:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:31:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:31:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:31:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:32:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:32:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:32:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:32:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:32:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:32:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:32:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:32:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:32:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:32:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:32:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:32:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:32:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:32:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:32:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:32:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:32:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:32:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:32:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:32:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:32:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:32:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:32:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:32:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:32:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:32:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:32:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:32:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:32:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:32:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:32:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:32:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:32:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:32:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:32:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:32:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:32:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:32:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:32:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:32:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:32:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:32:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:32:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:32:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:32:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:32:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:32:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:32:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:32:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:32:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:32:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:32:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:32:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:32:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:32:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:32:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:32:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:32:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:32:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:32:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:32:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:32:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:32:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:32:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:32:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:32:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:32:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:32:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:32:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:32:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:32:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:32:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:32:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:32:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:32:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:32:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:32:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:32:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:32:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:32:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:32:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:32:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:32:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:32:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:32:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:32:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:32:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:32:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:32:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:32:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:32:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:32:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:32:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:32:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:32:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:32:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:32:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:32:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:32:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:32:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:32:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:32:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:32:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:32:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:32:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:32:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:32:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:32:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:32:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:32:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:32:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:32:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:32:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:32:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:32:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:32:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:32:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:32:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:32:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:32:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:32:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:32:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:32:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:32:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:32:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:32:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:32:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:32:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:32:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:32:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:32:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:32:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:32:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:32:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:32:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:32:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:32:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:32:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:32:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:32:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:32:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:32:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:32:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:32:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:32:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:32:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:32:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:32:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:32:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:32:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:32:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:32:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:32:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:32:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:32:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:32:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:32:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:32:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:32:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:32:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:32:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:32:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:32:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:32:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:32:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:32:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:32:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:32:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:32:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:32:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:32:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:32:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:32:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:32:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:32:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:32:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:32:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:32:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:32:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:32:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:32:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:32:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:32:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:32:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:32:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:32:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:32:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:32:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:32:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:32:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:32:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:32:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:32:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:32:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:32:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:32:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:32:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:32:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:32:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:32:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:32:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:32:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:32:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:32:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:32:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:32:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:32:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:32:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:32:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:32:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:32:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:32:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:32:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:32:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:32:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:32:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:33:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:33:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:33:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:33:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:33:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:33:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:33:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:33:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:33:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:33:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:33:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:33:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:33:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:33:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:33:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:33:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:33:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:33:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:33:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:33:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:33:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:33:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:33:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:33:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:33:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:33:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:33:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:33:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:33:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:33:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:33:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:33:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:33:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:33:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:33:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:33:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:33:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:33:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:33:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:33:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:33:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:33:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:33:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:33:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:33:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:33:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:33:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:33:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:33:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:33:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:33:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:33:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:33:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:33:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:33:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:33:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:33:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:33:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:33:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:33:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:33:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:33:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:33:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:33:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:33:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:33:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:33:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:33:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:33:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:33:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:33:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:33:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:33:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:33:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:33:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:33:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:33:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:33:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:33:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:33:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:33:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:33:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:33:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:33:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:33:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:33:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:33:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:33:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:33:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:33:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:33:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:33:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:33:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:33:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:33:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:33:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:33:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:33:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:33:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:33:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:33:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:33:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:33:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:33:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:33:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:33:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:33:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:33:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:33:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:33:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:33:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:33:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:33:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:33:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:33:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:33:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:33:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:33:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:33:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:33:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:33:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:33:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:33:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:33:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:33:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:33:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:33:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:33:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:33:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:33:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:33:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:33:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:33:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:33:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:33:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:33:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:33:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:33:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:33:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:33:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:33:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:33:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:33:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:33:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:33:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:33:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:33:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:33:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:33:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:33:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:33:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:33:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:33:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:33:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:33:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:33:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:33:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:33:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:33:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:33:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:33:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:33:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:33:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:33:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:33:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:33:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:33:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:33:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:33:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:33:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:33:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:33:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:33:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:33:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:33:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:33:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:33:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:33:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:33:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:33:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:33:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:33:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:33:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:33:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:33:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:33:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:33:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:33:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:33:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:33:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:33:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:33:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:33:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:33:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:33:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:33:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:33:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:33:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:33:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:33:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:33:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:33:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:33:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:33:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:33:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:33:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:33:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:33:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:33:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:33:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:33:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:33:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:33:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:33:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:33:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:33:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:34:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:34:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:34:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:34:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:34:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:34:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:34:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:34:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:34:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:34:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:34:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:34:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:34:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:34:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:34:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:34:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:34:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:34:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:34:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:34:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:34:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:34:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:34:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:34:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:34:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:34:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:34:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:34:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:34:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:34:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:34:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:34:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:34:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:34:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:34:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:34:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:34:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:34:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:34:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:34:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:34:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:34:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:34:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:34:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:34:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:34:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:34:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:34:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:34:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:34:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:34:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:34:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:34:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:34:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:34:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:34:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:34:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:34:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:34:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:34:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:34:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:34:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:34:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:34:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:34:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:34:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:34:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:34:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:34:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:34:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:34:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:34:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:34:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:34:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:34:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:34:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:34:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:34:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:34:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:34:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:34:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:34:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:34:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:34:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:34:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:34:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:34:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:34:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:34:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:34:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:34:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:34:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:34:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:34:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:34:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:34:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:34:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:34:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:34:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:34:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:34:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:34:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:34:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:34:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:34:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:34:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:34:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:34:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:34:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:34:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:34:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:34:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:34:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:34:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:34:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:34:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:34:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:34:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:34:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:34:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:34:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:34:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:34:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:34:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:34:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:34:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:34:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:34:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:34:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:34:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:34:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:34:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:34:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:34:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:34:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:34:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:34:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:34:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:34:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:34:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:34:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:34:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:34:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:34:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:36:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:36:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:36:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:36:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:36:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:36:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:36:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:36:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:36:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:36:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:36:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:36:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:36:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:36:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:36:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:36:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:36:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:36:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:36:45 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 15:36:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:36:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:36:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:36:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:36:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:36:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:36:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:36:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:36:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:36:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:36:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:36:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:36:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:36:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:36:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:36:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:36:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:36:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:36:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:36:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:36:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:36:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:36:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:36:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:36:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:36:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:36:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:36:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:36:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:36:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:36:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:36:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:36:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:36:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:36:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:36:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:37:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:37:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:37:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:37:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:37:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:37:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:37:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:37:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:37:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:37:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:37:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:37:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:37:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:37:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:37:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:37:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:37:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:37:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:37:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:37:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:37:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:37:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:37:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:37:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:37:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:37:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:37:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:37:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:37:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:37:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:37:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:37:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:37:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:37:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:37:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:37:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:37:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:37:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:37:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:37:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:37:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:37:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:37:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:37:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:37:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:37:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:37:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:37:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:37:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:37:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:37:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:37:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:37:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:37:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:37:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:37:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:37:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:37:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:37:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:37:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:37:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:37:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:37:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:37:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:37:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:37:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:37:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:37:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:37:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:37:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:37:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:37:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:37:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:37:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:37:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:37:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:37:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:37:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:37:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:37:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:37:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:37:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:37:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:37:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:37:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:37:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:37:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:37:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:37:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:37:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:37:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:37:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:37:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:37:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:37:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:37:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:37:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:37:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:37:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:37:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:37:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:37:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:37:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:37:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:37:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:37:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:37:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:37:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:37:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:37:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:37:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:37:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:37:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:37:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:37:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:37:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:37:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:37:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:37:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:37:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:37:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:37:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:37:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:37:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:37:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:37:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:37:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:37:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:37:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:37:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:37:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:37:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:37:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:37:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:37:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:37:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:37:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:37:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:37:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:37:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:37:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:37:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:37:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:37:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:37:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:37:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:37:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:37:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:37:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:37:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:37:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:37:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:37:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:37:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:37:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:37:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:37:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:37:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:37:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:37:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:37:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:37:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:37:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:37:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:37:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:37:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:37:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:37:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:37:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:37:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:37:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:37:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:37:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:37:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:37:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:37:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:37:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:37:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:37:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:37:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:37:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:37:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:37:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:37:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:37:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:37:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:37:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:37:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:37:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:37:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:37:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:37:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:37:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:37:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:37:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:37:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:37:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:37:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:37:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:37:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:37:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:37:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:37:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:37:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:37:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:37:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:37:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:37:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:37:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:37:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:37:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:37:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:37:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:37:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:37:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:37:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:38:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:38:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:38:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:38:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:38:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:38:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:38:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:38:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:38:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:38:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:38:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:38:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:38:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:38:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:38:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:38:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:38:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:38:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:38:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:38:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:38:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:38:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:38:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:38:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:38:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:38:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:38:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:38:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:38:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:38:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:38:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:38:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:38:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:38:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:38:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:38:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:38:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:38:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:38:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:38:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:38:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:38:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:38:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:38:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:38:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:38:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:38:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:38:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:38:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:38:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:38:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:38:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:38:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:38:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:38:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:38:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:38:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:38:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:38:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:38:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:38:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:38:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:38:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:38:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:38:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:38:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:38:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:38:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:38:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:38:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:38:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:38:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:38:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:38:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:38:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:38:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:38:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:38:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:38:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:38:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:38:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:38:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:38:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:38:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:38:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:38:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:38:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:38:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:38:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:38:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:38:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:38:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:38:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:38:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:38:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:38:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:38:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:38:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:38:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:38:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:38:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:38:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:38:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:38:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:38:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:38:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:38:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:38:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:38:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:38:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:38:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:38:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:38:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:38:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:38:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:38:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:38:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:38:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:38:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:38:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:38:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:38:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:38:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:38:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:38:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:38:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:38:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:38:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:38:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:38:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:38:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:38:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:38:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:38:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:38:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:38:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:38:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:38:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:38:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:38:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:38:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:38:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:38:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:38:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:38:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:38:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:38:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:38:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:38:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:38:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:38:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:38:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:38:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:38:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:38:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:38:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:38:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:38:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:38:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:38:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:38:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:38:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:38:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:38:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:38:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:38:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:38:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:38:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:38:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:38:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:38:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:38:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:38:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:38:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:38:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:38:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:38:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:38:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:38:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:38:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:38:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:38:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:38:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:38:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:38:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:38:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:38:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:38:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:38:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:38:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:38:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:38:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:38:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:38:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:38:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:38:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:38:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:38:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:38:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:38:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:38:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:38:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:38:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:38:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:38:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:38:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:38:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:38:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:38:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:38:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:38:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:38:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:38:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:38:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:38:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:38:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:39:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:39:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:39:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:39:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:39:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:39:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:39:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:39:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:39:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:39:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:39:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:39:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:39:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:39:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:39:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:39:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:39:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:39:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:39:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:39:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:39:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:39:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:39:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:39:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:39:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:39:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:39:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:39:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:39:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:39:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:39:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:39:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:39:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:39:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:39:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:39:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:39:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:39:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:39:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:39:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:39:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:39:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:39:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:39:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:39:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:39:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:39:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:39:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:39:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:39:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:39:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:39:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:39:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:39:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:39:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:39:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:39:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:39:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:39:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:39:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:39:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:39:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:39:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:39:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:39:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:39:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:39:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:39:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:39:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:39:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:39:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:39:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:39:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:39:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:39:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:39:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:39:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:39:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:39:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:39:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:39:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:39:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:39:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:39:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:39:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:39:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:39:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:39:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:39:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:39:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:39:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:39:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:39:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:39:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:39:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:39:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:39:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:39:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:39:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:39:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:39:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:39:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:39:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:39:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:39:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:39:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:39:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:39:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:39:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:39:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:39:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:39:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:39:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:39:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:39:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:39:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:39:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:39:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:39:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:39:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:39:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:39:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:39:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:39:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:39:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:39:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:39:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:39:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:39:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:39:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:39:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:39:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:39:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:39:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:39:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:39:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:39:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:39:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:39:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:39:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:39:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:39:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:39:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:39:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:39:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:39:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:39:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:39:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:39:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:39:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:39:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:39:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:39:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:39:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:39:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:39:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:39:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:39:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:39:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:39:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:39:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:39:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:39:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:39:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:39:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:39:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:39:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:39:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:39:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:39:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:39:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:39:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:39:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:39:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:39:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:39:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:39:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:39:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:39:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:39:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:39:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:39:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:39:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:39:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:39:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:39:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:39:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:39:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:39:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:39:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:39:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:39:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:39:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:39:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:39:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:39:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:39:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:39:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:39:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:39:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:39:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:39:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:39:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:39:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:39:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:39:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:39:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:39:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:39:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:39:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:39:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:39:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:39:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:39:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:39:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:39:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:40:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:40:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:40:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:40:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:40:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:40:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:40:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:40:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:40:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:40:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:40:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:40:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:40:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:40:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:40:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:40:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:40:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:40:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:40:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:40:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:40:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:40:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:40:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:40:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:40:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:40:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:40:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:40:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:40:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:40:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:40:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:40:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:40:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:40:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:40:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:40:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:40:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:40:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:40:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:40:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:40:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:40:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:40:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:40:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:40:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:40:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:40:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:40:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:40:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:40:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:40:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:40:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:40:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:40:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:40:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:40:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:40:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:40:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:40:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:40:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:40:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:40:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:40:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:40:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:40:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:40:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:40:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:40:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:40:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:40:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:40:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:40:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:40:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:40:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:40:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:40:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:40:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:40:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:40:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:40:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:40:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:40:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:40:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:40:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:40:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:40:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:40:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:40:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:40:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:40:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:40:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:40:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:40:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:40:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:40:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:40:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:40:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:40:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:40:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:40:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:40:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:40:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:40:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:40:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:40:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:40:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:40:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:40:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:40:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:40:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:40:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:40:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:40:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:40:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:40:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:40:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:40:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:40:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:40:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:40:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:40:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:40:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:40:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:40:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:40:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:40:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:40:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:40:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:40:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:40:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:40:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:40:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:40:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:40:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:40:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:40:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:40:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:40:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:40:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:40:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:40:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:40:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:40:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:40:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:40:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:40:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:40:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:40:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:40:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:40:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:40:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:40:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:40:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:40:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:40:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:40:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:40:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:40:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:40:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:40:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:40:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:40:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:40:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:40:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:40:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:40:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:40:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:40:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:40:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:40:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:40:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:40:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:40:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:40:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:40:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:40:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:40:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:40:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:40:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:40:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:40:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:40:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:40:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:40:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:40:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:40:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:40:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:40:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:40:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:40:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:40:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:40:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:40:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:40:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:40:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:40:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:40:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:40:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:40:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:40:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:40:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:40:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:40:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:40:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:40:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:40:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:40:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:40:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:40:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:40:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:40:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:40:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:40:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:40:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:40:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:40:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:41:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:41:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:41:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:41:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:41:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:41:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:41:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:41:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:41:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:41:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:41:00 rdma06 systemd-journald[790]: Data hash table of /run/log/journal/95379d2f723847b68d9e976dee8967a9/system.journal has a fill level at 75.0 (163751 of 218332 items, 75497472 file size, 461 bytes per hash table item), suggesting rotation.
Apr 18 15:41:00 rdma06 systemd-journald[790]: /run/log/journal/95379d2f723847b68d9e976dee8967a9/system.journal: Journal header limits reached or header out-of-date, rotating.
Apr 18 15:41:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:41:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:41:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:41:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:41:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:41:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:41:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:41:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:41:00 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 18 15:41:00 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 18 15:41:00 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 18 15:41:01 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 18 15:41:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:41:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:41:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:41:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:41:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:41:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:41:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:41:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:41:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:41:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:41:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:41:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:41:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:41:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:41:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:41:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:41:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:41:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:41:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:41:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:41:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:41:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:41:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:41:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:41:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:41:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:41:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:41:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:41:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:41:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:41:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:41:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:41:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:41:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:41:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:41:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:41:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:41:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:41:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:41:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:41:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:41:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:41:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:41:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:41:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:41:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:41:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:41:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:41:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:41:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:41:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:41:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:41:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:41:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:41:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:41:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:41:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:41:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:41:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:41:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:41:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:41:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:41:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:41:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:41:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:41:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:41:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:41:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:41:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:41:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:41:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:41:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:41:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:41:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:41:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:41:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:41:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:41:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:41:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:41:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:41:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:41:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:41:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:41:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:41:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:41:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:41:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:41:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:41:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:41:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:41:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:41:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:41:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:41:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:41:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:41:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:41:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:41:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:41:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:41:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:41:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:41:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:41:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:41:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:41:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:41:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:41:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:41:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:41:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:41:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:41:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:41:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:41:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:41:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:41:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:41:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:41:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:41:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:41:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:41:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:41:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:41:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:41:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:41:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:41:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:41:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:41:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:41:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:41:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:41:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:41:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:41:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:41:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:41:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:41:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:41:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:41:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:41:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:41:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:41:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:41:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:41:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:41:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:41:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:41:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:41:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:41:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:41:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:41:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:41:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:41:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:41:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:41:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:41:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:41:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:41:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:41:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:41:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:41:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:41:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:41:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:41:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:41:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:41:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:41:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:41:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:41:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:41:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:41:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:41:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:41:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:41:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:41:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:41:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:41:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:41:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:41:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:41:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:41:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:41:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:41:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:41:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:41:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:41:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:41:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:41:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:41:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:41:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:41:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:41:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:41:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:41:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:41:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:41:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:41:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:41:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:41:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:41:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:42:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:42:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:42:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:42:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:42:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:42:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:42:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:42:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:42:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:42:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:42:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:42:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:42:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:42:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:42:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:42:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:42:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:42:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:42:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:42:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:42:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:42:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:42:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:42:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:42:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:42:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:42:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:42:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:42:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:42:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:42:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:42:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:42:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:42:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:42:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:42:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:42:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:42:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:42:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:42:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:42:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:42:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:42:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:42:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:42:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:42:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:42:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:42:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:42:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:42:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:42:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:42:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:42:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:42:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:42:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:42:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:42:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:42:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:42:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:42:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:42:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:42:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:42:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:42:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:42:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:42:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:42:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:42:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:42:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:42:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:42:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:42:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:42:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:42:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:42:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:42:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:42:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:42:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:42:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:42:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:42:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:42:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:42:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:42:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:42:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:42:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:42:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:42:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:42:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:42:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:42:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:42:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:42:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:42:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:42:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:42:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:42:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:42:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:42:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:42:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:42:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:42:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:42:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:42:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:42:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:42:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:42:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:42:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:42:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:42:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:42:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:42:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:42:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:42:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:42:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:42:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:42:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:42:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:42:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:42:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:42:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:42:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:42:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:42:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:42:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:42:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:42:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:42:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:42:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:42:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:42:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:42:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:42:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:42:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:42:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:42:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:42:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:42:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:42:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:42:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:42:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:42:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:42:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:42:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:42:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:42:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:42:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:42:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:42:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:42:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:42:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:42:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:42:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:42:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:42:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:42:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:42:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:42:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:42:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:42:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:42:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:42:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:42:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:42:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:42:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:42:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:42:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:42:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:42:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:42:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:42:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:42:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:42:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:42:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:42:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:42:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:42:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:42:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:42:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:42:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:42:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:42:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:42:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:42:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:42:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:42:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:42:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:42:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:42:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:42:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:42:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:42:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:42:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:42:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:42:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:42:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:42:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:42:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:42:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:42:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:42:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:42:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:42:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:42:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:42:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:42:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:42:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:42:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:42:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:42:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:42:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:42:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:42:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:42:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:42:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:42:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:43:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:43:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:43:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:43:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:43:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:43:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:43:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:43:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:43:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:43:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:43:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:43:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:43:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:43:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:43:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:43:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:43:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:43:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:43:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:43:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:43:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:43:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:43:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:43:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:43:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:43:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:43:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:43:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:43:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:43:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:43:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:43:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:43:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:43:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:43:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:43:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:43:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:43:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:43:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:43:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:43:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:43:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:43:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:43:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:43:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:43:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:43:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:43:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:43:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:43:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:43:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:43:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:43:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:43:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:43:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:43:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:43:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:43:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:43:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:43:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:43:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:43:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:43:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:43:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:43:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:43:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:43:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:43:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:43:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:43:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:43:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:43:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:43:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:43:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:43:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:43:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:43:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:43:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:43:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:43:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:43:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:43:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:43:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:43:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:43:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:43:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:43:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:43:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:43:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:43:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:43:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:43:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:43:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:43:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:43:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:43:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:43:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:43:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:43:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:43:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:43:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:43:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:43:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:43:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:43:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:43:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:43:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:43:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:43:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:43:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:43:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:43:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:43:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:43:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:43:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:43:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:43:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:43:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:43:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:43:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:43:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:43:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:43:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:43:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:43:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:43:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:43:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:43:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:43:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:43:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:43:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:43:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:43:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:43:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:43:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:43:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:43:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:43:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:43:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:43:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:43:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:43:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:43:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:43:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:43:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:43:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:43:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:43:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:43:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:43:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:43:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:43:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:43:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:43:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:43:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:43:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:43:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:43:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:43:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:43:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:43:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:43:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:43:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:43:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:43:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:43:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:43:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:43:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:43:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:43:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:43:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:43:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:43:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:43:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:43:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:43:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:43:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:43:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:43:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:43:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:43:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:43:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:43:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:43:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:43:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:43:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:43:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:43:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:43:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:43:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:43:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:43:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:43:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:43:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:43:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:43:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:43:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:43:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:43:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:43:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:43:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:43:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:43:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:43:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:43:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:43:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:43:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:43:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:43:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:43:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:43:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:43:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:43:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:43:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:43:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:43:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:44:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:44:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:44:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:44:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:44:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:44:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:44:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:44:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:44:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:44:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:44:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:44:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:44:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:44:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:44:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:44:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:44:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:44:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:44:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:44:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:44:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:44:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:44:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:44:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:44:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:44:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:44:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:44:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:44:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:44:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:44:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:44:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:44:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:44:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:44:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:44:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:44:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:44:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:44:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:44:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:44:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:44:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:44:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:44:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:44:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:44:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:44:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:44:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:44:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:44:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:44:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:44:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:44:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:44:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:44:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:44:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:44:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:44:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:44:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:44:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:44:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:44:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:44:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:44:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:44:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:44:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:44:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:44:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:44:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:44:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:44:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:44:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:44:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:44:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:44:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:44:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:44:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:44:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:44:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:44:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:44:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:44:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:44:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:44:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:44:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:44:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:44:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:44:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:44:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:44:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:44:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:44:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:44:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:44:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:44:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:44:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:44:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:44:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:44:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:44:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:44:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:44:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:44:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:44:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:44:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:44:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:44:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:44:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:44:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:44:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:44:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:44:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:44:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:44:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:44:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:44:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:44:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:44:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:44:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:44:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:44:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:44:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:44:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:44:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:44:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:44:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:44:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:44:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:44:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:44:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:44:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:44:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:44:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:44:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:44:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:44:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:44:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:44:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:44:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:44:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:44:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:44:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:44:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:44:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:44:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:44:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:44:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:44:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:44:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:44:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:44:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:44:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:44:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:44:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:44:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:44:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:44:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:44:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:44:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:44:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:44:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:44:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:46:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:46:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:46:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:46:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:46:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:46:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:46:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:46:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:46:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:46:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:46:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:46:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:46:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:46:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:46:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:46:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:46:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:46:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:46:51 rdma06 rsyslogd[1041]: imjournal: 5297 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 15:46:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:46:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:46:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:46:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:46:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:46:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:46:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:46:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:46:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:46:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:46:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:46:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:46:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:46:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:46:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:46:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:46:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:46:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:47:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:47:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:47:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:47:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:47:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:47:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:47:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:47:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:47:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:47:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:47:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:47:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:47:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:47:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:47:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:47:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:47:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:47:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:47:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:47:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:47:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:47:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:47:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:47:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:47:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:47:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:47:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:47:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:47:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:47:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:47:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:47:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:47:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:47:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:47:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:47:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:47:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:47:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:47:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:47:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:47:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:47:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:47:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:47:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:47:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:47:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:47:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:47:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:47:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:47:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:47:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:47:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:47:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:47:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:47:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:47:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:47:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:47:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:47:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:47:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:47:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:47:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:47:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:47:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:47:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:47:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:47:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:47:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:47:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:47:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:47:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:47:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:47:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:47:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:47:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:47:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:47:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:47:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:47:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:47:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:47:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:47:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:47:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:47:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:47:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:47:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:47:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:47:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:47:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:47:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:47:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:47:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:47:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:47:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:47:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:47:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:47:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:47:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:47:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:47:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:47:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:47:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:47:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:47:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:47:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:47:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:47:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:47:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:47:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:47:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:47:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:47:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:47:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:47:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:47:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:47:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:47:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:47:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:47:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:47:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:47:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:47:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:47:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:47:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:47:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:47:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:47:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:47:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:47:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:47:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:47:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:47:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:47:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:47:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:47:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:47:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:47:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:47:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:47:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:47:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:47:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:47:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:47:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:47:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:47:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:47:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:47:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:47:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:47:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:47:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:47:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:47:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:47:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:47:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:47:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:47:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:47:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:47:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:47:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:47:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:47:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:47:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:47:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:47:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:47:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:47:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:47:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:47:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:47:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:47:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:47:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:47:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:47:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:47:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:47:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:47:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:47:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:47:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:47:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:47:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:47:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:47:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:47:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:47:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:47:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:47:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:47:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:47:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:47:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:47:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:47:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:47:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:47:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:47:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:47:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:47:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:47:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:47:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:47:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:47:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:47:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:47:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:47:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:47:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:47:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:47:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:47:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:47:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:47:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:47:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:47:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:47:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:47:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:47:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:47:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:47:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:48:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:48:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:48:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:48:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:48:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:48:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:48:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:48:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:48:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:48:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:48:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:48:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:48:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:48:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:48:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:48:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:48:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:48:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:48:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:48:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:48:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:48:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:48:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:48:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:48:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:48:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:48:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:48:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:48:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:48:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:48:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:48:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:48:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:48:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:48:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:48:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:48:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:48:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:48:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:48:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:48:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:48:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:48:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:48:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:48:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:48:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:48:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:48:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:48:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:48:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:48:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:48:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:48:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:48:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:48:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:48:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:48:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:48:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:48:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:48:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:48:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:48:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:48:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:48:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:48:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:48:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:48:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:48:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:48:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:48:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:48:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:48:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:48:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:48:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:48:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:48:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:48:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:48:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:48:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:48:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:48:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:48:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:48:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:48:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:48:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:48:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:48:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:48:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:48:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:48:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:48:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:48:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:48:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:48:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:48:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:48:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:48:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:48:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:48:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:48:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:48:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:48:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:48:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:48:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:48:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:48:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:48:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:48:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:48:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:48:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:48:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:48:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:48:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:48:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:48:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:48:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:48:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:48:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:48:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:48:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:48:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:48:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:48:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:48:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:48:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:48:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:48:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:48:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:48:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:48:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:48:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:48:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:48:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:48:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:48:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:48:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:48:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:48:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:48:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:48:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:48:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:48:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:48:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:48:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:48:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:48:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:48:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:48:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:48:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:48:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:48:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:48:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:48:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:48:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:48:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:48:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:48:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:48:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:48:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:48:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:48:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:48:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:48:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:48:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:48:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:48:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:48:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:48:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:48:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:48:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:48:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:48:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:48:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:48:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:48:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:48:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:48:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:48:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:48:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:48:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:48:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:48:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:48:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:48:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:48:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:48:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:48:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:48:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:48:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:48:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:48:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:48:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:48:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:48:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:48:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:48:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:48:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:48:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:48:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:48:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:48:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:48:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:48:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:48:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:48:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:48:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:48:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:48:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:48:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:48:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:48:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:48:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:48:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:48:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:48:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:48:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:49:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:49:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:49:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:49:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:49:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:49:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:49:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:49:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:49:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:49:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:49:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:49:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:49:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:49:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:49:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:49:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:49:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:49:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:49:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:49:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:49:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:49:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:49:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:49:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:49:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:49:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:49:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:49:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:49:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:49:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:49:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:49:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:49:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:49:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:49:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:49:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:49:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:49:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:49:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:49:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:49:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:49:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:49:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:49:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:49:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:49:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:49:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:49:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:49:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:49:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:49:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:49:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:49:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:49:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:49:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:49:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:49:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:49:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:49:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:49:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:49:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:49:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:49:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:49:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:49:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:49:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:49:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:49:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:49:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:49:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:49:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:49:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:49:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:49:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:49:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:49:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:49:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:49:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:49:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:49:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:49:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:49:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:49:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:49:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:49:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:49:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:49:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:49:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:49:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:49:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:49:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:49:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:49:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:49:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:49:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:49:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:49:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:49:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:49:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:49:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:49:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:49:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:49:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:49:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:49:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:49:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:49:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:49:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:49:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:49:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:49:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:49:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:49:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:49:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:49:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:49:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:49:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:49:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:49:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:49:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:49:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:49:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:49:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:49:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:49:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:49:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:49:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:49:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:49:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:49:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:49:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:49:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:49:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:49:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:49:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:49:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:49:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:49:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:49:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:49:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:49:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:49:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:49:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:49:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:49:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:49:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:49:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:49:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:49:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:49:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:49:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:49:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:49:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:49:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:49:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:49:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:49:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:49:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:49:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:49:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:49:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:49:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:49:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:49:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:49:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:49:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:49:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:49:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:49:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:49:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:49:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:49:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:49:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:49:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:49:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:49:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:49:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:49:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:49:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:49:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:49:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:49:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:49:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:49:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:49:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:49:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:49:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:49:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:49:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:49:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:49:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:49:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:49:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:49:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:49:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:49:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:49:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:49:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:49:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:49:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:49:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:49:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:49:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:49:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:49:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:49:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:49:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:49:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:49:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:49:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:49:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:49:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:49:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:49:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:49:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:49:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:50:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:50:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:50:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:50:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:50:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:50:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:50:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:50:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:50:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:50:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:50:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:50:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:50:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:50:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:50:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:50:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:50:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:50:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:50:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:50:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:50:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:50:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:50:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:50:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:50:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:50:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:50:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:50:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:50:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:50:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:50:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:50:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:50:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:50:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:50:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:50:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:50:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:50:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:50:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:50:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:50:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:50:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:50:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:50:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:50:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:50:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:50:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:50:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:50:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:50:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:50:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:50:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:50:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:50:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:50:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:50:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:50:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:50:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:50:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:50:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:50:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:50:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:50:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:50:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:50:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:50:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:50:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:50:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:50:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:50:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:50:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:50:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:50:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:50:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:50:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:50:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:50:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:50:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:50:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:50:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:50:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:50:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:50:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:50:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:50:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:50:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:50:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:50:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:50:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:50:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:50:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:50:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:50:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:50:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:50:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:50:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:50:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:50:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:50:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:50:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:50:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:50:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:50:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:50:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:50:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:50:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:50:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:50:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:50:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:50:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:50:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:50:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:50:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:50:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:50:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:50:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:50:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:50:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:50:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:50:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:50:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:50:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:50:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:50:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:50:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:50:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:50:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:50:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:50:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:50:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:50:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:50:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:50:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:50:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:50:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:50:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:50:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:50:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:50:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:50:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:50:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:50:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:50:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:50:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:50:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:50:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:50:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:50:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:50:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:50:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:50:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:50:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:50:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:50:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:50:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:50:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:50:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:50:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:50:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:50:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:50:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:50:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:50:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:50:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:50:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:50:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:50:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:50:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:50:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:50:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:50:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:50:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:50:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:50:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:50:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:50:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:50:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:50:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:50:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:50:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:50:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:50:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:50:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:50:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:50:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:50:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:50:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:50:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:50:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:50:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:50:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:50:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:50:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:50:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:50:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:50:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:50:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:50:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:50:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:50:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:50:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:50:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:50:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:50:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:50:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:50:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:50:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:50:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:50:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:50:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:50:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:50:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:50:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:50:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:50:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:50:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:51:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:51:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:51:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:51:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:51:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:51:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:51:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:51:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:51:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:51:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:51:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:51:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:51:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:51:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:51:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:51:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:51:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:51:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:51:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:51:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:51:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:51:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:51:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:51:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:51:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:51:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:51:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:51:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:51:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:51:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:51:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:51:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:51:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:51:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:51:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:51:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:51:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:51:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:51:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:51:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:51:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:51:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:51:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:51:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:51:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:51:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:51:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:51:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:51:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:51:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:51:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:51:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:51:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:51:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:51:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:51:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:51:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:51:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:51:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:51:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:51:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:51:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:51:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:51:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:51:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:51:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:51:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:51:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:51:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:51:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:51:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:51:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:51:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:51:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:51:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:51:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:51:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:51:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:51:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:51:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:51:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:51:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:51:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:51:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:51:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:51:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:51:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:51:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:51:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:51:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:51:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:51:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:51:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:51:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:51:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:51:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:51:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:51:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:51:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:51:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:51:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:51:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:51:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:51:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:51:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:51:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:51:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:51:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:51:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:51:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:51:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:51:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:51:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:51:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:51:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:51:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:51:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:51:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:51:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:51:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:51:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:51:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:51:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:51:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:51:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:51:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:51:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:51:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:51:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:51:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:51:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:51:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:51:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:51:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:51:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:51:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:51:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:51:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:51:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:51:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:51:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:51:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:51:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:51:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:51:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:51:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:51:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:51:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:51:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:51:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:51:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:51:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:51:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:51:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:51:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:51:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:51:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:51:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:51:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:51:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:51:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:51:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:51:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:51:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:51:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:51:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:51:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:51:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:51:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:51:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:51:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:51:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:51:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:51:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:51:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:51:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:51:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:51:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:51:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:51:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:51:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:51:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:51:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:51:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:51:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:51:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:52:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:52:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:52:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:52:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:52:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:52:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:52:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:52:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:52:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:52:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:52:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:52:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:52:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:52:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:52:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:52:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:52:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:52:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:52:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:52:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:52:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:52:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:52:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:52:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:52:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:52:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:52:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:52:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:52:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:52:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:52:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:52:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:52:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:52:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:52:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:52:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:52:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:52:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:52:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:52:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:52:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:52:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:52:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:52:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:52:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:52:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:52:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:52:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:52:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:52:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:52:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:52:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:52:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:52:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:52:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:52:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:52:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:52:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:52:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:52:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:52:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:52:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:52:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:52:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:52:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:52:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:52:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:52:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:52:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:52:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:52:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:52:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:52:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:52:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:52:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:52:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:52:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:52:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:52:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:52:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:52:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:52:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:52:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:52:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:52:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:52:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:52:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:52:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:52:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:52:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:52:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:52:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:52:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:52:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:52:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:52:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:52:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:52:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:52:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:52:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:52:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:52:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:52:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:52:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:52:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:52:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:52:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:52:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:52:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:52:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:52:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:52:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:52:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:52:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:52:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:52:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:52:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:52:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:52:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:52:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:52:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:52:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:52:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:52:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:52:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:52:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:52:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:52:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:52:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:52:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:52:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:52:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:52:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:52:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:52:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:52:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:52:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:52:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:52:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:52:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:52:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:52:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:52:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:52:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:52:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:52:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:52:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:52:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:52:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:52:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:52:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:52:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:52:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:52:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:52:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:52:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:52:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:52:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:52:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:52:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:52:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:52:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:52:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:52:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:52:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:52:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:52:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:52:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:52:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:52:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:52:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:52:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:52:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:52:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:52:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:52:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:52:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:52:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:52:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:52:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:52:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:52:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:52:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:52:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:52:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:52:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:52:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:52:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:52:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:52:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:52:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:52:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:52:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:52:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:52:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:52:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:52:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:52:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:52:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:52:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:52:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:52:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:52:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:52:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:52:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:52:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:52:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:52:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:52:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:52:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:52:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:52:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:52:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:52:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:52:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:52:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:53:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:53:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:53:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:53:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:53:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:53:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:53:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:53:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:53:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:53:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:53:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:53:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:53:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:53:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:53:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:53:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:53:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:53:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:53:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:53:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:53:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:53:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:53:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:53:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:53:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:53:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:53:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:53:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:53:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:53:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:53:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:53:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:53:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:53:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:53:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:53:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:53:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:53:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:53:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:53:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:53:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:53:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:53:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:53:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:53:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:53:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:53:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:53:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:53:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:53:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:53:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:53:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:53:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:53:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:53:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:53:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:53:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:53:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:53:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:53:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:53:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:53:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:53:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:53:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:53:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:53:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:53:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:53:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:53:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:53:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:53:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:53:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:53:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:53:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:53:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:53:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:53:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:53:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:53:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:53:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:53:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:53:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:53:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:53:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:53:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:53:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:53:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:53:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:53:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:53:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:53:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:53:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:53:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:53:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:53:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:53:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:53:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:53:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:53:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:53:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:53:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:53:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:53:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:53:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:53:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:53:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:53:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:53:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:53:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:53:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:53:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:53:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:53:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:53:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:53:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:53:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:53:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:53:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:53:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:53:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:53:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:53:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:53:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:53:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:53:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:53:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:53:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:53:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:53:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:53:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:53:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:53:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:53:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:53:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:53:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:53:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:53:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:53:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:53:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:53:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:53:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:53:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:53:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:53:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:53:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:53:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:53:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:53:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:53:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:53:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:53:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:53:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:53:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:53:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:53:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:53:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:53:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:53:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:53:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:53:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:53:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:53:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:53:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:53:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:53:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:53:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:53:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:53:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:53:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:53:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:53:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:53:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:53:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:53:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:53:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:53:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:53:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:53:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:53:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:53:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:53:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:53:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:53:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:53:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:53:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:53:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:53:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:53:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:53:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:53:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:53:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:53:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:53:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:53:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:53:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:53:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:53:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:53:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:53:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:53:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:53:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:53:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:53:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:53:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:53:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:53:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:53:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:53:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:53:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:53:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:53:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:53:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:53:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:53:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:53:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:53:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:54:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:54:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:54:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:54:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:54:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:54:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:54:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:54:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:54:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:54:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:54:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:54:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:54:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:54:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:54:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:54:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:54:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:54:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:54:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:54:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:54:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:54:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:54:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:54:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:54:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:54:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:54:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:54:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:54:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:54:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:54:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:54:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:54:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:54:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:54:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:54:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:54:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:54:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:54:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:54:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:54:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:54:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:54:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:54:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:54:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:54:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:54:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:54:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:54:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:54:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:54:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:54:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:54:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:54:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:54:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:54:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:54:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:54:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:54:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:54:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:54:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:54:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:54:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:54:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:54:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:54:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:54:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:54:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:54:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:54:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:54:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:54:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:54:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:54:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:54:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:54:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:54:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:54:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:54:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:54:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:54:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:54:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:54:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:54:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:54:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:54:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:54:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:54:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:54:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:54:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:54:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:54:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:54:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:54:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:54:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:54:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:54:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:54:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:54:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:54:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:54:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:54:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:54:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:54:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:54:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:54:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:54:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:54:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:54:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:54:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:54:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:54:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:54:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:54:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:54:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:54:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:54:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:54:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:54:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:54:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:54:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:54:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:54:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:54:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:54:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:54:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:54:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:54:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:54:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:54:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:54:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:54:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:54:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:54:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:54:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:54:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:54:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:54:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:54:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:54:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:54:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:54:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:54:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:54:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:54:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:54:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:54:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:54:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:54:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:54:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:54:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:54:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:54:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:54:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:54:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:54:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:54:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:54:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:54:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:54:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:54:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:54:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:54:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:54:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:54:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:54:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:54:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:54:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:54:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:54:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:54:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:54:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:54:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:54:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:54:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:54:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:54:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:54:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:54:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:54:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:56:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:56:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:56:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:56:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:56:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:56:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:56:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:56:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:56:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:56:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:56:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:56:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:56:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:56:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:56:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:56:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:56:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:56:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:56:56 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 15:57:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:57:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:57:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:57:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:57:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:57:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:57:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:57:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:57:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:57:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:57:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:57:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:57:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:57:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:57:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:57:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:57:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:57:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:57:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:57:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:57:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:57:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:57:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:57:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:57:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:57:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:57:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:57:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:57:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:57:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:57:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:57:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:57:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:57:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:57:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:57:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:57:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:57:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:57:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:57:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:57:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:57:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:57:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:57:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:57:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:57:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:57:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:57:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:57:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:57:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:57:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:57:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:57:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:57:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:57:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:57:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:57:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:57:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:57:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:57:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:57:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:57:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:57:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:57:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:57:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:57:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:57:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:57:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:57:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:57:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:57:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:57:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:57:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:57:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:57:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:57:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:57:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:57:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:57:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:57:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:57:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:57:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:57:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:57:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:57:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:57:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:57:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:57:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:57:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:57:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:57:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:57:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:57:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:57:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:57:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:57:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:57:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:57:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:57:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:57:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:57:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:57:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:57:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:57:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:57:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:57:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:57:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:57:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:57:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:57:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:57:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:57:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:57:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:57:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:57:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:57:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:57:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:57:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:57:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:57:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:57:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:57:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:57:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:57:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:57:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:57:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:57:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:57:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:57:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:57:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:57:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:57:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:57:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:57:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:57:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:57:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:57:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:57:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:57:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:57:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:57:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:57:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:57:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:57:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:57:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:57:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:57:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:57:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:57:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:57:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:57:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:57:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:57:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:57:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:57:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:57:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:57:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:57:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:57:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:57:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:57:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:57:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:57:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:57:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:57:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:57:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:57:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:57:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:57:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:57:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:57:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:57:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:57:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:57:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:57:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:57:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:57:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:57:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:57:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:57:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:57:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:57:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:57:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:57:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:57:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:57:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:57:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:57:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:57:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:57:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:57:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:57:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:57:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:57:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:57:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:57:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:57:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:57:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:57:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:57:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:57:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:57:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:57:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:57:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:57:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:57:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:57:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:57:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:57:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:57:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:57:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:57:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:57:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:57:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:57:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:57:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:58:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:58:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:58:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:58:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:58:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:58:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:58:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:58:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:58:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:58:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:58:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:58:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:58:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:58:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:58:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:58:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:58:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:58:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:58:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:58:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:58:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:58:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:58:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:58:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:58:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:58:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:58:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:58:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:58:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:58:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:58:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:58:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:58:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:58:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:58:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:58:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:58:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:58:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:58:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:58:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:58:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:58:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:58:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:58:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:58:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:58:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:58:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:58:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:58:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:58:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:58:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:58:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:58:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:58:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:58:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:58:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:58:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:58:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:58:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:58:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:58:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:58:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:58:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:58:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:58:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:58:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:58:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:58:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:58:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:58:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:58:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:58:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:58:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:58:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:58:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:58:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:58:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:58:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:58:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:58:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:58:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:58:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:58:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:58:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:58:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:58:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:58:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:58:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:58:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:58:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:58:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:58:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:58:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:58:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:58:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:58:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:58:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:58:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:58:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:58:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:58:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:58:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:58:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:58:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:58:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:58:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:58:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:58:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:58:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:58:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:58:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:58:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:58:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:58:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:58:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:58:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:58:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:58:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:58:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:58:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:58:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:58:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:58:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:58:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:58:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:58:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:58:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:58:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:58:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:58:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:58:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:58:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:58:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:58:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:58:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:58:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:58:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:58:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:58:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:58:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:58:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:58:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:58:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:58:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:58:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:58:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:58:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:58:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:58:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:58:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:58:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:58:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:58:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:58:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:58:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:58:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:58:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:58:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:58:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:58:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:58:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:58:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:58:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:58:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:58:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:58:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:58:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:58:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:58:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:58:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:58:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:58:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:58:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:58:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:58:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:58:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:58:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:58:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:58:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:58:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:58:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:58:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:58:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:58:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:58:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:58:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:58:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:58:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:58:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:58:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:58:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:58:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:58:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:58:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:58:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:58:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:58:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:58:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:58:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:58:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:58:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:58:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:58:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:58:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:58:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:58:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:58:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:58:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:58:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:58:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:58:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:58:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:58:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:58:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:58:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:58:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:59:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:59:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:59:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:59:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:59:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:59:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:59:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:59:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:59:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:59:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:59:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:59:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:59:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:59:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:59:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:59:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:59:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:59:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:59:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:59:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:59:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:59:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:59:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:59:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:59:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:59:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:59:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:59:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:59:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:59:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:59:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:59:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:59:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:59:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:59:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:59:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:59:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:59:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:59:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:59:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:59:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:59:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:59:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:59:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:59:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:59:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:59:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:59:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:59:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:59:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:59:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:59:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:59:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:59:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:59:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:59:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:59:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:59:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:59:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:59:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:59:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:59:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:59:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:59:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:59:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:59:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:59:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:59:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:59:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:59:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:59:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:59:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:59:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:59:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:59:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:59:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:59:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:59:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:59:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:59:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:59:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:59:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:59:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:59:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:59:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:59:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:59:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:59:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:59:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:59:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:59:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:59:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:59:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:59:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:59:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:59:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:59:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:59:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:59:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:59:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:59:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:59:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:59:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:59:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:59:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:59:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:59:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:59:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:59:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:59:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:59:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:59:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:59:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:59:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:59:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:59:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:59:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:59:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:59:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:59:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:59:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:59:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:59:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:59:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:59:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:59:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:59:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:59:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:59:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:59:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:59:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:59:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:59:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:59:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:59:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:59:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:59:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:59:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:59:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:59:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:59:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:59:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:59:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:59:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:59:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:59:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:59:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:59:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:59:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:59:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:59:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:59:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:59:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:59:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:59:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:59:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:59:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:59:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:59:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:59:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:59:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:59:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:59:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:59:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:59:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:59:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:59:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:59:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:59:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:59:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:59:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:59:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:59:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:59:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:59:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:59:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:59:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:59:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:59:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:59:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:59:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:59:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:59:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:59:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:59:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:59:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:59:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:59:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:59:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:59:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:59:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:59:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:59:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:59:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:59:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:59:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:59:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:59:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 15:59:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 15:59:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:59:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:59:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:59:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 15:59:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 15:59:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 15:59:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 15:59:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 15:59:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 15:59:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 15:59:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 15:59:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 15:59:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 15:59:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 15:59:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 15:59:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 15:59:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:00:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:00:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:00:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:00:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:00:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:00:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:00:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:00:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:00:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:00:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:00:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:00:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:00:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:00:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:00:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:00:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:00:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:00:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:00:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:00:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:00:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:00:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:00:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:00:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:00:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:00:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:00:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:00:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:00:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:00:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:00:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:00:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:00:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:00:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:00:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:00:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:00:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:00:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:00:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:00:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:00:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:00:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:00:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:00:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:00:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:00:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:00:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:00:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:00:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:00:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:00:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:00:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:00:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:00:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:00:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:00:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:00:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:00:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:00:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:00:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:00:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:00:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:00:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:00:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:00:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:00:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:00:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:00:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:00:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:00:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:00:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:00:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:00:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:00:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:00:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:00:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:00:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:00:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:00:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:00:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:00:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:00:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:00:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:00:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:00:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:00:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:00:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:00:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:00:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:00:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:00:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:00:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:00:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:00:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:00:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:00:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:00:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:00:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:00:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:00:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:00:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:00:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:00:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:00:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:00:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:00:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:00:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:00:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:00:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:00:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:00:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:00:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:00:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:00:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:00:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:00:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:00:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:00:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:00:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:00:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:00:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:00:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:00:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:00:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:00:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:00:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:00:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:00:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:00:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:00:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:00:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:00:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:00:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:00:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:00:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:00:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:00:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:00:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:00:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:00:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:00:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:00:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:00:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:00:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:00:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:00:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:00:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:00:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:00:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:00:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:00:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:00:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:00:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:00:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:00:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:00:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:00:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:00:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:00:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:00:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:00:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:00:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:00:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:00:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:00:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:00:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:00:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:00:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:00:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:00:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:00:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:00:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:00:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:00:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:00:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:00:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:00:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:00:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:00:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:00:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:00:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:00:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:00:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:00:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:00:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:00:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:00:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:00:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:00:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:00:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:00:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:00:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:00:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:00:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:00:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:00:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:00:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:00:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:00:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:00:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:00:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:00:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:00:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:00:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:00:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:00:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:00:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:00:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:00:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:00:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:00:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:00:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:00:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:00:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:00:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:00:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:01:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:01:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:01:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:01:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:01:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:01:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:01:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:01:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:01:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:01:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:01:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:01:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:01:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:01:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:01:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:01:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:01:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:01:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:01:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:01:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:01:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:01:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:01:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:01:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:01:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:01:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:01:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:01:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:01:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:01:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:01:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:01:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:01:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:01:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:01:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:01:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:01:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:01:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:01:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:01:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:01:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:01:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:01:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:01:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:01:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:01:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:01:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:01:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:01:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:01:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:01:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:01:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:01:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:01:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:01:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:01:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:01:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:01:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:01:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:01:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:01:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:01:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:01:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:01:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:01:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:01:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:01:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:01:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:01:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:01:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:01:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:01:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:01:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:01:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:01:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:01:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:01:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:01:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:01:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:01:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:01:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:01:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:01:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:01:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:01:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:01:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:01:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:01:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:01:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:01:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:01:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:01:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:01:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:01:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:01:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:01:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:01:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:01:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:01:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:01:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:01:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:01:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:01:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:01:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:01:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:01:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:01:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:01:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:01:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:01:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:01:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:01:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:01:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:01:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:01:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:01:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:01:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:01:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:01:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:01:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:01:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:01:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:01:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:01:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:01:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:01:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:01:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:01:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:01:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:01:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:01:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:01:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:01:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:01:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:01:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:01:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:01:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:01:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:01:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:01:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:01:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:01:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:01:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:01:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:01:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:01:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:01:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:01:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:01:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:01:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:01:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:01:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:01:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:01:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:01:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:01:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:01:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:01:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:01:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:01:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:01:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:01:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:01:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:01:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:01:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:01:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:01:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:01:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:01:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:01:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:01:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:01:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:01:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:01:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:01:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:01:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:01:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:01:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:01:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:01:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:01:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:01:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:01:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:01:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:01:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:01:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:01:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:01:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:01:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:01:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:01:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:01:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:01:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:01:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:01:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:01:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:01:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:01:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:01:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:01:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:01:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:01:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:01:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:01:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:01:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:01:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:01:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:01:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:01:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:01:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:01:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:01:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:01:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:01:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:01:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:01:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:02:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:02:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:02:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:02:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:02:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:02:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:02:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:02:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:02:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:02:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:02:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:02:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:02:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:02:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:02:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:02:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:02:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:02:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:02:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:02:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:02:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:02:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:02:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:02:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:02:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:02:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:02:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:02:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:02:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:02:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:02:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:02:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:02:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:02:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:02:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:02:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:02:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:02:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:02:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:02:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:02:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:02:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:02:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:02:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:02:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:02:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:02:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:02:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:02:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:02:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:02:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:02:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:02:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:02:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:02:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:02:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:02:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:02:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:02:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:02:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:02:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:02:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:02:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:02:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:02:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:02:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:02:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:02:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:02:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:02:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:02:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:02:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:02:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:02:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:02:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:02:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:02:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:02:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:02:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:02:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:02:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:02:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:02:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:02:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:02:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:02:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:02:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:02:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:02:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:02:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:02:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:02:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:02:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:02:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:02:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:02:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:02:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:02:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:02:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:02:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:02:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:02:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:02:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:02:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:02:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:02:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:02:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:02:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:02:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:02:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:02:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:02:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:02:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:02:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:02:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:02:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:02:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:02:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:02:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:02:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:02:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:02:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:02:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:02:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:02:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:02:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:02:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:02:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:02:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:02:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:02:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:02:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:02:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:02:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:02:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:02:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:02:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:02:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:02:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:02:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:02:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:02:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:02:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:02:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:02:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:02:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:02:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:02:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:02:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:02:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:02:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:02:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:02:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:02:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:02:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:02:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:02:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:02:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:02:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:02:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:02:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:02:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:02:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:02:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:02:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:02:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:02:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:02:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:02:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:02:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:02:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:02:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:02:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:02:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:02:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:02:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:02:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:02:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:02:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:02:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:02:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:02:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:02:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:02:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:02:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:02:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:02:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:02:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:02:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:02:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:02:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:02:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:02:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:02:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:02:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:02:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:02:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:02:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:02:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:02:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:02:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:02:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:02:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:02:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:02:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:02:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:02:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:02:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:02:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:02:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:02:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:02:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:02:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:02:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:02:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:02:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:03:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:03:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:03:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:03:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:03:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:03:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:03:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:03:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:03:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:03:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:03:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:03:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:03:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:03:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:03:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:03:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:03:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:03:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:03:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:03:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:03:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:03:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:03:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:03:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:03:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:03:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:03:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:03:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:03:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:03:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:03:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:03:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:03:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:03:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:03:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:03:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:03:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:03:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:03:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:03:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:03:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:03:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:03:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:03:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:03:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:03:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:03:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:03:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:03:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:03:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:03:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:03:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:03:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:03:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:03:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:03:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:03:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:03:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:03:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:03:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:03:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:03:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:03:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:03:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:03:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:03:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:03:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:03:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:03:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:03:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:03:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:03:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:03:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:03:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:03:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:03:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:03:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:03:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:03:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:03:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:03:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:03:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:03:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:03:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:03:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:03:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:03:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:03:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:03:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:03:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:03:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:03:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:03:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:03:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:03:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:03:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:03:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:03:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:03:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:03:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:03:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:03:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:03:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:03:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:03:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:03:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:03:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:03:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:03:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:03:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:03:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:03:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:03:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:03:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:03:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:03:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:03:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:03:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:03:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:03:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:03:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:03:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:03:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:03:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:03:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:03:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:03:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:03:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:03:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:03:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:03:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:03:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:03:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:03:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:03:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:03:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:03:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:03:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:03:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:03:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:03:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:03:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:03:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:03:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:03:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:03:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:03:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:03:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:03:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:03:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:03:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:03:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:03:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:03:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:03:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:03:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:03:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:03:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:03:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:03:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:03:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:03:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:03:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:03:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:03:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:03:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:03:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:03:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:03:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:03:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:03:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:03:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:03:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:03:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:03:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:03:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:03:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:03:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:03:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:03:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:03:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:03:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:03:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:03:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:03:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:03:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:03:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:03:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:03:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:03:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:03:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:03:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:03:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:03:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:03:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:03:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:03:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:03:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:03:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:03:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:03:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:03:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:03:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:03:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:03:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:03:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:03:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:03:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:03:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:03:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:03:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:03:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:03:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:03:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:03:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:03:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:04:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:04:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:04:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:04:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:04:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:04:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:04:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:04:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:04:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:04:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:04:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:04:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:04:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:04:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:04:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:04:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:04:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:04:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:04:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:04:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:04:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:04:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:04:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:04:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:04:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:04:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:04:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:04:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:04:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:04:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:04:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:04:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:04:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:04:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:04:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:04:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:04:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:04:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:04:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:04:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:04:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:04:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:04:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:04:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:04:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:04:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:04:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:04:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:04:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:04:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:04:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:04:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:04:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:04:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:04:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:04:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:04:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:04:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:04:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:04:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:04:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:04:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:04:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:04:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:04:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:04:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:04:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:04:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:04:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:04:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:04:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:04:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:04:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:04:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:04:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:04:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:04:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:04:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:04:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:04:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:04:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:04:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:04:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:04:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:04:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:04:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:04:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:04:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:04:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:04:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:04:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:04:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:04:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:04:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:04:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:04:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:04:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:04:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:04:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:04:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:04:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:04:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:04:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:04:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:04:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:04:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:04:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:04:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:04:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:04:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:04:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:04:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:04:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:04:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:04:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:04:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:04:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:04:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:04:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:04:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:04:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:04:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:04:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:04:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:04:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:04:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:04:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:04:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:04:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:04:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:04:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:04:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:04:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:04:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:04:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:04:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:04:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:04:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:04:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:04:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:04:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:04:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:04:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:04:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:04:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:04:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:04:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:04:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:04:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:04:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:04:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:04:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:04:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:04:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:04:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:04:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:04:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:04:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:04:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:04:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:04:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:04:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:04:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:04:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:04:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:04:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:04:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:04:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:04:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:04:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:04:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:04:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:04:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:04:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:04:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:04:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:04:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:04:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:04:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:04:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:04:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:04:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:04:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:04:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:04:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:04:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:04:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:04:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:04:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:04:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:04:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:04:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:04:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:04:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:04:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:04:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:04:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:04:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:07:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:07:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:07:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:07:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:07:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:07:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:07:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:07:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:07:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:07:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:07:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:07:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:07:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:07:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:07:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:07:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:07:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:07:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:07:01 rdma06 rsyslogd[1041]: imjournal: 5295 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 16:07:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:07:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:07:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:07:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:07:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:07:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:07:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:07:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:07:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:07:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:07:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:07:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:07:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:07:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:07:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:07:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:07:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:07:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:07:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:07:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:07:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:07:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:07:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:07:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:07:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:07:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:07:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:07:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:07:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:07:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:07:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:07:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:07:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:07:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:07:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:07:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:07:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:07:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:07:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:07:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:07:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:07:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:07:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:07:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:07:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:07:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:07:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:07:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:07:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:07:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:07:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:07:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:07:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:07:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:07:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:07:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:07:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:07:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:07:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:07:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:07:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:07:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:07:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:07:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:07:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:07:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:07:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:07:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:07:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:07:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:07:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:07:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:07:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:07:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:07:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:07:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:07:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:07:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:07:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:07:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:07:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:07:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:07:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:07:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:07:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:07:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:07:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:07:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:07:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:07:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:07:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:07:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:07:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:07:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:07:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:07:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:07:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:07:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:07:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:07:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:07:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:07:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:07:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:07:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:07:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:07:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:07:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:07:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:07:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:07:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:07:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:07:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:07:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:07:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:07:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:07:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:07:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:07:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:07:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:07:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:07:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:07:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:07:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:07:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:07:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:07:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:07:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:07:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:07:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:07:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:07:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:07:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:07:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:07:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:07:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:07:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:07:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:07:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:07:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:07:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:07:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:07:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:07:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:07:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:07:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:07:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:07:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:07:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:07:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:07:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:07:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:07:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:07:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:07:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:07:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:07:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:07:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:07:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:07:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:07:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:07:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:07:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:07:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:07:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:07:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:07:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:07:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:07:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:07:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:07:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:07:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:07:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:07:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:07:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:07:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:07:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:07:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:07:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:07:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:07:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:07:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:07:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:07:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:07:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:07:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:07:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:07:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:07:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:07:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:07:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:07:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:07:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:07:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:07:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:07:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:07:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:07:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:07:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:08:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:08:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:08:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:08:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:08:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:08:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:08:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:08:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:08:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:08:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:08:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:08:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:08:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:08:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:08:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:08:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:08:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:08:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:08:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:08:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:08:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:08:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:08:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:08:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:08:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:08:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:08:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:08:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:08:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:08:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:08:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:08:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:08:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:08:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:08:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:08:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:08:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:08:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:08:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:08:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:08:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:08:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:08:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:08:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:08:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:08:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:08:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:08:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:08:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:08:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:08:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:08:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:08:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:08:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:08:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:08:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:08:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:08:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:08:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:08:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:08:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:08:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:08:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:08:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:08:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:08:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:08:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:08:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:08:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:08:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:08:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:08:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:08:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:08:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:08:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:08:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:08:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:08:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:08:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:08:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:08:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:08:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:08:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:08:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:08:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:08:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:08:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:08:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:08:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:08:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:08:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:08:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:08:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:08:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:08:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:08:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:08:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:08:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:08:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:08:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:08:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:08:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:08:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:08:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:08:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:08:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:08:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:08:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:08:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:08:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:08:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:08:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:08:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:08:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:08:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:08:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:08:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:08:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:08:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:08:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:08:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:08:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:08:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:08:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:08:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:08:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:08:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:08:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:08:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:08:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:08:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:08:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:08:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:08:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:08:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:08:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:08:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:08:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:08:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:08:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:08:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:08:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:08:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:08:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:08:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:08:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:08:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:08:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:08:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:08:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:08:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:08:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:08:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:08:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:08:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:08:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:08:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:08:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:08:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:08:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:08:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:08:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:08:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:08:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:08:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:08:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:08:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:08:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:08:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:08:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:08:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:08:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:08:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:08:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:08:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:08:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:08:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:08:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:08:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:08:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:08:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:08:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:08:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:08:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:08:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:08:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:08:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:08:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:08:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:08:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:08:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:08:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:08:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:08:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:08:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:08:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:08:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:08:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:08:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:08:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:08:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:08:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:08:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:08:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:08:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:08:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:08:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:08:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:08:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:08:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:08:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:08:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:08:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:08:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:08:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:08:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:09:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:09:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:09:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:09:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:09:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:09:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:09:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:09:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:09:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:09:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:09:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:09:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:09:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:09:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:09:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:09:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:09:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:09:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:09:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:09:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:09:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:09:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:09:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:09:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:09:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:09:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:09:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:09:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:09:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:09:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:09:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:09:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:09:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:09:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:09:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:09:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:09:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:09:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:09:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:09:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:09:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:09:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:09:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:09:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:09:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:09:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:09:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:09:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:09:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:09:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:09:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:09:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:09:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:09:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:09:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:09:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:09:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:09:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:09:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:09:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:09:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:09:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:09:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:09:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:09:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:09:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:09:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:09:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:09:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:09:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:09:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:09:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:09:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:09:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:09:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:09:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:09:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:09:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:09:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:09:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:09:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:09:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:09:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:09:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:09:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:09:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:09:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:09:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:09:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:09:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:09:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:09:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:09:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:09:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:09:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:09:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:09:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:09:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:09:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:09:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:09:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:09:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:09:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:09:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:09:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:09:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:09:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:09:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:09:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:09:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:09:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:09:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:09:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:09:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:09:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:09:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:09:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:09:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:09:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:09:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:09:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:09:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:09:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:09:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:09:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:09:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:09:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:09:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:09:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:09:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:09:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:09:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:09:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:09:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:09:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:09:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:09:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:09:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:09:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:09:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:09:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:09:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:09:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:09:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:09:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:09:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:09:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:09:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:09:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:09:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:09:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:09:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:09:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:09:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:09:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:09:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:09:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:09:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:09:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:09:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:09:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:09:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:09:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:09:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:09:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:09:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:09:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:09:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:09:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:09:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:09:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:09:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:09:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:09:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:09:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:09:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:09:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:09:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:09:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:09:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:09:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:09:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:09:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:09:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:09:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:09:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:09:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:09:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:09:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:09:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:09:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:09:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:09:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:09:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:09:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:09:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:09:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:09:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:09:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:09:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:09:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:09:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:09:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:09:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:09:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:09:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:09:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:09:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:09:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:09:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:09:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:09:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:09:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:09:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:09:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:09:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:10:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:10:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:10:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:10:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:10:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:10:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:10:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:10:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:10:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:10:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:10:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:10:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:10:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:10:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:10:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:10:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:10:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:10:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:10:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:10:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:10:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:10:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:10:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:10:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:10:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:10:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:10:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:10:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:10:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:10:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:10:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:10:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:10:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:10:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:10:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:10:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:10:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:10:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:10:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:10:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:10:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:10:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:10:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:10:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:10:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:10:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:10:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:10:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:10:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:10:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:10:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:10:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:10:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:10:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:10:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:10:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:10:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:10:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:10:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:10:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:10:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:10:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:10:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:10:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:10:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:10:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:10:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:10:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:10:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:10:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:10:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:10:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:10:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:10:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:10:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:10:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:10:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:10:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:10:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:10:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:10:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:10:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:10:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:10:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:10:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:10:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:10:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:10:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:10:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:10:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:10:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:10:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:10:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:10:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:10:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:10:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:10:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:10:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:10:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:10:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:10:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:10:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:10:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:10:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:10:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:10:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:10:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:10:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:10:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:10:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:10:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:10:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:10:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:10:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:10:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:10:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:10:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:10:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:10:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:10:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:10:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:10:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:10:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:10:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:10:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:10:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:10:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:10:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:10:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:10:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:10:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:10:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:10:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:10:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:10:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:10:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:10:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:10:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:10:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:10:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:10:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:10:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:10:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:10:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:10:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:10:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:10:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:10:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:10:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:10:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:10:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:10:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:10:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:10:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:10:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:10:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:10:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:10:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:10:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:10:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:10:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:10:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:10:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:10:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:10:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:10:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:10:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:10:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:10:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:10:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:10:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:10:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:10:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:10:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:10:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:10:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:10:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:10:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:10:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:10:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:10:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:10:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:10:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:10:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:10:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:10:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:10:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:10:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:10:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:10:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:10:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:10:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:10:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:10:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:10:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:10:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:10:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:10:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:10:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:10:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:10:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:10:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:10:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:10:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:10:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:10:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:10:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:10:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:10:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:10:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:10:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:10:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:10:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:10:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:10:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:10:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:11:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:11:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:11:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:11:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:11:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:11:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:11:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:11:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:11:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:11:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:11:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:11:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:11:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:11:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:11:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:11:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:11:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:11:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:11:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:11:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:11:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:11:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:11:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:11:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:11:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:11:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:11:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:11:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:11:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:11:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:11:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:11:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:11:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:11:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:11:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:11:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:11:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:11:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:11:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:11:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:11:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:11:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:11:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:11:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:11:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:11:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:11:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:11:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:11:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:11:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:11:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:11:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:11:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:11:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:11:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:11:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:11:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:11:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:11:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:11:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:11:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:11:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:11:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:11:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:11:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:11:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:11:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:11:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:11:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:11:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:11:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:11:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:11:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:11:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:11:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:11:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:11:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:11:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:11:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:11:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:11:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:11:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:11:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:11:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:11:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:11:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:11:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:11:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:11:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:11:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:11:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:11:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:11:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:11:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:11:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:11:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:11:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:11:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:11:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:11:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:11:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:11:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:11:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:11:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:11:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:11:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:11:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:11:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:11:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:11:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:11:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:11:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:11:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:11:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:11:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:11:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:11:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:11:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:11:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:11:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:11:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:11:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:11:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:11:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:11:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:11:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:11:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:11:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:11:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:11:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:11:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:11:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:11:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:11:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:11:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:11:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:11:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:11:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:11:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:11:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:11:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:11:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:11:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:11:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:11:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:11:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:11:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:11:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:11:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:11:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:11:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:11:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:11:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:11:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:11:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:11:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:11:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:11:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:11:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:11:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:11:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:11:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:11:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:11:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:11:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:11:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:11:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:11:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:11:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:11:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:11:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:11:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:11:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:11:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:11:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:11:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:11:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:11:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:11:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:11:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:11:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:11:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:11:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:11:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:11:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:11:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:11:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:11:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:11:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:11:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:11:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:11:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:11:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:11:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:11:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:11:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:11:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:11:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:11:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:11:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:11:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:11:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:11:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:11:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:11:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:11:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:11:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:11:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:11:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:11:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:11:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:11:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:11:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:11:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:11:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:11:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:12:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:12:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:12:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:12:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:12:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:12:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:12:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:12:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:12:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:12:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:12:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:12:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:12:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:12:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:12:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:12:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:12:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:12:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:12:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:12:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:12:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:12:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:12:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:12:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:12:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:12:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:12:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:12:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:12:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:12:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:12:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:12:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:12:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:12:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:12:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:12:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:12:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:12:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:12:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:12:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:12:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:12:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:12:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:12:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:12:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:12:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:12:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:12:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:12:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:12:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:12:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:12:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:12:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:12:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:12:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:12:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:12:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:12:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:12:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:12:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:12:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:12:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:12:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:12:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:12:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:12:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:12:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:12:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:12:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:12:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:12:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:12:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:12:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:12:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:12:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:12:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:12:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:12:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:12:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:12:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:12:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:12:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:12:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:12:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:12:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:12:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:12:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:12:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:12:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:12:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:12:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:12:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:12:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:12:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:12:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:12:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:12:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:12:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:12:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:12:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:12:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:12:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:12:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:12:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:12:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:12:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:12:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:12:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:12:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:12:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:12:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:12:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:12:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:12:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:12:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:12:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:12:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:12:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:12:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:12:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:12:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:12:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:12:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:12:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:12:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:12:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:12:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:12:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:12:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:12:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:12:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:12:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:12:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:12:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:12:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:12:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:12:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:12:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:12:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:12:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:12:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:12:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:12:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:12:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:12:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:12:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:12:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:12:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:12:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:12:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:12:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:12:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:12:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:12:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:12:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:12:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:12:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:12:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:12:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:12:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:12:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:12:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:12:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:12:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:12:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:12:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:12:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:12:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:12:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:12:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:12:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:12:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:12:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:12:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:12:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:12:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:12:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:12:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:12:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:12:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:12:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:12:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:12:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:12:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:12:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:12:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:12:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:12:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:12:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:12:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:12:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:12:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:12:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:12:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:12:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:12:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:12:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:12:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:12:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:12:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:12:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:12:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:12:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:12:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:12:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:12:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:12:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:12:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:12:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:12:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:12:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:12:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:12:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:12:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:12:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:12:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:13:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:13:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:13:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:13:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:13:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:13:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:13:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:13:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:13:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:13:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:13:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:13:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:13:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:13:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:13:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:13:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:13:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:13:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:13:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:13:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:13:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:13:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:13:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:13:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:13:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:13:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:13:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:13:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:13:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:13:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:13:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:13:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:13:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:13:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:13:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:13:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:13:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:13:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:13:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:13:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:13:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:13:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:13:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:13:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:13:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:13:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:13:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:13:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:13:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:13:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:13:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:13:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:13:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:13:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:13:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:13:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:13:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:13:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:13:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:13:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:13:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:13:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:13:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:13:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:13:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:13:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:13:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:13:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:13:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:13:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:13:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:13:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:13:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:13:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:13:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:13:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:13:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:13:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:13:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:13:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:13:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:13:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:13:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:13:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:13:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:13:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:13:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:13:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:13:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:13:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:13:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:13:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:13:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:13:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:13:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:13:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:13:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:13:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:13:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:13:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:13:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:13:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:13:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:13:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:13:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:13:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:13:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:13:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:13:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:13:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:13:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:13:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:13:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:13:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:13:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:13:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:13:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:13:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:13:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:13:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:13:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:13:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:13:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:13:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:13:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:13:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:13:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:13:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:13:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:13:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:13:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:13:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:13:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:13:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:13:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:13:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:13:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:13:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:13:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:13:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:13:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:13:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:13:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:13:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:13:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:13:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:13:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:13:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:13:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:13:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:13:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:13:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:13:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:13:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:13:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:13:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:13:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:13:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:13:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:13:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:13:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:13:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:13:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:13:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:13:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:13:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:13:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:13:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:13:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:13:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:13:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:13:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:13:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:13:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:13:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:13:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:13:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:13:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:13:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:13:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:13:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:13:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:13:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:13:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:13:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:13:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:13:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:13:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:13:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:13:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:13:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:13:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:13:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:13:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:13:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:13:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:13:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:13:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:13:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:13:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:13:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:13:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:13:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:13:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:13:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:13:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:13:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:13:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:13:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:13:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:13:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:13:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:13:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:13:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:13:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:13:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:14:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:14:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:14:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:14:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:14:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:14:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:14:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:14:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:14:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:14:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:14:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:14:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:14:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:14:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:14:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:14:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:14:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:14:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:14:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:14:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:14:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:14:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:14:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:14:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:14:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:14:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:14:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:14:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:14:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:14:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:14:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:14:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:14:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:14:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:14:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:14:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:14:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:14:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:14:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:14:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:14:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:14:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:14:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:14:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:14:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:14:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:14:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:14:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:14:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:14:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:14:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:14:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:14:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:14:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:14:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:14:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:14:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:14:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:14:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:14:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:14:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:14:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:14:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:14:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:14:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:14:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:14:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:14:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:14:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:14:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:14:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:14:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:14:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:14:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:14:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:14:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:14:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:14:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:14:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:14:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:14:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:14:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:14:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:14:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:14:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:14:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:14:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:14:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:14:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:14:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:14:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:14:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:14:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:14:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:14:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:14:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:14:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:14:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:14:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:14:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:14:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:14:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:14:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:14:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:14:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:14:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:14:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:14:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:14:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:14:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:14:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:14:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:14:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:14:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:14:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:14:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:14:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:14:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:14:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:14:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:14:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:14:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:14:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:14:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:14:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:14:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:14:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:14:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:14:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:14:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:14:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:14:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:14:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:14:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:14:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:14:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:14:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:14:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:14:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:14:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:14:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:14:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:14:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:14:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:14:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:14:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:14:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:14:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:14:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:14:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:14:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:14:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:14:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:14:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:14:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:14:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:14:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:14:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:14:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:14:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:14:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:14:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:14:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:14:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:14:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:14:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:14:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:14:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:14:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:14:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:14:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:14:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:14:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:14:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:14:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:14:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:14:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:14:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:14:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:14:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:14:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:14:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:14:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:14:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:14:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:14:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:14:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:14:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:14:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:14:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:14:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:14:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:14:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:14:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:14:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:14:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:14:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:14:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:14:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:14:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:14:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:14:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:14:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:14:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:14:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:14:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:14:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:14:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:14:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:14:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:14:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:14:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:14:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:14:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:14:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:14:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:17:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:17:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:17:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:17:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:17:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:17:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:17:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:17:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:17:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:17:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:17:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:17:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:17:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:17:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:17:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:17:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:17:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:17:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:17:02 rdma06 rsyslogd[1041]: imjournal: 5082 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 16:17:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:17:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:17:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:17:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:17:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:17:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:17:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:17:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:17:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:17:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:17:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:17:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:17:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:17:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:17:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:17:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:17:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:17:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:17:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:17:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:17:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:17:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:17:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:17:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:17:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:17:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:17:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:17:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:17:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:17:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:17:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:17:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:17:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:17:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:17:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:17:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:17:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:17:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:17:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:17:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:17:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:17:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:17:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:17:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:17:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:17:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:17:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:17:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:17:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:17:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:17:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:17:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:17:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:17:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:17:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:17:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:17:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:17:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:17:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:17:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:17:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:17:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:17:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:17:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:17:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:17:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:17:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:17:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:17:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:17:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:17:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:17:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:17:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:17:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:17:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:17:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:17:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:17:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:17:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:17:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:17:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:17:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:17:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:17:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:17:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:17:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:17:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:17:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:17:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:17:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:17:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:17:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:17:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:17:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:17:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:17:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:17:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:17:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:17:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:17:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:17:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:17:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:17:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:17:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:17:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:17:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:17:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:17:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:17:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:17:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:17:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:17:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:17:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:17:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:17:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:17:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:17:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:17:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:17:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:17:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:17:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:17:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:17:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:17:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:17:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:17:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:17:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:17:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:17:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:17:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:17:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:17:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:17:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:17:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:17:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:17:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:17:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:17:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:17:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:17:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:17:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:17:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:17:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:17:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:17:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:17:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:17:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:17:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:17:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:17:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:17:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:17:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:17:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:17:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:17:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:17:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:17:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:17:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:17:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:17:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:17:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:17:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:17:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:17:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:17:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:17:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:17:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:17:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:17:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:17:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:17:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:17:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:17:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:17:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:17:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:17:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:17:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:17:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:17:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:17:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:17:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:17:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:17:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:17:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:17:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:17:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:17:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:17:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:17:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:17:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:17:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:17:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:17:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:17:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:17:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:17:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:17:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:17:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:18:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:18:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:18:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:18:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:18:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:18:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:18:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:18:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:18:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:18:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:18:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:18:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:18:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:18:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:18:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:18:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:18:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:18:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:18:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:18:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:18:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:18:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:18:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:18:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:18:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:18:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:18:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:18:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:18:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:18:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:18:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:18:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:18:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:18:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:18:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:18:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:18:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:18:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:18:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:18:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:18:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:18:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:18:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:18:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:18:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:18:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:18:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:18:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:18:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:18:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:18:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:18:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:18:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:18:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:18:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:18:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:18:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:18:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:18:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:18:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:18:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:18:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:18:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:18:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:18:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:18:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:18:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:18:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:18:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:18:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:18:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:18:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:18:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:18:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:18:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:18:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:18:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:18:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:18:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:18:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:18:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:18:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:18:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:18:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:18:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:18:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:18:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:18:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:18:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:18:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:18:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:18:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:18:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:18:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:18:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:18:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:18:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:18:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:18:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:18:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:18:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:18:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:18:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:18:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:18:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:18:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:18:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:18:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:18:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:18:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:18:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:18:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:18:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:18:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:18:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:18:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:18:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:18:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:18:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:18:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:18:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:18:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:18:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:18:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:18:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:18:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:18:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:18:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:18:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:18:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:18:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:18:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:18:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:18:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:18:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:18:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:18:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:18:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:18:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:18:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:18:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:18:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:18:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:18:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:18:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:18:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:18:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:18:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:18:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:18:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:18:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:18:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:18:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:18:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:18:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:18:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:18:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:18:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:18:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:18:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:18:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:18:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:18:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:18:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:18:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:18:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:18:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:18:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:18:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:18:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:18:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:18:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:18:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:18:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:18:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:18:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:18:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:18:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:18:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:18:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:18:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:18:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:18:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:18:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:18:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:18:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:18:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:18:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:18:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:18:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:18:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:18:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:18:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:18:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:18:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:18:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:18:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:18:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:18:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:18:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:18:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:18:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:18:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:18:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:18:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:18:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:18:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:18:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:18:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:18:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:18:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:18:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:18:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:18:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:18:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:18:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:19:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:19:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:19:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:19:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:19:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:19:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:19:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:19:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:19:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:19:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:19:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:19:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:19:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:19:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:19:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:19:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:19:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:19:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:19:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:19:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:19:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:19:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:19:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:19:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:19:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:19:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:19:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:19:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:19:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:19:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:19:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:19:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:19:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:19:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:19:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:19:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:19:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:19:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:19:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:19:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:19:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:19:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:19:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:19:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:19:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:19:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:19:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:19:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:19:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:19:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:19:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:19:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:19:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:19:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:19:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:19:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:19:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:19:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:19:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:19:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:19:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:19:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:19:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:19:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:19:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:19:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:19:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:19:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:19:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:19:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:19:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:19:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:19:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:19:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:19:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:19:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:19:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:19:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:19:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:19:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:19:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:19:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:19:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:19:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:19:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:19:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:19:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:19:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:19:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:19:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:19:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:19:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:19:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:19:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:19:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:19:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:19:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:19:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:19:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:19:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:19:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:19:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:19:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:19:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:19:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:19:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:19:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:19:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:19:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:19:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:19:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:19:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:19:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:19:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:19:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:19:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:19:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:19:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:19:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:19:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:19:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:19:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:19:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:19:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:19:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:19:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:19:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:19:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:19:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:19:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:19:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:19:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:19:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:19:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:19:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:19:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:19:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:19:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:19:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:19:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:19:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:19:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:19:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:19:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:19:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:19:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:19:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:19:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:19:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:19:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:19:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:19:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:19:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:19:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:19:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:19:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:19:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:19:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:19:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:19:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:19:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:19:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:19:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:19:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:19:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:19:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:19:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:19:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:19:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:19:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:19:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:19:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:19:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:19:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:19:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:19:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:19:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:19:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:19:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:19:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:19:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:19:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:19:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:19:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:19:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:19:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:19:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:19:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:19:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:19:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:19:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:19:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:19:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:19:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:19:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:19:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:19:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:19:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:19:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:19:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:19:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:19:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:19:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:19:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:19:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:19:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:19:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:19:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:19:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:19:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:19:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:19:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:19:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:19:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:19:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:19:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:20:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:20:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:20:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:20:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:20:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:20:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:20:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:20:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:20:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:20:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:20:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:20:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:20:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:20:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:20:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:20:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:20:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:20:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:20:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:20:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:20:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:20:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:20:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:20:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:20:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:20:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:20:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:20:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:20:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:20:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:20:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:20:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:20:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:20:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:20:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:20:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:20:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:20:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:20:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:20:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:20:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:20:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:20:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:20:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:20:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:20:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:20:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:20:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:20:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:20:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:20:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:20:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:20:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:20:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:20:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:20:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:20:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:20:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:20:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:20:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:20:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:20:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:20:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:20:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:20:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:20:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:20:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:20:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:20:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:20:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:20:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:20:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:20:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:20:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:20:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:20:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:20:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:20:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:20:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:20:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:20:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:20:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:20:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:20:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:20:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:20:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:20:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:20:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:20:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:20:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:20:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:20:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:20:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:20:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:20:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:20:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:20:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:20:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:20:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:20:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:20:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:20:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:20:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:20:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:20:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:20:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:20:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:20:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:20:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:20:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:20:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:20:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:20:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:20:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:20:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:20:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:20:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:20:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:20:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:20:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:20:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:20:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:20:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:20:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:20:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:20:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:20:32 rdma06 NetworkManager[1025]: <info>  [1681849232.9131] dhcp4 (eno1): state changed new lease, address=10.73.131.45
Apr 18 16:20:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:20:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:20:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:20:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:20:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:20:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:20:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:20:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:20:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:20:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:20:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:20:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:20:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:20:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:20:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:20:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:20:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:20:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:20:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:20:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:20:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:20:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:20:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:20:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:20:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:20:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:20:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:20:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:20:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:20:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:20:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:20:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:20:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:20:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:20:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:20:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:20:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:20:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:20:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:20:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:20:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:20:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:20:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:20:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:20:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:20:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:20:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:20:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:20:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:20:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:20:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:20:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:20:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:20:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:20:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:20:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:20:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:20:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:20:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:20:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:20:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:20:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:20:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:20:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:20:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:20:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:20:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:20:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:20:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:20:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:20:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:20:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:20:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:20:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:20:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:20:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:20:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:20:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:20:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:20:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:20:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:20:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:20:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:20:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:20:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:20:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:20:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:20:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:20:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:20:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:21:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:21:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:21:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:21:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:21:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:21:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:21:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:21:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:21:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:21:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:21:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:21:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:21:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:21:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:21:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:21:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:21:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:21:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:21:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:21:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:21:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:21:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:21:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:21:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:21:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:21:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:21:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:21:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:21:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:21:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:21:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:21:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:21:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:21:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:21:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:21:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:21:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:21:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:21:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:21:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:21:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:21:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:21:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:21:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:21:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:21:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:21:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:21:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:21:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:21:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:21:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:21:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:21:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:21:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:21:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:21:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:21:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:21:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:21:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:21:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:21:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:21:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:21:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:21:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:21:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:21:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:21:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:21:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:21:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:21:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:21:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:21:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:21:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:21:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:21:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:21:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:21:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:21:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:21:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:21:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:21:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:21:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:21:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:21:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:21:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:21:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:21:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:21:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:21:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:21:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:21:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:21:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:21:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:21:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:21:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:21:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:21:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:21:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:21:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:21:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:21:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:21:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:21:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:21:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:21:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:21:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:21:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:21:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:21:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:21:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:21:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:21:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:21:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:21:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:21:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:21:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:21:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:21:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:21:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:21:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:21:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:21:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:21:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:21:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:21:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:21:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:21:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:21:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:21:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:21:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:21:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:21:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:21:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:21:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:21:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:21:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:21:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:21:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:21:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:21:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:21:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:21:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:21:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:21:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:21:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:21:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:21:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:21:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:21:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:21:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:21:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:21:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:21:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:21:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:21:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:21:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:21:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:21:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:21:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:21:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:21:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:21:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:21:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:21:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:21:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:21:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:21:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:21:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:21:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:21:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:21:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:21:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:21:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:21:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:21:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:21:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:21:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:21:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:21:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:21:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:21:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:21:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:21:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:21:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:21:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:21:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:21:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:21:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:21:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:21:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:21:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:21:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:21:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:21:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:21:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:21:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:21:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:21:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:21:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:21:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:21:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:21:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:21:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:21:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:21:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:21:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:21:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:21:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:21:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:21:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:21:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:21:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:21:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:22:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:22:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:22:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:22:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:22:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:22:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:22:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:22:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:22:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:22:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:22:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:22:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:22:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:22:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:22:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:22:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:22:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:22:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:22:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:22:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:22:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:22:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:22:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:22:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:22:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:22:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:22:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:22:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:22:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:22:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:22:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:22:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:22:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:22:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:22:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:22:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:22:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:22:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:22:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:22:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:22:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:22:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:22:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:22:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:22:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:22:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:22:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:22:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:22:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:22:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:22:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:22:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:22:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:22:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:22:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:22:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:22:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:22:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:22:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:22:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:22:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:22:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:22:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:22:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:22:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:22:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:22:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:22:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:22:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:22:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:22:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:22:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:22:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:22:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:22:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:22:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:22:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:22:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:22:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:22:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:22:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:22:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:22:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:22:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:22:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:22:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:22:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:22:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:22:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:22:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:22:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:22:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:22:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:22:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:22:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:22:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:22:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:22:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:22:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:22:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:22:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:22:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:22:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:22:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:22:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:22:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:22:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:22:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:22:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:22:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:22:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:22:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:22:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:22:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:22:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:22:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:22:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:22:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:22:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:22:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:22:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:22:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:22:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:22:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:22:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:22:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:22:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:22:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:22:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:22:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:22:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:22:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:22:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:22:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:22:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:22:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:22:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:22:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:22:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:22:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:22:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:22:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:22:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:22:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:22:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:22:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:22:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:22:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:22:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:22:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:22:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:22:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:22:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:22:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:22:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:22:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:22:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:22:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:22:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:22:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:22:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:22:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:22:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:22:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:22:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:22:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:22:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:22:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:22:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:22:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:22:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:22:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:22:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:22:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:22:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:22:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:22:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:22:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:22:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:22:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:22:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:22:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:22:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:22:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:22:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:22:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:22:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:22:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:22:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:22:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:22:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:22:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:22:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:22:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:22:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:22:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:23:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:23:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:23:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:23:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:23:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:23:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:23:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:23:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:23:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:23:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:23:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:23:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:23:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:23:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:23:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:23:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:23:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:23:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:23:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:23:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:23:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:23:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:23:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:23:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:23:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:23:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:23:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:23:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:23:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:23:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:23:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:23:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:23:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:23:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:23:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:23:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:23:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:23:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:23:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:23:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:23:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:23:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:23:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:23:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:23:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:23:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:23:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:23:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:23:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:23:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:23:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:23:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:23:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:23:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:23:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:23:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:23:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:23:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:23:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:23:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:23:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:23:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:23:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:23:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:23:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:23:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:23:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:23:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:23:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:23:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:23:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:23:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:23:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:23:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:23:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:23:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:23:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:23:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:23:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:23:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:23:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:23:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:23:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:23:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:23:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:23:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:23:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:23:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:23:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:23:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:23:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:23:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:23:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:23:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:23:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:23:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:23:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:23:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:23:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:23:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:23:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:23:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:23:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:23:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:23:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:23:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:23:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:23:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:23:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:23:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:23:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:23:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:23:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:23:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:23:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:23:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:23:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:23:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:23:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:23:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:23:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:23:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:23:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:23:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:23:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:23:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:23:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:23:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:23:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:23:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:23:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:23:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:23:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:23:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:23:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:23:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:23:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:23:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:23:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:23:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:23:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:23:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:23:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:23:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:23:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:23:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:23:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:23:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:23:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:23:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:23:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:23:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:23:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:23:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:23:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:23:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:23:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:23:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:23:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:23:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:23:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:23:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:23:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:23:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:23:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:23:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:23:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:23:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:23:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:23:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:23:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:23:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:23:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:23:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:23:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:23:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:23:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:23:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:23:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:23:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:23:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:23:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:23:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:23:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:23:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:23:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:23:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:23:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:23:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:23:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:23:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:23:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:23:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:23:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:23:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:23:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:23:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:23:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:23:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:23:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:23:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:23:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:23:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:23:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:23:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:23:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:23:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:23:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:23:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:23:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:23:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:23:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:23:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:23:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:23:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:23:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:24:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:24:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:24:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:24:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:24:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:24:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:24:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:24:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:24:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:24:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:24:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:24:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:24:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:24:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:24:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:24:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:24:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:24:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:24:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:24:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:24:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:24:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:24:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:24:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:24:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:24:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:24:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:24:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:24:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:24:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:24:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:24:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:24:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:24:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:24:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:24:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:24:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:24:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:24:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:24:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:24:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:24:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:24:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:24:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:24:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:24:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:24:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:24:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:24:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:24:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:24:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:24:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:24:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:24:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:24:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:24:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:24:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:24:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:24:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:24:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:24:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:24:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:24:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:24:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:24:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:24:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:24:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:24:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:24:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:24:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:24:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:24:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:24:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:24:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:24:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:24:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:24:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:24:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:24:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:24:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:24:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:24:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:24:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:24:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:24:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:24:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:24:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:24:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:24:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:24:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:24:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:24:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:24:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:24:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:24:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:24:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:24:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:24:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:24:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:24:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:24:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:24:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:24:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:24:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:24:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:24:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:24:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:24:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:24:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:24:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:24:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:24:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:24:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:24:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:24:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:24:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:24:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:24:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:24:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:24:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:24:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:24:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:24:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:24:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:24:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:24:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:24:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:24:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:24:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:24:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:24:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:24:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:24:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:24:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:24:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:24:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:24:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:24:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:24:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:24:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:24:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:24:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:24:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:24:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:24:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:24:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:24:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:24:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:24:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:24:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:24:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:24:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:24:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:24:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:24:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:24:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:24:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:24:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:24:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:24:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:24:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:24:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:24:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:24:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:24:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:24:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:24:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:24:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:24:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:24:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:24:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:24:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:24:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:24:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:24:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:24:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:24:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:24:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:24:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:24:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:24:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:24:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:24:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:24:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:24:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:24:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:24:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:24:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:24:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:24:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:24:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:24:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:24:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:24:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:24:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:24:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:24:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:24:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:24:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:24:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:24:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:24:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:24:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:24:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:24:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:24:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:24:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:24:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:24:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:24:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:24:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:24:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:24:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:24:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:24:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:24:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:27:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:27:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:27:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:27:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:27:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:27:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:27:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:27:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:27:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:27:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:27:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:27:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:27:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:27:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:27:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:27:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:27:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:27:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:27:07 rdma06 rsyslogd[1041]: imjournal: 5292 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 16:27:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:27:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:27:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:27:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:27:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:27:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:27:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:27:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:27:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:27:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:27:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:27:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:27:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:27:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:27:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:27:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:27:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:27:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:27:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:27:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:27:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:27:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:27:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:27:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:27:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:27:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:27:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:27:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:27:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:27:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:27:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:27:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:27:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:27:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:27:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:27:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:27:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:27:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:27:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:27:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:27:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:27:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:27:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:27:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:27:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:27:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:27:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:27:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:27:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:27:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:27:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:27:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:27:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:27:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:27:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:27:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:27:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:27:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:27:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:27:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:27:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:27:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:27:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:27:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:27:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:27:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:27:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:27:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:27:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:27:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:27:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:27:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:27:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:27:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:27:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:27:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:27:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:27:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:27:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:27:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:27:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:27:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:27:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:27:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:27:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:27:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:27:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:27:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:27:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:27:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:27:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:27:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:27:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:27:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:27:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:27:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:27:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:27:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:27:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:27:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:27:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:27:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:27:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:27:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:27:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:27:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:27:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:27:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:27:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:27:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:27:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:27:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:27:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:27:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:27:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:27:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:27:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:27:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:27:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:27:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:27:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:27:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:27:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:27:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:27:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:27:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:27:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:27:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:27:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:27:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:27:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:27:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:27:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:27:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:27:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:27:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:27:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:27:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:27:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:27:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:27:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:27:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:27:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:27:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:27:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:27:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:27:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:27:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:27:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:27:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:27:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:27:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:27:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:27:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:27:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:27:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:27:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:27:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:27:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:27:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:27:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:27:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:27:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:27:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:27:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:27:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:27:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:27:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:27:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:27:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:27:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:27:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:27:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:27:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:27:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:27:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:27:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:27:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:27:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:27:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:28:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:28:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:28:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:28:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:28:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:28:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:28:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:28:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:28:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:28:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:28:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:28:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:28:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:28:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:28:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:28:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:28:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:28:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:28:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:28:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:28:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:28:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:28:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:28:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:28:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:28:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:28:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:28:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:28:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:28:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:28:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:28:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:28:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:28:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:28:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:28:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:28:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:28:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:28:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:28:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:28:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:28:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:28:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:28:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:28:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:28:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:28:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:28:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:28:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:28:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:28:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:28:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:28:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:28:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:28:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:28:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:28:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:28:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:28:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:28:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:28:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:28:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:28:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:28:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:28:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:28:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:28:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:28:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:28:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:28:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:28:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:28:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:28:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:28:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:28:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:28:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:28:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:28:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:28:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:28:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:28:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:28:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:28:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:28:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:28:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:28:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:28:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:28:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:28:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:28:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:28:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:28:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:28:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:28:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:28:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:28:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:28:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:28:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:28:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:28:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:28:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:28:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:28:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:28:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:28:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:28:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:28:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:28:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:28:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:28:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:28:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:28:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:28:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:28:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:28:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:28:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:28:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:28:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:28:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:28:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:28:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:28:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:28:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:28:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:28:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:28:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:28:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:28:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:28:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:28:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:28:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:28:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:28:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:28:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:28:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:28:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:28:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:28:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:28:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:28:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:28:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:28:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:28:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:28:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:28:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:28:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:28:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:28:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:28:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:28:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:28:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:28:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:28:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:28:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:28:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:28:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:28:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:28:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:28:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:28:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:28:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:28:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:28:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:28:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:28:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:28:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:28:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:28:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:28:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:28:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:28:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:28:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:28:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:28:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:28:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:28:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:28:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:28:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:28:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:28:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:28:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:28:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:28:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:28:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:28:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:28:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:28:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:28:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:28:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:28:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:28:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:28:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:28:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:28:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:28:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:28:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:28:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:28:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:28:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:28:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:28:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:28:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:28:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:28:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:28:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:28:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:28:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:28:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:28:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:28:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:28:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:28:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:28:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:28:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:28:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:28:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:29:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:29:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:29:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:29:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:29:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:29:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:29:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:29:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:29:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:29:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:29:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:29:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:29:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:29:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:29:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:29:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:29:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:29:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:29:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:29:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:29:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:29:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:29:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:29:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:29:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:29:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:29:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:29:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:29:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:29:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:29:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:29:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:29:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:29:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:29:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:29:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:29:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:29:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:29:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:29:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:29:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:29:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:29:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:29:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:29:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:29:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:29:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:29:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:29:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:29:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:29:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:29:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:29:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:29:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:29:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:29:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:29:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:29:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:29:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:29:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:29:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:29:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:29:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:29:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:29:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:29:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:29:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:29:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:29:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:29:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:29:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:29:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:29:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:29:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:29:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:29:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:29:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:29:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:29:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:29:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:29:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:29:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:29:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:29:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:29:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:29:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:29:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:29:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:29:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:29:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:29:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:29:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:29:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:29:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:29:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:29:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:29:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:29:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:29:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:29:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:29:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:29:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:29:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:29:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:29:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:29:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:29:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:29:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:29:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:29:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:29:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:29:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:29:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:29:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:29:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:29:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:29:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:29:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:29:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:29:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:29:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:29:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:29:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:29:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:29:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:29:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:29:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:29:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:29:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:29:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:29:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:29:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:29:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:29:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:29:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:29:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:29:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:29:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:29:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:29:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:29:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:29:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:29:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:29:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:29:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:29:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:29:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:29:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:29:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:29:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:29:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:29:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:29:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:29:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:29:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:29:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:29:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:29:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:29:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:29:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:29:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:29:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:29:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:29:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:29:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:29:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:29:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:29:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:29:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:29:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:29:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:29:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:29:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:29:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:29:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:29:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:29:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:29:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:29:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:29:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:29:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:29:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:29:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:29:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:29:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:29:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:29:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:29:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:29:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:29:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:29:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:29:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:29:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:29:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:29:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:29:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:29:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:29:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:29:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:29:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:29:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:29:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:29:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:29:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:29:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:29:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:29:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:29:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:29:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:29:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:29:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:29:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:29:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:29:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:29:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:29:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:30:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:30:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:30:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:30:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:30:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:30:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:30:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:30:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:30:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:30:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:30:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:30:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:30:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:30:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:30:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:30:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:30:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:30:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:30:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:30:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:30:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:30:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:30:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:30:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:30:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:30:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:30:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:30:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:30:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:30:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:30:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:30:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:30:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:30:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:30:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:30:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:30:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:30:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:30:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:30:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:30:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:30:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:30:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:30:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:30:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:30:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:30:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:30:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:30:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:30:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:30:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:30:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:30:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:30:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:30:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:30:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:30:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:30:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:30:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:30:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:30:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:30:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:30:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:30:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:30:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:30:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:30:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:30:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:30:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:30:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:30:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:30:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:30:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:30:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:30:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:30:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:30:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:30:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:30:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:30:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:30:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:30:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:30:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:30:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:30:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:30:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:30:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:30:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:30:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:30:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:30:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:30:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:30:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:30:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:30:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:30:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:30:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:30:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:30:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:30:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:30:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:30:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:30:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:30:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:30:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:30:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:30:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:30:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:30:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:30:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:30:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:30:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:30:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:30:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:30:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:30:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:30:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:30:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:30:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:30:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:30:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:30:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:30:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:30:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:30:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:30:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:30:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:30:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:30:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:30:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:30:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:30:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:30:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:30:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:30:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:30:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:30:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:30:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:30:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:30:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:30:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:30:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:30:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:30:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:30:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:30:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:30:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:30:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:30:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:30:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:30:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:30:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:30:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:30:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:30:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:30:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:30:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:30:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:30:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:30:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:30:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:30:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:30:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:30:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:30:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:30:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:30:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:30:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:30:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:30:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:30:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:30:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:30:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:30:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:30:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:30:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:30:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:30:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:30:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:30:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:30:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:30:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:30:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:30:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:30:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:30:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:30:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:30:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:30:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:30:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:30:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:30:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:30:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:30:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:30:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:30:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:30:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:30:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:30:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:30:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:30:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:30:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:30:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:30:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:30:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:30:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:30:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:30:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:30:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:30:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:30:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:30:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:30:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:30:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:30:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:30:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:31:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:31:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:31:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:31:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:31:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:31:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:31:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:31:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:31:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:31:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:31:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:31:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:31:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:31:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:31:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:31:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:31:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:31:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:31:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:31:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:31:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:31:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:31:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:31:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:31:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:31:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:31:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:31:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:31:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:31:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:31:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:31:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:31:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:31:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:31:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:31:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:31:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:31:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:31:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:31:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:31:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:31:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:31:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:31:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:31:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:31:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:31:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:31:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:31:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:31:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:31:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:31:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:31:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:31:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:31:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:31:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:31:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:31:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:31:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:31:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:31:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:31:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:31:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:31:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:31:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:31:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:31:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:31:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:31:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:31:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:31:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:31:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:31:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:31:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:31:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:31:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:31:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:31:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:31:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:31:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:31:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:31:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:31:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:31:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:31:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:31:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:31:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:31:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:31:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:31:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:31:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:31:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:31:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:31:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:31:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:31:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:31:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:31:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:31:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:31:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:31:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:31:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:31:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:31:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:31:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:31:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:31:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:31:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:31:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:31:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:31:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:31:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:31:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:31:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:31:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:31:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:31:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:31:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:31:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:31:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:31:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:31:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:31:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:31:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:31:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:31:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:31:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:31:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:31:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:31:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:31:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:31:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:31:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:31:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:31:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:31:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:31:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:31:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:31:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:31:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:31:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:31:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:31:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:31:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:31:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:31:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:31:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:31:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:31:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:31:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:31:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:31:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:31:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:31:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:31:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:31:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:31:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:31:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:31:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:31:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:31:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:31:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:31:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:31:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:31:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:31:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:31:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:31:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:31:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:31:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:31:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:31:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:31:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:31:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:31:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:31:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:31:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:31:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:31:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:31:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:31:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:31:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:31:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:31:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:31:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:31:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:31:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:31:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:31:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:31:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:31:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:31:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:31:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:31:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:31:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:31:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:31:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:31:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:31:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:31:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:31:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:31:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:31:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:31:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:31:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:31:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:31:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:31:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:31:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:31:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:31:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:31:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:31:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:31:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:31:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:31:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:32:00 rdma06 systemd[1]: Starting dnf makecache...
Apr 18 16:32:01 rdma06 dnf[1196586]: Updating Subscription Management repositories.
Apr 18 16:32:01 rdma06 dnf[1196586]: Unable to read consumer identity
Apr 18 16:32:01 rdma06 dnf[1196586]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 18 16:32:01 rdma06 dnf[1196586]: Metadata cache refreshed recently.
Apr 18 16:32:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 18 16:32:01 rdma06 systemd[1]: Finished dnf makecache.
Apr 18 16:32:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:32:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:32:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:32:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:32:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:32:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:32:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:32:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:32:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:32:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:32:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:32:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:32:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:32:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:32:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:32:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:32:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:32:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:32:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:32:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:32:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:32:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:32:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:32:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:32:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:32:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:32:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:32:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:32:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:32:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:32:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:32:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:32:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:32:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:32:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:32:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:32:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:32:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:32:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:32:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:32:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:32:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:32:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:32:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:32:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:32:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:32:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:32:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:32:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:32:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:32:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:32:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:32:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:32:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:32:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:32:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:32:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:32:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:32:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:32:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:32:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:32:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:32:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:32:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:32:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:32:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:32:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:32:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:32:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:32:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:32:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:32:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:32:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:32:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:32:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:32:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:32:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:32:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:32:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:32:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:32:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:32:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:32:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:32:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:32:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:32:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:32:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:32:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:32:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:32:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:32:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:32:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:32:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:32:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:32:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:32:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:32:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:32:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:32:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:32:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:32:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:32:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:32:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:32:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:32:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:32:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:32:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:32:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:32:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:32:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:32:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:32:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:32:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:32:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:32:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:32:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:32:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:32:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:32:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:32:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:32:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:32:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:32:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:32:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:32:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:32:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:32:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:32:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:32:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:32:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:32:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:32:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:32:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:32:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:32:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:32:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:32:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:32:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:32:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:32:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:32:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:32:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:32:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:32:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:32:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:32:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:32:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:32:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:32:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:32:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:32:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:32:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:32:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:32:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:32:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:32:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:32:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:32:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:32:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:32:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:32:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:32:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:32:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:32:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:32:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:32:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:32:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:32:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:32:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:32:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:32:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:32:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:32:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:32:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:32:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:32:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:32:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:32:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:32:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:32:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:32:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:32:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:32:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:32:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:32:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:32:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:32:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:32:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:32:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:32:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:32:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:32:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:32:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:32:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:32:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:32:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:32:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:32:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:32:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:32:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:32:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:32:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:32:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:32:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:32:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:32:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:32:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:32:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:32:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:32:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:32:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:32:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:32:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:32:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:32:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:32:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:33:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:33:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:33:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:33:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:33:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:33:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:33:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:33:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:33:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:33:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:33:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:33:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:33:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:33:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:33:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:33:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:33:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:33:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:33:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:33:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:33:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:33:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:33:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:33:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:33:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:33:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:33:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:33:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:33:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:33:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:33:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:33:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:33:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:33:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:33:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:33:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:33:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:33:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:33:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:33:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:33:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:33:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:33:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:33:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:33:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:33:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:33:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:33:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:33:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:33:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:33:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:33:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:33:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:33:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:33:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:33:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:33:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:33:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:33:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:33:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:33:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:33:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:33:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:33:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:33:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:33:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:33:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:33:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:33:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:33:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:33:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:33:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:33:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:33:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:33:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:33:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:33:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:33:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:33:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:33:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:33:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:33:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:33:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:33:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:33:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:33:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:33:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:33:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:33:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:33:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:33:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:33:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:33:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:33:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:33:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:33:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:33:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:33:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:33:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:33:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:33:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:33:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:33:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:33:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:33:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:33:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:33:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:33:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:33:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:33:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:33:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:33:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:33:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:33:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:33:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:33:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:33:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:33:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:33:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:33:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:33:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:33:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:33:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:33:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:33:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:33:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:33:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:33:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:33:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:33:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:33:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:33:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:33:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:33:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:33:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:33:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:33:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:33:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:33:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:33:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:33:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:33:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:33:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:33:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:33:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:33:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:33:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:33:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:33:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:33:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:33:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:33:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:33:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:33:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:33:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:33:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:33:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:33:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:33:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:33:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:33:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:33:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:33:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:33:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:33:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:33:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:33:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:33:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:33:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:33:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:33:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:33:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:33:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:33:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:33:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:33:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:33:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:33:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:33:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:33:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:33:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:33:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:33:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:33:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:33:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:33:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:33:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:33:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:33:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:33:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:33:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:33:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:33:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:33:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:33:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:33:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:33:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:33:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:33:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:33:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:33:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:33:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:33:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:33:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:33:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:33:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:33:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:33:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:33:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:33:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:33:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:33:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:33:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:33:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:33:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:33:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:34:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:34:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:34:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:34:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:34:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:34:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:34:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:34:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:34:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:34:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:34:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:34:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:34:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:34:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:34:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:34:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:34:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:34:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:34:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:34:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:34:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:34:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:34:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:34:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:34:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:34:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:34:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:34:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:34:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:34:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:34:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:34:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:34:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:34:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:34:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:34:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:34:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:34:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:34:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:34:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:34:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:34:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:34:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:34:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:34:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:34:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:34:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:34:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:34:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:34:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:34:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:34:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:34:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:34:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:34:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:34:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:34:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:34:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:34:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:34:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:34:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:34:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:34:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:34:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:34:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:34:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:34:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:34:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:34:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:34:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:34:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:34:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:34:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:34:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:34:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:34:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:34:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:34:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:34:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:34:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:34:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:34:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:34:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:34:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:34:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:34:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:34:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:34:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:34:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:34:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:34:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:34:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:34:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:34:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:34:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:34:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:34:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:34:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:34:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:34:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:34:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:34:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:34:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:34:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:34:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:34:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:34:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:34:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:34:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:34:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:34:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:34:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:34:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:34:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:34:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:34:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:34:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:34:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:34:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:34:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:34:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:34:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:34:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:34:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:34:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:34:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:34:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:34:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:34:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:34:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:34:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:34:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:34:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:34:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:34:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:34:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:34:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:34:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:34:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:34:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:34:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:34:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:34:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:34:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:34:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:34:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:34:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:34:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:34:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:34:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:34:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:34:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:34:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:34:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:34:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:34:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:34:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:34:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:34:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:34:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:34:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:34:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:34:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:34:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:34:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:34:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:34:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:34:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:34:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:34:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:34:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:34:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:34:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:34:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:34:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:34:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:34:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:34:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:34:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:34:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:34:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:34:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:34:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:34:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:34:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:34:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:34:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:34:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:34:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:34:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:34:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:34:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:34:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:34:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:34:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:34:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:34:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:34:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:34:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:34:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:34:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:34:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:34:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:34:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:34:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:34:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:34:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:34:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:34:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:34:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:34:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:34:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:34:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:34:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:34:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:34:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:35:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:35:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:35:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:35:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:35:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:35:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:35:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:35:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:35:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:35:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:35:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:35:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:35:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:35:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:35:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:35:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:35:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:35:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:37:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:37:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:37:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:37:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:37:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:37:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:37:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:37:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:37:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:37:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:37:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:37:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:37:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:37:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:37:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:37:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:37:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:37:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:37:12 rdma06 rsyslogd[1041]: imjournal: 5298 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 16:37:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:37:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:37:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:37:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:37:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:37:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:37:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:37:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:37:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:37:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:37:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:37:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:37:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:37:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:37:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:37:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:37:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:37:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:37:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:37:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:37:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:37:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:37:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:37:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:37:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:37:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:37:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:37:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:37:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:37:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:37:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:37:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:37:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:37:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:37:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:37:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:37:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:37:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:37:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:37:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:37:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:37:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:37:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:37:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:37:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:37:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:37:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:37:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:37:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:37:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:37:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:37:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:37:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:37:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:37:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:37:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:37:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:37:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:37:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:37:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:37:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:37:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:37:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:37:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:37:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:37:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:37:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:37:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:37:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:37:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:37:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:37:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:37:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:37:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:37:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:37:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:37:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:37:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:37:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:37:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:37:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:37:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:37:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:37:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:37:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:37:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:37:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:37:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:37:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:37:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:37:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:37:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:37:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:37:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:37:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:37:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:37:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:37:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:37:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:37:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:37:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:37:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:37:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:37:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:37:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:37:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:37:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:37:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:37:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:37:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:37:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:37:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:37:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:37:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:37:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:37:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:37:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:37:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:37:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:37:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:37:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:37:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:37:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:37:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:37:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:37:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:37:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:37:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:37:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:37:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:37:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:37:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:37:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:37:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:37:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:37:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:37:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:37:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:37:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:37:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:37:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:37:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:37:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:37:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:37:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:37:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:37:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:37:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:37:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:37:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:37:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:37:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:37:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:37:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:37:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:37:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:37:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:37:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:37:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:37:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:37:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:37:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:38:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:38:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:38:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:38:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:38:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:38:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:38:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:38:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:38:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:38:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:38:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:38:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:38:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:38:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:38:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:38:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:38:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:38:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:38:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:38:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:38:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:38:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:38:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:38:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:38:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:38:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:38:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:38:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:38:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:38:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:38:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:38:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:38:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:38:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:38:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:38:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:38:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:38:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:38:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:38:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:38:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:38:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:38:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:38:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:38:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:38:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:38:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:38:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:38:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:38:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:38:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:38:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:38:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:38:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:38:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:38:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:38:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:38:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:38:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:38:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:38:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:38:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:38:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:38:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:38:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:38:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:38:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:38:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:38:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:38:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:38:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:38:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:38:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:38:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:38:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:38:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:38:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:38:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:38:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:38:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:38:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:38:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:38:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:38:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:38:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:38:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:38:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:38:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:38:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:38:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:38:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:38:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:38:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:38:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:38:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:38:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:38:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:38:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:38:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:38:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:38:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:38:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:38:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:38:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:38:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:38:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:38:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:38:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:38:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:38:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:38:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:38:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:38:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:38:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:38:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:38:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:38:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:38:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:38:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:38:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:38:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:38:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:38:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:38:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:38:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:38:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:38:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:38:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:38:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:38:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:38:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:38:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:38:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:38:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:38:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:38:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:38:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:38:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:38:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:38:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:38:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:38:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:38:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:38:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:38:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:38:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:38:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:38:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:38:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:38:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:38:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:38:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:38:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:38:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:38:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:38:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:38:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:38:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:38:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:38:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:38:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:38:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:38:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:38:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:38:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:38:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:38:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:38:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:38:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:38:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:38:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:38:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:38:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:38:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:38:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:38:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:38:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:38:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:38:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:38:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:38:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:38:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:38:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:38:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:38:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:38:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:38:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:38:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:38:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:38:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:38:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:38:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:38:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:38:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:38:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:38:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:38:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:38:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:38:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:38:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:38:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:38:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:38:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:38:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:38:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:38:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:38:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:38:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:38:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:38:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:38:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:38:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:38:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:38:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:38:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:38:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:39:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:39:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:39:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:39:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:39:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:39:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:39:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:39:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:39:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:39:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:39:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:39:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:39:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:39:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:39:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:39:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:39:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:39:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:39:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:39:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:39:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:39:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:39:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:39:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:39:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:39:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:39:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:39:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:39:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:39:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:39:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:39:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:39:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:39:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:39:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:39:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:39:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:39:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:39:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:39:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:39:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:39:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:39:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:39:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:39:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:39:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:39:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:39:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:39:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:39:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:39:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:39:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:39:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:39:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:39:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:39:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:39:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:39:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:39:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:39:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:39:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:39:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:39:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:39:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:39:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:39:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:39:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:39:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:39:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:39:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:39:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:39:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:39:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:39:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:39:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:39:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:39:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:39:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:39:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:39:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:39:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:39:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:39:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:39:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:39:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:39:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:39:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:39:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:39:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:39:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:39:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:39:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:39:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:39:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:39:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:39:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:39:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:39:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:39:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:39:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:39:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:39:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:39:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:39:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:39:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:39:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:39:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:39:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:39:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:39:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:39:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:39:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:39:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:39:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:39:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:39:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:39:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:39:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:39:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:39:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:39:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:39:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:39:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:39:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:39:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:39:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:39:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:39:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:39:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:39:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:39:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:39:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:39:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:39:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:39:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:39:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:39:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:39:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:39:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:39:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:39:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:39:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:39:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:39:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:39:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:39:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:39:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:39:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:39:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:39:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:39:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:39:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:39:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:39:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:39:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:39:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:39:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:39:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:39:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:39:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:39:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:39:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:39:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:39:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:39:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:39:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:39:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:39:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:39:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:39:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:39:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:39:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:39:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:39:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:39:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:39:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:39:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:39:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:39:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:39:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:39:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:39:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:39:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:39:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:39:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:39:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:39:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:39:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:39:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:39:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:39:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:39:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:39:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:39:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:39:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:39:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:39:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:39:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:39:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:39:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:39:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:39:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:39:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:39:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:39:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:39:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:39:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:39:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:39:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:39:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:39:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:39:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:39:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:39:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:39:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:39:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:40:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:40:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:40:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:40:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:40:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:40:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:40:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:40:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:40:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:40:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:40:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:40:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:40:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:40:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:40:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:40:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:40:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:40:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:40:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:40:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:40:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:40:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:40:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:40:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:40:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:40:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:40:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:40:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:40:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:40:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:40:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:40:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:40:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:40:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:40:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:40:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:40:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:40:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:40:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:40:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:40:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:40:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:40:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:40:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:40:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:40:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:40:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:40:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:40:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:40:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:40:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:40:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:40:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:40:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:40:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:40:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:40:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:40:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:40:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:40:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:40:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:40:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:40:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:40:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:40:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:40:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:40:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:40:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:40:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:40:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:40:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:40:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:40:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:40:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:40:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:40:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:40:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:40:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:40:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:40:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:40:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:40:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:40:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:40:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:40:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:40:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:40:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:40:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:40:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:40:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:40:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:40:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:40:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:40:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:40:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:40:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:40:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:40:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:40:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:40:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:40:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:40:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:40:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:40:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:40:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:40:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:40:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:40:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:40:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:40:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:40:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:40:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:40:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:40:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:40:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:40:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:40:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:40:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:40:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:40:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:40:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:40:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:40:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:40:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:40:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:40:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:40:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:40:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:40:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:40:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:40:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:40:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:40:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:40:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:40:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:40:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:40:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:40:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:40:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:40:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:40:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:40:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:40:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:40:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:40:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:40:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:40:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:40:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:40:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:40:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:40:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:40:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:40:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:40:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:40:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:40:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:40:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:40:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:40:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:40:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:40:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:40:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:40:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:40:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:40:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:40:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:40:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:40:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:40:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:40:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:40:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:40:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:40:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:40:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:40:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:40:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:40:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:40:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:40:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:40:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:40:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:40:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:40:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:40:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:40:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:40:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:40:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:40:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:40:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:40:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:40:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:40:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:40:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:40:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:40:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:40:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:40:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:40:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:40:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:40:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:40:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:40:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:40:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:40:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:40:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:40:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:40:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:40:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:40:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:40:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:40:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:40:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:40:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:40:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:40:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:40:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:41:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:41:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:41:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:41:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:41:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:41:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:41:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:41:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:41:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:41:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:41:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:41:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:41:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:41:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:41:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:41:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:41:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:41:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:41:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:41:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:41:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:41:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:41:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:41:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:41:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:41:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:41:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:41:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:41:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:41:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:41:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:41:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:41:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:41:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:41:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:41:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:41:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:41:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:41:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:41:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:41:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:41:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:41:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:41:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:41:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:41:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:41:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:41:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:41:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:41:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:41:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:41:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:41:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:41:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:41:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:41:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:41:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:41:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:41:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:41:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:41:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:41:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:41:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:41:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:41:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:41:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:41:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:41:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:41:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:41:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:41:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:41:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:41:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:41:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:41:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:41:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:41:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:41:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:41:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:41:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:41:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:41:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:41:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:41:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:41:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:41:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:41:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:41:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:41:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:41:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:41:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:41:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:41:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:41:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:41:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:41:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:41:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:41:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:41:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:41:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:41:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:41:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:41:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:41:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:41:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:41:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:41:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:41:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:41:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:41:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:41:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:41:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:41:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:41:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:41:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:41:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:41:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:41:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:41:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:41:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:41:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:41:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:41:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:41:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:41:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:41:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:41:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:41:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:41:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:41:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:41:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:41:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:41:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:41:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:41:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:41:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:41:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:41:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:41:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:41:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:41:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:41:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:41:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:41:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:41:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:41:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:41:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:41:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:41:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:41:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:41:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:41:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:41:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:41:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:41:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:41:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:41:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:41:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:41:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:41:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:41:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:41:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:41:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:41:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:41:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:41:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:41:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:41:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:41:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:41:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:41:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:41:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:41:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:41:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:41:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:41:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:41:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:41:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:41:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:41:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:41:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:41:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:41:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:41:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:41:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:41:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:41:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:41:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:41:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:41:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:41:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:41:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:41:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:41:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:41:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:41:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:41:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:41:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:41:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:41:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:41:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:41:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:41:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:41:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:41:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:41:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:41:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:41:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:41:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:41:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:41:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:41:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:41:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:41:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:41:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:41:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:42:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:42:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:42:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:42:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:42:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:42:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:42:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:42:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:42:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:42:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:42:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:42:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:42:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:42:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:42:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:42:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:42:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:42:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:42:02 rdma06 systemd-journald[790]: Data hash table of /run/log/journal/95379d2f723847b68d9e976dee8967a9/system.journal has a fill level at 75.0 (163750 of 218332 items, 75497472 file size, 461 bytes per hash table item), suggesting rotation.
Apr 18 16:42:02 rdma06 systemd-journald[790]: /run/log/journal/95379d2f723847b68d9e976dee8967a9/system.journal: Journal header limits reached or header out-of-date, rotating.
Apr 18 16:42:02 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 18 16:42:03 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 18 16:42:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:42:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:42:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:42:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:42:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:42:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:42:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:42:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:42:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:42:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:42:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:42:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:42:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:42:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:42:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:42:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:42:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:42:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:42:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:42:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:42:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:42:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:42:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:42:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:42:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:42:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:42:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:42:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:42:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:42:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:42:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:42:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:42:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:42:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:42:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:42:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:42:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:42:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:42:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:42:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:42:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:42:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:42:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:42:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:42:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:42:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:42:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:42:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:42:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:42:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:42:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:42:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:42:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:42:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:42:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:42:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:42:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:42:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:42:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:42:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:42:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:42:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:42:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:42:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:42:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:42:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:42:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:42:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:42:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:42:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:42:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:42:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:42:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:42:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:42:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:42:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:42:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:42:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:42:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:42:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:42:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:42:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:42:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:42:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:42:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:42:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:42:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:42:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:42:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:42:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:42:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:42:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:42:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:42:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:42:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:42:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:42:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:42:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:42:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:42:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:42:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:42:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:42:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:42:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:42:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:42:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:42:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:42:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:42:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:42:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:42:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:42:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:42:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:42:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:42:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:42:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:42:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:42:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:42:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:42:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:42:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:42:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:42:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:42:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:42:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:42:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:42:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:42:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:42:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:42:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:42:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:42:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:42:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:42:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:42:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:42:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:42:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:42:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:42:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:42:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:42:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:42:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:42:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:42:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:42:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:42:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:42:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:42:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:42:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:42:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:42:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:42:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:42:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:42:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:42:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:42:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:42:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:42:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:42:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:42:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:42:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:42:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:42:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:42:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:42:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:42:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:42:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:42:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:42:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:42:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:42:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:42:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:42:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:42:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:42:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:42:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:42:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:42:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:42:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:42:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:42:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:42:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:42:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:42:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:42:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:42:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:42:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:42:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:42:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:42:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:42:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:42:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:42:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:42:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:42:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:42:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:42:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:42:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:43:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:43:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:43:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:43:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:43:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:43:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:43:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:43:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:43:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:43:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:43:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:43:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:43:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:43:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:43:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:43:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:43:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:43:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:43:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:43:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:43:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:43:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:43:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:43:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:43:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:43:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:43:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:43:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:43:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:43:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:43:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:43:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:43:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:43:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:43:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:43:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:43:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:43:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:43:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:43:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:43:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:43:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:43:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:43:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:43:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:43:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:43:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:43:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:43:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:43:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:43:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:43:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:43:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:43:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:43:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:43:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:43:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:43:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:43:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:43:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:43:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:43:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:43:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:43:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:43:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:43:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:43:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:43:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:43:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:43:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:43:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:43:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:43:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:43:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:43:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:43:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:43:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:43:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:43:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:43:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:43:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:43:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:43:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:43:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:43:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:43:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:43:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:43:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:43:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:43:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:43:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:43:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:43:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:43:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:43:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:43:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:43:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:43:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:43:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:43:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:43:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:43:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:43:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:43:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:43:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:43:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:43:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:43:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:43:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:43:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:43:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:43:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:43:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:43:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:43:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:43:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:43:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:43:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:43:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:43:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:43:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:43:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:43:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:43:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:43:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:43:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:43:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:43:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:43:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:43:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:43:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:43:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:43:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:43:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:43:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:43:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:43:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:43:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:43:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:43:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:43:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:43:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:43:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:43:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:43:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:43:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:43:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:43:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:43:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:43:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:43:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:43:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:43:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:43:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:43:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:43:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:43:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:43:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:43:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:43:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:43:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:43:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:43:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:43:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:43:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:43:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:43:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:43:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:43:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:43:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:43:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:43:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:43:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:43:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:43:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:43:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:43:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:43:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:43:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:43:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:43:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:43:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:43:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:43:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:43:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:43:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:43:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:43:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:43:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:43:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:43:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:43:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:43:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:43:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:43:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:43:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:43:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:43:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:43:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:43:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:43:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:43:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:43:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:43:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:43:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:43:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:43:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:43:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:43:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:43:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:43:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:43:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:43:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:43:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:43:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:43:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:44:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:44:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:44:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:44:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:44:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:44:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:44:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:44:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:44:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:44:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:44:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:44:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:44:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:44:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:44:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:44:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:44:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:44:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:44:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:44:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:44:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:44:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:44:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:44:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:44:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:44:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:44:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:44:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:44:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:44:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:44:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:44:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:44:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:44:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:44:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:44:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:44:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:44:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:44:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:44:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:44:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:44:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:44:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:44:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:44:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:44:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:44:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:44:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:44:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:44:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:44:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:44:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:44:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:44:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:44:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:44:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:44:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:44:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:44:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:44:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:44:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:44:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:44:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:44:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:44:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:44:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:44:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:44:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:44:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:44:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:44:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:44:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:44:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:44:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:44:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:44:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:44:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:44:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:44:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:44:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:44:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:44:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:44:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:44:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:44:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:44:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:44:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:44:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:44:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:44:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:44:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:44:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:44:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:44:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:44:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:44:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:44:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:44:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:44:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:44:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:44:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:44:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:44:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:44:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:44:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:44:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:44:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:44:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:44:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:44:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:44:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:44:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:44:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:44:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:44:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:44:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:44:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:44:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:44:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:44:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:44:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:44:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:44:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:44:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:44:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:44:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:44:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:44:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:44:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:44:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:44:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:44:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:44:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:44:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:44:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:44:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:44:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:44:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:44:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:44:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:44:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:44:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:44:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:44:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:44:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:44:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:44:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:44:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:44:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:44:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:44:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:44:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:44:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:44:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:44:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:44:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:44:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:44:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:44:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:44:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:44:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:44:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:44:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:44:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:44:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:44:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:44:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:44:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:44:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:44:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:44:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:44:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:44:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:44:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:44:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:44:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:44:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:44:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:44:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:44:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:44:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:44:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:44:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:44:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:44:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:44:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:44:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:44:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:44:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:44:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:44:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:44:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:44:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:44:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:44:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:44:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:44:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:44:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:44:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:44:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:44:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:44:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:44:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:44:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:44:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:44:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:44:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:44:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:44:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:44:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:44:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:44:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:44:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:44:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:44:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:44:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:45:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:45:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:45:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:45:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:45:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:45:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:45:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:45:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:45:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:45:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:45:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:45:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:45:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:45:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:45:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:45:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:45:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:45:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:45:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:45:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:45:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:45:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:45:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:45:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:45:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:45:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:45:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:45:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:45:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:45:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:45:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:45:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:45:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:45:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:45:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:45:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:47:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:47:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:47:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:47:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:47:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:47:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:47:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:47:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:47:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:47:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:47:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:47:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:47:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:47:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:47:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:47:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:47:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:47:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:47:13 rdma06 rsyslogd[1041]: imjournal: 5086 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 16:47:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:47:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:47:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:47:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:47:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:47:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:47:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:47:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:47:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:47:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:47:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:47:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:47:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:47:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:47:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:47:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:47:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:47:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:47:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:47:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:47:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:47:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:47:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:47:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:47:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:47:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:47:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:47:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:47:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:47:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:47:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:47:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:47:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:47:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:47:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:47:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:47:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:47:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:47:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:47:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:47:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:47:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:47:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:47:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:47:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:47:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:47:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:47:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:47:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:47:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:47:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:47:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:47:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:47:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:47:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:47:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:47:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:47:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:47:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:47:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:47:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:47:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:47:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:47:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:47:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:47:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:47:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:47:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:47:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:47:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:47:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:47:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:47:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:47:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:47:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:47:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:47:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:47:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:47:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:47:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:47:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:47:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:47:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:47:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:47:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:47:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:47:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:47:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:47:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:47:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:47:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:47:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:47:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:47:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:47:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:47:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:47:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:47:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:47:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:47:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:47:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:47:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:47:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:47:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:47:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:47:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:47:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:47:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:47:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:47:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:47:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:47:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:47:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:47:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:47:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:47:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:47:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:47:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:47:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:47:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:47:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:47:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:47:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:47:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:47:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:47:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:47:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:47:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:47:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:47:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:47:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:47:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:47:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:47:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:47:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:47:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:47:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:47:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:47:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:47:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:47:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:47:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:47:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:47:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:47:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:47:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:47:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:47:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:47:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:47:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:47:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:47:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:47:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:47:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:47:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:47:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:47:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:47:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:47:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:47:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:47:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:47:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:48:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:48:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:48:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:48:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:48:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:48:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:48:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:48:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:48:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:48:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:48:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:48:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:48:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:48:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:48:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:48:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:48:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:48:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:48:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:48:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:48:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:48:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:48:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:48:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:48:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:48:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:48:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:48:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:48:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:48:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:48:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:48:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:48:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:48:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:48:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:48:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:48:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:48:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:48:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:48:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:48:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:48:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:48:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:48:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:48:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:48:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:48:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:48:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:48:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:48:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:48:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:48:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:48:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:48:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:48:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:48:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:48:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:48:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:48:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:48:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:48:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:48:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:48:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:48:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:48:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:48:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:48:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:48:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:48:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:48:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:48:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:48:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:48:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:48:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:48:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:48:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:48:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:48:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:48:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:48:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:48:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:48:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:48:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:48:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:48:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:48:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:48:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:48:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:48:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:48:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:48:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:48:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:48:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:48:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:48:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:48:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:48:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:48:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:48:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:48:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:48:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:48:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:48:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:48:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:48:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:48:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:48:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:48:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:48:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:48:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:48:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:48:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:48:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:48:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:48:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:48:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:48:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:48:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:48:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:48:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:48:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:48:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:48:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:48:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:48:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:48:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:48:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:48:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:48:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:48:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:48:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:48:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:48:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:48:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:48:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:48:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:48:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:48:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:48:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:48:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:48:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:48:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:48:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:48:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:48:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:48:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:48:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:48:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:48:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:48:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:48:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:48:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:48:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:48:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:48:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:48:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:48:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:48:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:48:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:48:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:48:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:48:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:48:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:48:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:48:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:48:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:48:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:48:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:48:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:48:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:48:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:48:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:48:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:48:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:48:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:48:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:48:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:48:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:48:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:48:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:48:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:48:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:48:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:48:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:48:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:48:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:48:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:48:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:48:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:48:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:48:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:48:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:48:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:48:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:48:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:48:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:48:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:48:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:48:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:48:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:48:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:48:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:48:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:48:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:48:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:48:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:48:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:48:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:48:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:48:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:48:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:48:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:48:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:48:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:48:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:48:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:49:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:49:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:49:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:49:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:49:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:49:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:49:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:49:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:49:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:49:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:49:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:49:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:49:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:49:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:49:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:49:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:49:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:49:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:49:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:49:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:49:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:49:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:49:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:49:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:49:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:49:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:49:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:49:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:49:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:49:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:49:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:49:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:49:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:49:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:49:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:49:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:49:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:49:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:49:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:49:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:49:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:49:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:49:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:49:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:49:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:49:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:49:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:49:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:49:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:49:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:49:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:49:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:49:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:49:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:49:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:49:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:49:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:49:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:49:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:49:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:49:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:49:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:49:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:49:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:49:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:49:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:49:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:49:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:49:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:49:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:49:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:49:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:49:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:49:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:49:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:49:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:49:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:49:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:49:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:49:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:49:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:49:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:49:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:49:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:49:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:49:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:49:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:49:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:49:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:49:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:49:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:49:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:49:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:49:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:49:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:49:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:49:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:49:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:49:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:49:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:49:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:49:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:49:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:49:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:49:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:49:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:49:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:49:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:49:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:49:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:49:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:49:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:49:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:49:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:49:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:49:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:49:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:49:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:49:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:49:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:49:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:49:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:49:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:49:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:49:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:49:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:49:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:49:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:49:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:49:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:49:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:49:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:49:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:49:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:49:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:49:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:49:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:49:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:49:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:49:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:49:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:49:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:49:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:49:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:49:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:49:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:49:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:49:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:49:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:49:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:49:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:49:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:49:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:49:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:49:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:49:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:49:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:49:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:49:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:49:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:49:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:49:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:49:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:49:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:49:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:49:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:49:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:49:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:49:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:49:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:49:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:49:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:49:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:49:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:49:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:49:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:49:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:49:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:49:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:49:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:49:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:49:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:49:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:49:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:49:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:49:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:49:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:49:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:49:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:49:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:49:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:49:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:49:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:49:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:49:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:49:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:49:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:49:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:49:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:49:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:49:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:49:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:49:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:49:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:49:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:49:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:49:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:49:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:49:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:49:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:49:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:49:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:49:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:49:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:49:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:49:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:50:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:50:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:50:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:50:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:50:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:50:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:50:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:50:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:50:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:50:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:50:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:50:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:50:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:50:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:50:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:50:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:50:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:50:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:50:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:50:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:50:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:50:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:50:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:50:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:50:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:50:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:50:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:50:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:50:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:50:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:50:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:50:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:50:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:50:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:50:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:50:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:50:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:50:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:50:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:50:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:50:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:50:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:50:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:50:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:50:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:50:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:50:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:50:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:50:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:50:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:50:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:50:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:50:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:50:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:50:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:50:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:50:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:50:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:50:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:50:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:50:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:50:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:50:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:50:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:50:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:50:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:50:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:50:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:50:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:50:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:50:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:50:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:50:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:50:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:50:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:50:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:50:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:50:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:50:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:50:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:50:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:50:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:50:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:50:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:50:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:50:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:50:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:50:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:50:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:50:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:50:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:50:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:50:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:50:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:50:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:50:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:50:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:50:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:50:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:50:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:50:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:50:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:50:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:50:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:50:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:50:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:50:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:50:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:50:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:50:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:50:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:50:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:50:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:50:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:50:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:50:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:50:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:50:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:50:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:50:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:50:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:50:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:50:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:50:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:50:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:50:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:50:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:50:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:50:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:50:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:50:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:50:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:50:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:50:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:50:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:50:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:50:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:50:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:50:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:50:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:50:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:50:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:50:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:50:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:50:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:50:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:50:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:50:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:50:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:50:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:50:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:50:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:50:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:50:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:50:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:50:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:50:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:50:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:50:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:50:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:50:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:50:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:50:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:50:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:50:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:50:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:50:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:50:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:50:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:50:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:50:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:50:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:50:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:50:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:50:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:50:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:50:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:50:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:50:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:50:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:50:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:50:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:50:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:50:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:50:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:50:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:50:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:50:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:50:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:50:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:50:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:50:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:50:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:50:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:50:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:50:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:50:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:50:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:50:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:50:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:50:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:50:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:50:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:50:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:50:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:50:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:50:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:50:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:50:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:50:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:50:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:50:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:50:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:50:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:50:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:50:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:51:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:51:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:51:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:51:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:51:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:51:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:51:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:51:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:51:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:51:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:51:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:51:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:51:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:51:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:51:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:51:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:51:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:51:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:51:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:51:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:51:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:51:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:51:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:51:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:51:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:51:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:51:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:51:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:51:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:51:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:51:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:51:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:51:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:51:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:51:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:51:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:51:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:51:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:51:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:51:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:51:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:51:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:51:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:51:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:51:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:51:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:51:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:51:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:51:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:51:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:51:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:51:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:51:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:51:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:51:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:51:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:51:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:51:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:51:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:51:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:51:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:51:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:51:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:51:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:51:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:51:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:51:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:51:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:51:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:51:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:51:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:51:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:51:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:51:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:51:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:51:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:51:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:51:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:51:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:51:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:51:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:51:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:51:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:51:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:51:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:51:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:51:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:51:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:51:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:51:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:51:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:51:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:51:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:51:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:51:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:51:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:51:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:51:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:51:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:51:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:51:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:51:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:51:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:51:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:51:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:51:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:51:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:51:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:51:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:51:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:51:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:51:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:51:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:51:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:51:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:51:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:51:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:51:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:51:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:51:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:51:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:51:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:51:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:51:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:51:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:51:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:51:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:51:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:51:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:51:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:51:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:51:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:51:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:51:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:51:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:51:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:51:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:51:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:51:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:51:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:51:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:51:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:51:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:51:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:51:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:51:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:51:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:51:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:51:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:51:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:51:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:51:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:51:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:52:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:52:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:52:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:52:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:52:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:52:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:52:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:52:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:52:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:52:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:52:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:52:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:52:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:52:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:52:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:52:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:52:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:52:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:52:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:52:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:52:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:52:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:52:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:52:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:52:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:52:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:52:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:52:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:52:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:52:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:52:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:52:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:52:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:52:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:52:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:52:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:52:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:52:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:52:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:52:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:52:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:52:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:52:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:52:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:52:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:52:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:52:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:52:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:52:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:52:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:52:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:52:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:52:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:52:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:52:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:52:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:52:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:52:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:52:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:52:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:52:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:52:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:52:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:52:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:52:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:52:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:52:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:52:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:52:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:52:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:52:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:52:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:52:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:52:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:52:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:52:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:52:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:52:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:52:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:52:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:52:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:52:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:52:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:52:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:52:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:52:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:52:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:52:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:52:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:52:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:52:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:52:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:52:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:52:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:52:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:52:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:52:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:52:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:52:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:52:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:52:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:52:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:52:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:52:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:52:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:52:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:52:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:52:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:52:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:52:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:52:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:52:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:52:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:52:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:52:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:52:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:52:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:52:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:52:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:52:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:52:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:52:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:52:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:52:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:52:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:52:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:52:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:52:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:52:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:52:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:52:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:52:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:52:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:52:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:52:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:52:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:52:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:52:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:52:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:52:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:52:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:52:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:52:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:52:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:52:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:52:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:52:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:52:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:52:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:52:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:52:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:52:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:52:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:52:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:52:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:52:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:52:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:52:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:52:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:52:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:52:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:52:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:52:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:52:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:52:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:52:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:52:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:52:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:52:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:52:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:52:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:52:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:52:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:52:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:52:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:52:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:53:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:53:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:53:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:53:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:53:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:53:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:53:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:53:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:53:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:53:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:53:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:53:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:53:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:53:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:53:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:53:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:53:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:53:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:53:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:53:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:53:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:53:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:53:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:53:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:53:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:53:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:53:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:53:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:53:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:53:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:53:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:53:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:53:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:53:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:53:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:53:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:53:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:53:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:53:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:53:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:53:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:53:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:53:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:53:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:53:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:53:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:53:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:53:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:53:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:53:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:53:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:53:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:53:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:53:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:53:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:53:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:53:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:53:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:53:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:53:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:53:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:53:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:53:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:53:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:53:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:53:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:53:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:53:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:53:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:53:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:53:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:53:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:53:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:53:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:53:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:53:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:53:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:53:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:53:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:53:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:53:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:53:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:53:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:53:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:53:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:53:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:53:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:53:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:53:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:53:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:53:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:53:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:53:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:53:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:53:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:53:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:53:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:53:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:53:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:53:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:53:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:53:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:53:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:53:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:53:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:53:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:53:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:53:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:53:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:53:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:53:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:53:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:53:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:53:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:53:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:53:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:53:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:53:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:53:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:53:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:53:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:53:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:53:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:53:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:53:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:53:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:53:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:53:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:53:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:53:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:53:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:53:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:53:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:53:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:53:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:53:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:53:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:53:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:53:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:53:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:53:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:53:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:53:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:53:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:53:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:53:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:53:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:53:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:53:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:53:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:53:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:53:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:53:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:53:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:53:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:53:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:53:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:53:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:53:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:53:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:53:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:53:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:53:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:53:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:53:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:53:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:53:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:53:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:53:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:53:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:53:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:53:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:53:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:53:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:53:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:53:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:53:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:53:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:53:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:53:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:53:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:53:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:53:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:53:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:53:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:53:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:53:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:53:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:53:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:53:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:53:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:53:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:53:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:53:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:53:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:53:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:53:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:53:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:53:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:53:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:53:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:53:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:53:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:53:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:53:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:53:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:53:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:53:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:53:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:53:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:53:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:53:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:53:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:53:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:53:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:53:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:54:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:54:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:54:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:54:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:54:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:54:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:54:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:54:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:54:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:54:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:54:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:54:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:54:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:54:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:54:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:54:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:54:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:54:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:54:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:54:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:54:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:54:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:54:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:54:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:54:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:54:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:54:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:54:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:54:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:54:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:54:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:54:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:54:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:54:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:54:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:54:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:54:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:54:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:54:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:54:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:54:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:54:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:54:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:54:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:54:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:54:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:54:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:54:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:54:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:54:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:54:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:54:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:54:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:54:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:54:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:54:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:54:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:54:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:54:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:54:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:54:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:54:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:54:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:54:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:54:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:54:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:54:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:54:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:54:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:54:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:54:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:54:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:54:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:54:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:54:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:54:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:54:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:54:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:54:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:54:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:54:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:54:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:54:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:54:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:54:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:54:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:54:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:54:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:54:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:54:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:54:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:54:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:54:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:54:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:54:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:54:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:54:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:54:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:54:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:54:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:54:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:54:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:54:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:54:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:54:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:54:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:54:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:54:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:54:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:54:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:54:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:54:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:54:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:54:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:54:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:54:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:54:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:54:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:54:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:54:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:54:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:54:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:54:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:54:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:54:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:54:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:54:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:54:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:54:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:54:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:54:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:54:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:54:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:54:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:54:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:54:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:54:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:54:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:54:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:54:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:54:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:54:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:54:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:54:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:54:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:54:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:54:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:54:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:54:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:54:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:54:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:54:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:54:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:54:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:54:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:54:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:54:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:54:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:54:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:54:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:54:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:54:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:54:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:54:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:54:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:54:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:54:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:54:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:54:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:54:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:54:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:54:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:54:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:54:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:54:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:54:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:54:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:54:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:54:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:54:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:54:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:54:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:54:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:54:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:54:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:54:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:54:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:54:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:54:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:54:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:54:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:54:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:54:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:54:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:54:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:54:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:54:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:54:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:54:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:54:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:54:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:54:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:54:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:54:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:54:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:54:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:54:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:54:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:54:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:54:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:54:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:54:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:54:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:54:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:54:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:54:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:55:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:55:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:55:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:55:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:55:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:55:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:55:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:55:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:55:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:55:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:55:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:55:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:55:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:55:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:55:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:55:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:55:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:55:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:55:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:55:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:55:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:55:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:55:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:55:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:55:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:55:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:55:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:55:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:55:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:55:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:55:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:55:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:55:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:55:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:55:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:55:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:57:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:57:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:57:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:57:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:57:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:57:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:57:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:57:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:57:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:57:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:57:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:57:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:57:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:57:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:57:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:57:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:57:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:57:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:57:18 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 16:57:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:57:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:57:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:57:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:57:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:57:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:57:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:57:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:57:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:57:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:57:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:57:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:57:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:57:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:57:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:57:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:57:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:57:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:57:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:57:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:57:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:57:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:57:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:57:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:57:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:57:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:57:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:57:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:57:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:57:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:57:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:57:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:57:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:57:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:57:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:57:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:57:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:57:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:57:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:57:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:57:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:57:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:57:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:57:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:57:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:57:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:57:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:57:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:57:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:57:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:57:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:57:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:57:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:57:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:57:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:57:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:57:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:57:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:57:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:57:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:57:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:57:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:57:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:57:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:57:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:57:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:57:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:57:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:57:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:57:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:57:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:57:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:57:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:57:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:57:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:57:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:57:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:57:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:57:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:57:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:57:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:57:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:57:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:57:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:57:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:57:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:57:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:57:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:57:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:57:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:57:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:57:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:57:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:57:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:57:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:57:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:57:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:57:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:57:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:57:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:57:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:57:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:57:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:57:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:57:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:57:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:57:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:57:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:57:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:57:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:57:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:57:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:57:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:57:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:57:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:57:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:57:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:57:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:57:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:57:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:57:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:57:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:57:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:57:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:57:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:57:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:57:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:57:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:57:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:57:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:57:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:57:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:57:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:57:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:57:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:57:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:57:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:57:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:57:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:57:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:57:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:57:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:57:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:57:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:58:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:58:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:58:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:58:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:58:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:58:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:58:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:58:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:58:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:58:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:58:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:58:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:58:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:58:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:58:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:58:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:58:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:58:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:58:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:58:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:58:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:58:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:58:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:58:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:58:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:58:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:58:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:58:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:58:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:58:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:58:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:58:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:58:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:58:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:58:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:58:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:58:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:58:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:58:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:58:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:58:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:58:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:58:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:58:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:58:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:58:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:58:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:58:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:58:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:58:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:58:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:58:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:58:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:58:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:58:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:58:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:58:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:58:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:58:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:58:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:58:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:58:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:58:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:58:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:58:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:58:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:58:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:58:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:58:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:58:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:58:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:58:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:58:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:58:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:58:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:58:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:58:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:58:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:58:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:58:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:58:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:58:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:58:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:58:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:58:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:58:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:58:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:58:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:58:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:58:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:58:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:58:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:58:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:58:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:58:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:58:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:58:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:58:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:58:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:58:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:58:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:58:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:58:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:58:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:58:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:58:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:58:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:58:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:58:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:58:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:58:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:58:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:58:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:58:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:58:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:58:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:58:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:58:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:58:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:58:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:58:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:58:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:58:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:58:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:58:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:58:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:58:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:58:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:58:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:58:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:58:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:58:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:58:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:58:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:58:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:58:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:58:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:58:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:58:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:58:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:58:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:58:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:58:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:58:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:58:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:58:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:58:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:58:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:58:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:58:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:58:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:58:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:58:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:58:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:58:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:58:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:58:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:58:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:58:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:58:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:58:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:58:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:58:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:58:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:58:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:58:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:58:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:58:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:58:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:58:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:58:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:58:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:58:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:58:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:58:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:58:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:58:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:58:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:58:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:58:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:58:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:58:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:58:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:58:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:58:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:58:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:58:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:58:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:58:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:58:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:58:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:58:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:58:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:58:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:58:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:58:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:58:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:58:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:58:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:58:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:58:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:58:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:58:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:58:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:58:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:58:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:58:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:58:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:58:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:58:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:58:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:58:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:58:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:58:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:58:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:58:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:59:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:59:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:59:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:59:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:59:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:59:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:59:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:59:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:59:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:59:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:59:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:59:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:59:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:59:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:59:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:59:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:59:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:59:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:59:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:59:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:59:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:59:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:59:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:59:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:59:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:59:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:59:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:59:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:59:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:59:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:59:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:59:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:59:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:59:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:59:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:59:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:59:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:59:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:59:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:59:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:59:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:59:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:59:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:59:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:59:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:59:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:59:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:59:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:59:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:59:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:59:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:59:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:59:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:59:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:59:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:59:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:59:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:59:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:59:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:59:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:59:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:59:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:59:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:59:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:59:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:59:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:59:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:59:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:59:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:59:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:59:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:59:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:59:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:59:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:59:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:59:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:59:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:59:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:59:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:59:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:59:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:59:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:59:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:59:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:59:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:59:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:59:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:59:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:59:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:59:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:59:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:59:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:59:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:59:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:59:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:59:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:59:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:59:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:59:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:59:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:59:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:59:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:59:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:59:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:59:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:59:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:59:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:59:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:59:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:59:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:59:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:59:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:59:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:59:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:59:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:59:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:59:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:59:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:59:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:59:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:59:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:59:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:59:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:59:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:59:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:59:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:59:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:59:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:59:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:59:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:59:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:59:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:59:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:59:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:59:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:59:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:59:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:59:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:59:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:59:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:59:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:59:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:59:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:59:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:59:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:59:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:59:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:59:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:59:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:59:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:59:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:59:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:59:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:59:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:59:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:59:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:59:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:59:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:59:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:59:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:59:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:59:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:59:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:59:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:59:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:59:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:59:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:59:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:59:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:59:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:59:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:59:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:59:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:59:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:59:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:59:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:59:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:59:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:59:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:59:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:59:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:59:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:59:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:59:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:59:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:59:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:59:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:59:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:59:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:59:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:59:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:59:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:59:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:59:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:59:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:59:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:59:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:59:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 16:59:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 16:59:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:59:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:59:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:59:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 16:59:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 16:59:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 16:59:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 16:59:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 16:59:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 16:59:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 16:59:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 16:59:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 16:59:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 16:59:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 16:59:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 16:59:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 16:59:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:00:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:00:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:00:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:00:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:00:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:00:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:00:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:00:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:00:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:00:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:00:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:00:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:00:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:00:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:00:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:00:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:00:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:00:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:00:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:00:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:00:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:00:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:00:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:00:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:00:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:00:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:00:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:00:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:00:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:00:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:00:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:00:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:00:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:00:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:00:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:00:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:00:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:00:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:00:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:00:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:00:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:00:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:00:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:00:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:00:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:00:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:00:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:00:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:00:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:00:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:00:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:00:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:00:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:00:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:00:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:00:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:00:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:00:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:00:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:00:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:00:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:00:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:00:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:00:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:00:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:00:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:00:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:00:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:00:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:00:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:00:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:00:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:00:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:00:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:00:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:00:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:00:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:00:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:00:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:00:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:00:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:00:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:00:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:00:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:00:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:00:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:00:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:00:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:00:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:00:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:00:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:00:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:00:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:00:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:00:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:00:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:00:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:00:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:00:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:00:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:00:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:00:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:00:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:00:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:00:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:00:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:00:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:00:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:00:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:00:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:00:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:00:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:00:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:00:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:00:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:00:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:00:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:00:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:00:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:00:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:00:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:00:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:00:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:00:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:00:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:00:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:00:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:00:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:00:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:00:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:00:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:00:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:00:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:00:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:00:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:00:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:00:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:00:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:00:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:00:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:00:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:00:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:00:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:00:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:00:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:00:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:00:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:00:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:00:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:00:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:00:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:00:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:00:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:00:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:00:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:00:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:00:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:00:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:00:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:00:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:00:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:00:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:00:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:00:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:00:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:00:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:00:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:00:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:00:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:00:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:00:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:00:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:00:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:00:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:00:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:00:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:00:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:00:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:00:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:00:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:00:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:00:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:00:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:00:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:00:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:00:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:00:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:00:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:00:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:00:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:00:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:00:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:00:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:00:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:00:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:00:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:00:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:00:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:00:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:00:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:00:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:00:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:00:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:00:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:00:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:00:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:00:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:00:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:00:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:00:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:00:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:00:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:00:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:00:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:00:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:00:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:01:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:01:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:01:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:01:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:01:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:01:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:01:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:01:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:01:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:01:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:01:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:01:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:01:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:01:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:01:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:01:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:01:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:01:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:01:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:01:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:01:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:01:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:01:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:01:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:01:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:01:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:01:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:01:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:01:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:01:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:01:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:01:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:01:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:01:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:01:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:01:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:01:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:01:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:01:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:01:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:01:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:01:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:01:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:01:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:01:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:01:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:01:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:01:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:01:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:01:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:01:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:01:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:01:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:01:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:01:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:01:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:01:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:01:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:01:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:01:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:01:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:01:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:01:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:01:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:01:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:01:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:01:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:01:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:01:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:01:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:01:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:01:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:01:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:01:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:01:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:01:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:01:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:01:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:01:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:01:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:01:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:01:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:01:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:01:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:01:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:01:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:01:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:01:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:01:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:01:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:01:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:01:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:01:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:01:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:01:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:01:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:01:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:01:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:01:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:01:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:01:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:01:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:01:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:01:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:01:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:01:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:01:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:01:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:01:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:01:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:01:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:01:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:01:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:01:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:01:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:01:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:01:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:01:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:01:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:01:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:01:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:01:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:01:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:01:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:01:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:01:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:01:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:01:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:01:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:01:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:01:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:01:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:01:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:01:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:01:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:01:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:01:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:01:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:01:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:01:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:01:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:01:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:01:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:01:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:01:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:01:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:01:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:01:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:01:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:01:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:01:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:01:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:01:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:01:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:01:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:01:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:01:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:01:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:01:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:01:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:01:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:01:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:01:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:01:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:01:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:01:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:01:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:01:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:01:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:01:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:01:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:01:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:01:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:01:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:01:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:01:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:01:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:01:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:01:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:01:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:01:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:01:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:01:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:01:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:01:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:01:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:01:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:01:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:01:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:01:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:01:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:01:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:01:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:01:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:01:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:01:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:01:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:01:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:01:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:01:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:01:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:01:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:01:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:01:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:01:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:01:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:01:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:01:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:01:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:01:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:01:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:01:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:01:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:01:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:01:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:01:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:02:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:02:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:02:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:02:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:02:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:02:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:02:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:02:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:02:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:02:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:02:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:02:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:02:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:02:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:02:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:02:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:02:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:02:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:02:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:02:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:02:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:02:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:02:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:02:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:02:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:02:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:02:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:02:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:02:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:02:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:02:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:02:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:02:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:02:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:02:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:02:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:02:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:02:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:02:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:02:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:02:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:02:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:02:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:02:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:02:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:02:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:02:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:02:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:02:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:02:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:02:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:02:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:02:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:02:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:02:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:02:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:02:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:02:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:02:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:02:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:02:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:02:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:02:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:02:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:02:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:02:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:02:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:02:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:02:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:02:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:02:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:02:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:02:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:02:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:02:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:02:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:02:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:02:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:02:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:02:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:02:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:02:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:02:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:02:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:02:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:02:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:02:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:02:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:02:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:02:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:02:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:02:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:02:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:02:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:02:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:02:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:02:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:02:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:02:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:02:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:02:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:02:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:02:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:02:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:02:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:02:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:02:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:02:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:02:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:02:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:02:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:02:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:02:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:02:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:02:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:02:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:02:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:02:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:02:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:02:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:02:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:02:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:02:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:02:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:02:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:02:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:02:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:02:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:02:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:02:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:02:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:02:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:02:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:02:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:02:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:02:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:02:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:02:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:02:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:02:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:02:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:02:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:02:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:02:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:02:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:02:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:02:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:02:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:02:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:02:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:02:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:02:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:02:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:02:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:02:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:02:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:02:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:02:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:02:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:02:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:02:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:02:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:02:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:02:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:02:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:02:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:02:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:02:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:02:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:02:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:02:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:02:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:02:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:02:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:02:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:02:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:02:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:02:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:02:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:02:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:02:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:02:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:02:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:02:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:02:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:02:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:02:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:02:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:02:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:02:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:02:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:02:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:02:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:02:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:02:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:02:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:02:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:02:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:02:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:02:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:02:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:02:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:02:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:02:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:02:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:02:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:02:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:02:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:02:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:02:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:02:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:02:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:02:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:02:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:02:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:02:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:03:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:03:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:03:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:03:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:03:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:03:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:03:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:03:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:03:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:03:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:03:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:03:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:03:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:03:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:03:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:03:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:03:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:03:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:03:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:03:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:03:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:03:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:03:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:03:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:03:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:03:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:03:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:03:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:03:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:03:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:03:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:03:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:03:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:03:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:03:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:03:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:03:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:03:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:03:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:03:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:03:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:03:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:03:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:03:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:03:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:03:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:03:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:03:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:03:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:03:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:03:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:03:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:03:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:03:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:03:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:03:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:03:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:03:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:03:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:03:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:03:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:03:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:03:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:03:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:03:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:03:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:03:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:03:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:03:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:03:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:03:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:03:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:03:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:03:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:03:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:03:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:03:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:03:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:03:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:03:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:03:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:03:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:03:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:03:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:03:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:03:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:03:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:03:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:03:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:03:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:03:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:03:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:03:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:03:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:03:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:03:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:03:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:03:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:03:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:03:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:03:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:03:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:03:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:03:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:03:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:03:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:03:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:03:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:03:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:03:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:03:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:03:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:03:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:03:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:03:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:03:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:03:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:03:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:03:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:03:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:03:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:03:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:03:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:03:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:03:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:03:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:03:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:03:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:03:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:03:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:03:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:03:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:03:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:03:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:03:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:03:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:03:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:03:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:03:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:03:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:03:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:03:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:03:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:03:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:03:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:03:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:03:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:03:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:03:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:03:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:03:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:03:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:03:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:03:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:03:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:03:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:03:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:03:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:03:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:03:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:03:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:03:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:03:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:03:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:03:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:03:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:03:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:03:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:03:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:03:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:03:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:03:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:03:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:03:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:03:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:03:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:03:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:03:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:03:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:03:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:03:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:03:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:03:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:03:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:03:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:03:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:03:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:03:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:03:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:03:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:03:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:03:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:03:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:03:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:03:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:03:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:03:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:03:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:03:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:03:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:03:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:03:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:03:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:03:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:03:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:03:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:03:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:03:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:03:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:03:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:03:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:03:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:03:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:03:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:03:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:03:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:04:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:04:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:04:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:04:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:04:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:04:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:04:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:04:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:04:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:04:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:04:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:04:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:04:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:04:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:04:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:04:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:04:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:04:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:04:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:04:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:04:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:04:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:04:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:04:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:04:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:04:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:04:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:04:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:04:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:04:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:04:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:04:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:04:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:04:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:04:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:04:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:04:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:04:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:04:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:04:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:04:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:04:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:04:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:04:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:04:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:04:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:04:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:04:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:04:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:04:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:04:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:04:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:04:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:04:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:04:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:04:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:04:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:04:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:04:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:04:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:04:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:04:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:04:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:04:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:04:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:04:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:04:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:04:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:04:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:04:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:04:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:04:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:04:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:04:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:04:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:04:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:04:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:04:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:04:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:04:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:04:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:04:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:04:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:04:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:04:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:04:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:04:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:04:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:04:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:04:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:04:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:04:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:04:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:04:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:04:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:04:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:04:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:04:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:04:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:04:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:04:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:04:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:04:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:04:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:04:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:04:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:04:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:04:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:04:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:04:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:04:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:04:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:04:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:04:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:04:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:04:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:04:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:04:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:04:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:04:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:04:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:04:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:04:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:04:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:04:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:04:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:04:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:04:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:04:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:04:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:04:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:04:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:04:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:04:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:04:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:04:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:04:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:04:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:04:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:04:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:04:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:04:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:04:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:04:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:04:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:04:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:04:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:04:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:04:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:04:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:04:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:04:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:04:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:04:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:04:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:04:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:04:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:04:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:04:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:04:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:04:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:04:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:04:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:04:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:04:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:04:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:04:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:04:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:04:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:04:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:04:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:04:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:04:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:04:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:04:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:04:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:04:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:04:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:04:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:04:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:04:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:04:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:04:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:04:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:04:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:04:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:04:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:04:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:04:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:04:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:04:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:04:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:04:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:04:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:04:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:04:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:04:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:04:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:04:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:04:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:04:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:04:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:04:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:04:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:04:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:04:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:04:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:04:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:04:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:04:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:04:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:04:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:04:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:04:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:04:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:04:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:05:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:05:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:05:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:05:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:05:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:05:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:05:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:05:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:05:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:05:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:05:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:05:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:05:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:05:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:05:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:05:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:05:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:05:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:05:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:05:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:05:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:05:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:05:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:05:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:05:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:05:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:05:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:05:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:05:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:05:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:05:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:05:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:05:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:05:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:05:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:05:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:05:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:05:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:05:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:05:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:05:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:05:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:05:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:05:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:05:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:05:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:05:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:05:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:05:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:05:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:05:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:05:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:05:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:05:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:07:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:07:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:07:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:07:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:07:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:07:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:07:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:07:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:07:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:07:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:07:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:07:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:07:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:07:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:07:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:07:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:07:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:07:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:07:23 rdma06 rsyslogd[1041]: imjournal: 5295 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 17:07:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:07:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:07:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:07:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:07:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:07:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:07:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:07:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:07:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:07:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:07:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:07:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:07:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:07:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:07:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:07:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:07:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:07:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:07:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:07:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:07:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:07:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:07:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:07:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:07:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:07:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:07:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:07:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:07:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:07:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:07:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:07:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:07:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:07:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:07:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:07:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:07:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:07:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:07:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:07:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:07:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:07:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:07:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:07:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:07:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:07:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:07:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:07:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:07:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:07:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:07:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:07:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:07:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:07:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:07:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:07:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:07:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:07:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:07:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:07:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:07:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:07:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:07:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:07:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:07:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:07:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:07:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:07:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:07:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:07:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:07:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:07:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:07:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:07:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:07:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:07:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:07:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:07:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:07:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:07:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:07:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:07:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:07:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:07:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:07:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:07:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:07:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:07:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:07:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:07:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:07:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:07:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:07:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:07:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:07:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:07:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:07:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:07:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:07:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:07:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:07:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:07:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:07:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:07:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:07:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:07:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:07:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:07:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:07:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:07:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:07:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:07:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:07:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:07:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:07:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:07:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:07:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:07:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:07:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:07:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:07:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:07:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:07:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:07:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:07:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:07:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:08:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:08:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:08:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:08:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:08:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:08:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:08:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:08:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:08:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:08:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:08:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:08:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:08:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:08:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:08:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:08:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:08:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:08:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:08:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:08:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:08:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:08:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:08:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:08:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:08:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:08:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:08:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:08:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:08:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:08:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:08:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:08:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:08:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:08:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:08:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:08:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:08:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:08:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:08:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:08:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:08:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:08:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:08:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:08:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:08:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:08:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:08:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:08:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:08:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:08:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:08:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:08:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:08:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:08:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:08:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:08:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:08:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:08:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:08:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:08:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:08:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:08:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:08:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:08:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:08:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:08:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:08:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:08:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:08:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:08:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:08:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:08:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:08:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:08:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:08:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:08:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:08:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:08:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:08:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:08:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:08:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:08:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:08:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:08:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:08:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:08:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:08:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:08:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:08:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:08:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:08:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:08:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:08:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:08:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:08:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:08:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:08:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:08:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:08:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:08:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:08:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:08:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:08:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:08:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:08:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:08:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:08:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:08:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:08:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:08:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:08:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:08:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:08:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:08:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:08:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:08:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:08:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:08:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:08:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:08:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:08:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:08:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:08:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:08:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:08:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:08:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:08:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:08:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:08:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:08:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:08:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:08:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:08:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:08:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:08:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:08:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:08:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:08:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:08:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:08:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:08:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:08:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:08:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:08:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:08:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:08:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:08:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:08:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:08:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:08:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:08:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:08:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:08:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:08:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:08:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:08:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:08:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:08:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:08:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:08:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:08:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:08:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:08:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:08:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:08:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:08:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:08:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:08:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:08:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:08:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:08:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:08:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:08:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:08:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:08:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:08:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:08:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:08:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:08:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:08:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:08:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:08:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:08:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:08:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:08:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:08:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:08:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:08:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:08:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:08:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:08:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:08:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:08:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:08:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:08:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:08:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:08:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:08:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:08:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:08:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:08:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:08:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:08:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:08:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:08:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:08:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:08:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:08:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:08:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:08:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:08:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:08:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:08:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:08:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:08:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:08:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:09:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:09:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:09:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:09:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:09:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:09:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:09:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:09:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:09:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:09:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:09:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:09:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:09:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:09:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:09:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:09:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:09:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:09:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:09:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:09:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:09:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:09:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:09:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:09:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:09:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:09:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:09:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:09:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:09:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:09:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:09:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:09:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:09:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:09:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:09:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:09:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:09:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:09:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:09:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:09:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:09:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:09:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:09:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:09:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:09:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:09:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:09:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:09:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:09:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:09:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:09:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:09:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:09:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:09:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:09:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:09:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:09:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:09:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:09:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:09:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:09:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:09:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:09:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:09:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:09:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:09:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:09:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:09:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:09:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:09:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:09:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:09:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:09:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:09:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:09:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:09:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:09:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:09:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:09:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:09:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:09:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:09:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:09:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:09:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:09:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:09:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:09:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:09:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:09:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:09:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:09:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:09:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:09:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:09:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:09:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:09:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:09:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:09:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:09:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:09:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:09:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:09:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:09:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:09:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:09:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:09:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:09:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:09:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:09:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:09:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:09:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:09:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:09:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:09:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:09:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:09:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:09:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:09:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:09:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:09:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:09:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:09:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:09:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:09:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:09:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:09:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:09:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:09:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:09:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:09:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:09:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:09:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:09:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:09:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:09:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:09:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:09:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:09:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:09:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:09:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:09:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:09:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:09:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:09:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:09:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:09:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:09:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:09:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:09:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:09:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:09:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:09:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:09:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:09:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:09:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:09:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:09:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:09:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:09:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:09:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:09:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:09:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:09:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:09:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:09:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:09:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:09:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:09:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:09:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:09:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:09:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:09:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:09:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:09:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:09:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:09:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:09:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:09:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:09:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:09:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:09:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:09:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:09:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:09:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:09:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:09:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:09:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:09:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:09:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:09:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:09:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:09:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:09:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:09:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:09:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:09:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:09:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:09:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:09:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:09:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:09:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:09:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:09:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:09:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:09:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:09:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:09:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:09:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:09:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:09:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:09:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:09:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:09:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:09:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:09:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:09:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:10:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:10:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:10:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:10:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:10:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:10:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:10:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:10:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:10:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:10:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:10:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:10:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:10:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:10:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:10:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:10:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:10:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:10:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:10:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:10:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:10:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:10:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:10:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:10:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:10:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:10:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:10:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:10:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:10:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:10:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:10:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:10:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:10:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:10:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:10:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:10:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:10:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:10:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:10:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:10:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:10:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:10:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:10:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:10:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:10:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:10:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:10:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:10:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:10:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:10:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:10:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:10:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:10:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:10:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:10:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:10:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:10:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:10:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:10:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:10:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:10:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:10:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:10:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:10:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:10:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:10:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:10:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:10:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:10:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:10:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:10:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:10:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:10:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:10:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:10:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:10:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:10:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:10:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:10:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:10:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:10:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:10:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:10:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:10:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:10:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:10:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:10:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:10:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:10:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:10:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:10:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:10:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:10:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:10:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:10:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:10:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:10:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:10:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:10:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:10:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:10:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:10:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:10:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:10:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:10:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:10:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:10:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:10:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:10:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:10:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:10:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:10:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:10:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:10:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:10:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:10:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:10:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:10:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:10:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:10:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:10:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:10:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:10:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:10:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:10:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:10:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:10:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:10:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:10:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:10:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:10:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:10:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:10:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:10:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:10:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:10:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:10:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:10:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:10:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:10:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:10:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:10:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:10:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:10:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:10:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:10:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:10:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:10:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:10:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:10:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:10:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:10:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:10:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:10:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:10:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:10:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:10:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:10:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:10:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:10:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:10:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:10:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:10:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:10:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:10:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:10:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:10:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:10:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:10:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:10:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:10:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:10:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:10:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:10:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:10:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:10:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:10:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:10:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:10:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:10:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:10:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:10:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:10:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:10:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:10:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:10:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:10:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:10:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:10:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:10:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:10:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:10:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:10:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:10:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:10:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:10:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:10:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:10:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:10:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:10:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:10:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:10:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:10:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:10:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:10:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:10:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:10:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:10:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:10:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:10:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:10:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:10:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:10:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:10:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:10:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:10:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:11:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:11:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:11:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:11:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:11:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:11:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:11:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:11:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:11:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:11:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:11:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:11:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:11:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:11:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:11:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:11:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:11:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:11:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:11:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:11:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:11:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:11:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:11:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:11:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:11:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:11:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:11:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:11:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:11:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:11:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:11:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:11:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:11:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:11:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:11:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:11:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:11:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:11:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:11:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:11:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:11:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:11:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:11:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:11:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:11:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:11:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:11:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:11:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:11:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:11:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:11:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:11:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:11:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:11:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:11:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:11:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:11:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:11:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:11:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:11:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:11:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:11:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:11:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:11:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:11:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:11:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:11:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:11:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:11:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:11:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:11:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:11:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:11:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:11:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:11:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:11:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:11:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:11:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:11:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:11:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:11:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:11:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:11:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:11:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:11:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:11:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:11:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:11:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:11:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:11:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:11:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:11:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:11:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:11:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:11:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:11:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:11:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:11:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:11:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:11:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:11:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:11:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:11:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:11:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:11:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:11:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:11:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:11:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:11:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:11:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:11:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:11:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:11:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:11:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:11:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:11:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:11:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:11:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:11:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:11:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:11:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:11:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:11:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:11:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:11:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:11:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:11:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:11:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:11:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:11:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:11:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:11:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:11:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:11:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:11:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:11:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:11:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:11:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:11:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:11:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:11:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:11:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:11:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:11:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:11:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:11:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:11:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:11:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:11:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:11:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:11:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:11:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:11:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:11:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:11:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:11:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:11:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:11:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:11:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:11:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:11:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:11:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:11:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:11:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:11:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:11:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:11:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:11:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:11:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:11:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:11:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:11:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:11:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:11:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:11:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:11:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:11:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:11:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:11:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:11:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:11:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:11:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:11:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:11:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:11:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:11:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:11:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:11:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:11:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:11:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:11:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:11:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:11:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:11:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:11:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:11:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:11:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:11:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:11:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:11:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:11:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:11:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:11:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:11:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:11:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:11:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:11:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:11:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:11:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:11:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:11:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:11:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:11:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:11:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:11:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:11:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:12:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:12:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:12:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:12:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:12:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:12:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:12:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:12:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:12:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:12:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:12:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:12:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:12:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:12:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:12:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:12:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:12:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:12:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:12:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:12:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:12:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:12:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:12:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:12:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:12:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:12:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:12:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:12:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:12:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:12:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:12:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:12:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:12:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:12:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:12:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:12:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:12:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:12:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:12:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:12:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:12:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:12:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:12:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:12:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:12:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:12:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:12:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:12:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:12:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:12:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:12:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:12:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:12:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:12:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:12:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:12:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:12:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:12:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:12:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:12:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:12:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:12:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:12:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:12:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:12:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:12:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:12:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:12:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:12:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:12:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:12:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:12:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:12:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:12:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:12:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:12:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:12:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:12:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:12:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:12:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:12:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:12:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:12:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:12:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:12:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:12:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:12:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:12:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:12:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:12:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:12:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:12:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:12:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:12:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:12:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:12:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:12:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:12:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:12:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:12:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:12:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:12:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:12:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:12:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:12:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:12:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:12:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:12:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:12:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:12:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:12:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:12:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:12:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:12:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:12:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:12:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:12:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:12:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:12:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:12:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:12:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:12:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:12:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:12:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:12:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:12:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:12:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:12:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:12:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:12:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:12:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:12:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:12:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:12:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:12:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:12:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:12:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:12:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:12:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:12:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:12:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:12:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:12:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:12:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:12:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:12:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:12:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:12:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:12:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:12:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:12:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:12:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:12:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:12:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:12:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:12:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:12:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:12:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:12:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:12:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:12:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:12:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:12:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:12:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:12:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:12:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:12:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:12:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:12:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:12:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:12:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:12:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:12:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:12:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:12:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:12:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:12:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:12:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:12:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:12:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:12:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:12:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:12:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:12:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:12:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:12:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:12:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:12:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:12:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:12:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:12:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:12:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:12:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:12:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:12:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:12:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:12:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:12:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:12:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:12:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:12:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:12:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:12:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:12:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:12:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:12:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:12:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:12:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:12:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:12:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:12:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:12:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:12:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:12:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:12:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:12:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:13:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:13:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:13:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:13:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:13:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:13:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:13:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:13:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:13:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:13:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:13:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:13:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:13:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:13:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:13:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:13:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:13:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:13:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:13:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:13:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:13:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:13:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:13:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:13:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:13:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:13:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:13:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:13:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:13:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:13:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:13:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:13:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:13:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:13:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:13:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:13:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:13:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:13:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:13:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:13:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:13:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:13:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:13:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:13:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:13:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:13:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:13:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:13:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:13:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:13:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:13:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:13:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:13:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:13:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:13:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:13:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:13:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:13:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:13:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:13:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:13:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:13:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:13:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:13:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:13:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:13:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:13:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:13:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:13:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:13:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:13:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:13:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:13:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:13:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:13:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:13:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:13:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:13:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:13:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:13:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:13:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:13:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:13:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:13:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:13:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:13:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:13:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:13:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:13:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:13:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:13:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:13:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:13:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:13:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:13:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:13:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:13:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:13:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:13:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:13:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:13:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:13:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:13:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:13:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:13:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:13:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:13:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:13:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:13:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:13:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:13:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:13:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:13:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:13:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:13:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:13:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:13:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:13:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:13:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:13:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:13:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:13:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:13:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:13:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:13:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:13:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:13:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:13:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:13:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:13:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:13:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:13:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:13:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:13:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:13:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:13:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:13:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:13:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:13:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:13:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:13:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:13:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:13:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:13:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:13:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:13:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:13:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:13:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:13:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:13:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:13:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:13:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:13:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:13:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:13:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:13:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:13:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:13:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:13:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:13:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:13:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:13:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:13:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:13:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:13:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:13:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:13:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:13:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:13:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:13:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:13:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:13:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:13:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:13:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:13:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:13:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:13:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:13:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:13:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:13:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:13:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:13:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:13:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:13:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:13:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:13:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:13:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:13:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:13:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:13:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:13:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:13:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:13:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:13:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:13:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:13:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:13:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:13:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:13:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:13:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:13:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:13:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:13:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:13:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:13:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:13:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:13:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:13:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:13:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:13:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:13:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:13:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:13:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:13:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:13:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:13:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:14:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:14:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:14:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:14:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:14:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:14:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:14:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:14:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:14:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:14:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:14:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:14:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:14:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:14:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:14:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:14:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:14:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:14:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:14:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:14:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:14:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:14:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:14:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:14:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:14:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:14:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:14:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:14:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:14:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:14:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:14:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:14:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:14:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:14:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:14:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:14:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:14:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:14:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:14:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:14:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:14:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:14:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:14:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:14:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:14:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:14:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:14:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:14:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:14:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:14:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:14:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:14:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:14:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:14:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:14:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:14:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:14:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:14:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:14:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:14:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:14:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:14:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:14:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:14:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:14:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:14:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:14:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:14:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:14:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:14:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:14:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:14:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:14:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:14:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:14:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:14:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:14:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:14:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:14:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:14:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:14:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:14:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:14:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:14:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:14:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:14:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:14:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:14:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:14:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:14:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:14:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:14:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:14:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:14:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:14:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:14:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:14:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:14:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:14:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:14:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:14:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:14:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:14:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:14:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:14:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:14:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:14:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:14:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:14:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:14:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:14:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:14:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:14:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:14:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:14:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:14:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:14:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:14:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:14:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:14:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:14:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:14:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:14:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:14:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:14:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:14:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:14:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:14:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:14:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:14:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:14:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:14:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:14:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:14:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:14:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:14:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:14:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:14:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:14:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:14:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:14:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:14:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:14:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:14:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:14:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:14:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:14:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:14:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:14:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:14:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:14:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:14:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:14:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:14:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:14:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:14:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:14:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:14:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:14:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:14:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:14:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:14:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:14:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:14:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:14:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:14:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:14:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:14:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:14:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:14:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:14:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:14:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:14:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:14:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:14:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:14:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:14:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:14:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:14:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:14:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:14:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:14:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:14:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:14:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:14:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:14:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:14:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:14:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:14:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:14:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:14:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:14:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:14:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:14:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:14:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:14:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:14:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:14:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:14:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:14:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:14:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:14:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:14:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:14:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:14:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:14:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:14:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:14:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:14:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:14:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:14:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:14:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:14:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:14:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:14:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:14:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:15:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:15:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:15:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:15:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:15:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:15:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:15:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:15:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:15:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:15:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:15:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:15:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:15:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:15:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:15:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:15:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:15:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:15:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:15:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:15:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:15:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:15:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:15:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:15:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:15:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:15:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:15:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:15:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:15:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:15:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:15:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:15:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:15:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:15:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:15:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:15:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:15:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:15:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:15:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:15:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:15:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:15:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:15:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:15:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:15:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:15:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:15:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:15:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:15:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:15:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:15:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:15:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:15:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:15:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:15:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:15:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:15:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:15:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:15:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:15:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:15:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:15:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:15:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:15:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:15:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:15:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:15:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:15:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:15:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:15:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:15:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:15:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:17:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:17:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:17:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:17:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:17:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:17:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:17:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:17:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:17:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:17:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:17:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:17:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:17:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:17:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:17:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:17:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:17:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:17:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:17:24 rdma06 rsyslogd[1041]: imjournal: 5082 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 17:17:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:17:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:17:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:17:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:17:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:17:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:17:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:17:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:17:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:17:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:17:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:17:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:17:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:17:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:17:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:17:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:17:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:17:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:17:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:17:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:17:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:17:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:17:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:17:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:17:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:17:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:17:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:17:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:17:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:17:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:17:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:17:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:17:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:17:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:17:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:17:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:17:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:17:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:17:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:17:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:17:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:17:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:17:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:17:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:17:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:17:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:17:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:17:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:17:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:17:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:17:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:17:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:17:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:17:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:17:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:17:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:17:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:17:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:17:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:17:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:17:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:17:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:17:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:17:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:17:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:17:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:17:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:17:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:17:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:17:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:17:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:17:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:17:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:17:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:17:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:17:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:17:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:17:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:17:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:17:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:17:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:17:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:17:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:17:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:17:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:17:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:17:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:17:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:17:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:17:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:17:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:17:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:17:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:17:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:17:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:17:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:17:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:17:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:17:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:17:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:17:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:17:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:17:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:17:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:17:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:17:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:17:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:17:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:17:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:17:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:17:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:17:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:17:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:17:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:17:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:17:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:17:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:17:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:17:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:17:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:17:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:17:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:17:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:17:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:17:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:17:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:18:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:18:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:18:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:18:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:18:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:18:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:18:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:18:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:18:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:18:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:18:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:18:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:18:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:18:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:18:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:18:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:18:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:18:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:18:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:18:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:18:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:18:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:18:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:18:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:18:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:18:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:18:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:18:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:18:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:18:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:18:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:18:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:18:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:18:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:18:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:18:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:18:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:18:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:18:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:18:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:18:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:18:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:18:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:18:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:18:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:18:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:18:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:18:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:18:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:18:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:18:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:18:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:18:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:18:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:18:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:18:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:18:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:18:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:18:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:18:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:18:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:18:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:18:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:18:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:18:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:18:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:18:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:18:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:18:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:18:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:18:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:18:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:18:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:18:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:18:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:18:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:18:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:18:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:18:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:18:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:18:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:18:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:18:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:18:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:18:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:18:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:18:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:18:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:18:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:18:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:18:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:18:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:18:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:18:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:18:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:18:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:18:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:18:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:18:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:18:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:18:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:18:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:18:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:18:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:18:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:18:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:18:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:18:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:18:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:18:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:18:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:18:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:18:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:18:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:18:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:18:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:18:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:18:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:18:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:18:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:18:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:18:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:18:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:18:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:18:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:18:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:18:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:18:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:18:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:18:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:18:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:18:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:18:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:18:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:18:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:18:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:18:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:18:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:18:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:18:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:18:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:18:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:18:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:18:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:18:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:18:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:18:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:18:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:18:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:18:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:18:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:18:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:18:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:18:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:18:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:18:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:18:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:18:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:18:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:18:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:18:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:18:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:18:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:18:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:18:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:18:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:18:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:18:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:18:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:18:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:18:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:18:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:18:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:18:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:18:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:18:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:18:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:18:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:18:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:18:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:18:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:18:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:18:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:18:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:18:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:18:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:18:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:18:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:18:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:18:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:18:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:18:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:18:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:18:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:18:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:18:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:18:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:18:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:18:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:18:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:18:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:18:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:18:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:18:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:18:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:18:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:18:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:18:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:18:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:18:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:18:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:18:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:18:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:18:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:18:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:18:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:19:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:19:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:19:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:19:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:19:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:19:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:19:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:19:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:19:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:19:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:19:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:19:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:19:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:19:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:19:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:19:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:19:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:19:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:19:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:19:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:19:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:19:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:19:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:19:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:19:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:19:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:19:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:19:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:19:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:19:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:19:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:19:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:19:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:19:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:19:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:19:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:19:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:19:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:19:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:19:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:19:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:19:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:19:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:19:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:19:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:19:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:19:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:19:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:19:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:19:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:19:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:19:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:19:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:19:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:19:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:19:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:19:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:19:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:19:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:19:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:19:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:19:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:19:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:19:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:19:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:19:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:19:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:19:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:19:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:19:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:19:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:19:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:19:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:19:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:19:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:19:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:19:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:19:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:19:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:19:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:19:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:19:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:19:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:19:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:19:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:19:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:19:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:19:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:19:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:19:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:19:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:19:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:19:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:19:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:19:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:19:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:19:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:19:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:19:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:19:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:19:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:19:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:19:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:19:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:19:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:19:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:19:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:19:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:19:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:19:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:19:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:19:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:19:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:19:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:19:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:19:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:19:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:19:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:19:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:19:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:19:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:19:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:19:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:19:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:19:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:19:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:19:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:19:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:19:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:19:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:19:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:19:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:19:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:19:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:19:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:19:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:19:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:19:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:19:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:19:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:19:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:19:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:19:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:19:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:19:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:19:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:19:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:19:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:19:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:19:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:19:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:19:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:19:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:19:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:19:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:19:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:19:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:19:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:19:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:19:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:19:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:19:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:19:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:19:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:19:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:19:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:19:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:19:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:19:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:19:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:19:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:19:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:19:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:19:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:19:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:19:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:19:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:19:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:19:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:19:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:19:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:19:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:19:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:19:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:19:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:19:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:19:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:19:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:19:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:19:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:19:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:19:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:19:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:19:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:19:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:19:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:19:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:19:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:19:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:19:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:19:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:19:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:19:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:19:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:19:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:19:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:19:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:19:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:19:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:19:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:19:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:19:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:19:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:19:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:19:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:19:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:20:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:20:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:20:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:20:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:20:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:20:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:20:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:20:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:20:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:20:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:20:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:20:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:20:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:20:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:20:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:20:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:20:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:20:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:20:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:20:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:20:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:20:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:20:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:20:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:20:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:20:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:20:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:20:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:20:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:20:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:20:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:20:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:20:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:20:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:20:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:20:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:20:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:20:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:20:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:20:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:20:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:20:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:20:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:20:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:20:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:20:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:20:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:20:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:20:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:20:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:20:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:20:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:20:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:20:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:20:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:20:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:20:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:20:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:20:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:20:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:20:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:20:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:20:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:20:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:20:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:20:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:20:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:20:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:20:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:20:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:20:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:20:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:20:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:20:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:20:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:20:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:20:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:20:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:20:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:20:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:20:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:20:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:20:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:20:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:20:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:20:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:20:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:20:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:20:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:20:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:20:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:20:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:20:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:20:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:20:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:20:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:20:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:20:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:20:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:20:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:20:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:20:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:20:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:20:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:20:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:20:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:20:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:20:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:20:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:20:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:20:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:20:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:20:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:20:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:20:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:20:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:20:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:20:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:20:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:20:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:20:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:20:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:20:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:20:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:20:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:20:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:20:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:20:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:20:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:20:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:20:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:20:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:20:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:20:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:20:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:20:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:20:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:20:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:20:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:20:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:20:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:20:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:20:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:20:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:20:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:20:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:20:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:20:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:20:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:20:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:20:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:20:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:20:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:20:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:20:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:20:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:20:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:20:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:20:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:20:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:20:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:20:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:20:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:20:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:20:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:20:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:20:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:20:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:20:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:20:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:20:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:20:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:20:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:20:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:20:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:20:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:20:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:20:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:20:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:20:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:20:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:20:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:20:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:20:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:20:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:20:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:20:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:20:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:20:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:20:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:20:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:20:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:20:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:20:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:20:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:20:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:20:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:20:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:20:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:20:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:20:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:20:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:20:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:20:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:20:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:20:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:20:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:20:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:20:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:20:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:20:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:20:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:20:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:20:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:20:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:20:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:21:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:21:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:21:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:21:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:21:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:21:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:21:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:21:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:21:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:21:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:21:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:21:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:21:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:21:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:21:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:21:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:21:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:21:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:21:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:21:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:21:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:21:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:21:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:21:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:21:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:21:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:21:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:21:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:21:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:21:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:21:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:21:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:21:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:21:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:21:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:21:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:21:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:21:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:21:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:21:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:21:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:21:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:21:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:21:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:21:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:21:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:21:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:21:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:21:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:21:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:21:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:21:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:21:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:21:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:21:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:21:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:21:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:21:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:21:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:21:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:21:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:21:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:21:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:21:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:21:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:21:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:21:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:21:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:21:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:21:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:21:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:21:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:21:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:21:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:21:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:21:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:21:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:21:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:21:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:21:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:21:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:21:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:21:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:21:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:21:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:21:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:21:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:21:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:21:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:21:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:21:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:21:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:21:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:21:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:21:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:21:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:21:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:21:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:21:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:21:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:21:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:21:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:21:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:21:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:21:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:21:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:21:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:21:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:21:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:21:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:21:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:21:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:21:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:21:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:21:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:21:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:22:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:22:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:22:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:22:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:22:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:22:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:22:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:22:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:22:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:22:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:22:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:22:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:22:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:22:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:22:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:22:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:22:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:22:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:22:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:22:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:22:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:22:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:22:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:22:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:22:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:22:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:22:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:22:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:22:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:22:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:22:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:22:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:22:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:22:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:22:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:22:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:22:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:22:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:22:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:22:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:22:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:22:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:22:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:22:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:22:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:22:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:22:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:22:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:22:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:22:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:22:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:22:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:22:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:22:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:22:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:22:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:22:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:22:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:22:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:22:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:22:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:22:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:22:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:22:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:22:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:22:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:22:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:22:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:22:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:22:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:22:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:22:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:22:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:22:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:22:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:22:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:22:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:22:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:22:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:22:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:22:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:22:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:22:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:22:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:22:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:22:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:22:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:22:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:22:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:22:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:22:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:22:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:22:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:22:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:22:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:22:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:22:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:22:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:22:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:22:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:22:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:22:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:22:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:22:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:22:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:22:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:22:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:22:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:22:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:22:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:22:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:22:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:22:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:22:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:22:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:22:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:22:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:22:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:22:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:22:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:22:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:22:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:22:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:22:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:22:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:22:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:22:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:22:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:22:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:22:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:22:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:22:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:22:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:22:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:22:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:22:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:22:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:22:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:22:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:22:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:22:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:22:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:22:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:22:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:22:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:22:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:22:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:22:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:22:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:22:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:22:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:22:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:22:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:22:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:22:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:22:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:22:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:22:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:22:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:22:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:22:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:22:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:22:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:22:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:22:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:22:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:22:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:22:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:22:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:22:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:22:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:22:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:22:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:22:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:22:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:22:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:22:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:22:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:22:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:22:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:22:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:22:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:22:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:22:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:22:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:22:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:22:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:22:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:22:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:22:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:22:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:22:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:22:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:22:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:22:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:22:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:22:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:22:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:22:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:22:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:22:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:22:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:22:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:22:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:22:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:22:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:22:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:22:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:22:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:22:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:22:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:22:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:22:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:22:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:22:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:22:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:23:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:23:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:23:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:23:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:23:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:23:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:23:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:23:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:23:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:23:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:23:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:23:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:23:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:23:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:23:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:23:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:23:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:23:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:23:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:23:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:23:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:23:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:23:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:23:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:23:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:23:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:23:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:23:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:23:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:23:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:23:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:23:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:23:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:23:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:23:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:23:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:23:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:23:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:23:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:23:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:23:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:23:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:23:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:23:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:23:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:23:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:23:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:23:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:23:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:23:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:23:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:23:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:23:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:23:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:23:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:23:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:23:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:23:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:23:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:23:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:23:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:23:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:23:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:23:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:23:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:23:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:23:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:23:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:23:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:23:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:23:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:23:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:23:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:23:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:23:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:23:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:23:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:23:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:23:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:23:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:23:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:23:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:23:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:23:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:23:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:23:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:23:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:23:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:23:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:23:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:23:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:23:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:23:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:23:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:23:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:23:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:23:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:23:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:23:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:23:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:23:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:23:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:23:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:23:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:23:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:23:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:23:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:23:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:23:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:23:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:23:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:23:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:23:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:23:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:23:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:23:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:23:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:23:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:23:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:23:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:23:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:23:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:23:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:23:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:23:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:23:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:23:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:23:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:23:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:23:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:23:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:23:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:23:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:23:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:23:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:23:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:23:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:23:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:23:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:23:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:23:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:23:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:23:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:23:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:23:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:23:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:23:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:23:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:23:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:23:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:23:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:23:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:23:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:23:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:23:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:23:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:23:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:23:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:23:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:23:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:23:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:23:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:23:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:23:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:23:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:23:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:23:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:23:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:23:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:23:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:23:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:23:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:23:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:23:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:23:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:23:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:23:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:23:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:23:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:23:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:23:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:23:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:23:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:23:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:23:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:23:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:23:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:23:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:23:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:23:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:23:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:23:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:23:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:23:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:23:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:23:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:23:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:23:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:23:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:23:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:23:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:23:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:23:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:23:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:23:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:23:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:23:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:23:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:23:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:23:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:23:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:23:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:23:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:23:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:23:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:23:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:24:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:24:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:24:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:24:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:24:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:24:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:24:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:24:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:24:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:24:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:24:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:24:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:24:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:24:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:24:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:24:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:24:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:24:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:24:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:24:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:24:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:24:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:24:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:24:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:24:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:24:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:24:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:24:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:24:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:24:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:24:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:24:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:24:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:24:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:24:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:24:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:24:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:24:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:24:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:24:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:24:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:24:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:24:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:24:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:24:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:24:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:24:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:24:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:24:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:24:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:24:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:24:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:24:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:24:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:24:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:24:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:24:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:24:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:24:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:24:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:24:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:24:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:24:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:24:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:24:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:24:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:24:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:24:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:24:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:24:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:24:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:24:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:24:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:24:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:24:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:24:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:24:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:24:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:24:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:24:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:24:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:24:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:24:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:24:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:24:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:24:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:24:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:24:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:24:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:24:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:24:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:24:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:24:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:24:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:24:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:24:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:24:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:24:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:24:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:24:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:24:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:24:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:24:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:24:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:24:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:24:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:24:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:24:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:24:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:24:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:24:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:24:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:24:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:24:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:24:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:24:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:24:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:24:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:24:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:24:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:24:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:24:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:24:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:24:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:24:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:24:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:24:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:24:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:24:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:24:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:24:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:24:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:24:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:24:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:24:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:24:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:24:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:24:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:24:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:24:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:24:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:24:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:24:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:24:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:24:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:24:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:24:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:24:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:24:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:24:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:24:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:24:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:24:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:24:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:24:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:24:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:24:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:24:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:24:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:24:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:24:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:24:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:24:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:24:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:24:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:24:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:24:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:24:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:24:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:24:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:24:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:24:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:24:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:24:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:24:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:24:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:24:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:24:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:24:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:24:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:24:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:24:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:24:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:24:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:24:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:24:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:24:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:24:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:24:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:24:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:24:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:24:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:24:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:24:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:24:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:24:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:24:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:24:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:24:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:24:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:24:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:24:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:24:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:24:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:24:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:24:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:24:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:24:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:24:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:24:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:24:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:24:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:24:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:24:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:24:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:24:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:25:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:25:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:25:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:25:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:25:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:25:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:25:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:25:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:25:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:25:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:25:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:25:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:25:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:25:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:25:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:25:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:25:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:25:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:25:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:25:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:25:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:25:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:25:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:25:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:25:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:25:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:25:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:25:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:25:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:25:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:25:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:25:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:25:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:25:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:25:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:25:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:25:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:25:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:25:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:25:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:25:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:25:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:25:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:25:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:25:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:25:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:25:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:25:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:25:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:25:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:25:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:25:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:25:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:25:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:25:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:25:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:25:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:25:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:25:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:25:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:25:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:25:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:25:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:25:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:25:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:25:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:25:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:25:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:25:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:25:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:25:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:25:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:27:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:27:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:27:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:27:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:27:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:27:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:27:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:27:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:27:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:27:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:27:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:27:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:27:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:27:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:27:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:27:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:27:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:27:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:27:29 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 17:27:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:27:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:27:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:27:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:27:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:27:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:27:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:27:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:27:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:27:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:27:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:27:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:27:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:27:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:27:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:27:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:27:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:27:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:27:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:27:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:27:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:27:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:27:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:27:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:27:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:27:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:27:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:27:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:27:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:27:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:27:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:27:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:27:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:27:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:27:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:27:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:27:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:27:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:27:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:27:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:27:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:27:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:27:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:27:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:27:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:27:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:27:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:27:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:27:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:27:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:27:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:27:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:27:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:27:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:27:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:27:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:27:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:27:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:27:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:27:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:27:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:27:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:27:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:27:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:27:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:27:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:27:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:27:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:27:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:27:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:27:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:27:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:27:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:27:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:27:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:27:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:27:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:27:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:27:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:27:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:27:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:27:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:27:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:27:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:27:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:27:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:27:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:27:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:27:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:27:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:27:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:27:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:27:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:27:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:27:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:27:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:27:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:27:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:27:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:27:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:27:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:27:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:27:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:27:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:27:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:27:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:27:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:27:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:28:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:28:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:28:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:28:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:28:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:28:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:28:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:28:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:28:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:28:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:28:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:28:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:28:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:28:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:28:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:28:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:28:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:28:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:28:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:28:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:28:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:28:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:28:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:28:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:28:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:28:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:28:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:28:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:28:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:28:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:28:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:28:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:28:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:28:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:28:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:28:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:28:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:28:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:28:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:28:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:28:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:28:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:28:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:28:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:28:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:28:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:28:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:28:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:28:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:28:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:28:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:28:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:28:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:28:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:28:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:28:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:28:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:28:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:28:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:28:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:28:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:28:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:28:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:28:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:28:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:28:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:28:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:28:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:28:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:28:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:28:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:28:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:28:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:28:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:28:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:28:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:28:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:28:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:28:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:28:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:28:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:28:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:28:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:28:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:28:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:28:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:28:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:28:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:28:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:28:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:28:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:28:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:28:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:28:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:28:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:28:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:28:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:28:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:28:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:28:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:28:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:28:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:28:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:28:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:28:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:28:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:28:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:28:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:28:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:28:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:28:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:28:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:28:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:28:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:28:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:28:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:28:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:28:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:28:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:28:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:28:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:28:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:28:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:28:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:28:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:28:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:28:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:28:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:28:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:28:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:28:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:28:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:28:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:28:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:28:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:28:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:28:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:28:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:28:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:28:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:28:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:28:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:28:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:28:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:28:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:28:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:28:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:28:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:28:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:28:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:28:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:28:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:28:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:28:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:28:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:28:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:28:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:28:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:28:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:28:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:28:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:28:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:28:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:28:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:28:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:28:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:28:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:28:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:28:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:28:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:28:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:28:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:28:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:28:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:28:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:28:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:28:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:28:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:28:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:28:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:28:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:28:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:28:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:28:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:28:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:28:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:28:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:28:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:28:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:28:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:28:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:28:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:28:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:28:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:28:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:28:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:28:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:28:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:28:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:28:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:28:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:28:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:28:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:28:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:28:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:28:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:28:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:28:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:28:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:28:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:28:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:28:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:28:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:28:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:28:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:28:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:29:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:29:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:29:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:29:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:29:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:29:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:29:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:29:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:29:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:29:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:29:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:29:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:29:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:29:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:29:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:29:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:29:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:29:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:29:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:29:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:29:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:29:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:29:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:29:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:29:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:29:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:29:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:29:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:29:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:29:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:29:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:29:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:29:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:29:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:29:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:29:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:29:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:29:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:29:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:29:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:29:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:29:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:29:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:29:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:29:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:29:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:29:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:29:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:29:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:29:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:29:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:29:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:29:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:29:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:29:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:29:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:29:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:29:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:29:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:29:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:29:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:29:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:29:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:29:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:29:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:29:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:29:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:29:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:29:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:29:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:29:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:29:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:29:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:29:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:29:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:29:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:29:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:29:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:29:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:29:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:29:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:29:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:29:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:29:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:29:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:29:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:29:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:29:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:29:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:29:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:29:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:29:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:29:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:29:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:29:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:29:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:29:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:29:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:29:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:29:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:29:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:29:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:29:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:29:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:29:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:29:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:29:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:29:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:29:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:29:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:29:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:29:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:29:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:29:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:29:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:29:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:29:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:29:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:29:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:29:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:29:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:29:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:29:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:29:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:29:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:29:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:29:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:29:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:29:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:29:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:29:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:29:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:29:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:29:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:29:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:29:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:29:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:29:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:29:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:29:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:29:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:29:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:29:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:29:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:29:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:29:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:29:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:29:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:29:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:29:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:29:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:29:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:29:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:29:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:29:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:29:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:29:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:29:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:29:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:29:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:29:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:29:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:29:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:29:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:29:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:29:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:29:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:29:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:29:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:29:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:29:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:29:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:29:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:29:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:29:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:29:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:29:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:29:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:29:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:29:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:29:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:29:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:29:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:29:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:29:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:29:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:29:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:29:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:29:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:29:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:29:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:29:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:29:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:29:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:29:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:29:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:29:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:29:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:29:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:29:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:29:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:29:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:29:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:29:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:29:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:29:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:29:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:29:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:29:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:29:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:29:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:29:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:29:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:29:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:29:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:29:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:30:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:30:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:30:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:30:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:30:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:30:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:30:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:30:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:30:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:30:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:30:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:30:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:30:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:30:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:30:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:30:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:30:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:30:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:30:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:30:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:30:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:30:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:30:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:30:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:30:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:30:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:30:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:30:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:30:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:30:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:30:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:30:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:30:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:30:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:30:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:30:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:30:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:30:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:30:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:30:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:30:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:30:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:30:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:30:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:30:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:30:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:30:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:30:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:30:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:30:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:30:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:30:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:30:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:30:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:30:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:30:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:30:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:30:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:30:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:30:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:30:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:30:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:30:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:30:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:30:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:30:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:30:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:30:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:30:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:30:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:30:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:30:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:30:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:30:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:30:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:30:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:30:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:30:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:30:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:30:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:30:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:30:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:30:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:30:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:30:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:30:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:30:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:30:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:30:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:30:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:30:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:30:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:30:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:30:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:30:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:30:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:30:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:30:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:30:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:30:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:30:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:30:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:30:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:30:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:30:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:30:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:30:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:30:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:30:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:30:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:30:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:30:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:30:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:30:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:30:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:30:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:30:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:30:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:30:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:30:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:30:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:30:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:30:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:30:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:30:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:30:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:30:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:30:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:30:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:30:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:30:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:30:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:30:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:30:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:30:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:30:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:30:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:30:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:30:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:30:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:30:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:30:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:30:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:30:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:30:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:30:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:30:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:30:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:30:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:30:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:30:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:30:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:30:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:30:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:30:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:30:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:30:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:30:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:30:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:30:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:30:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:30:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:30:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:30:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:30:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:30:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:30:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:30:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:30:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:30:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:30:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:30:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:30:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:30:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:30:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:30:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:30:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:30:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:30:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:30:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:30:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:30:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:30:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:30:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:30:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:30:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:30:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:30:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:30:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:30:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:30:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:30:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:30:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:30:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:30:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:30:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:30:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:30:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:30:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:30:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:30:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:30:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:30:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:30:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:30:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:30:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:30:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:30:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:30:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:30:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:30:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:30:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:30:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:30:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:30:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:30:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:31:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:31:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:31:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:31:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:31:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:31:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:31:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:31:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:31:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:31:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:31:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:31:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:31:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:31:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:31:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:31:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:31:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:31:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:31:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:31:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:31:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:31:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:31:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:31:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:31:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:31:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:31:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:31:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:31:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:31:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:31:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:31:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:31:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:31:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:31:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:31:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:31:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:31:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:31:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:31:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:31:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:31:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:31:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:31:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:31:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:31:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:31:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:31:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:31:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:31:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:31:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:31:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:31:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:31:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:31:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:31:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:31:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:31:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:31:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:31:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:31:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:31:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:31:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:31:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:31:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:31:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:31:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:31:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:31:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:31:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:31:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:31:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:31:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:31:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:31:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:31:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:31:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:31:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:31:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:31:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:31:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:31:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:31:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:31:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:31:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:31:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:31:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:31:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:31:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:31:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:31:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:31:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:31:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:31:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:31:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:31:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:31:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:31:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:31:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:31:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:31:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:31:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:31:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:31:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:31:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:31:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:31:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:31:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:31:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:31:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:31:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:31:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:31:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:31:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:31:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:31:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:31:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:31:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:31:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:31:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:31:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:31:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:31:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:31:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:31:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:31:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:31:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:31:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:31:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:31:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:31:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:31:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:31:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:31:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:31:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:31:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:31:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:31:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:31:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:31:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:31:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:31:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:31:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:31:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:31:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:31:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:31:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:31:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:31:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:31:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:31:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:31:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:31:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:31:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:31:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:31:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:31:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:31:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:31:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:31:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:31:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:31:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:31:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:31:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:31:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:31:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:31:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:31:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:31:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:31:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:31:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:31:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:31:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:31:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:31:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:31:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:31:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:31:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:31:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:31:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:31:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:31:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:31:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:31:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:31:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:31:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:31:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:31:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:31:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:31:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:31:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:31:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:31:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:31:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:31:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:31:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:31:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:31:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:31:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:31:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:31:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:31:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:31:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:31:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:31:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:31:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:32:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:32:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:32:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:32:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:32:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:32:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:32:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:32:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:32:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:32:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:32:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:32:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:32:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:32:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:32:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:32:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:32:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:32:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:32:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:32:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:32:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:32:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:32:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:32:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:32:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:32:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:32:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:32:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:32:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:32:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:32:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:32:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:32:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:32:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:32:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:32:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:32:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:32:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:32:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:32:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:32:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:32:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:32:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:32:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:32:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:32:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:32:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:32:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:32:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:32:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:32:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:32:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:32:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:32:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:32:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:32:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:32:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:32:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:32:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:32:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:32:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:32:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:32:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:32:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:32:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:32:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:32:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:32:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:32:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:32:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:32:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:32:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:32:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:32:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:32:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:32:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:32:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:32:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:32:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:32:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:32:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:32:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:32:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:32:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:32:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:32:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:32:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:32:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:32:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:32:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:32:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:32:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:32:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:32:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:32:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:32:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:32:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:32:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:32:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:32:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:32:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:32:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:32:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:32:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:32:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:32:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:32:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:32:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:32:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:32:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:32:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:32:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:32:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:32:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:32:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:32:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:32:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:32:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:32:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:32:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:32:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:32:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:32:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:32:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:32:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:32:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:32:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:32:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:32:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:32:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:32:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:32:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:32:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:32:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:32:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:32:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:32:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:32:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:32:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:32:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:32:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:32:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:32:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:32:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:32:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:32:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:32:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:32:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:32:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:32:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:32:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:32:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:32:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:32:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:32:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:32:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:32:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:32:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:32:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:32:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:32:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:32:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:32:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:32:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:32:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:32:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:32:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:32:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:32:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:32:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:32:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:32:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:32:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:32:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:32:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:32:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:32:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:32:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:32:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:32:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:32:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:32:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:32:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:32:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:32:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:32:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:32:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:32:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:32:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:32:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:32:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:32:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:32:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:32:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:32:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:32:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:32:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:32:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:32:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:32:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:32:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:32:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:32:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:32:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:32:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:32:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:32:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:32:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:32:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:32:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:32:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:32:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:32:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:32:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:32:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:32:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:33:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:33:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:33:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:33:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:33:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:33:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:33:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:33:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:33:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:33:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:33:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:33:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:33:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:33:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:33:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:33:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:33:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:33:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:33:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:33:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:33:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:33:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:33:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:33:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:33:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:33:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:33:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:33:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:33:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:33:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:33:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:33:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:33:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:33:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:33:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:33:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:33:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:33:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:33:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:33:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:33:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:33:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:33:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:33:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:33:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:33:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:33:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:33:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:33:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:33:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:33:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:33:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:33:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:33:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:33:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:33:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:33:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:33:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:33:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:33:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:33:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:33:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:33:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:33:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:33:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:33:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:33:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:33:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:33:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:33:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:33:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:33:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:33:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:33:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:33:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:33:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:33:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:33:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:33:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:33:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:33:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:33:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:33:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:33:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:33:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:33:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:33:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:33:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:33:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:33:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:33:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:33:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:33:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:33:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:33:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:33:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:33:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:33:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:33:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:33:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:33:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:33:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:33:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:33:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:33:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:33:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:33:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:33:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:33:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:33:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:33:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:33:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:33:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:33:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:33:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:33:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:33:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:33:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:33:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:33:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:33:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:33:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:33:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:33:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:33:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:33:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:33:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:33:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:33:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:33:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:33:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:33:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:33:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:33:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:33:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:33:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:33:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:33:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:33:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:33:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:33:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:33:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:33:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:33:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:33:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:33:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:33:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:33:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:33:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:33:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:33:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:33:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:33:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:33:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:33:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:33:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:33:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:33:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:33:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:33:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:33:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:33:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:33:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:33:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:33:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:33:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:33:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:33:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:33:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:33:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:33:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:33:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:33:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:33:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:33:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:33:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:33:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:33:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:33:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:33:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:33:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:33:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:33:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:33:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:33:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:33:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:33:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:33:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:33:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:33:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:33:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:33:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:33:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:33:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:33:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:33:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:33:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:33:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:33:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:33:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:33:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:33:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:33:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:33:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:33:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:33:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:33:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:33:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:33:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:33:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:33:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:33:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:33:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:33:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:33:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:33:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:34:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:34:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:34:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:34:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:34:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:34:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:34:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:34:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:34:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:34:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:34:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:34:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:34:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:34:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:34:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:34:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:34:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:34:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:34:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:34:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:34:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:34:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:34:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:34:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:34:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:34:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:34:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:34:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:34:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:34:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:34:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:34:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:34:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:34:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:34:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:34:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:34:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:34:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:34:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:34:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:34:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:34:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:34:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:34:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:34:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:34:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:34:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:34:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:34:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:34:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:34:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:34:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:34:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:34:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:34:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:34:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:34:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:34:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:34:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:34:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:34:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:34:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:34:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:34:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:34:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:34:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:34:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:34:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:34:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:34:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:34:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:34:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:34:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:34:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:34:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:34:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:34:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:34:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:34:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:34:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:34:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:34:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:34:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:34:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:34:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:34:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:34:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:34:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:34:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:34:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:34:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:34:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:34:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:34:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:34:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:34:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:34:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:34:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:34:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:34:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:34:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:34:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:34:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:34:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:34:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:34:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:34:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:34:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:34:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:34:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:34:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:34:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:34:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:34:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:34:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:34:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:34:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:34:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:34:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:34:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:34:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:34:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:34:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:34:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:34:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:34:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:34:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:34:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:34:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:34:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:34:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:34:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:34:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:34:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:34:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:34:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:34:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:34:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:34:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:34:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:34:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:34:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:34:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:34:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:34:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:34:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:34:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:34:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:34:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:34:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:34:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:34:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:34:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:34:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:34:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:34:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:34:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:34:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:34:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:34:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:34:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:34:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:34:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:34:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:34:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:34:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:34:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:34:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:34:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:34:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:34:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:34:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:34:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:34:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:34:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:34:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:34:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:34:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:34:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:34:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:34:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:34:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:34:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:34:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:34:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:34:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:34:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:34:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:34:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:34:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:34:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:34:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:34:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:34:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:34:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:34:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:34:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:34:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:34:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:34:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:34:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:34:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:34:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:34:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:34:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:34:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:34:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:34:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:34:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:34:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:34:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:34:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:34:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:34:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:34:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:34:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:35:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:35:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:35:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:35:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:35:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:35:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:35:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:35:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:35:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:35:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:35:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:35:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:35:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:35:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:35:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:35:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:35:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:35:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:35:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:35:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:35:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:35:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:35:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:35:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:35:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:35:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:35:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:35:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:35:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:35:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:35:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:35:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:35:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:35:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:35:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:35:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:35:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:35:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:35:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:35:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:35:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:35:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:35:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:35:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:35:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:35:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:35:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:35:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:35:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:35:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:35:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:35:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:35:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:35:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:35:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:35:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:35:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:35:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:35:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:35:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:35:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:35:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:35:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:35:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:35:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:35:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:35:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:35:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:35:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:35:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:35:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:35:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:35:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:35:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:35:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:35:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:35:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:35:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:35:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:35:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:35:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:35:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:35:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:35:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:35:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:35:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:35:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:35:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:35:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:35:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:37:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:37:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:37:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:37:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:37:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:37:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:37:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:37:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:37:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:37:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:37:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:37:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:37:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:37:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:37:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:37:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:37:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:37:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:37:30 rdma06 rsyslogd[1041]: imjournal: 5082 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 17:37:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:37:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:37:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:37:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:37:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:37:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:37:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:37:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:37:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:37:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:37:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:37:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:37:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:37:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:37:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:37:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:37:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:37:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:37:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:37:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:37:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:37:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:37:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:37:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:37:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:37:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:37:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:37:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:37:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:37:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:37:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:37:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:37:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:37:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:37:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:37:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:37:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:37:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:37:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:37:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:37:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:37:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:37:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:37:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:37:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:37:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:37:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:37:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:37:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:37:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:37:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:37:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:37:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:37:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:37:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:37:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:37:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:37:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:37:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:37:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:37:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:37:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:37:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:37:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:37:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:37:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:37:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:37:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:37:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:37:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:37:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:37:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:37:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:37:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:37:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:37:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:37:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:37:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:37:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:37:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:37:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:37:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:37:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:37:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:37:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:37:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:37:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:37:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:37:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:37:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:37:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:37:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:37:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:37:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:37:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:37:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:37:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:37:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:37:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:37:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:37:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:37:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:37:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:37:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:37:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:37:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:37:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:37:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:38:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:38:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:38:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:38:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:38:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:38:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:38:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:38:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:38:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:38:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:38:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:38:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:38:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:38:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:38:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:38:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:38:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:38:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:38:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:38:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:38:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:38:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:38:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:38:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:38:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:38:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:38:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:38:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:38:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:38:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:38:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:38:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:38:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:38:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:38:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:38:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:38:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:38:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:38:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:38:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:38:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:38:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:38:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:38:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:38:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:38:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:38:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:38:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:38:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:38:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:38:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:38:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:38:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:38:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:38:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:38:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:38:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:38:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:38:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:38:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:38:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:38:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:38:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:38:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:38:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:38:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:38:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:38:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:38:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:38:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:38:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:38:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:38:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:38:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:38:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:38:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:38:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:38:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:38:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:38:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:38:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:38:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:38:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:38:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:38:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:38:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:38:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:38:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:38:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:38:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:38:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:38:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:38:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:38:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:38:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:38:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:38:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:38:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:38:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:38:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:38:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:38:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:38:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:38:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:38:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:38:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:38:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:38:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:38:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:38:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:38:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:38:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:38:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:38:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:38:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:38:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:38:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:38:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:38:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:38:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:38:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:38:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:38:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:38:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:38:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:38:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:38:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:38:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:38:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:38:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:38:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:38:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:38:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:38:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:38:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:38:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:38:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:38:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:38:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:38:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:38:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:38:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:38:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:38:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:38:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:38:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:38:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:38:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:38:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:38:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:38:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:38:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:38:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:38:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:38:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:38:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:38:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:38:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:38:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:38:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:38:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:38:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:38:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:38:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:38:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:38:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:38:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:38:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:38:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:38:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:38:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:38:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:38:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:38:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:38:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:38:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:38:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:38:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:38:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:38:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:38:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:38:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:38:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:38:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:38:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:38:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:38:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:38:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:38:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:38:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:38:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:38:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:38:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:38:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:38:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:38:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:38:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:38:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:38:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:38:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:38:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:38:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:38:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:38:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:38:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:38:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:38:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:38:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:38:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:38:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:38:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:38:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:38:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:38:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:38:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:38:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:39:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:39:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:39:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:39:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:39:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:39:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:39:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:39:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:39:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:39:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:39:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:39:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:39:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:39:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:39:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:39:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:39:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:39:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:39:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:39:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:39:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:39:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:39:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:39:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:39:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:39:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:39:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:39:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:39:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:39:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:39:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:39:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:39:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:39:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:39:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:39:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:39:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:39:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:39:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:39:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:39:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:39:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:39:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:39:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:39:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:39:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:39:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:39:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:39:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:39:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:39:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:39:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:39:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:39:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:39:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:39:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:39:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:39:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:39:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:39:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:39:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:39:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:39:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:39:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:39:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:39:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:39:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:39:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:39:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:39:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:39:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:39:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:39:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:39:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:39:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:39:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:39:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:39:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:39:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:39:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:39:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:39:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:39:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:39:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:39:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:39:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:39:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:39:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:39:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:39:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:39:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:39:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:39:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:39:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:39:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:39:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:39:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:39:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:39:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:39:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:39:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:39:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:39:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:39:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:39:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:39:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:39:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:39:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:39:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:39:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:39:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:39:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:39:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:39:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:39:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:39:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:39:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:39:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:39:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:39:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:39:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:39:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:39:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:39:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:39:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:39:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:39:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:39:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:39:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:39:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:39:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:39:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:39:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:39:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:39:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:39:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:39:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:39:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:39:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:39:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:39:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:39:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:39:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:39:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:39:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:39:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:39:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:39:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:39:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:39:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:39:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:39:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:39:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:39:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:39:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:39:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:40:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:40:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:40:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:40:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:40:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:40:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:40:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:40:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:40:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:40:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:40:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:40:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:40:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:40:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:40:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:40:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:40:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:40:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:40:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:40:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:40:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:40:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:40:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:40:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:40:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:40:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:40:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:40:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:40:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:40:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:40:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:40:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:40:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:40:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:40:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:40:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:40:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:40:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:40:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:40:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:40:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:40:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:40:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:40:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:40:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:40:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:40:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:40:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:40:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:40:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:40:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:40:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:40:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:40:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:40:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:40:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:40:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:40:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:40:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:40:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:40:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:40:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:40:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:40:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:40:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:40:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:40:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:40:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:40:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:40:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:40:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:40:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:40:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:40:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:40:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:40:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:40:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:40:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:40:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:40:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:40:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:40:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:40:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:40:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:40:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:40:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:40:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:40:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:40:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:40:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:40:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:40:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:40:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:40:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:40:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:40:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:40:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:40:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:40:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:40:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:40:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:40:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:40:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:40:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:40:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:40:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:40:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:40:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:40:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:40:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:40:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:40:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:40:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:40:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:40:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:40:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:40:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:40:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:40:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:40:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:40:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:40:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:40:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:40:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:40:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:40:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:40:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:40:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:40:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:40:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:40:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:40:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:40:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:40:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:40:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:40:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:40:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:40:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:40:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:40:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:40:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:40:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:40:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:40:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:40:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:40:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:40:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:40:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:40:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:40:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:40:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:40:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:40:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:40:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:40:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:40:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:40:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:40:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:40:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:40:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:40:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:40:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:40:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:40:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:40:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:40:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:40:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:40:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:40:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:40:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:40:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:40:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:40:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:40:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:40:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:40:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:40:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:40:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:40:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:40:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:40:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:40:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:40:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:40:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:40:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:40:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:40:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:40:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:40:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:40:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:40:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:40:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:40:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:40:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:40:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:40:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:40:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:40:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:40:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:40:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:40:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:40:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:40:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:40:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:40:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:40:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:40:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:40:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:40:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:40:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:40:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:40:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:40:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:40:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:40:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:40:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:41:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:41:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:41:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:41:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:41:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:41:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:41:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:41:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:41:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:41:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:41:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:41:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:41:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:41:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:41:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:41:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:41:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:41:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:41:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:41:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:41:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:41:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:41:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:41:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:41:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:41:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:41:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:41:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:41:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:41:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:41:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:41:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:41:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:41:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:41:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:41:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:41:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:41:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:41:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:41:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:41:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:41:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:41:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:41:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:41:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:41:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:41:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:41:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:41:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:41:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:41:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:41:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:41:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:41:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:41:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:41:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:41:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:41:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:41:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:41:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:41:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:41:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:41:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:41:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:41:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:41:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:41:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:41:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:41:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:41:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:41:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:41:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:41:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:41:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:41:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:41:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:41:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:41:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:41:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:41:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:41:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:41:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:41:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:41:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:41:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:41:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:41:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:41:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:41:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:41:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:41:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:41:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:41:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:41:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:41:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:41:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:41:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:41:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:41:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:41:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:41:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:41:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:41:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:41:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:41:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:41:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:41:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:41:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:41:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:41:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:41:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:41:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:41:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:41:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:41:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:41:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:41:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:41:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:41:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:41:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:41:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:41:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:41:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:41:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:41:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:41:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:41:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:41:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:41:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:41:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:41:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:41:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:41:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:41:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:41:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:41:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:41:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:41:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:41:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:41:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:41:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:41:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:41:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:41:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:41:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:41:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:41:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:41:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:41:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:41:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:41:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:41:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:41:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:41:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:41:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:41:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:41:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:41:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:41:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:41:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:41:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:41:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:41:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:41:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:41:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:41:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:41:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:41:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:41:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:41:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:41:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:41:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:41:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:41:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:41:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:41:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:41:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:41:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:41:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:41:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:41:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:41:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:41:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:41:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:41:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:41:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:41:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:41:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:41:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:41:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:41:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:41:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:41:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:41:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:41:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:41:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:41:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:41:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:41:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:41:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:41:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:41:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:41:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:41:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:41:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:41:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:41:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:41:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:41:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:41:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:41:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:41:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:41:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:41:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:41:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:41:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:42:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:42:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:42:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:42:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:42:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:42:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:42:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:42:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:42:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:42:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:42:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:42:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:42:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:42:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:42:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:42:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:42:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:42:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:42:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:42:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:42:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:42:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:42:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:42:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:42:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:42:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:42:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:42:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:42:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:42:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:42:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:42:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:42:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:42:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:42:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:42:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:42:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:42:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:42:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:42:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:42:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:42:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:42:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:42:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:42:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:42:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:42:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:42:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:42:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:42:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:42:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:42:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:42:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:42:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:42:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:42:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:42:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:42:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:42:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:42:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:42:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:42:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:42:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:42:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:42:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:42:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:42:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:42:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:42:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:42:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:42:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:42:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:42:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:42:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:42:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:42:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:42:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:42:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:42:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:42:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:42:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:42:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:42:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:42:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:42:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:42:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:42:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:42:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:42:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:42:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:42:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:42:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:42:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:42:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:42:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:42:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:42:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:42:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:42:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:42:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:42:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:42:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:42:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:42:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:42:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:42:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:42:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:42:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:42:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:42:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:42:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:42:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:42:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:42:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:42:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:42:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:42:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:42:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:42:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:42:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:42:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:42:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:42:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:42:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:42:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:42:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:42:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:42:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:42:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:42:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:42:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:42:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:42:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:42:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:42:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:42:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:42:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:42:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:42:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:42:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:42:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:42:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:42:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:42:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:42:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:42:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:42:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:42:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:42:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:42:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:42:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:42:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:42:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:42:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:42:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:42:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:42:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:42:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:42:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:42:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:42:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:42:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:42:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:42:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:42:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:42:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:42:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:42:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:42:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:42:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:42:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:42:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:42:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:42:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:42:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:42:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:42:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:42:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:42:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:42:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:42:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:42:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:42:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:42:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:42:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:42:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:42:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:42:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:42:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:42:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:42:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:42:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:42:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:42:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:42:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:42:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:42:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:42:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:42:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:42:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:42:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:42:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:42:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:42:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:42:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:42:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:42:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:42:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:42:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:42:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:42:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:42:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:42:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:42:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:42:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:42:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:43:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:43:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:43:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:43:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:43:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:43:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:43:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:43:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:43:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:43:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:43:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:43:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:43:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:43:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:43:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:43:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:43:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:43:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:43:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:43:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:43:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:43:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:43:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:43:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:43:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:43:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:43:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:43:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:43:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:43:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:43:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:43:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:43:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:43:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:43:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:43:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:43:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:43:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:43:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:43:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:43:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:43:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:43:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:43:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:43:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:43:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:43:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:43:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:43:14 rdma06 systemd-journald[790]: Data hash table of /run/log/journal/95379d2f723847b68d9e976dee8967a9/system.journal has a fill level at 75.0 (163750 of 218332 items, 75497472 file size, 461 bytes per hash table item), suggesting rotation.
Apr 18 17:43:14 rdma06 systemd-journald[790]: /run/log/journal/95379d2f723847b68d9e976dee8967a9/system.journal: Journal header limits reached or header out-of-date, rotating.
Apr 18 17:43:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:43:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:43:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:43:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:43:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:43:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:43:14 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 18 17:43:14 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 18 17:43:14 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 18 17:43:15 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 18 17:43:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:43:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:43:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:43:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:43:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:43:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:43:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:43:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:43:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:43:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:43:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:43:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:43:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:43:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:43:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:43:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:43:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:43:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:43:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:43:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:43:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:43:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:43:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:43:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:43:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:43:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:43:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:43:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:43:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:43:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:43:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:43:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:43:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:43:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:43:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:43:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:43:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:43:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:43:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:43:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:43:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:43:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:43:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:43:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:43:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:43:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:43:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:43:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:43:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:43:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:43:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:43:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:43:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:43:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:43:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:43:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:43:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:43:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:43:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:43:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:43:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:43:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:43:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:43:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:43:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:43:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:43:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:43:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:43:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:43:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:43:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:43:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:43:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:43:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:43:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:43:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:43:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:43:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:43:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:43:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:43:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:43:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:43:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:43:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:43:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:43:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:43:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:43:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:43:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:43:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:43:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:43:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:43:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:43:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:43:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:43:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:43:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:43:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:43:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:43:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:43:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:43:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:43:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:43:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:43:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:43:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:43:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:43:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:43:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:43:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:43:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:43:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:43:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:43:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:43:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:43:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:43:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:43:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:43:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:43:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:43:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:43:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:43:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:43:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:43:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:43:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:43:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:43:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:43:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:43:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:43:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:43:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:43:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:43:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:43:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:43:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:43:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:43:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:43:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:43:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:43:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:43:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:43:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:43:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:43:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:43:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:43:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:43:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:43:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:43:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:43:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:43:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:43:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:43:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:43:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:43:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:43:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:43:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:43:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:43:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:43:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:43:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:44:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:44:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:44:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:44:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:44:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:44:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:44:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:44:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:44:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:44:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:44:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:44:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:44:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:44:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:44:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:44:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:44:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:44:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:44:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:44:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:44:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:44:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:44:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:44:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:44:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:44:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:44:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:44:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:44:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:44:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:44:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:44:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:44:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:44:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:44:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:44:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:44:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:44:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:44:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:44:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:44:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:44:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:44:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:44:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:44:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:44:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:44:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:44:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:44:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:44:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:44:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:44:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:44:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:44:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:44:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:44:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:44:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:44:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:44:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:44:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:44:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:44:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:44:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:44:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:44:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:44:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:44:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:44:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:44:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:44:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:44:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:44:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:44:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:44:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:44:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:44:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:44:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:44:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:44:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:44:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:44:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:44:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:44:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:44:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:44:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:44:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:44:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:44:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:44:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:44:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:44:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:44:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:44:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:44:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:44:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:44:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:44:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:44:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:44:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:44:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:44:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:44:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:44:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:44:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:44:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:44:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:44:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:44:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:44:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:44:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:44:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:44:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:44:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:44:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:44:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:44:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:44:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:44:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:44:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:44:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:44:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:44:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:44:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:44:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:44:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:44:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:44:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:44:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:44:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:44:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:44:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:44:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:44:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:44:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:44:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:44:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:44:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:44:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:44:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:44:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:44:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:44:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:44:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:44:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:44:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:44:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:44:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:44:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:44:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:44:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:44:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:44:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:44:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:44:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:44:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:44:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:44:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:44:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:44:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:44:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:44:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:44:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:44:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:44:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:44:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:44:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:44:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:44:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:44:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:44:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:44:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:44:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:44:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:44:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:44:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:44:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:44:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:44:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:44:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:44:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:44:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:44:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:44:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:44:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:44:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:44:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:44:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:44:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:44:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:44:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:44:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:44:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:44:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:44:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:44:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:44:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:44:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:44:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:44:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:44:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:44:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:44:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:44:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:44:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:44:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:44:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:44:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:44:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:44:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:44:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:44:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:44:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:44:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:44:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:44:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:44:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:45:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:45:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:45:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:45:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:45:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:45:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:45:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:45:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:45:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:45:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:45:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:45:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:45:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:45:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:45:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:45:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:45:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:45:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:45:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:45:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:45:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:45:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:45:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:45:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:45:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:45:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:45:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:45:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:45:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:45:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:45:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:45:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:45:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:45:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:45:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:45:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:45:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:45:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:45:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:45:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:45:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:45:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:45:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:45:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:45:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:45:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:45:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:45:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:45:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:45:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:45:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:45:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:45:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:45:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:45:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:45:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:45:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:45:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:45:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:45:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:45:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:45:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:45:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:45:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:45:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:45:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:45:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:45:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:45:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:45:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:45:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:45:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:45:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:45:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:45:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:45:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:45:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:45:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:45:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:45:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:45:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:45:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:45:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:45:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:45:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:45:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:45:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:45:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:45:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:45:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:47:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:47:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:47:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:47:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:47:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:47:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:47:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:47:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:47:35 rdma06 rsyslogd[1041]: imjournal: 5297 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 17:47:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:47:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:47:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:47:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:47:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:47:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:47:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:47:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:47:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:47:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:47:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:47:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:47:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:47:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:47:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:47:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:47:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:47:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:47:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:47:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:47:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:47:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:47:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:47:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:47:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:47:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:47:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:47:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:47:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:47:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:47:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:47:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:47:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:47:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:47:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:47:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:47:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:47:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:47:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:47:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:47:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:47:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:47:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:47:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:47:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:47:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:47:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:47:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:47:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:47:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:47:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:47:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:47:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:47:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:47:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:47:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:47:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:47:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:47:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:47:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:48:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:48:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:48:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:48:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:48:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:48:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:48:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:48:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:48:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:48:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:48:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:48:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:48:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:48:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:48:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:48:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:48:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:48:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:48:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:48:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:48:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:48:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:48:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:48:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:48:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:48:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:48:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:48:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:48:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:48:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:48:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:48:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:48:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:48:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:48:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:48:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:48:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:48:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:48:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:48:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:48:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:48:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:48:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:48:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:48:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:48:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:48:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:48:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:48:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:48:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:48:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:48:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:48:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:48:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:48:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:48:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:48:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:48:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:48:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:48:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:48:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:48:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:48:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:48:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:48:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:48:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:48:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:48:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:48:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:48:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:48:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:48:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:48:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:48:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:48:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:48:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:48:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:48:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:48:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:48:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:48:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:48:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:48:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:48:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:48:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:48:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:48:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:48:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:48:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:48:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:48:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:48:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:48:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:48:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:48:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:48:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:48:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:48:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:48:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:48:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:48:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:48:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:48:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:48:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:48:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:48:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:48:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:48:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:48:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:48:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:48:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:48:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:48:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:48:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:48:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:48:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:48:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:48:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:48:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:48:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:48:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:48:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:48:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:48:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:48:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:48:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:48:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:48:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:48:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:48:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:48:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:48:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:48:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:48:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:48:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:48:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:48:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:48:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:48:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:48:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:48:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:48:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:48:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:48:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:48:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:48:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:48:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:48:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:48:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:48:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:48:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:48:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:48:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:48:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:48:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:48:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:48:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:48:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:48:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:48:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:48:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:48:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:48:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:48:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:48:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:48:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:48:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:48:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:48:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:48:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:48:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:48:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:48:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:48:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:48:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:48:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:48:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:48:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:48:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:48:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:48:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:48:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:48:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:48:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:48:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:48:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:48:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:48:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:48:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:48:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:48:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:48:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:48:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:48:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:48:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:48:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:48:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:48:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:48:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:48:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:48:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:48:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:48:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:48:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:48:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:48:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:48:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:48:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:48:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:48:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:48:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:48:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:48:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:48:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:48:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:48:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:49:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:49:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:49:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:49:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:49:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:49:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:49:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:49:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:49:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:49:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:49:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:49:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:49:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:49:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:49:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:49:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:49:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:49:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:49:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:49:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:49:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:49:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:49:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:49:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:49:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:49:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:49:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:49:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:49:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:49:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:49:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:49:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:49:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:49:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:49:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:49:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:49:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:49:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:49:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:49:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:49:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:49:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:49:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:49:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:49:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:49:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:49:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:49:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:49:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:49:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:49:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:49:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:49:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:49:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:49:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:49:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:49:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:49:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:49:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:49:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:49:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:49:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:49:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:49:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:49:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:49:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:49:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:49:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:49:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:49:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:49:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:49:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:49:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:49:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:49:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:49:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:49:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:49:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:49:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:49:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:49:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:49:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:49:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:49:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:49:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:49:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:49:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:49:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:49:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:49:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:49:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:49:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:49:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:49:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:49:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:49:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:49:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:49:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:49:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:49:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:49:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:49:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:49:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:49:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:49:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:49:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:49:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:49:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:49:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:49:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:49:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:49:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:49:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:49:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:49:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:49:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:49:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:49:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:49:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:49:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:49:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:49:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:49:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:49:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:49:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:49:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:49:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:49:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:49:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:49:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:49:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:49:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:49:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:49:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:49:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:49:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:49:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:49:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:49:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:49:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:49:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:49:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:49:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:49:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:49:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:49:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:49:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:49:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:49:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:49:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:49:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:49:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:49:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:49:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:49:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:49:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:49:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:49:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:49:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:49:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:49:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:49:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:49:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:49:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:49:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:49:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:49:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:49:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:49:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:49:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:49:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:49:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:49:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:49:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:49:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:49:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:49:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:49:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:49:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:49:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:49:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:49:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:49:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:49:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:49:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:49:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:49:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:49:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:49:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:49:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:49:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:49:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:49:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:49:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:49:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:49:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:49:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:49:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:49:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:49:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:49:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:49:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:49:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:49:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:49:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:49:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:50:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:50:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:50:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:50:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:50:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:50:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:50:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:50:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:50:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:50:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:50:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:50:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:50:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:50:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:50:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:50:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:50:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:50:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:50:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:50:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:50:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:50:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:50:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:50:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:50:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:50:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:50:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:50:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:50:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:50:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:50:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:50:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:50:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:50:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:50:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:50:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:50:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:50:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:50:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:50:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:50:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:50:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:50:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:50:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:50:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:50:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:50:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:50:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:50:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:50:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:50:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:50:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:50:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:50:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:50:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:50:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:50:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:50:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:50:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:50:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:50:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:50:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:50:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:50:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:50:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:50:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:50:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:50:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:50:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:50:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:50:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:50:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:50:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:50:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:50:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:50:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:50:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:50:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:50:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:50:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:50:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:50:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:50:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:50:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:50:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:50:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:50:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:50:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:50:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:50:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:50:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:50:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:50:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:50:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:50:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:50:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:50:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:50:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:50:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:50:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:50:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:50:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:50:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:50:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:50:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:50:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:50:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:50:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:50:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:50:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:50:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:50:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:50:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:50:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:50:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:50:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:50:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:50:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:50:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:50:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:50:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:50:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:50:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:50:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:50:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:50:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:50:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:50:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:50:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:50:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:50:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:50:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:50:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:50:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:50:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:50:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:50:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:50:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:50:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:50:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:50:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:50:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:50:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:50:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:50:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:50:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:50:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:50:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:50:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:50:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:50:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:50:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:50:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:50:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:50:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:50:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:50:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:50:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:50:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:50:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:50:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:50:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:50:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:50:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:50:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:50:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:50:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:50:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:50:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:50:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:50:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:50:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:50:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:50:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:50:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:50:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:50:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:50:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:50:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:50:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:50:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:50:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:50:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:50:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:50:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:50:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:50:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:50:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:50:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:50:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:50:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:50:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:50:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:50:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:50:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:50:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:50:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:50:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:51:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:51:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:51:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:51:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:51:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:51:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:51:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:51:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:51:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:51:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:51:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:51:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:51:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:51:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:51:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:51:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:51:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:51:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:51:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:51:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:51:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:51:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:51:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:51:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:51:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:51:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:51:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:51:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:51:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:51:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:51:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:51:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:51:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:51:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:51:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:51:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:51:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:51:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:51:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:51:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:51:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:51:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:51:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:51:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:51:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:51:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:51:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:51:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:51:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:51:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:51:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:51:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:51:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:51:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:51:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:51:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:51:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:51:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:51:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:51:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:51:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:51:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:51:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:51:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:51:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:51:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:51:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:51:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:51:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:51:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:51:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:51:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:51:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:51:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:51:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:51:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:51:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:51:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:51:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:51:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:51:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:51:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:51:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:51:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:51:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:51:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:51:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:51:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:51:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:51:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:51:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:51:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:51:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:51:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:51:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:51:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:51:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:51:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:51:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:51:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:51:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:51:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:51:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:51:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:51:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:51:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:51:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:51:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:51:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:51:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:51:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:51:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:51:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:51:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:51:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:51:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:51:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:51:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:51:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:51:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:51:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:51:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:51:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:51:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:51:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:51:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:51:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:51:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:51:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:51:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:51:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:51:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:51:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:51:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:51:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:51:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:51:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:51:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:51:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:51:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:51:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:51:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:51:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:51:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:51:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:51:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:51:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:51:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:51:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:51:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:51:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:51:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:51:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:51:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:51:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:51:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:51:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:51:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:51:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:51:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:51:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:51:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:51:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:51:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:51:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:51:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:51:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:51:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:51:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:51:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:51:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:51:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:51:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:51:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:51:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:51:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:51:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:51:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:51:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:51:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:51:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:51:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:51:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:51:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:51:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:51:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:52:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:52:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:52:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:52:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:52:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:52:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:52:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:52:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:52:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:52:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:52:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:52:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:52:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:52:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:52:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:52:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:52:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:52:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:52:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:52:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:52:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:52:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:52:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:52:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:52:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:52:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:52:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:52:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:52:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:52:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:52:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:52:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:52:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:52:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:52:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:52:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:52:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:52:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:52:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:52:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:52:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:52:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:52:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:52:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:52:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:52:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:52:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:52:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:52:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:52:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:52:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:52:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:52:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:52:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:52:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:52:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:52:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:52:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:52:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:52:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:52:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:52:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:52:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:52:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:52:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:52:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:52:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:52:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:52:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:52:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:52:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:52:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:52:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:52:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:52:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:52:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:52:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:52:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:52:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:52:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:52:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:52:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:52:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:52:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:52:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:52:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:52:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:52:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:52:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:52:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:52:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:52:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:52:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:52:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:52:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:52:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:52:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:52:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:52:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:52:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:52:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:52:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:52:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:52:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:52:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:52:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:52:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:52:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:52:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:52:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:52:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:52:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:52:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:52:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:52:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:52:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:52:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:52:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:52:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:52:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:52:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:52:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:52:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:52:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:52:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:52:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:52:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:52:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:52:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:52:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:52:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:52:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:52:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:52:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:52:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:52:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:52:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:52:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:52:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:52:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:52:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:52:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:52:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:52:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:52:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:52:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:52:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:52:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:52:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:52:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:52:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:52:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:52:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:52:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:52:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:52:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:52:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:52:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:52:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:52:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:52:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:52:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:52:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:52:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:52:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:52:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:52:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:52:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:52:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:52:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:52:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:52:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:52:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:52:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:52:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:52:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:52:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:52:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:52:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:52:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:52:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:52:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:52:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:52:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:52:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:52:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:52:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:52:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:52:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:52:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:52:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:52:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:52:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:52:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:52:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:52:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:52:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:52:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:52:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:52:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:52:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:52:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:52:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:52:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:52:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:52:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:52:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:52:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:52:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:52:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:52:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:52:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:52:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:52:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:52:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:52:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:53:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:53:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:53:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:53:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:53:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:53:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:53:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:53:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:53:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:53:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:53:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:53:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:53:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:53:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:53:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:53:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:53:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:53:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:53:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:53:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:53:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:53:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:53:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:53:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:53:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:53:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:53:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:53:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:53:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:53:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:53:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:53:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:53:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:53:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:53:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:53:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:53:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:53:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:53:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:53:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:53:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:53:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:53:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:53:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:53:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:53:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:53:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:53:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:53:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:53:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:53:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:53:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:53:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:53:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:53:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:53:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:53:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:53:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:53:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:53:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:53:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:53:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:53:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:53:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:53:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:53:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:53:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:53:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:53:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:53:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:53:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:53:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:53:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:53:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:53:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:53:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:53:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:53:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:53:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:53:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:53:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:53:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:53:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:53:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:53:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:53:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:53:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:53:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:53:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:53:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:53:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:53:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:53:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:53:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:53:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:53:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:53:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:53:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:53:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:53:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:53:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:53:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:53:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:53:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:53:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:53:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:53:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:53:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:53:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:53:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:53:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:53:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:53:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:53:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:53:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:53:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:53:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:53:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:53:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:53:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:53:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:53:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:53:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:53:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:53:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:53:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:53:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:53:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:53:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:53:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:53:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:53:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:53:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:53:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:53:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:53:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:53:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:53:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:53:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:53:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:53:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:53:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:53:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:53:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:53:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:53:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:53:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:53:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:53:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:53:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:53:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:53:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:53:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:53:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:53:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:53:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:53:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:53:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:53:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:53:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:53:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:53:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:53:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:53:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:53:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:53:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:53:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:53:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:53:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:53:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:53:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:53:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:53:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:53:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:53:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:53:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:53:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:53:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:53:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:53:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:53:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:53:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:53:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:53:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:53:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:53:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:53:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:53:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:53:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:53:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:53:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:53:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:53:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:53:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:53:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:53:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:53:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:53:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:53:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:53:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:53:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:53:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:53:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:53:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:53:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:53:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:53:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:53:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:53:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:53:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:53:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:53:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:53:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:53:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:53:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:53:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:54:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:54:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:54:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:54:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:54:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:54:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:54:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:54:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:54:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:54:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:54:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:54:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:54:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:54:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:54:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:54:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:54:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:54:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:54:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:54:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:54:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:54:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:54:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:54:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:54:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:54:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:54:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:54:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:54:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:54:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:54:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:54:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:54:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:54:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:54:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:54:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:54:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:54:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:54:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:54:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:54:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:54:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:54:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:54:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:54:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:54:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:54:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:54:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:54:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:54:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:54:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:54:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:54:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:54:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:54:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:54:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:54:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:54:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:54:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:54:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:54:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:54:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:54:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:54:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:54:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:54:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:54:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:54:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:54:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:54:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:54:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:54:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:54:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:54:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:54:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:54:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:54:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:54:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:54:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:54:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:54:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:54:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:54:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:54:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:54:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:54:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:54:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:54:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:54:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:54:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:54:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:54:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:54:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:54:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:54:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:54:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:54:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:54:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:54:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:54:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:54:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:54:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:54:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:54:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:54:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:54:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:54:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:54:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:54:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:54:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:54:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:54:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:54:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:54:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:54:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:54:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:54:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:54:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:54:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:54:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:54:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:54:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:54:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:54:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:54:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:54:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:54:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:54:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:54:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:54:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:54:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:54:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:54:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:54:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:54:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:54:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:54:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:54:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:54:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:54:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:54:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:54:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:54:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:54:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:54:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:54:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:54:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:54:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:54:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:54:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:54:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:54:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:54:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:54:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:54:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:54:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:54:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:54:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:54:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:54:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:54:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:54:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:54:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:54:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:54:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:54:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:54:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:54:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:54:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:54:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:54:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:54:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:54:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:54:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:54:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:54:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:54:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:54:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:54:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:54:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:54:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:54:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:54:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:54:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:54:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:54:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:54:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:54:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:54:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:54:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:54:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:54:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:54:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:54:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:54:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:54:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:54:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:54:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:54:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:54:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:54:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:54:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:54:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:54:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:54:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:54:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:54:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:54:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:54:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:54:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:54:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:54:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:54:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:54:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:54:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:54:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:55:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:55:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:55:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:55:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:55:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:55:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:55:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:55:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:55:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:55:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:55:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:55:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:55:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:55:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:55:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:55:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:55:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:55:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:55:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:55:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:55:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:55:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:55:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:55:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:55:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:55:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:55:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:55:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:55:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:55:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:55:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:55:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:55:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:55:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:55:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:55:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:55:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:55:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:55:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:55:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:55:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:55:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:55:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:55:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:55:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:55:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:55:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:55:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:55:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:55:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:55:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:55:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:55:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:55:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:55:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:55:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:55:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:55:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:55:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:55:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:55:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:55:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:55:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:55:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:55:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:55:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:55:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:55:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:55:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:55:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:55:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:55:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:55:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:55:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:55:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:55:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:55:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:55:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:55:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:55:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:55:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:55:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:55:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:55:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:55:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:55:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:55:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:55:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:55:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:55:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:55:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:55:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:55:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:55:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:55:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:55:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:55:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:55:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:55:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:55:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:55:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:55:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:55:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:55:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:55:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:55:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:55:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:55:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:55:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:55:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:55:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:55:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:55:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:55:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:55:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:55:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:55:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:55:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:55:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:55:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:55:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:55:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:55:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:55:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:55:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:55:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:57:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:57:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:57:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:57:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:57:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:57:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:57:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:57:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:57:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:57:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:57:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:57:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:57:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:57:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:57:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:57:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:57:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:57:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:57:40 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 17:57:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:57:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:57:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:57:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:57:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:57:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:57:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:57:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:57:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:57:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:57:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:57:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:57:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:57:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:57:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:57:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:57:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:57:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:57:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:57:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:57:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:57:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:57:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:57:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:57:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:57:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:57:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:57:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:57:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:57:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:57:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:57:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:57:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:57:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:57:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:57:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:57:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:57:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:57:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:57:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:57:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:57:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:57:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:57:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:57:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:57:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:57:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:57:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:57:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:57:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:57:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:57:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:57:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:57:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:58:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:58:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:58:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:58:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:58:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:58:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:58:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:58:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:58:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:58:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:58:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:58:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:58:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:58:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:58:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:58:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:58:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:58:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:58:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:58:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:58:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:58:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:58:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:58:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:58:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:58:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:58:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:58:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:58:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:58:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:58:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:58:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:58:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:58:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:58:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:58:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:58:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:58:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:58:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:58:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:58:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:58:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:58:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:58:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:58:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:58:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:58:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:58:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:58:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:58:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:58:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:58:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:58:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:58:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:58:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:58:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:58:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:58:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:58:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:58:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:58:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:58:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:58:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:58:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:58:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:58:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:58:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:58:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:58:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:58:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:58:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:58:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:58:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:58:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:58:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:58:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:58:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:58:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:58:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:58:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:58:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:58:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:58:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:58:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:58:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:58:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:58:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:58:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:58:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:58:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:58:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:58:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:58:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:58:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:58:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:58:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:58:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:58:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:58:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:58:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:58:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:58:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:58:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:58:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:58:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:58:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:58:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:58:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:58:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:58:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:58:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:58:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:58:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:58:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:58:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:58:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:58:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:58:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:58:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:58:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:58:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:58:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:58:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:58:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:58:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:58:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:58:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:58:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:58:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:58:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:58:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:58:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:58:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:58:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:58:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:58:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:58:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:58:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:58:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:58:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:58:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:58:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:58:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:58:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:58:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:58:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:58:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:58:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:58:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:58:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:58:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:58:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:58:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:58:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:58:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:58:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:58:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:58:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:58:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:58:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:58:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:58:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:58:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:58:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:58:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:58:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:58:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:58:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:58:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:58:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:58:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:58:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:58:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:58:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:58:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:58:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:58:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:58:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:58:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:58:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:58:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:58:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:58:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:58:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:58:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:58:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:58:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:58:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:58:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:58:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:58:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:58:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:58:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:58:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:58:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:58:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:58:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:58:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:58:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:58:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:58:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:58:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:58:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:58:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:58:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:58:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:58:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:58:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:58:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:58:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:58:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:58:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:58:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:58:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:58:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:58:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:59:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:59:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:59:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:59:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:59:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:59:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:59:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:59:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:59:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:59:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:59:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:59:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:59:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:59:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:59:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:59:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:59:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:59:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:59:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:59:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:59:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:59:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:59:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:59:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:59:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:59:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:59:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:59:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:59:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:59:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:59:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:59:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:59:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:59:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:59:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:59:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:59:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:59:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:59:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:59:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:59:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:59:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:59:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:59:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:59:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:59:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:59:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:59:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:59:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:59:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:59:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:59:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:59:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:59:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:59:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:59:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:59:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:59:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:59:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:59:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:59:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:59:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:59:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:59:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:59:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:59:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:59:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:59:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:59:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:59:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:59:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:59:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:59:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:59:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:59:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:59:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:59:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:59:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:59:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:59:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:59:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:59:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:59:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:59:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:59:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:59:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:59:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:59:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:59:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:59:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:59:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:59:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:59:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:59:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:59:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:59:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:59:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:59:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:59:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:59:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:59:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:59:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:59:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:59:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:59:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:59:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:59:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:59:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:59:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:59:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:59:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:59:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:59:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:59:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:59:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:59:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:59:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:59:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:59:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:59:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:59:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:59:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:59:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:59:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:59:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:59:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:59:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:59:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:59:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:59:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:59:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:59:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:59:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:59:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:59:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:59:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:59:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:59:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:59:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:59:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:59:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:59:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:59:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:59:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:59:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:59:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:59:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:59:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:59:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:59:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:59:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:59:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:59:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:59:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:59:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:59:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:59:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:59:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:59:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:59:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:59:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:59:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:59:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:59:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:59:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:59:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:59:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:59:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:59:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:59:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:59:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:59:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:59:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:59:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:59:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:59:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:59:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:59:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:59:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:59:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:59:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:59:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:59:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:59:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:59:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:59:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:59:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:59:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:59:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:59:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:59:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:59:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:59:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:59:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:59:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:59:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:59:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:59:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 17:59:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 17:59:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:59:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:59:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:59:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 17:59:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 17:59:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 17:59:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 17:59:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 17:59:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 17:59:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 17:59:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 17:59:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 17:59:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 17:59:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 17:59:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 17:59:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 17:59:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:00:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:00:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:00:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:00:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:00:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:00:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:00:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:00:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:00:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:00:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:00:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:00:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:00:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:00:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:00:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:00:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:00:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:00:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:00:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:00:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:00:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:00:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:00:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:00:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:00:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:00:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:00:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:00:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:00:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:00:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:00:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:00:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:00:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:00:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:00:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:00:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:00:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:00:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:00:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:00:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:00:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:00:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:00:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:00:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:00:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:00:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:00:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:00:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:00:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:00:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:00:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:00:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:00:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:00:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:00:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:00:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:00:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:00:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:00:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:00:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:00:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:00:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:00:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:00:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:00:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:00:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:00:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:00:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:00:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:00:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:00:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:00:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:00:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:00:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:00:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:00:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:00:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:00:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:00:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:00:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:00:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:00:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:00:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:00:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:00:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:00:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:00:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:00:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:00:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:00:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:00:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:00:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:00:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:00:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:00:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:00:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:00:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:00:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:00:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:00:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:00:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:00:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:00:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:00:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:00:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:00:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:00:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:00:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:00:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:00:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:00:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:00:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:00:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:00:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:00:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:00:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:00:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:00:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:00:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:00:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:00:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:00:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:00:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:00:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:00:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:00:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:00:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:00:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:00:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:00:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:00:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:00:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:00:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:00:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:00:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:00:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:00:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:00:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:00:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:00:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:00:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:00:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:00:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:00:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:00:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:00:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:00:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:00:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:00:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:00:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:00:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:00:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:00:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:00:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:00:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:00:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:00:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:00:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:00:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:00:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:00:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:00:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:00:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:00:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:00:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:00:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:00:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:00:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:00:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:00:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:00:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:00:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:00:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:00:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:00:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:00:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:00:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:00:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:00:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:00:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:00:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:00:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:00:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:00:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:00:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:00:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:00:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:00:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:00:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:00:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:00:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:00:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:00:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:00:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:00:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:00:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:00:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:00:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:00:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:00:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:00:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:00:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:00:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:00:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:00:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:00:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:00:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:00:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:00:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:00:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:00:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:00:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:00:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:00:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:00:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:00:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:01:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:01:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:01:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:01:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:01:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:01:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:01:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:01:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:01:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:01:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:01:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:01:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:01:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:01:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:01:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:01:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:01:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:01:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:01:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:01:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:01:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:01:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:01:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:01:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:01:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:01:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:01:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:01:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:01:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:01:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:01:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:01:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:01:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:01:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:01:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:01:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:01:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:01:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:01:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:01:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:01:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:01:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:01:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:01:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:01:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:01:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:01:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:01:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:01:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:01:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:01:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:01:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:01:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:01:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:01:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:01:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:01:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:01:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:01:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:01:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:01:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:01:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:01:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:01:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:01:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:01:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:01:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:01:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:01:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:01:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:01:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:01:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:01:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:01:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:01:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:01:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:01:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:01:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:01:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:01:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:01:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:01:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:01:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:01:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:01:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:01:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:01:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:01:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:01:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:01:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:01:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:01:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:01:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:01:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:01:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:01:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:01:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:01:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:01:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:01:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:01:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:01:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:01:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:01:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:01:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:01:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:01:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:01:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:01:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:01:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:01:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:01:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:01:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:01:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:01:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:01:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:01:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:01:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:01:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:01:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:01:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:01:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:01:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:01:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:01:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:01:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:01:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:01:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:01:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:01:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:01:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:01:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:01:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:01:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:01:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:01:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:01:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:01:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:01:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:01:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:01:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:01:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:01:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:01:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:01:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:01:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:01:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:01:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:01:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:01:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:01:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:01:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:01:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:01:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:01:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:01:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:02:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:02:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:02:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:02:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:02:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:02:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:02:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:02:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:02:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:02:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:02:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:02:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:02:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:02:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:02:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:02:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:02:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:02:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:02:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:02:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:02:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:02:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:02:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:02:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:02:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:02:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:02:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:02:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:02:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:02:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:02:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:02:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:02:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:02:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:02:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:02:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:02:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:02:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:02:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:02:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:02:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:02:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:02:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:02:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:02:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:02:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:02:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:02:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:02:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:02:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:02:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:02:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:02:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:02:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:02:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:02:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:02:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:02:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:02:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:02:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:02:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:02:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:02:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:02:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:02:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:02:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:02:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:02:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:02:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:02:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:02:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:02:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:02:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:02:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:02:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:02:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:02:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:02:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:02:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:02:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:02:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:02:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:02:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:02:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:02:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:02:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:02:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:02:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:02:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:02:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:02:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:02:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:02:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:02:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:02:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:02:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:02:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:02:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:02:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:02:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:02:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:02:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:02:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:02:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:02:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:02:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:02:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:02:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:02:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:02:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:02:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:02:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:02:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:02:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:02:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:02:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:02:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:02:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:02:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:02:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:02:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:02:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:02:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:02:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:02:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:02:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:02:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:02:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:02:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:02:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:02:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:02:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:02:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:02:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:02:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:02:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:02:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:02:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:02:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:02:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:02:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:02:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:02:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:02:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:02:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:02:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:02:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:02:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:02:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:02:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:02:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:02:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:02:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:02:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:02:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:02:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:02:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:02:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:02:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:02:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:02:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:02:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:02:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:02:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:02:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:02:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:02:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:02:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:02:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:02:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:02:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:02:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:02:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:02:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:02:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:02:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:02:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:02:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:02:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:02:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:02:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:02:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:02:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:02:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:02:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:02:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:02:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:02:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:02:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:02:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:02:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:02:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:02:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:02:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:02:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:02:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:02:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:02:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:02:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:02:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:02:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:02:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:02:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:02:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:02:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:02:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:02:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:02:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:02:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:02:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:02:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:02:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:02:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:02:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:02:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:02:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:03:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:03:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:03:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:03:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:03:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:03:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:03:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:03:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:03:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:03:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:03:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:03:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:03:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:03:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:03:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:03:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:03:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:03:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:03:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:03:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:03:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:03:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:03:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:03:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:03:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:03:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:03:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:03:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:03:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:03:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:03:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:03:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:03:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:03:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:03:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:03:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:03:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:03:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:03:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:03:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:03:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:03:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:03:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:03:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:03:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:03:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:03:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:03:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:03:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:03:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:03:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:03:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:03:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:03:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:03:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:03:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:03:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:03:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:03:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:03:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:03:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:03:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:03:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:03:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:03:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:03:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:03:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:03:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:03:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:03:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:03:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:03:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:03:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:03:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:03:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:03:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:03:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:03:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:03:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:03:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:03:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:03:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:03:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:03:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:03:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:03:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:03:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:03:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:03:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:03:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:03:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:03:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:03:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:03:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:03:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:03:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:03:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:03:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:03:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:03:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:03:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:03:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:03:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:03:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:03:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:03:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:03:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:03:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:03:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:03:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:03:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:03:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:03:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:03:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:03:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:03:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:03:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:03:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:03:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:03:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:03:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:03:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:03:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:03:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:03:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:03:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:03:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:03:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:03:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:03:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:03:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:03:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:03:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:03:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:03:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:03:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:03:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:03:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:03:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:03:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:03:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:03:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:03:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:03:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:03:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:03:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:03:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:03:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:03:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:03:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:03:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:03:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:03:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:03:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:03:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:03:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:03:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:03:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:03:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:03:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:03:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:03:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:03:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:03:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:03:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:03:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:03:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:03:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:03:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:03:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:03:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:03:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:03:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:03:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:03:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:03:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:03:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:03:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:03:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:03:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:03:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:03:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:03:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:03:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:03:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:03:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:03:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:03:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:03:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:03:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:03:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:03:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:03:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:03:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:03:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:03:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:03:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:03:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:03:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:03:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:03:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:03:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:03:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:03:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:03:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:03:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:03:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:03:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:03:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:03:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:03:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:03:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:03:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:03:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:03:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:03:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:04:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:04:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:04:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:04:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:04:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:04:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:04:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:04:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:04:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:04:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:04:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:04:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:04:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:04:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:04:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:04:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:04:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:04:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:04:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:04:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:04:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:04:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:04:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:04:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:04:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:04:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:04:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:04:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:04:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:04:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:04:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:04:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:04:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:04:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:04:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:04:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:04:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:04:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:04:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:04:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:04:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:04:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:04:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:04:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:04:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:04:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:04:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:04:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:04:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:04:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:04:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:04:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:04:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:04:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:04:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:04:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:04:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:04:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:04:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:04:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:04:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:04:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:04:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:04:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:04:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:04:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:04:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:04:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:04:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:04:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:04:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:04:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:04:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:04:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:04:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:04:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:04:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:04:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:04:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:04:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:04:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:04:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:04:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:04:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:04:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:04:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:04:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:04:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:04:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:04:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:04:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:04:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:04:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:04:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:04:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:04:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:04:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:04:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:04:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:04:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:04:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:04:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:04:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:04:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:04:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:04:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:04:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:04:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:04:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:04:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:04:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:04:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:04:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:04:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:04:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:04:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:04:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:04:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:04:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:04:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:04:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:04:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:04:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:04:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:04:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:04:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:04:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:04:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:04:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:04:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:04:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:04:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:04:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:04:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:04:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:04:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:04:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:04:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:04:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:04:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:04:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:04:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:04:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:04:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:04:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:04:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:04:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:04:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:04:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:04:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:04:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:04:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:04:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:04:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:04:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:04:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:04:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:04:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:04:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:04:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:04:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:04:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:04:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:04:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:04:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:04:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:04:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:04:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:04:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:04:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:04:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:04:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:04:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:04:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:04:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:04:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:04:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:04:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:04:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:04:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:04:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:04:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:04:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:04:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:04:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:04:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:04:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:04:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:04:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:04:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:04:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:04:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:04:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:04:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:04:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:04:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:04:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:04:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:04:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:04:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:04:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:04:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:04:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:04:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:04:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:04:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:04:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:04:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:04:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:04:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:04:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:04:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:04:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:04:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:04:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:04:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:05:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:05:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:05:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:05:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:05:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:05:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:05:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:05:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:05:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:05:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:05:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:05:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:05:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:05:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:05:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:05:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:05:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:05:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:05:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:05:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:05:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:05:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:05:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:05:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:05:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:05:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:05:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:05:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:05:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:05:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:05:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:05:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:05:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:05:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:05:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:05:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:05:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:05:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:05:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:05:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:05:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:05:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:05:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:05:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:05:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:05:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:05:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:05:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:05:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:05:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:05:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:05:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:05:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:05:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:05:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:05:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:05:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:05:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:05:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:05:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:05:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:05:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:05:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:05:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:05:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:05:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:05:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:05:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:05:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:05:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:05:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:05:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:05:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:05:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:05:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:05:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:05:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:05:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:05:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:05:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:05:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:05:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:05:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:05:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:05:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:05:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:05:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:05:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:05:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:05:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:05:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:05:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:05:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:05:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:05:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:05:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:05:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:05:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:05:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:05:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:05:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:05:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:05:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:05:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:05:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:05:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:05:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:05:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:05:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:05:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:05:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:05:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:05:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:05:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:05:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:05:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:05:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:05:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:05:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:05:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:05:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:05:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:05:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:05:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:05:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:05:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:05:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:05:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:05:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:05:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:05:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:05:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:05:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:05:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:05:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:05:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:05:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:05:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:05:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:05:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:05:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:05:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:05:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:05:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:07:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:07:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:07:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:07:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:07:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:07:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:07:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:07:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:07:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:07:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:07:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:07:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:07:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:07:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:07:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:07:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:07:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:07:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:07:41 rdma06 rsyslogd[1041]: imjournal: 5086 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 18:07:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:07:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:07:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:07:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:07:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:07:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:07:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:07:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:07:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:07:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:07:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:07:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:07:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:07:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:07:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:07:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:07:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:07:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:07:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:07:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:07:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:07:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:07:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:07:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:07:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:07:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:07:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:07:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:07:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:07:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:07:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:07:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:07:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:07:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:07:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:07:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:07:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:07:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:07:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:07:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:07:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:07:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:07:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:07:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:07:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:07:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:07:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:07:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:07:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:07:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:07:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:07:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:07:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:07:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:08:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:08:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:08:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:08:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:08:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:08:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:08:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:08:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:08:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:08:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:08:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:08:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:08:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:08:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:08:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:08:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:08:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:08:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:08:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:08:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:08:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:08:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:08:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:08:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:08:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:08:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:08:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:08:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:08:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:08:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:08:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:08:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:08:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:08:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:08:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:08:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:08:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:08:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:08:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:08:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:08:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:08:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:08:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:08:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:08:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:08:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:08:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:08:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:08:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:08:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:08:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:08:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:08:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:08:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:08:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:08:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:08:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:08:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:08:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:08:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:08:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:08:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:08:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:08:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:08:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:08:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:08:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:08:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:08:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:08:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:08:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:08:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:08:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:08:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:08:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:08:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:08:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:08:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:08:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:08:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:08:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:08:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:08:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:08:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:08:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:08:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:08:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:08:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:08:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:08:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:08:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:08:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:08:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:08:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:08:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:08:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:08:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:08:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:08:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:08:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:08:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:08:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:08:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:08:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:08:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:08:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:08:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:08:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:08:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:08:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:08:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:08:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:08:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:08:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:08:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:08:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:08:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:08:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:08:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:08:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:08:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:08:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:08:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:08:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:08:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:08:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:08:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:08:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:08:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:08:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:08:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:08:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:08:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:08:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:08:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:08:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:08:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:08:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:08:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:08:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:08:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:08:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:08:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:08:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:08:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:08:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:08:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:08:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:08:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:08:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:08:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:08:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:08:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:08:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:08:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:08:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:08:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:08:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:08:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:08:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:08:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:08:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:08:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:08:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:08:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:08:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:08:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:08:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:08:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:08:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:08:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:08:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:08:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:08:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:08:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:08:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:08:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:08:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:08:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:08:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:08:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:08:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:08:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:08:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:08:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:08:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:08:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:08:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:08:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:08:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:08:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:08:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:08:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:08:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:08:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:08:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:08:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:08:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:08:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:08:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:08:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:08:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:08:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:08:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:08:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:08:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:08:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:08:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:08:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:08:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:08:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:08:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:08:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:08:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:08:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:08:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:09:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:09:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:09:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:09:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:09:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:09:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:09:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:09:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:09:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:09:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:09:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:09:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:09:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:09:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:09:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:09:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:09:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:09:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:09:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:09:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:09:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:09:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:09:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:09:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:09:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:09:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:09:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:09:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:09:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:09:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:09:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:09:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:09:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:09:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:09:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:09:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:09:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:09:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:09:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:09:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:09:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:09:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:09:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:09:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:09:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:09:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:09:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:09:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:09:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:09:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:09:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:09:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:09:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:09:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:09:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:09:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:09:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:09:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:09:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:09:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:09:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:09:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:09:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:09:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:09:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:09:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:09:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:09:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:09:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:09:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:09:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:09:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:09:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:09:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:09:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:09:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:09:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:09:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:09:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:09:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:09:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:09:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:09:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:09:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:09:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:09:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:09:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:09:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:09:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:09:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:09:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:09:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:09:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:09:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:09:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:09:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:09:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:09:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:09:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:09:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:09:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:09:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:09:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:09:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:09:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:09:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:09:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:09:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:09:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:09:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:09:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:09:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:09:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:09:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:09:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:09:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:09:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:09:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:09:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:09:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:09:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:09:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:09:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:09:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:09:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:09:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:09:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:09:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:09:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:09:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:09:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:09:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:09:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:09:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:09:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:09:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:09:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:09:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:09:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:09:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:09:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:09:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:09:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:09:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:09:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:09:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:09:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:09:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:09:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:09:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:09:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:09:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:09:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:09:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:09:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:09:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:09:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:09:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:09:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:09:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:09:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:09:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:09:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:09:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:09:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:09:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:09:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:09:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:09:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:09:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:09:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:09:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:09:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:09:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:09:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:09:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:09:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:09:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:09:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:09:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:09:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:09:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:09:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:09:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:09:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:09:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:09:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:09:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:09:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:09:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:09:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:09:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:09:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:09:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:09:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:09:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:09:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:09:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:09:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:09:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:09:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:09:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:09:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:09:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:09:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:09:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:09:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:09:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:09:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:09:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:09:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:09:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:09:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:09:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:09:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:09:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:10:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:10:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:10:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:10:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:10:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:10:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:10:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:10:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:10:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:10:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:10:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:10:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:10:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:10:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:10:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:10:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:10:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:10:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:10:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:10:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:10:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:10:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:10:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:10:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:10:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:10:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:10:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:10:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:10:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:10:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:10:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:10:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:10:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:10:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:10:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:10:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:10:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:10:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:10:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:10:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:10:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:10:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:10:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:10:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:10:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:10:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:10:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:10:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:10:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:10:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:10:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:10:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:10:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:10:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:10:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:10:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:10:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:10:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:10:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:10:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:10:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:10:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:10:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:10:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:10:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:10:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:10:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:10:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:10:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:10:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:10:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:10:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:10:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:10:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:10:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:10:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:10:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:10:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:10:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:10:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:10:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:10:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:10:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:10:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:10:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:10:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:10:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:10:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:10:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:10:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:10:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:10:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:10:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:10:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:10:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:10:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:10:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:10:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:10:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:10:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:10:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:10:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:10:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:10:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:10:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:10:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:10:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:10:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:10:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:10:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:10:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:10:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:10:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:10:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:10:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:10:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:10:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:10:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:10:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:10:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:10:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:10:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:10:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:10:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:10:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:10:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:10:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:10:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:10:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:10:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:10:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:10:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:10:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:10:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:10:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:10:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:10:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:10:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:10:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:10:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:10:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:10:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:10:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:10:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:10:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:10:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:10:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:10:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:10:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:10:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:10:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:10:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:10:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:10:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:10:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:10:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:10:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:10:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:10:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:10:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:10:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:10:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:10:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:10:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:10:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:10:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:10:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:10:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:10:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:10:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:10:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:10:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:10:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:10:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:10:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:10:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:10:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:10:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:10:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:10:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:10:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:10:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:10:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:10:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:10:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:10:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:10:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:10:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:10:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:10:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:10:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:10:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:10:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:10:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:10:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:10:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:10:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:10:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:10:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:10:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:10:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:10:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:10:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:10:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:10:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:10:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:10:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:10:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:10:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:10:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:10:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:10:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:10:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:10:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:10:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:10:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:11:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:11:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:11:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:11:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:11:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:11:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:11:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:11:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:11:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:11:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:11:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:11:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:11:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:11:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:11:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:11:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:11:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:11:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:11:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:11:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:11:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:11:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:11:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:11:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:11:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:11:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:11:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:11:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:11:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:11:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:11:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:11:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:11:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:11:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:11:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:11:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:11:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:11:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:11:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:11:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:11:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:11:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:11:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:11:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:11:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:11:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:11:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:11:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:11:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:11:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:11:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:11:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:11:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:11:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:11:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:11:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:11:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:11:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:11:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:11:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:11:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:11:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:11:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:11:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:11:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:11:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:11:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:11:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:11:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:11:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:11:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:11:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:11:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:11:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:11:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:11:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:11:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:11:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:11:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:11:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:11:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:11:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:11:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:11:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:11:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:11:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:11:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:11:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:11:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:11:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:11:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:11:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:11:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:11:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:11:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:11:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:11:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:11:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:11:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:11:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:11:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:11:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:11:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:11:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:11:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:11:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:11:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:11:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:11:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:11:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:11:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:11:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:11:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:11:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:11:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:11:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:11:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:11:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:11:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:11:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:11:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:11:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:11:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:11:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:11:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:11:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:11:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:11:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:11:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:11:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:11:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:11:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:11:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:11:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:11:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:11:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:11:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:11:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:11:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:11:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:11:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:11:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:11:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:11:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:11:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:11:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:11:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:11:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:11:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:11:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:11:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:11:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:11:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:11:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:11:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:11:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:11:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:11:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:11:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:11:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:11:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:11:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:11:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:11:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:11:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:11:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:11:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:11:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:11:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:11:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:11:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:11:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:11:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:11:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:11:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:11:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:11:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:11:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:11:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:11:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:11:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:11:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:11:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:11:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:11:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:11:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:12:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:12:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:12:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:12:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:12:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:12:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:12:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:12:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:12:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:12:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:12:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:12:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:12:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:12:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:12:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:12:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:12:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:12:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:12:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:12:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:12:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:12:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:12:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:12:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:12:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:12:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:12:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:12:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:12:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:12:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:12:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:12:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:12:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:12:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:12:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:12:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:12:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:12:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:12:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:12:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:12:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:12:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:12:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:12:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:12:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:12:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:12:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:12:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:12:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:12:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:12:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:12:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:12:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:12:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:12:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:12:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:12:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:12:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:12:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:12:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:12:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:12:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:12:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:12:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:12:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:12:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:12:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:12:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:12:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:12:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:12:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:12:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:12:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:12:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:12:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:12:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:12:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:12:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:12:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:12:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:12:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:12:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:12:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:12:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:12:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:12:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:12:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:12:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:12:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:12:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:12:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:12:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:12:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:12:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:12:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:12:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:12:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:12:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:12:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:12:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:12:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:12:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:12:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:12:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:12:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:12:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:12:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:12:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:12:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:12:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:12:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:12:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:12:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:12:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:12:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:12:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:12:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:12:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:12:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:12:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:12:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:12:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:12:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:12:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:12:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:12:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:12:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:12:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:12:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:12:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:12:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:12:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:12:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:12:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:12:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:12:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:12:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:12:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:12:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:12:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:12:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:12:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:12:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:12:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:12:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:12:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:12:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:12:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:12:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:12:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:12:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:12:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:12:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:12:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:12:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:12:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:12:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:12:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:12:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:12:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:12:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:12:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:12:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:12:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:12:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:12:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:12:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:12:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:12:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:12:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:12:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:12:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:12:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:12:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:12:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:12:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:12:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:12:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:12:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:12:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:12:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:12:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:12:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:12:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:12:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:12:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:12:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:12:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:12:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:12:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:12:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:12:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:12:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:12:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:12:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:12:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:12:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:12:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:12:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:12:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:12:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:12:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:12:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:12:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:12:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:12:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:12:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:12:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:12:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:12:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:12:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:12:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:12:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:12:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:12:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:12:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:13:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:13:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:13:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:13:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:13:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:13:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:13:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:13:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:13:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:13:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:13:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:13:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:13:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:13:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:13:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:13:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:13:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:13:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:13:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:13:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:13:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:13:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:13:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:13:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:13:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:13:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:13:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:13:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:13:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:13:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:13:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:13:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:13:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:13:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:13:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:13:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:13:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:13:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:13:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:13:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:13:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:13:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:13:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:13:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:13:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:13:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:13:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:13:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:13:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:13:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:13:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:13:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:13:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:13:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:13:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:13:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:13:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:13:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:13:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:13:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:13:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:13:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:13:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:13:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:13:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:13:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:13:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:13:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:13:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:13:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:13:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:13:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:13:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:13:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:13:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:13:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:13:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:13:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:13:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:13:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:13:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:13:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:13:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:13:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:13:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:13:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:13:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:13:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:13:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:13:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:13:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:13:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:13:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:13:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:13:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:13:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:13:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:13:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:13:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:13:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:13:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:13:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:13:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:13:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:13:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:13:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:13:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:13:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:13:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:13:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:13:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:13:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:13:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:13:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:13:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:13:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:13:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:13:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:13:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:13:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:13:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:13:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:13:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:13:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:13:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:13:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:13:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:13:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:13:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:13:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:13:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:13:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:13:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:13:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:13:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:13:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:13:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:13:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:13:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:13:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:13:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:13:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:13:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:13:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:13:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:13:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:13:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:13:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:13:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:13:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:13:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:13:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:13:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:13:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:13:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:13:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:13:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:13:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:13:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:13:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:13:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:13:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:13:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:13:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:13:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:13:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:13:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:13:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:13:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:13:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:13:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:13:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:13:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:13:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:13:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:13:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:13:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:13:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:13:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:13:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:13:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:13:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:13:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:13:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:13:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:13:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:13:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:13:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:13:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:13:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:13:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:13:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:13:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:13:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:13:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:13:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:13:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:13:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:13:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:13:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:13:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:13:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:13:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:13:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:13:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:13:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:13:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:13:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:13:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:13:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:13:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:13:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:13:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:13:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:13:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:13:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:14:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:14:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:14:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:14:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:14:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:14:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:14:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:14:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:14:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:14:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:14:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:14:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:14:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:14:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:14:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:14:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:14:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:14:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:14:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:14:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:14:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:14:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:14:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:14:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:14:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:14:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:14:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:14:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:14:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:14:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:14:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:14:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:14:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:14:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:14:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:14:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:14:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:14:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:14:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:14:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:14:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:14:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:14:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:14:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:14:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:14:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:14:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:14:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:14:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:14:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:14:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:14:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:14:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:14:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:14:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:14:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:14:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:14:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:14:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:14:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:14:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:14:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:14:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:14:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:14:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:14:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:14:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:14:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:14:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:14:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:14:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:14:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:14:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:14:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:14:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:14:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:14:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:14:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:14:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:14:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:14:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:14:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:14:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:14:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:14:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:14:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:14:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:14:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:14:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:14:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:14:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:14:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:14:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:14:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:14:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:14:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:14:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:14:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:14:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:14:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:14:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:14:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:14:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:14:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:14:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:14:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:14:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:14:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:14:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:14:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:14:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:14:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:14:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:14:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:14:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:14:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:14:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:14:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:14:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:14:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:14:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:14:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:14:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:14:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:14:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:14:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:14:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:14:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:14:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:14:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:14:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:14:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:14:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:14:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:14:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:14:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:14:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:14:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:14:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:14:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:14:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:14:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:14:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:14:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:14:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:14:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:14:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:14:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:14:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:14:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:14:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:14:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:14:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:14:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:14:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:14:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:14:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:14:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:14:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:14:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:14:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:14:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:14:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:14:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:14:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:14:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:14:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:14:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:14:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:14:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:14:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:14:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:14:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:14:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:14:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:14:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:14:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:14:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:14:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:14:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:14:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:14:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:14:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:14:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:14:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:14:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:14:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:14:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:14:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:14:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:14:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:14:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:14:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:14:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:14:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:14:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:14:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:14:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:14:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:14:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:14:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:14:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:14:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:14:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:14:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:14:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:14:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:14:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:14:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:14:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:14:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:14:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:14:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:14:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:14:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:14:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:15:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:15:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:15:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:15:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:15:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:15:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:15:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:15:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:15:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:15:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:15:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:15:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:15:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:15:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:15:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:15:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:15:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:15:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:15:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:15:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:15:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:15:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:15:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:15:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:15:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:15:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:15:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:15:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:15:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:15:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:15:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:15:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:15:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:15:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:15:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:15:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:15:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:15:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:15:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:15:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:15:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:15:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:15:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:15:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:15:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:15:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:15:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:15:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:15:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:15:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:15:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:15:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:15:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:15:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:15:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:15:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:15:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:15:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:15:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:15:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:15:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:15:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:15:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:15:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:15:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:15:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:15:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:15:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:15:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:15:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:15:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:15:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:15:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:15:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:15:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:15:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:15:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:15:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:15:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:15:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:15:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:15:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:15:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:15:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:15:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:15:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:15:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:15:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:15:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:15:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:15:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:15:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:15:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:15:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:15:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:15:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:15:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:15:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:15:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:15:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:15:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:15:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:15:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:15:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:15:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:15:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:15:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:15:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:15:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:15:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:15:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:15:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:15:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:15:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:15:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:15:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:15:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:15:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:15:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:15:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:15:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:15:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:15:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:15:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:15:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:15:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:15:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:15:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:15:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:15:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:15:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:15:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:15:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:15:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:15:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:15:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:15:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:15:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:15:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:15:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:15:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:15:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:15:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:15:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:17:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:17:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:17:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:17:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:17:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:17:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:17:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:17:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:17:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:17:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:17:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:17:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:17:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:17:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:17:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:17:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:17:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:17:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:17:46 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 18:17:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:17:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:17:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:17:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:17:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:17:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:17:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:17:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:17:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:17:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:17:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:17:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:17:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:17:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:17:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:17:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:17:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:17:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:17:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:17:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:17:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:17:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:17:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:17:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:17:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:17:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:17:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:17:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:17:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:17:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:17:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:17:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:17:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:17:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:17:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:17:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:18:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:18:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:18:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:18:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:18:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:18:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:18:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:18:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:18:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:18:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:18:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:18:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:18:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:18:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:18:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:18:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:18:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:18:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:18:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:18:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:18:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:18:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:18:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:18:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:18:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:18:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:18:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:18:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:18:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:18:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:18:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:18:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:18:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:18:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:18:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:18:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:18:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:18:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:18:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:18:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:18:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:18:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:18:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:18:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:18:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:18:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:18:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:18:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:18:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:18:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:18:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:18:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:18:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:18:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:18:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:18:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:18:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:18:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:18:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:18:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:18:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:18:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:18:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:18:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:18:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:18:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:18:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:18:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:18:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:18:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:18:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:18:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:18:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:18:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:18:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:18:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:18:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:18:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:18:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:18:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:18:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:18:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:18:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:18:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:18:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:18:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:18:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:18:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:18:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:18:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:18:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:18:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:18:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:18:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:18:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:18:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:18:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:18:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:18:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:18:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:18:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:18:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:18:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:18:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:18:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:18:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:18:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:18:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:18:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:18:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:18:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:18:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:18:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:18:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:18:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:18:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:18:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:18:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:18:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:18:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:18:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:18:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:18:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:18:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:18:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:18:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:18:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:18:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:18:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:18:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:18:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:18:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:18:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:18:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:18:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:18:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:18:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:18:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:18:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:18:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:18:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:18:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:18:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:18:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:18:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:18:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:18:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:18:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:18:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:18:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:18:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:18:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:18:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:18:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:18:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:18:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:18:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:18:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:18:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:18:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:18:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:18:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:18:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:18:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:18:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:18:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:18:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:18:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:18:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:18:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:18:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:18:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:18:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:18:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:18:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:18:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:18:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:18:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:18:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:18:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:18:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:18:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:18:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:18:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:18:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:18:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:18:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:18:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:18:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:18:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:18:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:18:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:18:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:18:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:18:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:18:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:18:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:18:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:18:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:18:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:18:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:18:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:18:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:18:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:18:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:18:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:18:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:18:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:18:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:18:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:18:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:18:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:18:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:18:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:18:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:18:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:19:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:19:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:19:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:19:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:19:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:19:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:19:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:19:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:19:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:19:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:19:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:19:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:19:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:19:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:19:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:19:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:19:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:19:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:19:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:19:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:19:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:19:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:19:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:19:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:19:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:19:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:19:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:19:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:19:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:19:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:19:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:19:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:19:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:19:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:19:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:19:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:19:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:19:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:19:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:19:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:19:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:19:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:19:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:19:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:19:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:19:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:19:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:19:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:19:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:19:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:19:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:19:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:19:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:19:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:19:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:19:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:19:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:19:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:19:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:19:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:19:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:19:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:19:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:19:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:19:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:19:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:19:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:19:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:19:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:19:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:19:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:19:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:19:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:19:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:19:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:19:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:19:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:19:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:19:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:19:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:19:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:19:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:19:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:19:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:19:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:19:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:19:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:19:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:19:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:19:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:19:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:19:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:19:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:19:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:19:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:19:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:19:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:19:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:19:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:19:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:19:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:19:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:19:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:19:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:19:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:19:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:19:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:19:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:19:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:19:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:19:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:19:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:19:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:19:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:19:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:19:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:19:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:19:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:19:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:19:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:19:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:19:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:19:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:19:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:19:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:19:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:19:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:19:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:19:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:19:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:19:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:19:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:19:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:19:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:19:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:19:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:19:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:19:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:19:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:19:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:19:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:19:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:19:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:19:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:19:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:19:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:19:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:19:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:19:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:19:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:19:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:19:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:19:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:19:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:19:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:19:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:19:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:19:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:19:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:19:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:19:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:19:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:19:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:19:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:19:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:19:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:19:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:19:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:19:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:19:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:19:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:19:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:19:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:19:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:19:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:19:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:19:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:19:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:19:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:19:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:19:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:19:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:19:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:19:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:19:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:19:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:19:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:19:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:19:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:19:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:19:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:19:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:19:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:19:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:19:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:19:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:19:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:19:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:19:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:19:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:19:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:19:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:19:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:19:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:19:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:19:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:19:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:19:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:19:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:19:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:19:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:19:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:19:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:19:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:19:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:19:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:20:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:20:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:20:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:20:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:20:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:20:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:20:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:20:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:20:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:20:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:20:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:20:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:20:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:20:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:20:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:20:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:20:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:20:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:20:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:20:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:20:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:20:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:20:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:20:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:20:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:20:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:20:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:20:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:20:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:20:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:20:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:20:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:20:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:20:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:20:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:20:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:20:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:20:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:20:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:20:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:20:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:20:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:20:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:20:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:20:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:20:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:20:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:20:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:20:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:20:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:20:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:20:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:20:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:20:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:20:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:20:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:20:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:20:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:20:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:20:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:20:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:20:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:20:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:20:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:20:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:20:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:20:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:20:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:20:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:20:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:20:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:20:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:20:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:20:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:20:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:20:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:20:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:20:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:20:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:20:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:20:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:20:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:20:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:20:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:20:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:20:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:20:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:20:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:20:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:20:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:20:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:20:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:20:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:20:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:20:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:20:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:20:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:20:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:20:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:20:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:20:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:20:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:20:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:20:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:20:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:20:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:20:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:20:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:20:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:20:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:20:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:20:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:20:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:20:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:20:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:20:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:20:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:20:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:20:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:20:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:20:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:20:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:20:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:20:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:20:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:20:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:20:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:20:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:20:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:20:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:20:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:20:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:20:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:20:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:20:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:20:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:20:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:20:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:20:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:20:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:20:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:20:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:20:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:20:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:20:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:20:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:20:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:20:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:20:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:20:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:20:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:20:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:20:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:20:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:20:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:20:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:20:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:20:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:20:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:20:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:20:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:20:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:20:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:20:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:20:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:20:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:20:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:20:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:20:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:20:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:20:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:20:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:20:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:20:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:20:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:20:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:20:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:20:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:20:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:20:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:20:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:20:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:20:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:20:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:20:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:20:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:20:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:20:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:20:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:20:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:20:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:20:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:20:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:20:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:20:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:20:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:20:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:20:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:20:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:20:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:20:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:20:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:20:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:20:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:20:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:20:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:20:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:20:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:20:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:20:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:20:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:20:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:20:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:20:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:20:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:20:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:21:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:21:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:21:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:21:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:21:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:21:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:21:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:21:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:21:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:21:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:21:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:21:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:21:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:21:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:21:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:21:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:21:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:21:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:21:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:21:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:21:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:21:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:21:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:21:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:21:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:21:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:21:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:21:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:21:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:21:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:21:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:21:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:21:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:21:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:21:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:21:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:21:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:21:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:21:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:21:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:21:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:21:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:21:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:21:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:21:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:21:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:21:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:21:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:21:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:21:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:21:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:21:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:21:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:21:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:21:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:21:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:21:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:21:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:21:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:21:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:21:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:21:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:21:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:21:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:21:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:21:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:21:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:21:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:21:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:21:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:21:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:21:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:21:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:21:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:21:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:21:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:21:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:21:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:21:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:21:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:21:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:21:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:21:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:21:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:21:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:21:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:21:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:21:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:21:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:21:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:21:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:21:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:21:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:21:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:21:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:21:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:21:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:21:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:21:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:21:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:21:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:21:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:21:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:21:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:21:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:21:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:21:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:21:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:21:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:21:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:21:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:21:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:21:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:21:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:21:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:21:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:21:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:21:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:21:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:21:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:21:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:21:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:21:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:21:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:21:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:21:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:21:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:21:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:21:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:21:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:21:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:21:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:21:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:21:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:21:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:21:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:21:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:21:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:21:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:21:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:21:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:21:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:21:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:21:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:21:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:21:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:21:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:21:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:21:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:21:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:21:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:21:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:21:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:21:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:21:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:21:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:21:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:21:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:21:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:21:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:21:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:21:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:21:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:21:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:21:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:21:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:21:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:21:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:21:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:21:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:21:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:21:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:21:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:21:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:21:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:21:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:21:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:21:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:21:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:21:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:21:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:21:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:21:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:21:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:21:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:21:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:21:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:21:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:21:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:21:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:21:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:21:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:21:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:21:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:21:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:21:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:21:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:21:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:21:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:21:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:21:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:21:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:21:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:21:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:21:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:21:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:21:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:21:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:21:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:21:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:21:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:21:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:21:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:21:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:21:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:21:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:22:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:22:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:22:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:22:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:22:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:22:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:22:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:22:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:22:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:22:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:22:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:22:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:22:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:22:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:22:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:22:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:22:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:22:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:22:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:22:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:22:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:22:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:22:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:22:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:22:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:22:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:22:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:22:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:22:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:22:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:22:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:22:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:22:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:22:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:22:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:22:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:22:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:22:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:22:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:22:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:22:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:22:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:22:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:22:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:22:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:22:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:22:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:22:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:22:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:22:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:22:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:22:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:22:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:22:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:22:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:22:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:22:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:22:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:22:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:22:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:22:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:22:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:22:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:22:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:22:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:22:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:22:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:22:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:22:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:22:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:22:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:22:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:22:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:22:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:22:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:22:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:22:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:22:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:22:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:22:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:22:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:22:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:22:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:22:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:22:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:22:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:22:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:22:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:22:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:22:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:22:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:22:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:22:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:22:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:22:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:22:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:22:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:22:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:22:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:22:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:22:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:22:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:22:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:22:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:22:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:22:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:22:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:22:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:22:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:22:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:22:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:22:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:22:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:22:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:22:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:22:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:22:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:22:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:22:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:22:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:22:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:22:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:22:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:22:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:22:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:22:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:22:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:22:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:22:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:22:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:22:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:22:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:22:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:22:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:22:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:22:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:22:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:22:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:22:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:22:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:22:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:22:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:22:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:22:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:22:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:22:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:22:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:22:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:22:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:22:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:22:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:22:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:22:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:22:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:22:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:22:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:22:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:22:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:22:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:22:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:22:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:22:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:22:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:22:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:22:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:22:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:22:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:22:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:22:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:22:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:22:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:22:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:22:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:22:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:22:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:22:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:22:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:22:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:22:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:22:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:22:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:22:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:22:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:22:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:22:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:22:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:22:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:22:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:22:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:22:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:22:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:22:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:22:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:22:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:22:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:22:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:22:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:22:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:22:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:22:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:22:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:22:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:22:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:22:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:22:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:22:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:22:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:22:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:22:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:22:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:22:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:22:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:22:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:22:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:22:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:22:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:23:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:23:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:23:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:23:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:23:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:23:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:23:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:23:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:23:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:23:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:23:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:23:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:23:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:23:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:23:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:23:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:23:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:23:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:23:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:23:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:23:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:23:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:23:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:23:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:23:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:23:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:23:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:23:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:23:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:23:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:23:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:23:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:23:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:23:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:23:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:23:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:23:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:23:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:23:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:23:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:23:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:23:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:23:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:23:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:23:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:23:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:23:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:23:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:23:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:23:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:23:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:23:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:23:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:23:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:23:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:23:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:23:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:23:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:23:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:23:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:23:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:23:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:23:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:23:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:23:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:23:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:23:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:23:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:23:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:23:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:23:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:23:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:23:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:23:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:23:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:23:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:23:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:23:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:23:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:23:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:23:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:23:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:23:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:23:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:23:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:23:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:23:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:23:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:23:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:23:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:23:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:23:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:23:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:23:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:23:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:23:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:23:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:23:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:23:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:23:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:23:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:23:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:23:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:23:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:23:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:23:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:23:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:23:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:23:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:23:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:23:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:23:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:23:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:23:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:23:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:23:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:23:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:23:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:23:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:23:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:23:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:23:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:23:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:23:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:23:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:23:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:23:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:23:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:23:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:23:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:23:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:23:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:23:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:23:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:23:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:23:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:23:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:23:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:23:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:23:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:23:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:23:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:23:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:23:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:23:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:23:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:23:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:23:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:23:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:23:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:23:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:23:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:23:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:23:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:23:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:23:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:23:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:23:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:23:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:23:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:23:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:23:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:23:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:23:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:23:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:23:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:23:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:23:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:23:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:23:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:23:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:23:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:23:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:23:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:23:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:23:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:23:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:23:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:23:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:23:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:23:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:23:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:23:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:23:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:23:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:23:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:23:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:23:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:23:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:23:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:23:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:23:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:23:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:23:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:23:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:23:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:23:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:23:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:23:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:23:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:23:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:23:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:23:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:23:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:23:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:23:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:23:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:23:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:23:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:23:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:23:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:23:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:23:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:23:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:23:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:23:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:24:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:24:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:24:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:24:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:24:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:24:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:24:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:24:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:24:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:24:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:24:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:24:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:24:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:24:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:24:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:24:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:24:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:24:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:24:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:24:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:24:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:24:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:24:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:24:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:24:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:24:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:24:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:24:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:24:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:24:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:24:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:24:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:24:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:24:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:24:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:24:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:24:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:24:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:24:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:24:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:24:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:24:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:24:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:24:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:24:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:24:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:24:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:24:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:24:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:24:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:24:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:24:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:24:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:24:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:24:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:24:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:24:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:24:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:24:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:24:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:24:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:24:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:24:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:24:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:24:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:24:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:24:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:24:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:24:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:24:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:24:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:24:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:24:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:24:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:24:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:24:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:24:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:24:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:24:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:24:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:24:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:24:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:24:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:24:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:24:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:24:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:24:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:24:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:24:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:24:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:24:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:24:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:24:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:24:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:24:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:24:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:24:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:24:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:24:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:24:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:24:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:24:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:24:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:24:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:24:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:24:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:24:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:24:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:24:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:24:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:24:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:24:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:24:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:24:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:24:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:24:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:24:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:24:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:24:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:24:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:24:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:24:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:24:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:24:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:24:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:24:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:24:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:24:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:24:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:24:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:24:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:24:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:24:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:24:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:24:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:24:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:24:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:24:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:24:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:24:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:24:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:24:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:24:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:24:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:24:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:24:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:24:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:24:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:24:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:24:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:24:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:24:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:24:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:24:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:24:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:24:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:24:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:24:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:24:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:24:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:24:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:24:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:24:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:24:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:24:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:24:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:24:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:24:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:24:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:24:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:24:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:24:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:24:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:24:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:24:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:24:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:24:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:24:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:24:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:24:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:24:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:24:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:24:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:24:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:24:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:24:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:24:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:24:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:24:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:24:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:24:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:24:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:24:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:24:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:24:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:24:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:24:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:24:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:24:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:24:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:24:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:24:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:24:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:24:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:24:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:24:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:24:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:24:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:24:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:24:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:24:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:24:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:24:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:24:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:24:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:24:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:25:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:25:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:25:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:25:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:25:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:25:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:25:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:25:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:25:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:25:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:25:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:25:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:25:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:25:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:25:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:25:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:25:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:25:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:25:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:25:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:25:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:25:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:25:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:25:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:25:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:25:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:25:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:25:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:25:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:25:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:25:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:25:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:25:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:25:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:25:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:25:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:25:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:25:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:25:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:25:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:25:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:25:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:25:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:25:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:25:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:25:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:25:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:25:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:25:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:25:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:25:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:25:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:25:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:25:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:25:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:25:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:25:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:25:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:25:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:25:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:25:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:25:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:25:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:25:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:25:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:25:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:25:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:25:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:25:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:25:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:25:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:25:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:25:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:25:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:25:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:25:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:25:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:25:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:25:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:25:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:25:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:25:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:25:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:25:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:25:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:25:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:25:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:25:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:25:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:25:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:25:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:25:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:25:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:25:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:25:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:25:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:25:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:25:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:25:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:25:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:25:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:25:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:25:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:25:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:25:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:25:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:25:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:25:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:25:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:25:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:25:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:25:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:25:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:25:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:25:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:25:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:25:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:25:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:25:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:25:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:25:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:25:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:25:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:25:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:25:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:25:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:25:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:25:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:25:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:25:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:25:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:25:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:25:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:25:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:25:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:25:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:25:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:25:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:25:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:25:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:25:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:25:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:25:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:25:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:25:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:25:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:25:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:25:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:25:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:25:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:25:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:25:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:25:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:25:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:25:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:25:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:25:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:25:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:25:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:25:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:25:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:25:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:27:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:27:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:27:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:27:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:27:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:27:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:27:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:27:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:27:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:27:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:27:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:27:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:27:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:27:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:27:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:27:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:27:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:27:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:27:51 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 18:27:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:27:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:27:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:27:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:27:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:27:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:27:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:27:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:27:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:27:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:27:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:27:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:27:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:27:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:27:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:27:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:27:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:27:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:28:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:28:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:28:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:28:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:28:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:28:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:28:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:28:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:28:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:28:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:28:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:28:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:28:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:28:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:28:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:28:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:28:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:28:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:28:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:28:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:28:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:28:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:28:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:28:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:28:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:28:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:28:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:28:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:28:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:28:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:28:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:28:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:28:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:28:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:28:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:28:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:28:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:28:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:28:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:28:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:28:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:28:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:28:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:28:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:28:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:28:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:28:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:28:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:28:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:28:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:28:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:28:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:28:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:28:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:28:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:28:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:28:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:28:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:28:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:28:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:28:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:28:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:28:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:28:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:28:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:28:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:28:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:28:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:28:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:28:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:28:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:28:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:28:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:28:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:28:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:28:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:28:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:28:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:28:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:28:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:28:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:28:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:28:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:28:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:28:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:28:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:28:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:28:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:28:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:28:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:28:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:28:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:28:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:28:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:28:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:28:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:28:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:28:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:28:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:28:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:28:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:28:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:28:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:28:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:28:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:28:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:28:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:28:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:28:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:28:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:28:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:28:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:28:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:28:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:28:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:28:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:28:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:28:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:28:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:28:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:28:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:28:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:28:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:28:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:28:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:28:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:28:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:28:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:28:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:28:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:28:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:28:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:28:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:28:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:28:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:28:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:28:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:28:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:28:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:28:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:28:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:28:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:28:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:28:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:28:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:28:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:28:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:28:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:28:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:28:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:28:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:28:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:28:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:28:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:28:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:28:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:28:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:28:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:28:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:28:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:28:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:28:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:28:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:28:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:28:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:28:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:28:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:28:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:28:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:28:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:28:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:28:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:28:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:28:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:28:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:28:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:28:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:28:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:28:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:28:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:28:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:28:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:28:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:28:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:28:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:28:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:28:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:28:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:28:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:28:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:28:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:28:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:28:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:28:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:28:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:28:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:28:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:28:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:28:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:28:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:28:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:28:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:28:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:28:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:28:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:28:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:28:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:28:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:28:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:28:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:28:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:28:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:28:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:28:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:28:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:28:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:29:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:29:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:29:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:29:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:29:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:29:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:29:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:29:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:29:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:29:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:29:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:29:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:29:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:29:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:29:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:29:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:29:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:29:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:29:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:29:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:29:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:29:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:29:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:29:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:29:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:29:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:29:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:29:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:29:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:29:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:29:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:29:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:29:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:29:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:29:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:29:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:29:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:29:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:29:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:29:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:29:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:29:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:29:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:29:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:29:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:29:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:29:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:29:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:29:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:29:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:29:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:29:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:29:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:29:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:29:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:29:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:29:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:29:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:29:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:29:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:29:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:29:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:29:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:29:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:29:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:29:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:29:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:29:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:29:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:29:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:29:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:29:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:29:16 rdma06 systemd[1]: Starting dnf makecache...
Apr 18 18:29:16 rdma06 dnf[1199467]: Updating Subscription Management repositories.
Apr 18 18:29:16 rdma06 dnf[1199467]: Unable to read consumer identity
Apr 18 18:29:16 rdma06 dnf[1199467]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 18 18:29:17 rdma06 dnf[1199467]: Copr repo for fmf owned by psss                 3.1 kB/s | 3.3 kB     00:01
Apr 18 18:29:17 rdma06 dnf[1199467]: beaker-AppStream                                 39 kB/s | 168  B     00:00
Apr 18 18:29:17 rdma06 dnf[1199467]: Errors during downloading metadata for repository 'beaker-AppStream':
Apr 18 18:29:17 rdma06 dnf[1199467]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 18:29:17 rdma06 dnf[1199467]: Error: Failed to download metadata for repo 'beaker-AppStream': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 18:29:17 rdma06 dnf[1199467]: beaker-AppStream-debuginfo                       53 kB/s | 168  B     00:00
Apr 18 18:29:17 rdma06 dnf[1199467]: Errors during downloading metadata for repository 'beaker-AppStream-debuginfo':
Apr 18 18:29:17 rdma06 dnf[1199467]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 18:29:17 rdma06 dnf[1199467]: Error: Failed to download metadata for repo 'beaker-AppStream-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 18:29:17 rdma06 dnf[1199467]: beaker-BaseOS                                    58 kB/s | 168  B     00:00
Apr 18 18:29:17 rdma06 dnf[1199467]: Errors during downloading metadata for repository 'beaker-BaseOS':
Apr 18 18:29:17 rdma06 dnf[1199467]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 18:29:17 rdma06 dnf[1199467]: Error: Failed to download metadata for repo 'beaker-BaseOS': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 18:29:17 rdma06 dnf[1199467]: beaker-BaseOS-debuginfo                          63 kB/s | 168  B     00:00
Apr 18 18:29:17 rdma06 dnf[1199467]: Errors during downloading metadata for repository 'beaker-BaseOS-debuginfo':
Apr 18 18:29:17 rdma06 dnf[1199467]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 18:29:17 rdma06 dnf[1199467]: Error: Failed to download metadata for repo 'beaker-BaseOS-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 18:29:17 rdma06 dnf[1199467]: beaker-CRB                                       63 kB/s | 168  B     00:00
Apr 18 18:29:17 rdma06 dnf[1199467]: Errors during downloading metadata for repository 'beaker-CRB':
Apr 18 18:29:17 rdma06 dnf[1199467]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 18:29:17 rdma06 dnf[1199467]: Error: Failed to download metadata for repo 'beaker-CRB': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 18:29:17 rdma06 dnf[1199467]: beaker-CRB-debuginfo                             60 kB/s | 168  B     00:00
Apr 18 18:29:17 rdma06 dnf[1199467]: Errors during downloading metadata for repository 'beaker-CRB-debuginfo':
Apr 18 18:29:17 rdma06 dnf[1199467]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 18:29:17 rdma06 dnf[1199467]: Error: Failed to download metadata for repo 'beaker-CRB-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 18:29:18 rdma06 dnf[1199467]: beaker-HighAvailability                          61 kB/s | 168  B     00:00
Apr 18 18:29:18 rdma06 dnf[1199467]: Errors during downloading metadata for repository 'beaker-HighAvailability':
Apr 18 18:29:18 rdma06 dnf[1199467]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 18:29:18 rdma06 dnf[1199467]: Error: Failed to download metadata for repo 'beaker-HighAvailability': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 18:29:18 rdma06 dnf[1199467]: beaker-HighAvailability-debuginfo                60 kB/s | 168  B     00:00
Apr 18 18:29:18 rdma06 dnf[1199467]: Errors during downloading metadata for repository 'beaker-HighAvailability-debuginfo':
Apr 18 18:29:18 rdma06 dnf[1199467]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 18:29:18 rdma06 dnf[1199467]: Error: Failed to download metadata for repo 'beaker-HighAvailability-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 18:29:18 rdma06 dnf[1199467]: beaker-NFV                                       55 kB/s | 168  B     00:00
Apr 18 18:29:18 rdma06 dnf[1199467]: Errors during downloading metadata for repository 'beaker-NFV':
Apr 18 18:29:18 rdma06 dnf[1199467]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 18:29:18 rdma06 dnf[1199467]: Error: Failed to download metadata for repo 'beaker-NFV': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 18:29:18 rdma06 dnf[1199467]: beaker-NFV-debuginfo                             52 kB/s | 168  B     00:00
Apr 18 18:29:18 rdma06 dnf[1199467]: Errors during downloading metadata for repository 'beaker-NFV-debuginfo':
Apr 18 18:29:18 rdma06 dnf[1199467]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 18:29:18 rdma06 dnf[1199467]: Error: Failed to download metadata for repo 'beaker-NFV-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 18:29:18 rdma06 dnf[1199467]: beaker-RT                                        60 kB/s | 168  B     00:00
Apr 18 18:29:18 rdma06 dnf[1199467]: Errors during downloading metadata for repository 'beaker-RT':
Apr 18 18:29:18 rdma06 dnf[1199467]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 18:29:18 rdma06 dnf[1199467]: Error: Failed to download metadata for repo 'beaker-RT': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 18:29:18 rdma06 dnf[1199467]: beaker-RT-debuginfo                              58 kB/s | 168  B     00:00
Apr 18 18:29:18 rdma06 dnf[1199467]: Errors during downloading metadata for repository 'beaker-RT-debuginfo':
Apr 18 18:29:18 rdma06 dnf[1199467]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 18:29:18 rdma06 dnf[1199467]: Error: Failed to download metadata for repo 'beaker-RT-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 18:29:18 rdma06 dnf[1199467]: beaker-ResilientStorage                          68 kB/s | 168  B     00:00
Apr 18 18:29:18 rdma06 dnf[1199467]: Errors during downloading metadata for repository 'beaker-ResilientStorage':
Apr 18 18:29:18 rdma06 dnf[1199467]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 18:29:18 rdma06 dnf[1199467]: Error: Failed to download metadata for repo 'beaker-ResilientStorage': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 18:29:18 rdma06 dnf[1199467]: beaker-ResilientStorage-debuginfo                59 kB/s | 168  B     00:00
Apr 18 18:29:18 rdma06 dnf[1199467]: Errors during downloading metadata for repository 'beaker-ResilientStorage-debuginfo':
Apr 18 18:29:18 rdma06 dnf[1199467]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 18:29:18 rdma06 dnf[1199467]: Error: Failed to download metadata for repo 'beaker-ResilientStorage-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 18:29:18 rdma06 dnf[1199467]: beaker-SAP                                       58 kB/s | 168  B     00:00
Apr 18 18:29:18 rdma06 dnf[1199467]: Errors during downloading metadata for repository 'beaker-SAP':
Apr 18 18:29:18 rdma06 dnf[1199467]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 18:29:18 rdma06 dnf[1199467]: Error: Failed to download metadata for repo 'beaker-SAP': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 18:29:18 rdma06 dnf[1199467]: beaker-SAP-debuginfo                             57 kB/s | 168  B     00:00
Apr 18 18:29:18 rdma06 dnf[1199467]: Errors during downloading metadata for repository 'beaker-SAP-debuginfo':
Apr 18 18:29:18 rdma06 dnf[1199467]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 18:29:18 rdma06 dnf[1199467]: Error: Failed to download metadata for repo 'beaker-SAP-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 18:29:18 rdma06 dnf[1199467]: beaker-SAPHANA                                   56 kB/s | 168  B     00:00
Apr 18 18:29:18 rdma06 dnf[1199467]: Errors during downloading metadata for repository 'beaker-SAPHANA':
Apr 18 18:29:18 rdma06 dnf[1199467]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 18:29:18 rdma06 dnf[1199467]: Error: Failed to download metadata for repo 'beaker-SAPHANA': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 18:29:18 rdma06 dnf[1199467]: beaker-SAPHANA-debuginfo                         56 kB/s | 168  B     00:00
Apr 18 18:29:18 rdma06 dnf[1199467]: Errors during downloading metadata for repository 'beaker-SAPHANA-debuginfo':
Apr 18 18:29:18 rdma06 dnf[1199467]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 18:29:18 rdma06 dnf[1199467]: Error: Failed to download metadata for repo 'beaker-SAPHANA-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 18:29:18 rdma06 dnf[1199467]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr 18 18:29:19 rdma06 dnf[1199467]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr 18 18:29:19 rdma06 dnf[1199467]: Ignoring repositories: beaker-AppStream, beaker-AppStream-debuginfo, beaker-BaseOS, beaker-BaseOS-debuginfo, beaker-CRB, beaker-CRB-debuginfo, beaker-HighAvailability, beaker-HighAvailability-debuginfo, beaker-NFV, beaker-NFV-debuginfo, beaker-RT, beaker-RT-debuginfo, beaker-ResilientStorage, beaker-ResilientStorage-debuginfo, beaker-SAP, beaker-SAP-debuginfo, beaker-SAPHANA, beaker-SAPHANA-debuginfo
Apr 18 18:29:19 rdma06 dnf[1199467]: Metadata cache created.
Apr 18 18:29:19 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 18 18:29:19 rdma06 systemd[1]: Finished dnf makecache.
Apr 18 18:29:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:29:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:29:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:29:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:29:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:29:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:29:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:29:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:29:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:29:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:29:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:29:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:29:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:29:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:29:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:29:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:29:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:29:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:29:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:29:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:29:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:29:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:29:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:29:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:29:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:29:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:29:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:29:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:29:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:29:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:29:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:29:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:29:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:29:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:29:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:29:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:29:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:29:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:29:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:29:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:29:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:29:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:29:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:29:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:29:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:29:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:29:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:29:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:29:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:29:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:29:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:29:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:29:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:29:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:29:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:29:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:29:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:29:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:29:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:29:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:29:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:29:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:29:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:29:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:29:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:29:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:29:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:29:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:29:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:29:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:29:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:29:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:29:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:29:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:29:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:29:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:29:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:29:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:29:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:29:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:29:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:29:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:29:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:29:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:29:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:29:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:29:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:29:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:29:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:29:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:29:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:29:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:29:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:29:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:29:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:29:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:29:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:29:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:29:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:29:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:29:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:29:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:29:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:29:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:29:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:29:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:29:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:29:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:29:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:29:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:29:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:29:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:29:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:29:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:29:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:29:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:29:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:29:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:29:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:29:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:29:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:29:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:29:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:29:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:29:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:29:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:29:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:29:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:29:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:29:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:29:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:29:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:29:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:29:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:29:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:29:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:29:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:29:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:29:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:29:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:29:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:29:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:29:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:29:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:30:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:30:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:30:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:30:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:30:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:30:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:30:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:30:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:30:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:30:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:30:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:30:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:30:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:30:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:30:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:30:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:30:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:30:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:30:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:30:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:30:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:30:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:30:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:30:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:30:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:30:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:30:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:30:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:30:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:30:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:30:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:30:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:30:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:30:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:30:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:30:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:30:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:30:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:30:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:30:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:30:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:30:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:30:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:30:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:30:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:30:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:30:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:30:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:30:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:30:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:30:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:30:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:30:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:30:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:30:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:30:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:30:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:30:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:30:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:30:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:30:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:30:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:30:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:30:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:30:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:30:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:30:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:30:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:30:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:30:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:30:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:30:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:30:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:30:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:30:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:30:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:30:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:30:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:30:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:30:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:30:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:30:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:30:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:30:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:30:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:30:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:30:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:30:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:30:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:30:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:30:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:30:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:30:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:30:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:30:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:30:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:30:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:30:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:30:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:30:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:30:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:30:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:30:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:30:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:30:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:30:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:30:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:30:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:30:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:30:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:30:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:30:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:30:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:30:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:30:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:30:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:30:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:30:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:30:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:30:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:30:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:30:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:30:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:30:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:30:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:30:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:30:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:30:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:30:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:30:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:30:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:30:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:30:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:30:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:30:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:30:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:30:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:30:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:30:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:30:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:30:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:30:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:30:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:30:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:30:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:30:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:30:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:30:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:30:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:30:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:30:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:30:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:30:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:30:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:30:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:30:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:30:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:30:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:30:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:30:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:30:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:30:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:30:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:30:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:30:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:30:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:30:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:30:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:30:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:30:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:30:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:30:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:30:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:30:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:30:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:30:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:30:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:30:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:30:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:30:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:30:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:30:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:30:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:30:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:30:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:30:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:30:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:30:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:30:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:30:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:30:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:30:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:30:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:30:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:30:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:30:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:30:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:30:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:30:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:30:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:30:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:30:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:30:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:30:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:30:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:30:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:30:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:30:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:30:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:30:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:30:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:30:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:30:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:30:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:30:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:30:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:31:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:31:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:31:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:31:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:31:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:31:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:31:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:31:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:31:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:31:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:31:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:31:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:31:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:31:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:31:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:31:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:31:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:31:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:31:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:31:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:31:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:31:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:31:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:31:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:31:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:31:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:31:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:31:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:31:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:31:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:31:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:31:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:31:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:31:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:31:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:31:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:31:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:31:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:31:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:31:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:31:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:31:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:31:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:31:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:31:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:31:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:31:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:31:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:31:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:31:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:31:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:31:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:31:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:31:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:31:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:31:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:31:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:31:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:31:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:31:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:31:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:31:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:31:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:31:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:31:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:31:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:31:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:31:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:31:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:31:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:31:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:31:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:31:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:31:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:31:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:31:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:31:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:31:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:31:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:31:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:31:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:31:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:31:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:31:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:31:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:31:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:31:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:31:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:31:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:31:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:31:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:31:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:31:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:31:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:31:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:31:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:31:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:31:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:31:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:31:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:31:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:31:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:31:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:31:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:31:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:31:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:31:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:31:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:31:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:31:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:31:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:31:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:31:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:31:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:31:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:31:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:31:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:31:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:31:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:31:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:31:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:31:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:31:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:31:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:31:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:31:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:31:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:31:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:31:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:31:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:31:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:31:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:31:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:31:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:31:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:31:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:31:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:31:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:31:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:31:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:31:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:31:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:31:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:31:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:31:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:31:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:31:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:31:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:31:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:31:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:31:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:31:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:31:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:31:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:31:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:31:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:31:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:31:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:31:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:31:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:31:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:31:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:31:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:31:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:31:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:31:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:32:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:32:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:32:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:32:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:32:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:32:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:32:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:32:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:32:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:32:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:32:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:32:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:32:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:32:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:32:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:32:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:32:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:32:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:32:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:32:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:32:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:32:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:32:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:32:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:32:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:32:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:32:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:32:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:32:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:32:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:32:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:32:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:32:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:32:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:32:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:32:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:32:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:32:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:32:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:32:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:32:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:32:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:32:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:32:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:32:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:32:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:32:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:32:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:32:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:32:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:32:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:32:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:32:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:32:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:32:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:32:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:32:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:32:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:32:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:32:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:32:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:32:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:32:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:32:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:32:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:32:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:32:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:32:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:32:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:32:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:32:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:32:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:32:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:32:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:32:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:32:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:32:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:32:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:32:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:32:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:32:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:32:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:32:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:32:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:32:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:32:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:32:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:32:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:32:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:32:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:32:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:32:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:32:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:32:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:32:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:32:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:32:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:32:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:32:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:32:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:32:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:32:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:32:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:32:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:32:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:32:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:32:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:32:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:32:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:32:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:32:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:32:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:32:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:32:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:32:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:32:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:32:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:32:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:32:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:32:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:32:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:32:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:32:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:32:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:32:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:32:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:32:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:32:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:32:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:32:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:32:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:32:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:32:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:32:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:32:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:32:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:32:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:32:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:32:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:32:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:32:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:32:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:32:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:32:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:32:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:32:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:32:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:32:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:32:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:32:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:32:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:32:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:32:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:32:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:32:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:32:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:32:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:32:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:32:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:32:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:32:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:32:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:32:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:32:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:32:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:32:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:32:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:32:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:32:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:32:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:32:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:32:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:32:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:32:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:32:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:32:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:32:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:32:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:32:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:32:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:32:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:32:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:32:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:32:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:32:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:32:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:32:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:32:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:32:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:32:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:32:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:32:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:32:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:32:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:32:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:32:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:32:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:32:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:32:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:32:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:32:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:32:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:32:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:32:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:32:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:32:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:32:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:32:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:32:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:32:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:32:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:32:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:32:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:32:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:32:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:32:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:33:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:33:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:33:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:33:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:33:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:33:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:33:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:33:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:33:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:33:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:33:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:33:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:33:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:33:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:33:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:33:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:33:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:33:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:33:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:33:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:33:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:33:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:33:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:33:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:33:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:33:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:33:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:33:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:33:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:33:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:33:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:33:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:33:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:33:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:33:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:33:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:33:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:33:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:33:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:33:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:33:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:33:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:33:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:33:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:33:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:33:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:33:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:33:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:33:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:33:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:33:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:33:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:33:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:33:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:33:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:33:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:33:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:33:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:33:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:33:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:33:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:33:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:33:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:33:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:33:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:33:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:33:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:33:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:33:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:33:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:33:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:33:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:33:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:33:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:33:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:33:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:33:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:33:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:33:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:33:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:33:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:33:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:33:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:33:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:33:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:33:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:33:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:33:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:33:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:33:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:33:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:33:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:33:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:33:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:33:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:33:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:33:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:33:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:33:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:33:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:33:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:33:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:33:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:33:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:33:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:33:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:33:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:33:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:33:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:33:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:33:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:33:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:33:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:33:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:33:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:33:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:33:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:33:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:33:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:33:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:33:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:33:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:33:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:33:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:33:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:33:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:33:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:33:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:33:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:33:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:33:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:33:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:33:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:33:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:33:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:33:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:33:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:33:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:33:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:33:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:33:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:33:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:33:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:33:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:33:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:33:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:33:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:33:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:33:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:33:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:33:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:33:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:33:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:33:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:33:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:33:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:33:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:33:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:33:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:33:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:33:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:33:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:33:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:33:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:33:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:33:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:33:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:33:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:33:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:33:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:33:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:33:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:33:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:33:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:33:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:33:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:33:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:33:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:33:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:33:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:33:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:33:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:33:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:33:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:33:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:33:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:33:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:33:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:33:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:33:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:33:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:33:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:33:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:33:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:33:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:33:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:33:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:33:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:33:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:33:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:33:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:33:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:33:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:33:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:33:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:33:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:33:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:33:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:33:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:33:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:33:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:33:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:33:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:33:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:33:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:33:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:34:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:34:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:34:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:34:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:34:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:34:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:34:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:34:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:34:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:34:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:34:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:34:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:34:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:34:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:34:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:34:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:34:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:34:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:34:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:34:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:34:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:34:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:34:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:34:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:34:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:34:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:34:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:34:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:34:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:34:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:34:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:34:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:34:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:34:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:34:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:34:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:34:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:34:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:34:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:34:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:34:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:34:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:34:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:34:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:34:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:34:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:34:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:34:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:34:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:34:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:34:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:34:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:34:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:34:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:34:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:34:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:34:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:34:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:34:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:34:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:34:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:34:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:34:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:34:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:34:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:34:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:34:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:34:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:34:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:34:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:34:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:34:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:34:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:34:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:34:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:34:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:34:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:34:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:34:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:34:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:34:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:34:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:34:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:34:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:34:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:34:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:34:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:34:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:34:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:34:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:34:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:34:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:34:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:34:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:34:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:34:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:34:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:34:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:34:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:34:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:34:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:34:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:34:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:34:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:34:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:34:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:34:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:34:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:34:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:34:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:34:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:34:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:34:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:34:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:34:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:34:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:34:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:34:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:34:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:34:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:34:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:34:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:34:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:34:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:34:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:34:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:34:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:34:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:34:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:34:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:34:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:34:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:34:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:34:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:34:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:34:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:34:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:34:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:34:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:34:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:34:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:34:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:34:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:34:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:34:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:34:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:34:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:34:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:34:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:34:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:34:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:34:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:34:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:34:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:34:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:34:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:34:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:34:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:34:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:34:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:34:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:34:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:34:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:34:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:34:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:34:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:34:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:34:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:34:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:34:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:34:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:34:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:34:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:34:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:34:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:34:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:34:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:34:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:34:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:34:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:34:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:34:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:34:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:34:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:34:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:34:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:34:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:34:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:34:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:34:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:34:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:34:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:34:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:34:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:34:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:34:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:34:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:34:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:34:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:34:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:34:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:34:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:34:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:34:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:34:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:34:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:34:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:34:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:34:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:34:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:34:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:34:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:34:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:34:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:34:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:34:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:35:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:35:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:35:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:35:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:35:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:35:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:35:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:35:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:35:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:35:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:35:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:35:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:35:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:35:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:35:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:35:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:35:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:35:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:35:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:35:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:35:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:35:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:35:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:35:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:35:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:35:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:35:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:35:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:35:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:35:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:35:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:35:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:35:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:35:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:35:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:35:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:35:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:35:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:35:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:35:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:35:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:35:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:35:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:35:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:35:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:35:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:35:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:35:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:35:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:35:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:35:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:35:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:35:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:35:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:35:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:35:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:35:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:35:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:35:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:35:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:35:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:35:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:35:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:35:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:35:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:35:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:35:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:35:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:35:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:35:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:35:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:35:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:35:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:35:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:35:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:35:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:35:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:35:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:35:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:35:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:35:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:35:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:35:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:35:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:35:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:35:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:35:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:35:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:35:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:35:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:35:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:35:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:35:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:35:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:35:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:35:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:35:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:35:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:35:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:35:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:35:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:35:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:35:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:35:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:35:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:35:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:35:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:35:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:35:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:35:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:35:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:35:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:35:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:35:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:35:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:35:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:35:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:35:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:35:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:35:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:35:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:35:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:35:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:35:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:35:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:35:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:35:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:35:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:35:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:35:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:35:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:35:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:35:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:35:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:35:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:35:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:35:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:35:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:35:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:35:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:35:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:35:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:35:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:35:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:35:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:35:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:35:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:35:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:35:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:35:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:35:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:35:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:35:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:35:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:35:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:35:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:35:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:35:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:35:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:35:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:35:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:35:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:35:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:35:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:35:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:35:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:35:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:35:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:35:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:35:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:35:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:35:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:35:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:35:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:35:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:35:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:35:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:37:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:37:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:37:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:37:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:37:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:37:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:37:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:37:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:37:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:37:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:37:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:37:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:37:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:37:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:37:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:37:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:37:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:37:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:37:52 rdma06 rsyslogd[1041]: imjournal: 5165 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 18:37:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:37:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:37:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:37:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:37:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:37:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:37:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:37:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:37:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:37:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:37:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:37:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:37:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:37:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:37:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:37:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:37:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:37:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:38:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:38:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:38:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:38:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:38:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:38:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:38:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:38:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:38:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:38:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:38:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:38:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:38:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:38:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:38:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:38:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:38:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:38:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:38:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:38:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:38:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:38:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:38:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:38:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:38:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:38:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:38:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:38:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:38:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:38:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:38:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:38:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:38:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:38:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:38:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:38:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:38:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:38:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:38:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:38:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:38:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:38:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:38:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:38:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:38:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:38:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:38:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:38:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:38:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:38:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:38:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:38:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:38:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:38:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:38:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:38:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:38:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:38:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:38:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:38:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:38:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:38:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:38:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:38:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:38:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:38:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:38:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:38:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:38:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:38:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:38:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:38:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:38:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:38:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:38:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:38:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:38:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:38:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:38:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:38:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:38:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:38:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:38:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:38:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:38:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:38:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:38:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:38:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:38:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:38:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:38:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:38:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:38:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:38:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:38:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:38:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:38:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:38:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:38:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:38:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:38:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:38:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:38:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:38:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:38:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:38:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:38:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:38:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:38:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:38:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:38:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:38:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:38:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:38:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:38:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:38:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:38:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:38:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:38:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:38:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:38:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:38:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:38:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:38:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:38:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:38:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:38:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:38:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:38:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:38:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:38:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:38:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:38:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:38:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:38:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:38:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:38:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:38:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:38:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:38:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:38:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:38:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:38:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:38:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:38:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:38:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:38:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:38:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:38:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:38:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:38:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:38:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:38:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:38:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:38:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:38:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:38:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:38:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:38:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:38:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:38:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:38:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:38:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:38:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:38:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:38:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:38:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:38:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:38:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:38:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:38:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:38:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:38:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:38:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:38:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:38:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:38:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:38:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:38:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:38:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:38:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:38:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:38:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:38:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:38:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:38:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:38:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:38:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:38:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:38:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:38:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:38:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:38:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:38:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:38:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:38:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:38:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:38:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:38:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:38:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:38:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:38:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:38:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:38:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:38:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:38:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:38:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:38:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:38:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:38:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:38:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:38:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:38:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:38:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:38:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:38:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:39:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:39:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:39:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:39:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:39:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:39:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:39:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:39:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:39:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:39:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:39:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:39:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:39:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:39:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:39:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:39:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:39:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:39:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:39:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:39:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:39:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:39:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:39:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:39:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:39:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:39:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:39:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:39:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:39:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:39:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:39:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:39:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:39:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:39:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:39:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:39:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:39:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:39:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:39:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:39:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:39:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:39:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:39:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:39:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:39:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:39:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:39:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:39:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:39:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:39:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:39:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:39:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:39:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:39:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:39:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:39:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:39:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:39:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:39:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:39:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:39:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:39:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:39:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:39:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:39:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:39:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:39:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:39:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:39:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:39:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:39:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:39:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:39:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:39:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:39:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:39:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:39:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:39:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:39:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:39:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:39:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:39:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:39:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:39:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:39:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:39:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:39:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:39:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:39:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:39:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:39:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:39:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:39:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:39:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:39:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:39:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:39:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:39:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:39:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:39:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:39:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:39:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:39:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:39:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:39:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:39:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:39:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:39:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:39:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:39:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:39:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:39:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:39:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:39:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:39:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:39:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:39:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:39:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:39:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:39:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:39:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:39:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:39:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:39:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:39:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:39:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:39:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:39:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:39:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:39:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:39:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:39:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:39:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:39:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:39:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:39:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:39:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:39:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:39:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:39:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:39:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:39:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:39:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:39:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:39:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:39:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:39:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:39:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:39:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:39:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:39:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:39:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:39:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:39:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:39:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:39:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:39:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:39:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:39:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:39:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:39:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:39:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:39:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:39:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:39:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:39:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:39:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:39:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:39:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:39:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:39:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:39:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:39:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:39:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:39:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:39:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:39:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:39:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:39:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:39:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:39:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:39:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:39:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:39:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:39:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:39:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:39:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:39:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:39:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:39:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:39:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:39:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:39:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:39:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:39:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:39:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:39:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:39:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:39:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:39:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:39:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:39:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:39:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:39:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:39:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:39:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:39:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:39:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:39:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:39:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:39:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:39:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:39:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:39:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:39:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:39:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:40:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:40:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:40:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:40:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:40:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:40:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:40:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:40:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:40:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:40:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:40:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:40:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:40:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:40:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:40:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:40:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:40:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:40:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:40:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:40:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:40:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:40:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:40:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:40:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:40:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:40:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:40:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:40:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:40:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:40:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:40:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:40:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:40:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:40:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:40:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:40:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:40:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:40:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:40:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:40:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:40:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:40:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:40:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:40:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:40:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:40:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:40:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:40:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:40:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:40:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:40:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:40:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:40:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:40:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:40:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:40:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:40:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:40:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:40:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:40:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:40:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:40:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:40:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:40:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:40:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:40:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:40:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:40:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:40:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:40:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:40:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:40:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:40:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:40:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:40:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:40:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:40:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:40:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:40:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:40:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:40:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:40:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:40:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:40:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:40:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:40:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:40:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:40:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:40:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:40:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:40:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:40:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:40:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:40:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:40:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:40:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:40:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:40:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:40:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:40:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:40:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:40:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:40:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:40:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:40:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:40:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:40:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:40:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:40:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:40:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:40:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:40:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:40:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:40:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:40:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:40:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:40:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:40:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:40:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:40:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:40:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:40:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:40:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:40:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:40:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:40:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:40:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:40:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:40:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:40:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:40:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:40:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:40:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:40:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:40:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:40:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:40:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:40:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:40:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:40:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:40:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:40:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:40:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:40:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:40:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:40:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:40:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:40:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:40:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:40:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:40:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:40:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:40:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:40:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:40:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:40:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:40:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:40:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:40:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:40:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:40:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:40:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:40:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:40:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:40:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:40:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:40:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:40:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:40:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:40:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:40:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:40:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:40:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:40:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:40:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:40:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:40:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:40:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:40:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:40:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:40:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:40:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:40:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:40:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:40:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:40:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:40:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:40:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:40:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:40:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:40:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:40:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:40:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:40:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:40:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:40:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:40:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:40:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:40:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:40:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:40:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:40:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:40:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:40:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:40:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:40:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:40:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:40:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:40:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:40:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:40:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:40:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:40:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:40:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:40:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:40:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:41:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:41:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:41:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:41:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:41:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:41:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:41:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:41:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:41:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:41:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:41:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:41:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:41:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:41:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:41:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:41:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:41:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:41:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:41:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:41:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:41:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:41:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:41:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:41:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:41:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:41:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:41:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:41:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:41:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:41:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:41:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:41:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:41:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:41:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:41:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:41:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:41:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:41:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:41:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:41:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:41:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:41:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:41:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:41:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:41:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:41:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:41:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:41:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:41:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:41:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:41:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:41:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:41:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:41:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:41:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:41:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:41:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:41:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:41:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:41:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:41:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:41:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:41:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:41:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:41:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:41:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:41:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:41:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:41:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:41:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:41:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:41:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:41:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:41:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:41:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:41:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:41:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:41:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:41:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:41:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:41:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:41:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:41:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:41:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:41:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:41:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:41:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:41:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:41:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:41:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:41:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:41:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:41:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:41:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:41:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:41:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:41:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:41:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:41:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:41:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:41:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:41:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:41:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:41:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:41:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:41:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:41:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:41:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:41:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:41:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:41:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:41:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:41:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:41:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:41:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:41:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:41:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:41:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:41:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:41:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:41:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:41:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:41:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:41:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:41:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:41:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:41:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:41:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:41:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:41:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:41:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:41:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:41:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:41:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:41:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:41:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:41:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:41:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:41:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:41:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:41:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:41:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:41:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:41:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:41:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:41:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:41:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:41:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:41:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:41:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:41:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:41:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:41:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:41:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:41:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:41:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:41:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:41:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:41:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:41:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:41:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:41:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:41:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:41:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:41:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:41:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:41:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:41:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:41:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:41:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:41:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:41:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:41:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:41:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:41:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:41:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:41:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:41:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:41:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:41:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:41:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:41:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:41:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:41:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:41:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:41:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:41:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:41:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:41:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:41:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:41:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:41:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:41:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:41:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:41:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:41:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:41:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:41:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:41:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:41:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:41:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:41:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:41:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:41:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:41:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:41:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:41:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:41:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:41:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:41:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:41:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:41:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:41:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:41:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:41:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:41:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:42:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:42:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:42:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:42:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:42:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:42:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:42:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:42:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:42:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:42:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:42:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:42:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:42:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:42:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:42:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:42:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:42:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:42:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:42:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:42:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:42:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:42:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:42:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:42:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:42:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:42:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:42:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:42:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:42:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:42:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:42:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:42:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:42:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:42:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:42:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:42:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:42:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:42:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:42:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:42:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:42:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:42:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:42:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:42:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:42:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:42:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:42:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:42:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:42:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:42:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:42:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:42:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:42:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:42:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:42:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:42:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:42:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:42:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:42:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:42:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:42:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:42:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:42:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:42:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:42:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:42:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:42:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:42:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:42:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:42:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:42:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:42:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:42:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:42:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:42:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:42:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:42:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:42:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:42:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:42:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:42:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:42:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:42:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:42:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:42:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:42:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:42:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:42:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:42:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:42:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:42:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:42:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:42:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:42:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:42:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:42:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:42:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:42:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:42:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:42:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:42:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:42:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:42:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:42:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:42:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:42:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:42:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:42:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:42:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:42:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:42:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:42:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:42:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:42:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:42:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:42:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:42:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:42:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:42:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:42:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:42:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:42:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:42:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:42:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:42:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:42:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:42:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:42:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:42:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:42:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:42:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:42:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:42:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:42:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:42:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:42:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:42:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:42:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:42:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:42:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:42:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:42:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:42:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:42:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:42:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:42:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:42:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:42:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:42:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:42:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:42:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:42:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:42:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:42:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:42:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:42:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:42:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:42:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:42:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:42:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:42:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:42:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:42:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:42:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:42:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:42:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:42:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:42:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:42:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:42:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:42:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:42:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:42:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:42:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:42:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:42:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:42:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:42:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:42:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:42:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:42:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:42:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:42:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:42:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:42:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:42:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:42:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:42:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:42:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:42:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:42:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:42:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:42:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:42:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:42:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:42:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:42:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:42:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:42:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:42:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:42:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:42:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:42:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:42:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:42:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:42:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:42:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:42:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:42:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:42:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:42:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:42:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:42:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:42:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:42:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:42:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:43:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:43:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:43:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:43:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:43:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:43:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:43:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:43:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:43:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:43:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:43:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:43:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:43:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:43:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:43:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:43:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:43:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:43:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:43:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:43:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:43:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:43:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:43:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:43:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:43:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:43:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:43:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:43:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:43:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:43:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:43:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:43:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:43:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:43:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:43:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:43:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:43:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:43:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:43:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:43:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:43:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:43:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:43:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:43:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:43:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:43:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:43:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:43:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:43:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:43:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:43:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:43:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:43:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:43:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:43:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:43:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:43:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:43:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:43:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:43:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:43:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:43:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:43:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:43:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:43:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:43:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:43:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:43:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:43:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:43:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:43:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:43:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:43:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:43:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:43:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:43:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:43:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:43:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:43:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:43:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:43:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:43:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:43:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:43:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:43:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:43:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:43:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:43:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:43:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:43:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:43:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:43:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:43:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:43:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:43:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:43:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:43:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:43:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:43:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:43:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:43:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:43:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:43:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:43:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:43:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:43:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:43:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:43:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:43:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:43:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:43:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:43:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:43:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:43:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:43:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:43:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:43:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:43:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:43:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:43:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:43:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:43:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:43:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:43:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:43:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:43:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:43:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:43:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:43:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:43:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:43:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:43:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:43:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:43:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:43:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:43:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:43:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:43:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:43:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:43:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:43:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:43:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:43:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:43:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:43:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:43:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:43:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:43:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:43:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:43:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:43:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:43:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:43:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:43:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:43:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:43:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:43:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:43:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:43:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:43:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:43:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:43:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:43:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:43:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:43:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:43:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:43:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:43:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:43:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:43:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:43:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:43:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:43:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:43:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:43:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:43:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:43:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:43:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:43:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:43:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:43:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:43:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:43:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:43:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:43:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:43:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:43:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:43:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:43:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:43:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:43:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:43:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:43:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:43:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:43:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:43:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:43:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:43:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:43:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:43:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:43:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:43:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:43:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:43:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:43:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:43:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:43:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:43:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:43:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:43:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:43:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:43:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:43:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:43:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:43:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:43:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:44:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:44:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:44:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:44:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:44:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:44:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:44:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:44:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:44:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:44:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:44:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:44:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:44:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:44:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:44:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:44:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:44:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:44:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:44:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:44:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:44:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:44:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:44:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:44:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:44:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:44:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:44:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:44:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:44:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:44:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:44:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:44:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:44:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:44:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:44:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:44:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:44:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:44:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:44:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:44:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:44:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:44:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:44:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:44:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:44:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:44:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:44:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:44:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:44:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:44:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:44:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:44:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:44:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:44:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:44:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:44:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:44:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:44:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:44:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:44:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:44:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:44:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:44:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:44:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:44:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:44:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:44:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:44:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:44:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:44:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:44:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:44:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:44:16 rdma06 systemd-journald[790]: Data hash table of /run/log/journal/95379d2f723847b68d9e976dee8967a9/system.journal has a fill level at 75.0 (163750 of 218332 items, 75497472 file size, 461 bytes per hash table item), suggesting rotation.
Apr 18 18:44:16 rdma06 systemd-journald[790]: /run/log/journal/95379d2f723847b68d9e976dee8967a9/system.journal: Journal header limits reached or header out-of-date, rotating.
Apr 18 18:44:16 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 18 18:44:17 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 18 18:44:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:44:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:44:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:44:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:44:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:44:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:44:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:44:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:44:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:44:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:44:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:44:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:44:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:44:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:44:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:44:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:44:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:44:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:44:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:44:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:44:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:44:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:44:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:44:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:44:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:44:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:44:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:44:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:44:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:44:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:44:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:44:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:44:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:44:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:44:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:44:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:44:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:44:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:44:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:44:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:44:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:44:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:44:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:44:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:44:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:44:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:44:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:44:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:44:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:44:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:44:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:44:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:44:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:44:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:44:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:44:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:44:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:44:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:44:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:44:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:44:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:44:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:44:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:44:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:44:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:44:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:44:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:44:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:44:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:44:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:44:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:44:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:44:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:44:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:44:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:44:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:44:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:44:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:44:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:44:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:44:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:44:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:44:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:44:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:44:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:44:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:44:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:44:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:44:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:44:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:44:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:44:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:44:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:44:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:44:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:44:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:44:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:44:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:44:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:44:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:44:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:44:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:44:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:44:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:44:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:44:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:44:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:44:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:44:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:44:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:44:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:44:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:44:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:44:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:44:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:44:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:44:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:44:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:44:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:44:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:44:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:44:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:44:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:44:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:44:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:44:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:44:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:44:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:44:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:44:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:44:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:44:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:44:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:44:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:44:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:44:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:44:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:44:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:44:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:44:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:44:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:44:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:44:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:44:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:45:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:45:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:45:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:45:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:45:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:45:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:45:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:45:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:45:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:45:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:45:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:45:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:45:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:45:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:45:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:45:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:45:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:45:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:45:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:45:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:45:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:45:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:45:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:45:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:45:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:45:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:45:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:45:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:45:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:45:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:45:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:45:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:45:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:45:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:45:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:45:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:45:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:45:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:45:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:45:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:45:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:45:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:45:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:45:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:45:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:45:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:45:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:45:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:45:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:45:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:45:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:45:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:45:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:45:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:45:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:45:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:45:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:45:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:45:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:45:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:45:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:45:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:45:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:45:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:45:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:45:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:45:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:45:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:45:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:45:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:45:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:45:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:45:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:45:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:45:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:45:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:45:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:45:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:45:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:45:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:45:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:45:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:45:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:45:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:45:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:45:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:45:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:45:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:45:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:45:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:45:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:45:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:45:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:45:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:45:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:45:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:45:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:45:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:45:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:45:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:45:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:45:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:45:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:45:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:45:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:45:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:45:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:45:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:45:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:45:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:45:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:45:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:45:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:45:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:45:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:45:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:45:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:45:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:45:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:45:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:45:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:45:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:45:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:45:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:45:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:45:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:45:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:45:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:45:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:45:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:45:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:45:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:45:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:45:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:45:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:45:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:45:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:45:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:45:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:45:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:45:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:45:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:45:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:45:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:45:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:45:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:45:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:45:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:45:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:45:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:45:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:45:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:45:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:45:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:45:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:45:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:45:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:45:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:45:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:45:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:45:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:45:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:45:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:45:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:45:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:45:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:45:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:45:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:45:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:45:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:45:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:45:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:45:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:45:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:45:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:45:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:45:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:45:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:45:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:45:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:47:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:47:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:47:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:47:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:47:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:47:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:47:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:47:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:47:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:47:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:47:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:47:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:47:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:47:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:47:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:47:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:47:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:47:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:47:57 rdma06 rsyslogd[1041]: imjournal: 5295 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 18:48:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:48:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:48:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:48:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:48:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:48:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:48:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:48:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:48:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:48:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:48:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:48:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:48:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:48:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:48:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:48:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:48:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:48:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:48:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:48:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:48:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:48:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:48:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:48:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:48:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:48:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:48:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:48:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:48:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:48:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:48:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:48:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:48:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:48:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:48:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:48:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:48:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:48:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:48:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:48:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:48:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:48:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:48:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:48:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:48:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:48:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:48:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:48:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:48:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:48:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:48:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:48:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:48:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:48:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:48:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:48:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:48:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:48:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:48:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:48:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:48:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:48:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:48:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:48:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:48:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:48:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:48:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:48:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:48:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:48:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:48:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:48:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:48:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:48:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:48:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:48:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:48:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:48:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:48:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:48:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:48:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:48:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:48:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:48:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:48:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:48:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:48:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:48:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:48:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:48:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:48:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:48:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:48:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:48:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:48:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:48:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:48:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:48:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:48:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:48:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:48:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:48:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:48:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:48:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:48:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:48:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:48:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:48:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:48:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:48:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:48:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:48:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:48:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:48:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:48:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:48:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:48:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:48:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:48:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:48:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:48:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:48:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:48:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:48:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:48:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:48:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:48:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:48:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:48:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:48:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:48:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:48:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:48:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:48:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:48:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:48:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:48:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:48:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:48:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:48:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:48:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:48:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:48:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:48:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:48:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:48:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:48:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:48:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:48:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:48:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:48:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:48:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:48:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:48:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:48:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:48:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:48:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:48:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:48:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:48:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:48:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:48:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:48:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:48:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:48:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:48:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:48:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:48:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:48:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:48:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:48:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:48:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:48:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:48:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:48:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:48:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:48:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:48:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:48:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:48:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:48:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:48:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:48:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:48:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:48:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:48:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:48:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:48:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:48:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:48:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:48:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:48:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:48:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:48:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:48:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:48:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:48:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:48:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:48:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:48:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:48:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:48:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:48:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:48:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:48:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:48:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:48:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:48:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:48:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:48:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:48:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:48:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:48:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:48:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:48:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:48:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:49:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:49:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:49:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:49:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:49:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:49:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:49:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:49:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:49:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:49:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:49:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:49:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:49:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:49:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:49:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:49:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:49:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:49:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:49:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:49:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:49:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:49:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:49:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:49:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:49:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:49:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:49:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:49:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:49:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:49:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:49:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:49:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:49:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:49:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:49:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:49:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:49:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:49:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:49:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:49:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:49:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:49:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:49:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:49:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:49:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:49:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:49:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:49:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:49:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:49:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:49:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:49:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:49:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:49:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:49:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:49:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:49:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:49:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:49:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:49:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:49:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:49:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:49:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:49:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:49:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:49:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:49:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:49:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:49:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:49:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:49:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:49:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:49:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:49:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:49:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:49:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:49:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:49:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:49:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:49:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:49:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:49:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:49:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:49:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:49:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:49:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:49:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:49:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:49:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:49:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:49:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:49:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:49:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:49:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:49:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:49:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:49:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:49:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:49:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:49:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:49:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:49:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:49:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:49:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:49:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:49:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:49:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:49:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:49:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:49:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:49:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:49:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:49:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:49:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:49:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:49:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:49:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:49:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:49:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:49:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:49:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:49:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:49:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:49:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:49:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:49:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:49:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:49:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:49:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:49:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:49:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:49:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:49:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:49:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:49:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:49:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:49:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:49:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:49:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:49:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:49:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:49:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:49:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:49:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:49:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:49:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:49:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:49:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:49:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:49:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:49:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:49:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:49:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:49:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:49:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:49:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:49:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:49:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:49:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:49:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:49:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:49:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:49:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:49:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:49:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:49:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:49:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:49:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:49:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:49:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:49:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:49:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:49:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:49:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:49:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:49:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:49:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:49:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:49:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:49:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:49:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:49:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:49:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:49:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:49:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:49:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:49:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:49:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:49:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:49:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:49:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:49:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:49:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:49:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:49:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:49:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:49:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:49:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:49:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:49:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:49:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:49:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:49:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:49:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:49:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:49:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:49:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:49:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:49:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:49:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:49:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:49:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:49:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:49:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:49:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:49:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:50:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:50:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:50:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:50:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:50:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:50:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:50:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:50:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:50:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:50:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:50:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:50:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:50:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:50:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:50:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:50:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:50:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:50:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:50:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:50:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:50:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:50:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:50:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:50:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:50:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:50:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:50:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:50:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:50:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:50:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:50:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:50:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:50:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:50:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:50:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:50:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:50:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:50:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:50:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:50:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:50:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:50:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:50:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:50:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:50:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:50:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:50:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:50:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:50:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:50:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:50:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:50:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:50:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:50:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:50:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:50:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:50:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:50:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:50:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:50:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:50:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:50:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:50:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:50:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:50:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:50:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:50:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:50:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:50:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:50:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:50:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:50:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:50:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:50:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:50:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:50:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:50:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:50:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:50:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:50:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:50:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:50:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:50:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:50:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:50:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:50:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:50:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:50:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:50:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:50:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:50:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:50:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:50:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:50:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:50:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:50:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:50:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:50:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:50:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:50:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:50:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:50:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:50:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:50:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:50:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:50:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:50:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:50:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:50:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:50:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:50:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:50:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:50:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:50:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:50:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:50:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:50:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:50:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:50:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:50:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:50:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:50:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:50:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:50:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:50:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:50:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:50:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:50:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:50:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:50:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:50:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:50:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:50:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:50:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:50:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:50:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:50:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:50:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:50:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:50:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:50:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:50:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:50:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:50:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:50:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:50:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:50:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:50:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:50:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:50:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:50:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:50:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:50:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:50:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:50:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:50:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:50:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:50:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:50:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:50:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:50:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:50:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:50:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:50:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:50:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:50:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:50:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:50:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:50:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:50:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:50:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:50:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:50:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:50:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:50:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:50:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:50:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:50:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:50:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:50:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:50:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:50:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:50:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:50:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:50:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:50:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:50:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:50:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:50:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:50:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:50:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:50:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:50:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:50:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:50:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:50:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:50:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:50:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:50:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:50:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:50:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:50:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:50:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:50:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:50:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:50:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:50:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:50:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:50:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:50:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:50:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:50:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:50:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:50:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:50:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:50:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:51:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:51:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:51:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:51:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:51:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:51:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:51:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:51:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:51:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:51:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:51:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:51:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:51:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:51:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:51:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:51:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:51:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:51:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:51:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:51:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:51:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:51:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:51:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:51:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:51:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:51:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:51:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:51:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:51:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:51:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:51:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:51:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:51:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:51:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:51:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:51:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:51:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:51:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:51:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:51:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:51:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:51:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:51:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:51:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:51:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:51:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:51:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:51:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:51:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:51:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:51:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:51:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:51:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:51:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:51:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:51:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:51:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:51:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:51:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:51:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:51:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:51:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:51:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:51:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:51:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:51:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:51:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:51:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:51:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:51:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:51:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:51:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:51:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:51:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:51:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:51:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:51:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:51:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:51:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:51:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:51:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:51:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:51:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:51:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:51:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:51:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:51:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:51:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:51:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:51:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:51:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:51:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:51:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:51:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:51:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:51:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:51:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:51:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:51:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:51:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:51:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:51:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:51:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:51:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:51:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:51:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:51:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:51:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:51:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:51:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:51:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:51:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:51:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:51:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:51:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:51:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:51:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:51:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:51:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:51:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:51:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:51:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:51:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:51:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:51:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:51:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:51:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:51:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:51:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:51:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:51:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:51:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:51:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:51:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:51:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:51:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:51:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:51:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:51:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:51:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:51:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:51:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:51:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:51:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:51:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:51:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:51:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:51:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:51:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:51:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:51:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:51:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:51:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:51:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:51:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:51:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:51:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:51:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:51:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:51:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:51:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:51:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:51:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:51:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:51:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:51:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:51:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:51:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:51:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:51:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:51:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:51:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:51:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:51:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:51:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:51:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:51:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:51:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:51:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:51:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:51:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:51:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:51:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:51:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:51:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:51:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:51:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:51:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:51:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:51:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:51:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:51:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:51:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:51:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:51:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:51:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:51:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:51:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:51:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:51:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:51:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:51:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:51:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:51:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:51:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:51:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:51:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:51:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:51:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:51:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:51:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:51:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:51:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:51:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:51:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:51:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:52:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:52:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:52:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:52:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:52:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:52:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:52:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:52:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:52:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:52:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:52:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:52:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:52:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:52:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:52:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:52:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:52:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:52:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:52:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:52:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:52:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:52:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:52:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:52:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:52:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:52:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:52:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:52:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:52:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:52:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:52:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:52:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:52:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:52:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:52:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:52:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:52:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:52:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:52:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:52:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:52:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:52:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:52:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:52:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:52:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:52:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:52:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:52:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:52:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:52:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:52:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:52:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:52:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:52:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:52:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:52:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:52:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:52:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:52:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:52:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:52:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:52:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:52:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:52:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:52:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:52:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:52:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:52:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:52:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:52:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:52:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:52:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:52:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:52:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:52:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:52:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:52:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:52:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:52:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:52:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:52:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:52:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:52:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:52:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:52:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:52:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:52:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:52:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:52:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:52:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:52:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:52:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:52:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:52:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:52:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:52:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:52:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:52:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:52:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:52:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:52:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:52:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:52:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:52:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:52:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:52:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:52:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:52:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:52:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:52:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:52:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:52:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:52:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:52:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:52:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:52:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:52:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:52:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:52:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:52:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:52:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:52:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:52:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:52:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:52:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:52:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:52:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:52:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:52:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:52:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:52:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:52:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:52:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:52:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:52:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:52:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:52:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:52:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:52:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:52:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:52:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:52:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:52:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:52:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:52:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:52:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:52:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:52:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:52:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:52:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:52:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:52:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:52:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:52:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:52:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:52:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:52:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:52:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:52:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:52:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:52:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:52:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:52:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:52:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:52:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:52:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:52:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:52:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:52:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:52:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:52:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:52:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:52:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:52:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:52:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:52:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:52:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:52:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:52:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:52:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:52:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:52:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:52:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:52:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:52:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:52:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:52:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:52:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:52:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:52:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:52:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:52:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:52:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:52:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:52:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:52:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:52:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:52:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:52:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:52:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:52:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:52:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:52:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:52:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:52:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:52:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:52:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:52:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:52:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:52:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:52:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:52:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:52:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:52:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:52:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:52:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:53:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:53:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:53:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:53:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:53:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:53:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:53:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:53:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:53:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:53:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:53:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:53:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:53:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:53:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:53:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:53:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:53:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:53:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:53:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:53:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:53:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:53:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:53:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:53:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:53:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:53:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:53:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:53:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:53:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:53:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:53:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:53:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:53:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:53:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:53:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:53:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:53:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:53:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:53:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:53:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:53:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:53:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:53:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:53:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:53:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:53:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:53:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:53:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:53:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:53:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:53:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:53:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:53:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:53:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:53:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:53:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:53:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:53:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:53:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:53:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:53:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:53:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:53:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:53:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:53:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:53:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:53:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:53:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:53:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:53:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:53:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:53:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:53:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:53:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:53:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:53:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:53:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:53:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:53:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:53:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:53:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:53:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:53:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:53:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:53:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:53:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:53:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:53:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:53:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:53:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:53:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:53:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:53:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:53:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:53:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:53:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:53:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:53:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:53:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:53:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:53:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:53:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:53:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:53:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:53:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:53:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:53:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:53:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:53:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:53:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:53:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:53:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:53:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:53:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:53:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:53:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:53:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:53:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:53:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:53:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:53:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:53:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:53:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:53:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:53:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:53:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:53:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:53:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:53:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:53:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:53:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:53:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:53:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:53:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:53:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:53:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:53:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:53:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:53:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:53:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:53:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:53:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:53:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:53:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:53:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:53:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:53:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:53:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:53:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:53:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:53:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:53:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:53:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:53:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:53:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:53:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:53:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:53:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:53:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:53:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:53:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:53:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:53:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:53:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:53:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:53:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:53:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:53:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:53:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:53:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:53:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:53:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:53:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:53:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:53:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:53:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:53:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:53:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:53:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:53:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:53:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:53:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:53:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:53:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:53:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:53:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:53:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:53:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:53:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:53:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:53:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:53:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:53:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:53:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:53:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:53:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:53:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:53:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:53:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:53:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:53:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:53:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:53:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:53:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:53:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:53:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:53:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:53:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:53:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:53:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:53:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:53:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:53:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:53:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:53:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:53:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:54:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:54:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:54:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:54:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:54:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:54:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:54:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:54:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:54:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:54:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:54:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:54:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:54:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:54:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:54:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:54:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:54:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:54:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:54:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:54:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:54:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:54:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:54:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:54:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:54:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:54:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:54:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:54:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:54:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:54:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:54:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:54:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:54:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:54:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:54:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:54:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:54:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:54:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:54:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:54:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:54:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:54:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:54:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:54:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:54:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:54:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:54:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:54:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:54:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:54:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:54:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:54:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:54:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:54:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:54:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:54:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:54:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:54:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:54:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:54:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:54:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:54:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:54:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:54:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:54:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:54:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:54:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:54:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:54:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:54:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:54:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:54:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:54:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:54:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:54:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:54:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:54:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:54:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:54:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:54:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:54:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:54:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:54:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:54:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:54:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:54:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:54:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:54:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:54:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:54:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:54:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:54:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:54:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:54:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:54:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:54:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:54:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:54:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:54:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:54:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:54:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:54:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:54:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:54:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:54:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:54:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:54:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:54:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:54:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:54:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:54:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:54:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:54:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:54:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:54:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:54:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:54:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:54:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:54:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:54:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:54:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:54:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:54:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:54:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:54:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:54:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:54:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:54:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:54:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:54:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:54:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:54:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:54:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:54:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:54:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:54:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:54:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:54:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:54:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:54:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:54:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:54:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:54:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:54:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:54:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:54:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:54:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:54:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:54:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:54:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:54:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:54:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:54:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:54:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:54:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:54:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:54:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:54:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:54:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:54:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:54:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:54:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:54:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:54:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:54:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:54:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:54:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:54:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:54:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:54:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:54:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:54:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:54:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:54:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:54:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:54:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:54:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:54:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:54:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:54:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:54:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:54:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:54:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:54:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:54:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:54:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:54:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:54:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:54:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:54:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:54:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:54:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:54:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:54:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:54:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:54:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:54:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:54:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:54:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:54:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:54:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:54:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:54:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:54:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:54:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:54:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:54:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:54:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:54:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:54:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:54:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:54:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:54:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:54:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:54:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:54:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:55:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:55:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:55:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:55:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:55:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:55:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:55:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:55:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:55:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:55:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:55:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:55:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:55:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:55:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:55:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:55:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:55:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:55:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:55:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:55:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:55:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:55:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:55:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:55:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:55:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:55:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:55:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:55:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:55:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:55:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:55:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:55:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:55:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:55:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:55:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:55:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:55:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:55:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:55:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:55:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:55:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:55:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:55:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:55:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:55:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:55:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:55:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:55:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:55:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:55:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:55:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:55:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:55:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:55:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:55:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:55:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:55:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:55:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:55:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:55:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:55:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:55:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:55:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:55:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:55:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:55:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:55:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:55:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:55:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:55:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:55:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:55:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:55:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:55:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:55:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:55:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:55:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:55:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:55:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:55:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:55:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:55:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:55:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:55:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:55:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:55:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:55:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:55:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:55:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:55:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:55:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:55:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:55:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:55:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:55:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:55:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:55:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:55:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:55:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:55:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:55:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:55:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:55:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:55:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:55:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:55:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:55:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:55:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:55:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:55:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:55:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:55:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:55:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:55:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:55:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:55:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:55:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:55:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:55:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:55:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:55:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:55:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:55:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:55:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:55:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:55:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:55:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:55:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:55:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:55:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:55:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:55:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:55:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:55:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:55:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:55:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:55:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:55:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:55:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:55:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:55:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:55:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:55:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:55:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:55:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:55:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:55:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:55:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:55:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:55:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:55:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:55:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:55:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:55:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:55:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:55:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:55:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:55:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:55:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:55:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:55:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:55:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:55:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:55:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:55:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:55:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:55:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:55:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:55:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:55:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:55:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:55:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:55:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:55:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:55:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:55:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:55:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:55:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:55:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:55:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:55:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:55:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:55:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:55:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:55:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:55:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:55:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:55:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:55:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:55:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:55:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:55:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:55:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:55:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:55:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:55:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:55:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:55:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:58:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:58:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:58:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:58:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:58:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:58:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:58:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:58:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:58:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:58:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:58:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:58:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:58:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:58:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:58:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:58:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:58:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:58:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:58:02 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 18:58:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:58:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:58:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:58:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:58:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:58:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:58:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:58:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:58:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:58:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:58:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:58:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:58:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:58:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:58:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:58:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:58:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:58:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:58:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:58:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:58:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:58:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:58:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:58:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:58:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:58:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:58:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:58:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:58:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:58:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:58:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:58:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:58:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:58:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:58:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:58:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:58:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:58:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:58:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:58:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:58:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:58:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:58:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:58:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:58:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:58:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:58:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:58:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:58:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:58:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:58:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:58:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:58:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:58:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:58:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:58:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:58:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:58:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:58:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:58:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:58:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:58:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:58:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:58:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:58:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:58:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:58:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:58:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:58:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:58:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:58:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:58:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:58:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:58:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:58:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:58:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:58:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:58:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:58:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:58:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:58:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:58:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:58:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:58:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:58:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:58:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:58:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:58:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:58:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:58:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:58:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:58:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:58:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:58:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:58:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:58:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:58:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:58:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:58:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:58:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:58:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:58:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:58:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:58:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:58:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:58:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:58:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:58:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:58:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:58:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:58:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:58:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:58:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:58:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:58:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:58:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:58:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:58:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:58:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:58:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:58:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:58:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:58:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:58:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:58:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:58:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:58:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:58:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:58:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:58:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:58:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:58:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:58:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:58:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:58:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:58:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:58:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:58:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:58:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:58:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:58:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:58:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:58:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:58:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:58:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:58:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:58:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:58:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:58:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:58:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:58:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:58:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:58:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:58:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:58:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:58:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:58:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:58:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:58:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:58:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:58:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:58:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:58:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:58:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:58:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:58:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:58:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:58:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:58:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:58:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:58:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:58:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:58:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:58:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:58:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:58:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:58:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:58:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:58:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:58:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:58:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:58:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:58:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:58:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:58:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:58:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:58:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:58:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:58:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:58:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:58:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:58:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:58:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:58:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:58:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:58:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:58:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:58:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:59:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:59:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:59:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:59:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:59:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:59:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:59:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:59:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:59:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:59:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:59:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:59:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:59:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:59:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:59:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:59:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:59:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:59:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:59:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:59:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:59:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:59:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:59:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:59:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:59:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:59:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:59:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:59:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:59:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:59:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:59:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:59:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:59:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:59:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:59:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:59:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:59:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:59:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:59:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:59:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:59:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:59:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:59:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:59:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:59:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:59:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:59:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:59:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:59:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:59:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:59:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:59:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:59:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:59:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:59:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:59:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:59:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:59:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:59:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:59:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:59:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:59:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:59:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:59:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:59:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:59:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:59:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:59:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:59:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:59:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:59:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:59:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:59:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:59:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:59:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:59:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:59:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:59:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:59:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:59:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:59:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:59:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:59:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:59:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:59:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:59:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:59:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:59:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:59:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:59:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:59:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:59:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:59:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:59:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:59:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:59:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:59:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:59:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:59:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:59:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:59:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:59:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:59:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:59:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:59:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:59:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:59:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:59:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:59:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:59:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:59:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:59:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:59:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:59:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:59:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:59:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:59:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:59:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:59:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:59:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:59:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:59:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:59:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:59:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:59:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:59:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:59:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:59:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:59:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:59:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:59:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:59:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:59:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:59:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:59:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:59:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:59:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:59:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:59:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:59:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:59:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:59:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:59:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:59:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:59:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:59:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:59:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:59:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:59:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:59:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:59:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:59:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:59:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:59:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:59:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:59:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:59:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:59:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:59:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:59:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:59:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:59:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:59:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:59:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:59:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:59:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:59:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:59:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:59:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:59:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:59:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:59:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:59:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:59:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:59:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:59:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:59:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:59:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:59:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:59:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:59:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:59:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:59:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:59:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:59:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:59:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:59:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:59:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:59:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:59:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:59:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:59:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:59:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:59:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:59:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:59:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:59:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:59:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 18:59:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 18:59:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:59:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:59:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:59:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 18:59:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 18:59:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 18:59:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 18:59:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 18:59:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 18:59:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 18:59:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 18:59:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 18:59:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 18:59:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 18:59:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 18:59:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 18:59:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:00:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:00:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:00:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:00:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:00:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:00:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:00:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:00:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:00:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:00:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:00:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:00:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:00:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:00:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:00:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:00:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:00:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:00:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:00:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:00:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:00:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:00:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:00:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:00:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:00:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:00:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:00:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:00:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:00:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:00:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:00:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:00:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:00:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:00:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:00:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:00:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:00:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:00:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:00:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:00:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:00:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:00:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:00:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:00:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:00:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:00:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:00:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:00:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:00:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:00:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:00:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:00:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:00:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:00:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:00:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:00:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:00:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:00:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:00:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:00:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:00:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:00:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:00:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:00:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:00:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:00:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:00:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:00:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:00:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:00:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:00:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:00:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:00:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:00:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:00:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:00:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:00:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:00:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:00:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:00:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:00:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:00:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:00:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:00:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:00:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:00:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:00:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:00:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:00:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:00:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:00:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:00:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:00:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:00:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:00:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:00:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:00:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:00:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:00:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:00:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:00:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:00:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:00:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:00:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:00:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:00:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:00:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:00:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:00:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:00:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:00:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:00:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:00:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:00:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:00:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:00:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:00:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:00:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:00:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:00:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:00:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:00:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:00:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:00:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:00:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:00:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:00:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:00:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:00:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:00:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:00:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:00:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:00:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:00:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:00:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:00:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:00:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:00:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:00:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:00:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:00:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:00:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:00:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:00:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:00:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:00:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:00:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:00:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:00:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:00:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:00:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:00:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:00:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:00:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:00:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:00:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:00:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:00:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:00:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:00:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:00:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:00:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:00:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:00:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:00:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:00:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:00:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:00:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:00:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:00:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:00:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:00:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:00:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:00:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:00:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:00:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:00:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:00:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:00:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:00:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:00:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:00:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:00:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:00:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:00:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:00:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:00:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:00:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:00:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:00:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:00:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:00:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:00:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:00:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:00:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:00:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:00:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:00:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:00:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:00:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:00:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:00:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:00:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:00:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:00:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:00:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:00:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:00:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:00:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:00:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:00:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:00:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:00:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:00:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:00:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:00:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:01:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:01:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:01:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:01:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:01:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:01:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:01:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:01:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:01:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:01:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:01:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:01:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:01:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:01:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:01:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:01:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:01:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:01:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:01:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:01:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:01:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:01:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:01:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:01:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:01:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:01:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:01:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:01:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:01:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:01:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:01:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:01:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:01:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:01:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:01:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:01:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:01:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:01:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:01:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:01:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:01:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:01:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:01:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:01:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:01:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:01:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:01:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:01:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:01:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:01:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:01:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:01:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:01:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:01:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:01:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:01:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:01:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:01:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:01:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:01:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:01:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:01:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:01:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:01:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:01:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:01:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:01:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:01:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:01:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:01:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:01:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:01:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:01:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:01:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:01:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:01:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:01:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:01:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:01:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:01:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:01:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:01:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:01:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:01:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:01:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:01:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:01:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:01:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:01:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:01:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:01:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:01:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:01:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:01:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:01:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:01:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:01:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:01:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:01:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:01:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:01:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:01:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:01:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:01:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:01:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:01:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:01:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:01:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:01:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:01:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:01:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:01:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:01:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:01:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:01:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:01:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:01:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:01:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:01:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:01:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:01:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:01:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:01:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:01:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:01:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:01:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:01:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:01:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:01:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:01:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:01:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:01:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:01:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:01:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:01:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:01:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:01:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:01:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:01:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:01:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:01:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:01:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:01:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:01:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:01:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:01:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:01:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:01:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:01:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:01:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:01:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:01:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:01:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:01:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:01:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:01:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:01:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:01:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:01:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:01:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:01:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:01:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:01:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:01:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:01:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:01:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:01:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:01:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:01:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:01:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:01:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:01:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:01:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:01:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:01:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:01:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:01:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:01:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:01:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:01:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:01:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:01:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:01:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:01:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:01:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:01:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:01:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:01:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:01:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:01:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:01:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:01:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:01:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:01:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:01:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:01:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:01:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:01:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:01:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:01:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:01:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:01:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:01:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:01:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:01:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:01:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:01:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:01:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:01:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:01:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:01:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:01:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:01:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:01:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:01:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:01:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:02:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:02:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:02:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:02:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:02:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:02:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:02:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:02:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:02:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:02:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:02:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:02:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:02:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:02:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:02:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:02:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:02:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:02:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:02:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:02:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:02:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:02:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:02:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:02:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:02:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:02:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:02:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:02:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:02:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:02:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:02:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:02:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:02:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:02:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:02:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:02:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:02:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:02:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:02:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:02:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:02:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:02:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:02:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:02:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:02:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:02:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:02:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:02:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:02:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:02:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:02:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:02:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:02:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:02:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:02:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:02:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:02:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:02:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:02:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:02:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:02:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:02:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:02:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:02:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:02:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:02:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:02:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:02:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:02:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:02:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:02:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:02:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:02:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:02:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:02:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:02:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:02:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:02:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:02:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:02:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:02:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:02:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:02:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:02:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:02:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:02:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:02:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:02:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:02:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:02:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:02:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:02:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:02:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:02:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:02:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:02:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:02:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:02:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:02:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:02:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:02:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:02:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:02:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:02:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:02:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:02:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:02:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:02:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:02:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:02:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:02:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:02:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:02:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:02:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:02:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:02:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:02:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:02:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:02:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:02:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:02:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:02:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:02:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:02:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:02:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:02:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:02:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:02:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:02:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:02:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:02:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:02:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:02:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:02:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:02:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:02:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:02:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:02:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:02:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:02:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:02:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:02:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:02:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:02:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:02:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:02:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:02:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:02:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:02:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:02:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:02:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:02:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:02:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:02:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:02:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:02:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:02:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:02:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:02:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:02:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:02:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:02:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:02:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:02:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:02:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:02:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:02:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:02:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:02:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:02:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:02:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:02:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:02:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:02:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:02:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:02:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:02:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:02:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:02:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:02:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:02:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:02:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:02:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:02:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:02:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:02:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:02:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:02:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:02:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:02:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:02:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:02:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:02:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:02:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:02:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:02:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:02:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:02:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:02:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:02:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:02:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:02:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:02:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:02:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:02:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:02:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:02:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:02:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:02:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:02:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:02:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:02:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:02:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:02:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:02:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:02:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:03:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:03:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:03:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:03:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:03:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:03:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:03:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:03:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:03:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:03:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:03:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:03:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:03:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:03:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:03:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:03:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:03:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:03:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:03:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:03:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:03:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:03:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:03:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:03:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:03:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:03:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:03:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:03:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:03:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:03:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:03:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:03:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:03:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:03:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:03:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:03:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:03:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:03:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:03:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:03:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:03:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:03:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:03:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:03:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:03:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:03:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:03:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:03:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:03:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:03:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:03:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:03:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:03:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:03:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:03:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:03:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:03:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:03:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:03:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:03:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:03:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:03:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:03:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:03:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:03:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:03:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:03:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:03:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:03:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:03:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:03:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:03:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:03:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:03:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:03:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:03:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:03:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:03:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:03:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:03:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:03:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:03:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:03:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:03:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:03:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:03:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:03:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:03:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:03:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:03:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:03:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:03:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:03:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:03:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:03:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:03:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:03:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:03:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:03:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:03:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:03:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:03:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:03:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:03:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:03:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:03:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:03:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:03:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:03:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:03:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:03:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:03:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:03:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:03:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:03:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:03:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:03:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:03:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:03:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:03:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:03:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:03:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:03:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:03:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:03:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:03:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:03:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:03:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:03:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:03:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:03:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:03:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:03:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:03:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:03:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:03:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:03:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:03:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:03:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:03:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:03:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:03:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:03:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:03:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:03:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:03:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:03:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:03:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:03:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:03:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:03:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:03:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:03:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:03:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:03:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:03:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:03:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:03:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:03:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:03:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:03:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:03:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:03:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:03:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:03:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:03:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:03:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:03:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:03:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:03:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:03:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:03:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:03:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:03:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:03:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:03:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:03:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:03:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:03:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:03:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:03:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:03:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:03:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:03:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:03:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:03:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:03:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:03:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:03:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:03:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:03:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:03:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:03:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:03:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:03:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:03:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:03:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:03:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:03:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:03:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:03:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:03:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:03:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:03:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:03:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:03:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:03:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:03:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:03:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:03:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:03:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:03:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:03:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:03:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:03:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:03:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:04:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:04:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:04:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:04:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:04:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:04:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:04:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:04:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:04:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:04:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:04:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:04:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:04:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:04:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:04:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:04:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:04:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:04:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:04:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:04:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:04:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:04:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:04:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:04:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:04:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:04:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:04:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:04:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:04:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:04:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:04:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:04:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:04:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:04:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:04:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:04:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:04:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:04:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:04:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:04:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:04:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:04:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:04:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:04:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:04:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:04:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:04:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:04:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:04:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:04:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:04:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:04:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:04:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:04:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:04:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:04:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:04:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:04:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:04:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:04:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:04:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:04:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:04:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:04:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:04:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:04:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:04:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:04:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:04:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:04:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:04:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:04:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:04:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:04:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:04:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:04:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:04:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:04:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:04:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:04:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:04:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:04:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:04:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:04:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:04:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:04:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:04:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:04:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:04:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:04:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:04:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:04:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:04:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:04:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:04:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:04:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:04:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:04:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:04:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:04:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:04:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:04:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:04:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:04:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:04:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:04:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:04:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:04:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:04:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:04:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:04:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:04:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:04:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:04:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:04:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:04:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:04:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:04:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:04:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:04:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:04:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:04:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:04:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:04:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:04:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:04:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:04:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:04:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:04:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:04:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:04:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:04:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:04:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:04:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:04:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:04:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:04:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:04:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:04:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:04:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:04:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:04:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:04:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:04:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:04:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:04:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:04:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:04:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:04:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:04:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:04:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:04:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:04:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:04:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:04:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:04:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:04:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:04:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:04:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:04:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:04:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:04:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:04:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:04:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:04:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:04:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:04:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:04:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:04:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:04:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:04:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:04:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:04:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:04:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:04:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:04:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:04:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:04:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:04:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:04:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:04:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:04:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:04:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:04:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:04:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:04:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:04:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:04:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:04:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:04:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:04:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:04:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:04:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:04:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:04:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:04:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:04:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:04:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:04:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:04:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:04:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:04:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:04:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:04:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:04:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:04:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:04:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:04:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:04:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:04:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:04:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:04:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:04:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:04:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:04:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:04:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:05:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:05:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:05:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:05:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:05:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:05:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:05:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:05:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:05:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:05:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:05:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:05:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:05:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:05:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:05:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:05:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:05:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:05:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:05:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:05:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:05:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:05:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:05:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:05:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:05:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:05:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:05:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:05:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:05:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:05:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:05:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:05:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:05:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:05:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:05:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:05:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:05:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:05:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:05:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:05:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:05:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:05:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:05:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:05:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:05:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:05:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:05:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:05:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:05:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:05:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:05:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:05:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:05:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:05:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:05:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:05:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:05:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:05:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:05:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:05:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:05:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:05:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:05:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:05:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:05:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:05:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:05:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:05:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:05:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:05:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:05:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:05:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:05:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:05:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:05:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:05:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:05:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:05:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:05:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:05:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:05:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:05:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:05:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:05:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:05:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:05:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:05:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:05:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:05:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:05:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:05:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:05:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:05:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:05:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:05:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:05:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:05:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:05:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:05:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:05:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:05:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:05:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:05:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:05:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:05:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:05:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:05:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:05:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:05:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:05:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:05:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:05:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:05:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:05:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:05:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:05:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:05:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:05:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:05:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:05:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:05:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:05:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:05:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:05:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:05:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:05:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:05:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:05:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:05:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:05:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:05:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:05:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:05:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:05:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:05:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:05:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:05:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:05:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:05:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:05:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:05:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:05:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:05:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:05:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:05:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:05:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:05:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:05:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:05:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:05:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:05:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:05:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:05:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:05:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:05:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:05:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:05:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:05:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:05:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:05:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:05:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:05:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:05:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:05:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:05:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:05:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:05:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:05:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:05:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:05:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:05:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:05:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:05:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:05:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:05:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:05:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:05:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:05:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:05:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:05:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:05:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:05:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:05:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:05:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:05:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:05:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:05:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:05:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:05:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:05:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:05:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:05:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:05:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:05:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:05:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:05:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:05:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:05:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:05:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:05:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:05:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:05:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:05:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:05:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:05:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:05:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:05:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:05:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:05:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:05:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:05:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:05:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:05:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:05:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:05:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:05:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:08:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:08:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:08:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:08:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:08:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:08:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:08:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:08:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:08:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:08:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:08:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:08:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:08:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:08:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:08:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:08:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:08:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:08:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:08:03 rdma06 rsyslogd[1041]: imjournal: 5086 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 19:08:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:08:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:08:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:08:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:08:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:08:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:08:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:08:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:08:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:08:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:08:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:08:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:08:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:08:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:08:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:08:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:08:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:08:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:08:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:08:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:08:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:08:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:08:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:08:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:08:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:08:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:08:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:08:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:08:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:08:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:08:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:08:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:08:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:08:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:08:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:08:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:08:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:08:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:08:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:08:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:08:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:08:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:08:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:08:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:08:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:08:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:08:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:08:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:08:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:08:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:08:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:08:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:08:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:08:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:08:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:08:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:08:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:08:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:08:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:08:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:08:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:08:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:08:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:08:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:08:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:08:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:08:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:08:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:08:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:08:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:08:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:08:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:08:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:08:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:08:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:08:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:08:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:08:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:08:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:08:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:08:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:08:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:08:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:08:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:08:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:08:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:08:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:08:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:08:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:08:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:08:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:08:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:08:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:08:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:08:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:08:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:08:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:08:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:08:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:08:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:08:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:08:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:08:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:08:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:08:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:08:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:08:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:08:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:08:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:08:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:08:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:08:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:08:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:08:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:08:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:08:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:08:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:08:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:08:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:08:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:08:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:08:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:08:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:08:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:08:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:08:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:08:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:08:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:08:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:08:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:08:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:08:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:08:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:08:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:08:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:08:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:08:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:08:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:08:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:08:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:08:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:08:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:08:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:08:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:08:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:08:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:08:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:08:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:08:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:08:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:08:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:08:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:08:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:08:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:08:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:08:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:08:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:08:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:08:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:08:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:08:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:08:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:08:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:08:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:08:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:08:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:08:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:08:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:08:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:08:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:08:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:08:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:08:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:08:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:08:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:08:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:08:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:08:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:08:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:08:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:08:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:08:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:08:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:08:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:08:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:08:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:08:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:08:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:08:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:08:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:08:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:08:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:08:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:08:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:08:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:08:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:08:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:08:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:09:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:09:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:09:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:09:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:09:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:09:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:09:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:09:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:09:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:09:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:09:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:09:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:09:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:09:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:09:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:09:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:09:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:09:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:09:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:09:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:09:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:09:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:09:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:09:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:09:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:09:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:09:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:09:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:09:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:09:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:09:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:09:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:09:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:09:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:09:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:09:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:09:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:09:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:09:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:09:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:09:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:09:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:09:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:09:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:09:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:09:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:09:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:09:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:09:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:09:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:09:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:09:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:09:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:09:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:09:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:09:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:09:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:09:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:09:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:09:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:09:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:09:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:09:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:09:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:09:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:09:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:09:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:09:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:09:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:09:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:09:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:09:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:09:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:09:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:09:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:09:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:09:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:09:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:09:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:09:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:09:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:09:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:09:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:09:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:09:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:09:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:09:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:09:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:09:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:09:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:09:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:09:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:09:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:09:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:09:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:09:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:09:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:09:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:09:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:09:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:09:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:09:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:09:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:09:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:09:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:09:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:09:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:09:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:09:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:09:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:09:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:09:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:09:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:09:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:09:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:09:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:09:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:09:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:09:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:09:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:09:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:09:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:09:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:09:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:09:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:09:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:09:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:09:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:09:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:09:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:09:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:09:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:09:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:09:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:09:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:09:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:09:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:09:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:09:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:09:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:09:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:09:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:09:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:09:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:09:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:09:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:09:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:09:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:09:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:09:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:09:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:09:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:09:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:09:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:09:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:09:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:09:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:09:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:09:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:09:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:09:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:09:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:09:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:09:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:09:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:09:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:09:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:09:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:09:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:09:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:09:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:09:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:09:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:09:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:09:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:09:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:09:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:09:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:09:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:09:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:09:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:09:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:09:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:09:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:09:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:09:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:09:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:09:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:09:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:09:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:09:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:09:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:09:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:09:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:09:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:09:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:09:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:09:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:09:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:09:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:09:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:09:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:09:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:09:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:09:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:09:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:09:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:09:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:09:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:09:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:09:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:09:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:09:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:09:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:09:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:09:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:10:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:10:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:10:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:10:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:10:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:10:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:10:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:10:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:10:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:10:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:10:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:10:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:10:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:10:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:10:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:10:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:10:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:10:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:10:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:10:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:10:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:10:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:10:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:10:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:10:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:10:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:10:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:10:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:10:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:10:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:10:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:10:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:10:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:10:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:10:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:10:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:10:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:10:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:10:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:10:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:10:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:10:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:10:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:10:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:10:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:10:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:10:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:10:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:10:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:10:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:10:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:10:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:10:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:10:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:10:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:10:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:10:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:10:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:10:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:10:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:10:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:10:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:10:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:10:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:10:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:10:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:10:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:10:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:10:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:10:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:10:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:10:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:10:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:10:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:10:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:10:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:10:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:10:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:10:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:10:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:10:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:10:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:10:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:10:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:10:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:10:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:10:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:10:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:10:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:10:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:10:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:10:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:10:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:10:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:10:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:10:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:10:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:10:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:10:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:10:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:10:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:10:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:10:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:10:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:10:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:10:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:10:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:10:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:10:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:10:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:10:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:10:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:10:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:10:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:10:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:10:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:10:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:10:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:10:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:10:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:10:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:10:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:10:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:10:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:10:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:10:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:10:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:10:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:10:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:10:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:10:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:10:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:10:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:10:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:10:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:10:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:10:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:10:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:10:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:10:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:10:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:10:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:10:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:10:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:10:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:10:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:10:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:10:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:10:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:10:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:10:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:10:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:10:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:10:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:10:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:10:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:10:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:10:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:10:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:10:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:10:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:10:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:10:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:10:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:10:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:10:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:10:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:10:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:10:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:10:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:10:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:10:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:10:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:10:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:10:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:10:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:10:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:10:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:10:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:10:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:10:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:10:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:10:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:10:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:10:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:10:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:10:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:10:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:10:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:10:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:10:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:10:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:10:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:10:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:10:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:10:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:10:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:10:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:10:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:10:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:10:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:10:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:10:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:10:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:10:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:10:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:10:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:10:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:10:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:10:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:10:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:10:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:10:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:10:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:10:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:10:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:11:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:11:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:11:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:11:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:11:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:11:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:11:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:11:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:11:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:11:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:11:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:11:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:11:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:11:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:11:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:11:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:11:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:11:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:11:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:11:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:11:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:11:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:11:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:11:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:11:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:11:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:11:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:11:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:11:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:11:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:11:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:11:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:11:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:11:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:11:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:11:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:11:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:11:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:11:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:11:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:11:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:11:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:11:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:11:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:11:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:11:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:11:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:11:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:11:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:11:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:11:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:11:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:11:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:11:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:11:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:11:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:11:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:11:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:11:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:11:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:11:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:11:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:11:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:11:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:11:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:11:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:11:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:11:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:11:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:11:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:11:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:11:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:11:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:11:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:11:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:11:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:11:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:11:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:11:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:11:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:11:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:11:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:11:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:11:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:11:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:11:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:11:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:11:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:11:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:11:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:11:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:11:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:11:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:11:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:11:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:11:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:11:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:11:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:11:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:11:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:11:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:11:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:11:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:11:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:11:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:11:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:11:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:11:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:11:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:11:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:11:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:11:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:11:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:11:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:11:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:11:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:11:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:11:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:11:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:11:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:11:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:11:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:11:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:11:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:11:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:11:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:11:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:11:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:11:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:11:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:11:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:11:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:11:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:11:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:11:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:11:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:11:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:11:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:11:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:11:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:11:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:11:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:11:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:11:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:11:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:11:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:11:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:11:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:11:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:11:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:11:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:11:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:11:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:11:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:11:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:11:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:11:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:11:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:11:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:11:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:11:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:11:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:11:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:11:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:11:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:11:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:11:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:11:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:11:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:11:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:11:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:11:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:11:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:11:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:11:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:11:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:11:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:11:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:11:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:11:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:11:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:11:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:11:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:11:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:11:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:11:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:11:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:11:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:11:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:11:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:11:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:11:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:11:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:11:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:11:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:11:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:11:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:11:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:11:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:11:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:11:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:11:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:11:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:11:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:11:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:11:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:11:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:11:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:11:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:11:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:11:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:11:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:11:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:11:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:11:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:11:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:12:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:12:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:12:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:12:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:12:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:12:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:12:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:12:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:12:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:12:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:12:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:12:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:12:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:12:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:12:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:12:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:12:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:12:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:12:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:12:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:12:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:12:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:12:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:12:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:12:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:12:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:12:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:12:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:12:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:12:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:12:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:12:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:12:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:12:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:12:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:12:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:12:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:12:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:12:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:12:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:12:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:12:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:12:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:12:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:12:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:12:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:12:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:12:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:12:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:12:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:12:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:12:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:12:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:12:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:12:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:12:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:12:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:12:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:12:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:12:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:12:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:12:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:12:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:12:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:12:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:12:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:12:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:12:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:12:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:12:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:12:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:12:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:12:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:12:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:12:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:12:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:12:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:12:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:12:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:12:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:12:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:12:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:12:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:12:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:12:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:12:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:12:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:12:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:12:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:12:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:12:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:12:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:12:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:12:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:12:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:12:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:12:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:12:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:12:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:12:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:12:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:12:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:12:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:12:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:12:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:12:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:12:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:12:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:12:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:12:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:12:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:12:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:12:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:12:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:12:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:12:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:12:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:12:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:12:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:12:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:12:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:12:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:12:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:12:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:12:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:12:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:12:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:12:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:12:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:12:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:12:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:12:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:12:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:12:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:12:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:12:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:12:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:12:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:12:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:12:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:12:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:12:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:12:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:12:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:12:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:12:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:12:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:12:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:12:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:12:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:12:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:12:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:12:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:12:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:12:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:12:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:12:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:12:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:12:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:12:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:12:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:12:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:12:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:12:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:12:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:12:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:12:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:12:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:12:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:12:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:12:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:12:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:12:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:12:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:12:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:12:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:12:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:12:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:12:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:12:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:12:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:12:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:12:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:12:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:12:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:12:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:12:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:12:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:12:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:12:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:12:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:12:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:12:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:12:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:12:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:12:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:12:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:12:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:12:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:12:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:12:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:12:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:12:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:12:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:12:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:12:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:12:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:12:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:12:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:12:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:12:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:12:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:12:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:12:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:12:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:12:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:13:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:13:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:13:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:13:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:13:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:13:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:13:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:13:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:13:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:13:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:13:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:13:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:13:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:13:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:13:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:13:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:13:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:13:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:13:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:13:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:13:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:13:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:13:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:13:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:13:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:13:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:13:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:13:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:13:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:13:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:13:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:13:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:13:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:13:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:13:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:13:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:13:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:13:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:13:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:13:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:13:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:13:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:13:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:13:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:13:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:13:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:13:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:13:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:13:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:13:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:13:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:13:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:13:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:13:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:13:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:13:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:13:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:13:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:13:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:13:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:13:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:13:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:13:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:13:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:13:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:13:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:13:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:13:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:13:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:13:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:13:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:13:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:13:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:13:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:13:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:13:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:13:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:13:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:13:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:13:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:13:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:13:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:13:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:13:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:13:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:13:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:13:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:13:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:13:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:13:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:13:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:13:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:13:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:13:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:13:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:13:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:13:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:13:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:13:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:13:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:13:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:13:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:13:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:13:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:13:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:13:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:13:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:13:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:13:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:13:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:13:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:13:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:13:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:13:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:13:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:13:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:13:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:13:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:13:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:13:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:13:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:13:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:13:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:13:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:13:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:13:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:13:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:13:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:13:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:13:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:13:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:13:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:13:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:13:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:13:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:13:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:13:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:13:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:13:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:13:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:13:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:13:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:13:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:13:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:13:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:13:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:13:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:13:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:13:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:13:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:13:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:13:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:13:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:13:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:13:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:13:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:13:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:13:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:13:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:13:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:13:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:13:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:13:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:13:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:13:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:13:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:13:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:13:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:13:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:13:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:13:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:13:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:13:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:13:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:13:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:13:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:13:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:13:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:13:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:13:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:13:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:13:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:13:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:13:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:13:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:13:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:13:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:13:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:13:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:13:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:13:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:13:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:13:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:13:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:13:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:13:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:13:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:13:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:13:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:13:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:13:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:13:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:13:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:13:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:13:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:13:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:13:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:13:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:13:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:13:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:13:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:13:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:13:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:13:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:13:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:13:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:14:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:14:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:14:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:14:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:14:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:14:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:14:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:14:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:14:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:14:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:14:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:14:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:14:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:14:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:14:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:14:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:14:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:14:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:14:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:14:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:14:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:14:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:14:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:14:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:14:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:14:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:14:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:14:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:14:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:14:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:14:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:14:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:14:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:14:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:14:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:14:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:14:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:14:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:14:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:14:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:14:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:14:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:14:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:14:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:14:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:14:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:14:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:14:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:14:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:14:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:14:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:14:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:14:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:14:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:14:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:14:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:14:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:14:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:14:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:14:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:14:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:14:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:14:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:14:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:14:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:14:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:14:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:14:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:14:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:14:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:14:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:14:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:14:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:14:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:14:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:14:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:14:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:14:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:14:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:14:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:14:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:14:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:14:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:14:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:14:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:14:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:14:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:14:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:14:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:14:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:14:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:14:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:14:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:14:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:14:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:14:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:14:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:14:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:14:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:14:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:14:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:14:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:14:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:14:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:14:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:14:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:14:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:14:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:14:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:14:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:14:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:14:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:14:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:14:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:14:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:14:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:14:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:14:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:14:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:14:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:14:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:14:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:14:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:14:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:14:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:14:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:14:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:14:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:14:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:14:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:14:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:14:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:14:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:14:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:14:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:14:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:14:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:14:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:14:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:14:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:14:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:14:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:14:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:14:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:14:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:14:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:14:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:14:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:14:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:14:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:14:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:14:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:14:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:14:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:14:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:14:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:14:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:14:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:14:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:14:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:14:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:14:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:14:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:14:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:14:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:14:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:14:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:14:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:14:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:14:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:14:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:14:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:14:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:14:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:14:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:14:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:14:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:14:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:14:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:14:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:14:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:14:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:14:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:14:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:14:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:14:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:14:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:14:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:14:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:14:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:14:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:14:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:14:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:14:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:14:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:14:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:14:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:14:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:14:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:14:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:14:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:14:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:14:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:14:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:14:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:14:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:14:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:14:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:14:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:14:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:14:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:14:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:14:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:14:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:14:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:14:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:15:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:15:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:15:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:15:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:15:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:15:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:15:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:15:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:15:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:15:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:15:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:15:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:15:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:15:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:15:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:15:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:15:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:15:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:15:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:15:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:15:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:15:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:15:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:15:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:15:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:15:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:15:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:15:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:15:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:15:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:15:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:15:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:15:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:15:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:15:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:15:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:15:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:15:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:15:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:15:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:15:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:15:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:15:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:15:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:15:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:15:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:15:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:15:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:15:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:15:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:15:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:15:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:15:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:15:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:15:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:15:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:15:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:15:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:15:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:15:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:15:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:15:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:15:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:15:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:15:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:15:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:15:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:15:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:15:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:15:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:15:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:15:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:15:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:15:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:15:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:15:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:15:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:15:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:15:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:15:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:15:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:15:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:15:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:15:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:15:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:15:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:15:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:15:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:15:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:15:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:15:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:15:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:15:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:15:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:15:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:15:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:15:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:15:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:15:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:15:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:15:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:15:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:15:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:15:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:15:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:15:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:15:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:15:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:15:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:15:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:15:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:15:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:15:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:15:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:15:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:15:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:15:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:15:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:15:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:15:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:15:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:15:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:15:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:15:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:15:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:15:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:15:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:15:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:15:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:15:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:15:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:15:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:15:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:15:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:15:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:15:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:15:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:15:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:15:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:15:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:15:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:15:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:15:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:15:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:15:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:15:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:15:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:15:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:15:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:15:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:15:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:15:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:15:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:15:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:15:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:15:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:15:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:15:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:15:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:15:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:15:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:15:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:15:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:15:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:15:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:15:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:15:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:15:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:15:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:15:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:15:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:15:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:15:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:15:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:15:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:15:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:15:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:15:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:15:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:15:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:15:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:15:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:15:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:15:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:15:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:15:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:15:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:15:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:15:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:15:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:15:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:15:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:15:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:15:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:15:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:15:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:15:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:15:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:15:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:15:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:15:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:15:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:15:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:15:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:15:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:15:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:15:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:15:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:15:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:15:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:15:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:15:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:15:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:15:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:15:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:15:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:18:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:18:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:18:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:18:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:18:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:18:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:18:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:18:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:18:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:18:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:18:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:18:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:18:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:18:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:18:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:18:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:18:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:18:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:18:08 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 19:18:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:18:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:18:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:18:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:18:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:18:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:18:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:18:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:18:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:18:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:18:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:18:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:18:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:18:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:18:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:18:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:18:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:18:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:18:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:18:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:18:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:18:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:18:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:18:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:18:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:18:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:18:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:18:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:18:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:18:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:18:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:18:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:18:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:18:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:18:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:18:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:18:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:18:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:18:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:18:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:18:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:18:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:18:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:18:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:18:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:18:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:18:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:18:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:18:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:18:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:18:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:18:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:18:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:18:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:18:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:18:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:18:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:18:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:18:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:18:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:18:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:18:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:18:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:18:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:18:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:18:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:18:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:18:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:18:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:18:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:18:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:18:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:18:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:18:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:18:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:18:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:18:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:18:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:18:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:18:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:18:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:18:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:18:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:18:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:18:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:18:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:18:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:18:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:18:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:18:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:18:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:18:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:18:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:18:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:18:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:18:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:18:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:18:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:18:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:18:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:18:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:18:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:18:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:18:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:18:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:18:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:18:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:18:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:18:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:18:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:18:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:18:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:18:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:18:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:18:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:18:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:18:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:18:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:18:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:18:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:18:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:18:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:18:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:18:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:18:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:18:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:18:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:18:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:18:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:18:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:18:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:18:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:18:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:18:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:18:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:18:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:18:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:18:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:18:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:18:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:18:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:18:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:18:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:18:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:18:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:18:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:18:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:18:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:18:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:18:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:18:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:18:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:18:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:18:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:18:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:18:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:18:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:18:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:18:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:18:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:18:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:18:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:18:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:18:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:18:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:18:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:18:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:18:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:18:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:18:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:18:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:18:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:18:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:18:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:18:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:18:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:18:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:18:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:18:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:18:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:19:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:19:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:19:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:19:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:19:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:19:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:19:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:19:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:19:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:19:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:19:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:19:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:19:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:19:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:19:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:19:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:19:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:19:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:19:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:19:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:19:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:19:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:19:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:19:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:19:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:19:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:19:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:19:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:19:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:19:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:19:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:19:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:19:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:19:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:19:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:19:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:19:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:19:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:19:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:19:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:19:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:19:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:19:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:19:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:19:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:19:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:19:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:19:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:19:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:19:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:19:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:19:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:19:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:19:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:19:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:19:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:19:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:19:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:19:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:19:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:19:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:19:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:19:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:19:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:19:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:19:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:19:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:19:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:19:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:19:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:19:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:19:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:19:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:19:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:19:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:19:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:19:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:19:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:19:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:19:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:19:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:19:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:19:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:19:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:19:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:19:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:19:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:19:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:19:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:19:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:19:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:19:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:19:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:19:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:19:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:19:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:19:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:19:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:19:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:19:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:19:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:19:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:19:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:19:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:19:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:19:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:19:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:19:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:19:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:19:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:19:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:19:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:19:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:19:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:19:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:19:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:19:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:19:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:19:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:19:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:19:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:19:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:19:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:19:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:19:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:19:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:19:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:19:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:19:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:19:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:19:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:19:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:19:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:19:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:19:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:19:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:19:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:19:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:19:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:19:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:19:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:19:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:19:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:19:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:19:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:19:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:19:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:19:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:19:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:19:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:19:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:19:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:19:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:19:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:19:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:19:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:19:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:19:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:19:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:19:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:19:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:19:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:19:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:19:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:19:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:19:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:19:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:19:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:19:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:19:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:19:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:19:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:19:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:19:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:19:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:19:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:19:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:19:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:19:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:19:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:19:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:19:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:19:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:19:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:19:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:19:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:19:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:19:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:19:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:19:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:19:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:19:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:19:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:19:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:19:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:19:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:19:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:19:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:19:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:19:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:19:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:19:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:19:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:19:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:19:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:19:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:19:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:19:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:19:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:19:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:19:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:19:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:19:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:19:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:19:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:19:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:20:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:20:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:20:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:20:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:20:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:20:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:20:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:20:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:20:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:20:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:20:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:20:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:20:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:20:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:20:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:20:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:20:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:20:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:20:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:20:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:20:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:20:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:20:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:20:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:20:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:20:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:20:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:20:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:20:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:20:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:20:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:20:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:20:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:20:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:20:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:20:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:20:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:20:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:20:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:20:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:20:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:20:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:20:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:20:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:20:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:20:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:20:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:20:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:20:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:20:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:20:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:20:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:20:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:20:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:20:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:20:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:20:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:20:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:20:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:20:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:20:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:20:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:20:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:20:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:20:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:20:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:20:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:20:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:20:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:20:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:20:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:20:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:20:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:20:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:20:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:20:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:20:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:20:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:20:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:20:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:20:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:20:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:20:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:20:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:20:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:20:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:20:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:20:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:20:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:20:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:20:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:20:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:20:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:20:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:20:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:20:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:20:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:20:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:20:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:20:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:20:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:20:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:20:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:20:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:20:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:20:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:20:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:20:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:20:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:20:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:20:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:20:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:20:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:20:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:20:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:20:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:20:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:20:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:20:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:20:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:20:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:20:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:20:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:20:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:20:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:20:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:20:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:20:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:20:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:20:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:20:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:20:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:20:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:20:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:20:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:20:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:20:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:20:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:20:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:20:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:20:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:20:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:20:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:20:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:20:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:20:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:20:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:20:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:20:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:20:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:20:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:20:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:20:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:20:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:20:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:20:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:20:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:20:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:20:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:20:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:20:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:20:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:20:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:20:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:20:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:20:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:20:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:20:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:20:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:20:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:20:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:20:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:20:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:20:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:20:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:20:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:20:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:20:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:20:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:20:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:20:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:20:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:20:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:20:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:20:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:20:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:20:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:20:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:20:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:20:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:20:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:20:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:20:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:20:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:20:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:20:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:20:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:20:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:20:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:20:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:20:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:20:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:20:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:20:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:20:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:20:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:20:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:20:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:20:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:20:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:20:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:20:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:20:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:20:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:20:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:20:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:21:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:21:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:21:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:21:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:21:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:21:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:21:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:21:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:21:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:21:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:21:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:21:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:21:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:21:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:21:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:21:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:21:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:21:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:21:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:21:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:21:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:21:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:21:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:21:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:21:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:21:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:21:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:21:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:21:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:21:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:21:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:21:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:21:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:21:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:21:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:21:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:21:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:21:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:21:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:21:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:21:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:21:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:21:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:21:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:21:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:21:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:21:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:21:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:21:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:21:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:21:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:21:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:21:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:21:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:21:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:21:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:21:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:21:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:21:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:21:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:21:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:21:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:21:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:21:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:21:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:21:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:21:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:21:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:21:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:21:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:21:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:21:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:21:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:21:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:21:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:21:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:21:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:21:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:21:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:21:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:21:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:21:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:21:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:21:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:21:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:21:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:21:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:21:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:21:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:21:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:21:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:21:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:21:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:21:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:21:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:21:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:21:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:21:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:21:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:21:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:21:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:21:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:21:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:21:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:21:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:21:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:21:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:21:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:21:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:21:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:21:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:21:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:21:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:21:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:21:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:21:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:21:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:21:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:21:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:21:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:21:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:21:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:21:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:21:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:21:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:21:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:21:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:21:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:21:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:21:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:21:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:21:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:21:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:21:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:21:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:21:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:21:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:21:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:21:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:21:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:21:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:21:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:21:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:21:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:21:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:21:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:21:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:21:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:21:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:21:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:21:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:21:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:21:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:21:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:21:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:21:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:21:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:21:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:21:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:21:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:21:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:21:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:21:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:21:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:21:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:21:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:21:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:21:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:21:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:21:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:21:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:21:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:21:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:21:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:21:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:21:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:21:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:21:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:21:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:21:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:21:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:21:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:21:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:21:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:21:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:21:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:21:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:21:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:21:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:21:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:21:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:21:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:21:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:21:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:21:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:21:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:21:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:21:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:21:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:21:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:21:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:21:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:21:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:21:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:21:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:21:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:21:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:21:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:21:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:21:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:21:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:21:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:21:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:21:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:21:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:21:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:22:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:22:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:22:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:22:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:22:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:22:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:22:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:22:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:22:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:22:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:22:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:22:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:22:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:22:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:22:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:22:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:22:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:22:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:22:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:22:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:22:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:22:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:22:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:22:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:22:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:22:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:22:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:22:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:22:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:22:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:22:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:22:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:22:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:22:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:22:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:22:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:22:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:22:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:22:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:22:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:22:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:22:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:22:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:22:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:22:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:22:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:22:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:22:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:22:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:22:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:22:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:22:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:22:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:22:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:22:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:22:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:22:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:22:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:22:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:22:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:22:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:22:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:22:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:22:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:22:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:22:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:22:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:22:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:22:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:22:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:22:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:22:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:22:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:22:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:22:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:22:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:22:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:22:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:22:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:22:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:22:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:22:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:22:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:22:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:22:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:22:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:22:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:22:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:22:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:22:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:22:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:22:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:22:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:22:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:22:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:22:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:22:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:22:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:22:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:22:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:22:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:22:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:22:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:22:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:22:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:22:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:22:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:22:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:22:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:22:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:22:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:22:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:22:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:22:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:22:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:22:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:22:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:22:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:22:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:22:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:22:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:22:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:22:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:22:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:22:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:22:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:22:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:22:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:22:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:22:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:22:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:22:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:22:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:22:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:22:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:22:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:22:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:22:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:22:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:22:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:22:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:22:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:22:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:22:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:22:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:22:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:22:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:22:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:22:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:22:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:22:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:22:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:22:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:22:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:22:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:22:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:22:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:22:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:22:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:22:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:22:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:22:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:22:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:22:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:22:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:22:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:22:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:22:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:22:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:22:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:22:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:22:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:22:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:22:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:22:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:22:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:22:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:22:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:22:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:22:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:22:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:22:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:22:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:22:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:22:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:22:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:22:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:22:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:22:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:22:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:22:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:22:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:22:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:22:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:22:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:22:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:22:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:22:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:22:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:22:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:22:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:22:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:22:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:22:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:22:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:22:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:22:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:22:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:22:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:22:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:22:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:22:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:22:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:22:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:22:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:22:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:23:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:23:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:23:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:23:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:23:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:23:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:23:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:23:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:23:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:23:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:23:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:23:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:23:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:23:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:23:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:23:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:23:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:23:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:23:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:23:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:23:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:23:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:23:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:23:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:23:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:23:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:23:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:23:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:23:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:23:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:23:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:23:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:23:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:23:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:23:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:23:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:23:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:23:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:23:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:23:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:23:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:23:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:23:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:23:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:23:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:23:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:23:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:23:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:23:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:23:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:23:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:23:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:23:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:23:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:23:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:23:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:23:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:23:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:23:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:23:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:23:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:23:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:23:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:23:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:23:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:23:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:23:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:23:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:23:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:23:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:23:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:23:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:23:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:23:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:23:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:23:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:23:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:23:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:23:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:23:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:23:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:23:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:23:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:23:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:23:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:23:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:23:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:23:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:23:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:23:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:23:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:23:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:23:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:23:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:23:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:23:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:23:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:23:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:23:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:23:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:23:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:23:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:23:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:23:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:23:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:23:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:23:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:23:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:23:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:23:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:23:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:23:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:23:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:23:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:23:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:23:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:23:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:23:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:23:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:23:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:23:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:23:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:23:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:23:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:23:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:23:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:23:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:23:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:23:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:23:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:23:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:23:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:23:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:23:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:23:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:23:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:23:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:23:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:23:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:23:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:23:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:23:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:23:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:23:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:23:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:23:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:23:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:23:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:23:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:23:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:23:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:23:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:23:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:23:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:23:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:23:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:23:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:23:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:23:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:23:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:23:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:23:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:23:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:23:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:23:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:23:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:24:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:24:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:24:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:24:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:24:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:24:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:24:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:24:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:24:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:24:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:24:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:24:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:24:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:24:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:24:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:24:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:24:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:24:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:24:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:24:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:24:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:24:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:24:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:24:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:24:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:24:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:24:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:24:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:24:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:24:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:24:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:24:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:24:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:24:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:24:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:24:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:24:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:24:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:24:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:24:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:24:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:24:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:24:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:24:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:24:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:24:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:24:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:24:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:24:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:24:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:24:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:24:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:24:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:24:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:24:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:24:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:24:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:24:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:24:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:24:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:24:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:24:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:24:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:24:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:24:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:24:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:24:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:24:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:24:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:24:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:24:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:24:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:24:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:24:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:24:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:24:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:24:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:24:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:24:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:24:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:24:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:24:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:24:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:24:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:24:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:24:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:24:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:24:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:24:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:24:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:24:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:24:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:24:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:24:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:24:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:24:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:24:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:24:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:24:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:24:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:24:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:24:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:24:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:24:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:24:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:24:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:24:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:24:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:24:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:24:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:24:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:24:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:24:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:24:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:24:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:24:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:24:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:24:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:24:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:24:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:24:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:24:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:24:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:24:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:24:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:24:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:24:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:24:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:24:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:24:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:24:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:24:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:24:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:24:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:24:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:24:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:24:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:24:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:24:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:24:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:24:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:24:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:24:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:24:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:24:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:24:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:24:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:24:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:24:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:24:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:24:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:24:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:24:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:24:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:24:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:24:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:24:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:24:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:24:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:24:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:24:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:24:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:24:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:24:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:24:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:24:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:24:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:24:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:24:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:24:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:24:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:24:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:24:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:24:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:24:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:24:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:24:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:24:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:24:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:24:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:24:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:24:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:24:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:24:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:24:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:24:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:24:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:24:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:24:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:24:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:24:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:24:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:24:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:24:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:24:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:24:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:25:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:25:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:25:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:25:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:25:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:25:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:25:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:25:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:25:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:25:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:25:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:25:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:25:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:25:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:25:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:25:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:25:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:25:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:25:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:25:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:25:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:25:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:25:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:25:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:25:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:25:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:25:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:25:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:25:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:25:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:25:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:25:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:25:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:25:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:25:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:25:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:25:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:25:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:25:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:25:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:25:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:25:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:25:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:25:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:25:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:25:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:25:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:25:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:25:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:25:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:25:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:25:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:25:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:25:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:25:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:25:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:25:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:25:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:25:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:25:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:25:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:25:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:25:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:25:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:25:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:25:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:25:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:25:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:25:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:25:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:25:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:25:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:25:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:25:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:25:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:25:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:25:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:25:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:25:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:25:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:25:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:25:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:25:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:25:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:25:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:25:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:25:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:25:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:25:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:25:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:25:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:25:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:25:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:25:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:25:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:25:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:25:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:25:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:25:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:25:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:25:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:25:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:25:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:25:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:25:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:25:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:25:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:25:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:25:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:25:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:25:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:25:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:25:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:25:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:25:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:25:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:25:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:25:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:25:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:25:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:25:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:25:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:25:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:25:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:25:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:25:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:25:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:25:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:25:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:25:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:25:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:25:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:25:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:25:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:25:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:25:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:25:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:25:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:25:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:25:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:25:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:25:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:25:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:25:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:25:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:25:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:25:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:25:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:25:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:25:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:25:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:25:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:25:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:25:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:25:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:25:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:25:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:25:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:25:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:25:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:25:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:25:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:25:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:25:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:25:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:25:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:26:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:26:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:26:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:26:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:26:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:26:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:26:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:26:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:26:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:26:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:26:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:26:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:26:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:26:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:26:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:26:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:26:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:26:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:28:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:28:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:28:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:28:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:28:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:28:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:28:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:28:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:28:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:28:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:28:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:28:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:28:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:28:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:28:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:28:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:28:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:28:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:28:13 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 19:28:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:28:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:28:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:28:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:28:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:28:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:28:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:28:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:28:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:28:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:28:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:28:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:28:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:28:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:28:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:28:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:28:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:28:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:28:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:28:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:28:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:28:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:28:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:28:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:28:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:28:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:28:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:28:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:28:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:28:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:28:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:28:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:28:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:28:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:28:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:28:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:28:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:28:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:28:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:28:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:28:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:28:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:28:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:28:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:28:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:28:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:28:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:28:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:28:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:28:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:28:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:28:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:28:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:28:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:28:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:28:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:28:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:28:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:28:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:28:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:28:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:28:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:28:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:28:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:28:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:28:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:28:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:28:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:28:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:28:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:28:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:28:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:28:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:28:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:28:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:28:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:28:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:28:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:28:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:28:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:28:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:28:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:28:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:28:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:28:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:28:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:28:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:28:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:28:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:28:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:28:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:28:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:28:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:28:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:28:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:28:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:28:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:28:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:28:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:28:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:28:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:28:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:28:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:28:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:28:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:28:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:28:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:28:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:28:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:28:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:28:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:28:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:28:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:28:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:28:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:28:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:28:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:28:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:28:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:28:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:28:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:28:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:28:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:28:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:28:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:28:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:28:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:28:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:28:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:28:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:28:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:28:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:28:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:28:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:28:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:28:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:28:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:28:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:28:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:28:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:28:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:28:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:28:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:28:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:28:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:28:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:28:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:28:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:28:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:28:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:28:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:28:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:28:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:28:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:28:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:28:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:28:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:28:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:28:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:28:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:28:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:28:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:29:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:29:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:29:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:29:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:29:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:29:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:29:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:29:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:29:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:29:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:29:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:29:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:29:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:29:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:29:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:29:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:29:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:29:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:29:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:29:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:29:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:29:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:29:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:29:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:29:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:29:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:29:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:29:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:29:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:29:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:29:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:29:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:29:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:29:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:29:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:29:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:29:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:29:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:29:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:29:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:29:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:29:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:29:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:29:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:29:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:29:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:29:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:29:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:29:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:29:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:29:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:29:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:29:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:29:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:29:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:29:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:29:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:29:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:29:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:29:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:29:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:29:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:29:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:29:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:29:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:29:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:29:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:29:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:29:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:29:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:29:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:29:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:29:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:29:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:29:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:29:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:29:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:29:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:29:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:29:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:29:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:29:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:29:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:29:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:29:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:29:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:29:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:29:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:29:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:29:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:29:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:29:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:29:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:29:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:29:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:29:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:29:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:29:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:29:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:29:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:29:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:29:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:29:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:29:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:29:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:29:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:29:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:29:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:29:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:29:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:29:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:29:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:29:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:29:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:29:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:29:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:29:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:29:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:29:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:29:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:29:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:29:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:29:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:29:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:29:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:29:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:29:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:29:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:29:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:29:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:29:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:29:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:29:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:29:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:29:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:29:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:29:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:29:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:29:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:29:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:29:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:29:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:29:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:29:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:29:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:29:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:29:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:29:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:29:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:29:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:29:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:29:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:29:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:29:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:29:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:29:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:29:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:29:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:29:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:29:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:29:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:29:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:29:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:29:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:29:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:29:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:29:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:29:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:29:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:29:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:29:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:29:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:29:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:29:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:29:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:29:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:29:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:29:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:29:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:29:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:29:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:29:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:29:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:29:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:29:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:29:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:29:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:29:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:29:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:29:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:29:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:29:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:29:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:29:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:29:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:29:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:29:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:29:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:29:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:29:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:29:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:29:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:29:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:29:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:29:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:29:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:29:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:29:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:29:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:29:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:29:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:29:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:29:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:29:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:29:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:29:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:30:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:30:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:30:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:30:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:30:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:30:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:30:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:30:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:30:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:30:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:30:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:30:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:30:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:30:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:30:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:30:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:30:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:30:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:30:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:30:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:30:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:30:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:30:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:30:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:30:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:30:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:30:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:30:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:30:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:30:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:30:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:30:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:30:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:30:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:30:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:30:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:30:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:30:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:30:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:30:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:30:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:30:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:30:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:30:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:30:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:30:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:30:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:30:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:30:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:30:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:30:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:30:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:30:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:30:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:30:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:30:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:30:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:30:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:30:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:30:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:30:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:30:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:30:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:30:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:30:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:30:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:30:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:30:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:30:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:30:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:30:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:30:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:30:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:30:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:30:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:30:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:30:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:30:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:30:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:30:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:30:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:30:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:30:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:30:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:30:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:30:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:30:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:30:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:30:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:30:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:30:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:30:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:30:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:30:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:30:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:30:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:30:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:30:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:30:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:30:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:30:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:30:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:30:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:30:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:30:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:30:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:30:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:30:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:30:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:30:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:30:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:30:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:30:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:30:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:30:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:30:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:31:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:31:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:31:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:31:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:31:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:31:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:31:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:31:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:31:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:31:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:31:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:31:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:31:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:31:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:31:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:31:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:31:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:31:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:31:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:31:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:31:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:31:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:31:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:31:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:31:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:31:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:31:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:31:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:31:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:31:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:31:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:31:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:31:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:31:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:31:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:31:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:31:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:31:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:31:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:31:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:31:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:31:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:31:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:31:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:31:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:31:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:31:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:31:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:31:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:31:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:31:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:31:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:31:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:31:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:31:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:31:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:31:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:31:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:31:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:31:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:31:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:31:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:31:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:31:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:31:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:31:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:31:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:31:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:31:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:31:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:31:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:31:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:31:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:31:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:31:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:31:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:31:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:31:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:31:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:31:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:31:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:31:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:31:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:31:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:31:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:31:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:31:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:31:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:31:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:31:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:31:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:31:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:31:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:31:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:31:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:31:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:31:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:31:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:31:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:31:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:31:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:31:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:31:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:31:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:31:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:31:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:31:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:31:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:31:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:31:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:31:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:31:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:31:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:31:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:31:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:31:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:31:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:31:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:31:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:31:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:31:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:31:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:31:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:31:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:31:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:31:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:31:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:31:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:31:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:31:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:31:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:31:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:31:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:31:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:31:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:31:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:31:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:31:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:31:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:31:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:31:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:31:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:31:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:31:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:31:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:31:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:31:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:31:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:31:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:31:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:31:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:31:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:31:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:31:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:31:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:31:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:31:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:31:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:31:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:31:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:31:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:31:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:31:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:31:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:31:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:31:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:31:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:31:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:31:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:31:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:31:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:31:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:31:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:31:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:31:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:31:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:31:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:31:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:31:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:31:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:31:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:31:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:31:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:31:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:31:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:31:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:31:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:31:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:31:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:31:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:31:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:31:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:31:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:31:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:31:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:31:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:31:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:31:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:31:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:31:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:31:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:31:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:31:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:31:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:31:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:31:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:31:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:31:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:31:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:31:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:31:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:31:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:31:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:31:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:31:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:31:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:32:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:32:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:32:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:32:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:32:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:32:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:32:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:32:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:32:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:32:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:32:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:32:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:32:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:32:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:32:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:32:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:32:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:32:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:32:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:32:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:32:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:32:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:32:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:32:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:32:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:32:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:32:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:32:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:32:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:32:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:32:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:32:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:32:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:32:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:32:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:32:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:32:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:32:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:32:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:32:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:32:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:32:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:32:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:32:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:32:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:32:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:32:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:32:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:32:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:32:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:32:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:32:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:32:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:32:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:32:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:32:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:32:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:32:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:32:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:32:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:32:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:32:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:32:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:32:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:32:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:32:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:32:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:32:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:32:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:32:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:32:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:32:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:32:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:32:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:32:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:32:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:32:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:32:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:32:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:32:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:32:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:32:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:32:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:32:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:32:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:32:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:32:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:32:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:32:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:32:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:32:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:32:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:32:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:32:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:32:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:32:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:32:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:32:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:32:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:32:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:32:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:32:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:32:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:32:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:32:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:32:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:32:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:32:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:32:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:32:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:32:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:32:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:32:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:32:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:32:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:32:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:32:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:32:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:32:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:32:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:32:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:32:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:32:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:32:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:32:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:32:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:32:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:32:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:32:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:32:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:32:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:32:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:32:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:32:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:32:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:32:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:32:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:32:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:32:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:32:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:32:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:32:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:32:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:32:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:32:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:32:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:32:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:32:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:32:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:32:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:32:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:32:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:32:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:32:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:32:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:32:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:32:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:32:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:32:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:32:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:32:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:32:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:32:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:32:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:32:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:32:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:32:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:32:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:32:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:32:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:32:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:32:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:32:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:32:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:32:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:32:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:32:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:32:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:32:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:32:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:32:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:32:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:32:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:32:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:32:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:32:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:32:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:32:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:32:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:32:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:32:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:32:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:32:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:32:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:32:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:32:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:32:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:32:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:32:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:32:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:32:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:32:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:32:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:32:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:32:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:32:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:32:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:32:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:32:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:32:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:32:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:32:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:32:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:32:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:32:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:32:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:33:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:33:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:33:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:33:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:33:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:33:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:33:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:33:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:33:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:33:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:33:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:33:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:33:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:33:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:33:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:33:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:33:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:33:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:33:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:33:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:33:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:33:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:33:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:33:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:33:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:33:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:33:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:33:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:33:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:33:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:33:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:33:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:33:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:33:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:33:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:33:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:33:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:33:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:33:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:33:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:33:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:33:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:33:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:33:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:33:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:33:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:33:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:33:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:33:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:33:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:33:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:33:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:33:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:33:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:33:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:33:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:33:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:33:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:33:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:33:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:33:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:33:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:33:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:33:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:33:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:33:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:33:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:33:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:33:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:33:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:33:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:33:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:33:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:33:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:33:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:33:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:33:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:33:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:33:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:33:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:33:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:33:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:33:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:33:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:33:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:33:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:33:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:33:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:33:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:33:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:33:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:33:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:33:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:33:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:33:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:33:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:33:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:33:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:33:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:33:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:33:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:33:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:33:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:33:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:33:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:33:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:33:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:33:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:33:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:33:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:33:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:33:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:33:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:33:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:33:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:33:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:33:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:33:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:33:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:33:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:33:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:33:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:33:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:33:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:33:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:33:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:33:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:33:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:33:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:33:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:33:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:33:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:33:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:33:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:33:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:33:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:33:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:33:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:33:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:33:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:33:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:33:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:33:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:33:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:33:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:33:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:33:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:33:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:33:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:33:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:33:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:33:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:33:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:33:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:33:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:33:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:33:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:33:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:33:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:33:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:33:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:33:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:33:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:33:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:33:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:33:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:33:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:33:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:33:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:33:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:33:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:33:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:33:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:33:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:33:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:33:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:33:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:33:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:33:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:33:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:33:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:33:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:33:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:33:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:33:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:33:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:33:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:33:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:33:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:33:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:33:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:33:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:33:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:33:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:33:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:33:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:33:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:33:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:33:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:33:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:33:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:33:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:33:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:33:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:33:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:33:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:33:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:33:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:33:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:33:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:33:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:33:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:33:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:33:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:33:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:33:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:34:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:34:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:34:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:34:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:34:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:34:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:34:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:34:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:34:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:34:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:34:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:34:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:34:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:34:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:34:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:34:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:34:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:34:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:34:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:34:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:34:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:34:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:34:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:34:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:34:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:34:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:34:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:34:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:34:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:34:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:34:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:34:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:34:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:34:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:34:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:34:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:34:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:34:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:34:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:34:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:34:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:34:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:34:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:34:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:34:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:34:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:34:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:34:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:34:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:34:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:34:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:34:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:34:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:34:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:34:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:34:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:34:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:34:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:34:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:34:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:34:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:34:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:34:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:34:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:34:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:34:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:34:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:34:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:34:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:34:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:34:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:34:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:34:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:34:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:34:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:34:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:34:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:34:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:34:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:34:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:34:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:34:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:34:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:34:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:34:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:34:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:34:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:34:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:34:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:34:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:34:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:34:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:34:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:34:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:34:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:34:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:34:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:34:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:34:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:34:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:34:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:34:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:34:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:34:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:34:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:34:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:34:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:34:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:34:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:34:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:34:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:34:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:34:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:34:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:34:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:34:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:34:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:34:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:34:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:34:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:34:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:34:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:34:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:34:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:34:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:34:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:34:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:34:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:34:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:34:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:34:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:34:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:34:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:34:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:34:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:34:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:34:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:34:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:34:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:34:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:34:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:34:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:34:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:34:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:34:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:34:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:34:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:34:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:34:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:34:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:34:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:34:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:34:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:34:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:34:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:34:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:34:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:34:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:34:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:34:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:34:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:34:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:34:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:34:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:34:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:34:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:34:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:34:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:34:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:34:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:34:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:34:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:34:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:34:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:34:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:34:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:34:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:34:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:34:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:34:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:34:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:34:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:34:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:34:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:34:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:34:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:34:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:34:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:34:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:34:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:34:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:34:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:34:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:34:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:34:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:34:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:34:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:34:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:34:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:34:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:34:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:34:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:34:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:34:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:34:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:34:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:34:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:34:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:34:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:34:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:34:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:34:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:34:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:34:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:34:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:34:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:35:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:35:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:35:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:35:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:35:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:35:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:35:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:35:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:35:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:35:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:35:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:35:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:35:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:35:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:35:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:35:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:35:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:35:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:35:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:35:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:35:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:35:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:35:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:35:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:35:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:35:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:35:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:35:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:35:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:35:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:35:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:35:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:35:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:35:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:35:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:35:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:35:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:35:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:35:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:35:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:35:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:35:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:35:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:35:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:35:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:35:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:35:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:35:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:35:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:35:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:35:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:35:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:35:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:35:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:35:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:35:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:35:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:35:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:35:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:35:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:35:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:35:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:35:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:35:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:35:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:35:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:35:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:35:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:35:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:35:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:35:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:35:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:35:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:35:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:35:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:35:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:35:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:35:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:35:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:35:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:35:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:35:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:35:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:35:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:35:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:35:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:35:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:35:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:35:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:35:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:35:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:35:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:35:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:35:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:35:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:35:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:35:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:35:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:35:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:35:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:35:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:35:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:35:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:35:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:35:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:35:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:35:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:35:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:35:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:35:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:35:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:35:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:35:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:35:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:35:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:35:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:35:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:35:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:35:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:35:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:35:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:35:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:35:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:35:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:35:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:35:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:35:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:35:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:35:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:35:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:35:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:35:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:35:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:35:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:35:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:35:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:35:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:35:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:35:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:35:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:35:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:35:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:35:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:35:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:35:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:35:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:35:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:35:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:35:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:35:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:35:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:35:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:35:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:35:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:35:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:35:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:35:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:35:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:35:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:35:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:35:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:35:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:35:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:35:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:35:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:35:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:35:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:35:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:35:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:35:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:35:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:35:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:35:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:35:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:35:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:35:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:35:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:35:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:35:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:35:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:35:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:35:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:35:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:35:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:35:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:35:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:35:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:35:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:35:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:35:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:35:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:35:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:35:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:35:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:35:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:35:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:35:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:35:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:35:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:35:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:35:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:35:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:35:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:35:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:35:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:35:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:35:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:35:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:35:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:35:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:35:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:35:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:35:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:35:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:35:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:35:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:36:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:36:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:36:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:36:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:36:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:36:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:36:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:36:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:36:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:36:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:36:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:36:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:36:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:36:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:36:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:36:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:36:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:36:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:36:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:36:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:36:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:36:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:36:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:36:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:36:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:36:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:36:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:36:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:36:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:36:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:36:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:36:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:36:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:36:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:36:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:36:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:38:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:38:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:38:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:38:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:38:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:38:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:38:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:38:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:38:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:38:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:38:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:38:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:38:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:38:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:38:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:38:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:38:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:38:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:38:14 rdma06 rsyslogd[1041]: imjournal: 5082 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 19:38:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:38:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:38:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:38:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:38:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:38:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:38:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:38:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:38:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:38:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:38:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:38:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:38:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:38:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:38:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:38:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:38:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:38:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:38:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:38:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:38:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:38:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:38:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:38:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:38:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:38:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:38:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:38:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:38:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:38:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:38:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:38:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:38:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:38:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:38:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:38:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:38:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:38:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:38:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:38:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:38:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:38:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:38:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:38:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:38:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:38:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:38:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:38:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:38:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:38:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:38:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:38:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:38:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:38:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:38:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:38:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:38:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:38:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:38:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:38:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:38:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:38:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:38:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:38:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:38:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:38:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:38:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:38:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:38:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:38:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:38:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:38:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:38:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:38:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:38:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:38:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:38:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:38:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:38:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:38:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:38:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:38:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:38:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:38:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:38:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:38:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:38:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:38:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:38:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:38:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:38:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:38:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:38:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:38:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:38:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:38:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:38:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:38:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:38:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:38:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:38:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:38:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:38:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:38:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:38:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:38:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:38:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:38:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:38:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:38:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:38:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:38:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:38:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:38:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:38:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:38:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:38:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:38:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:38:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:38:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:38:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:38:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:38:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:38:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:38:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:38:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:38:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:38:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:38:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:38:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:38:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:38:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:38:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:38:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:38:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:38:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:38:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:38:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:38:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:38:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:38:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:38:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:38:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:38:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:38:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:38:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:38:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:38:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:38:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:38:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:38:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:38:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:38:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:38:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:38:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:38:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:38:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:38:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:38:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:38:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:38:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:38:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:39:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:39:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:39:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:39:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:39:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:39:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:39:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:39:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:39:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:39:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:39:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:39:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:39:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:39:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:39:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:39:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:39:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:39:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:39:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:39:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:39:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:39:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:39:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:39:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:39:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:39:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:39:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:39:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:39:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:39:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:39:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:39:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:39:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:39:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:39:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:39:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:39:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:39:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:39:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:39:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:39:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:39:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:39:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:39:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:39:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:39:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:39:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:39:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:39:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:39:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:39:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:39:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:39:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:39:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:39:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:39:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:39:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:39:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:39:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:39:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:39:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:39:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:39:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:39:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:39:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:39:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:39:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:39:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:39:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:39:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:39:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:39:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:39:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:39:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:39:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:39:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:39:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:39:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:39:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:39:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:39:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:39:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:39:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:39:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:39:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:39:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:39:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:39:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:39:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:39:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:39:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:39:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:39:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:39:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:39:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:39:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:39:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:39:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:39:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:39:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:39:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:39:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:39:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:39:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:39:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:39:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:39:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:39:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:39:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:39:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:39:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:39:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:39:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:39:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:39:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:39:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:39:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:39:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:39:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:39:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:39:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:39:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:39:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:39:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:39:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:39:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:39:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:39:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:39:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:39:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:39:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:39:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:39:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:39:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:39:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:39:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:39:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:39:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:39:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:39:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:39:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:39:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:39:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:39:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:39:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:39:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:39:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:39:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:39:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:39:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:39:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:39:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:39:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:39:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:39:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:39:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:39:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:39:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:39:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:39:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:39:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:39:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:39:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:39:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:39:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:39:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:39:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:39:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:39:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:39:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:39:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:39:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:39:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:39:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:39:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:39:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:39:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:39:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:39:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:39:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:39:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:39:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:39:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:39:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:39:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:39:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:39:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:39:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:39:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:39:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:39:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:39:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:39:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:39:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:39:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:39:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:39:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:39:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:39:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:39:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:39:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:39:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:39:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:39:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:39:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:39:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:39:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:39:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:39:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:39:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:39:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:39:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:39:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:39:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:39:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:39:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:40:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:40:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:40:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:40:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:40:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:40:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:40:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:40:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:40:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:40:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:40:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:40:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:40:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:40:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:40:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:40:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:40:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:40:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:40:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:40:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:40:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:40:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:40:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:40:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:40:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:40:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:40:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:40:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:40:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:40:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:40:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:40:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:40:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:40:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:40:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:40:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:40:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:40:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:40:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:40:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:40:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:40:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:40:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:40:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:40:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:40:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:40:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:40:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:40:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:40:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:40:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:40:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:40:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:40:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:40:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:40:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:40:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:40:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:40:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:40:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:40:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:40:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:40:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:40:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:40:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:40:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:40:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:40:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:40:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:40:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:40:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:40:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:40:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:40:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:40:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:40:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:40:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:40:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:40:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:40:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:40:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:40:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:40:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:40:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:40:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:40:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:40:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:40:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:40:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:40:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:40:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:40:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:40:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:40:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:40:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:40:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:40:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:40:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:40:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:40:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:40:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:40:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:40:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:40:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:40:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:40:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:40:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:40:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:40:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:40:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:40:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:40:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:40:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:40:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:40:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:40:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:40:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:40:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:40:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:40:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:40:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:40:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:40:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:40:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:40:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:40:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:40:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:40:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:40:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:40:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:40:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:40:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:40:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:40:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:40:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:40:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:40:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:40:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:40:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:40:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:40:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:40:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:40:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:40:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:40:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:40:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:40:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:40:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:40:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:40:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:40:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:40:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:40:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:40:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:40:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:40:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:40:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:40:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:40:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:40:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:40:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:40:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:40:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:40:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:40:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:40:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:40:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:40:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:40:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:40:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:40:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:40:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:40:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:40:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:40:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:40:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:40:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:40:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:40:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:40:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:40:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:40:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:40:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:40:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:40:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:40:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:40:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:40:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:40:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:40:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:40:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:40:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:40:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:40:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:40:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:40:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:40:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:40:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:40:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:40:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:40:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:40:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:40:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:40:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:40:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:40:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:40:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:40:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:40:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:40:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:40:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:40:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:40:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:40:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:40:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:40:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:41:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:41:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:41:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:41:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:41:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:41:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:41:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:41:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:41:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:41:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:41:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:41:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:41:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:41:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:41:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:41:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:41:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:41:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:41:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:41:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:41:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:41:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:41:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:41:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:41:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:41:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:41:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:41:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:41:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:41:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:41:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:41:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:41:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:41:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:41:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:41:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:41:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:41:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:41:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:41:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:41:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:41:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:41:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:41:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:41:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:41:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:41:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:41:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:41:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:41:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:41:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:41:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:41:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:41:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:41:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:41:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:41:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:41:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:41:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:41:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:41:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:41:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:41:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:41:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:41:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:41:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:41:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:41:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:41:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:41:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:41:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:41:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:41:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:41:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:41:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:41:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:41:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:41:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:41:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:41:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:41:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:41:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:41:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:41:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:41:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:41:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:41:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:41:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:41:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:41:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:41:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:41:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:41:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:41:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:41:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:41:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:41:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:41:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:41:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:41:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:41:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:41:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:41:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:41:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:41:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:41:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:41:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:41:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:41:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:41:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:41:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:41:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:41:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:41:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:41:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:41:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:41:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:41:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:41:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:41:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:41:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:41:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:41:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:41:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:41:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:41:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:41:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:41:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:41:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:41:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:41:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:41:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:41:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:41:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:41:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:41:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:41:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:41:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:41:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:41:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:41:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:41:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:41:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:41:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:41:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:41:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:41:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:41:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:41:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:41:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:41:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:41:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:41:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:41:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:41:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:41:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:41:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:41:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:41:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:41:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:41:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:41:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:41:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:41:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:41:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:41:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:41:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:41:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:41:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:41:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:41:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:41:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:41:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:41:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:41:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:41:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:41:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:41:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:41:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:41:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:41:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:41:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:41:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:41:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:41:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:41:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:41:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:41:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:41:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:41:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:41:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:41:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:41:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:41:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:41:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:41:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:41:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:41:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:41:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:41:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:41:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:41:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:41:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:41:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:41:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:41:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:41:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:41:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:41:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:41:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:41:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:41:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:41:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:41:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:41:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:41:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:42:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:42:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:42:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:42:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:42:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:42:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:42:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:42:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:42:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:42:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:42:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:42:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:42:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:42:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:42:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:42:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:42:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:42:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:42:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:42:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:42:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:42:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:42:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:42:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:42:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:42:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:42:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:42:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:42:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:42:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:42:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:42:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:42:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:42:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:42:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:42:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:42:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:42:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:42:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:42:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:42:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:42:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:42:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:42:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:42:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:42:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:42:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:42:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:42:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:42:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:42:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:42:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:42:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:42:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:42:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:42:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:42:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:42:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:42:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:42:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:42:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:42:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:42:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:42:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:42:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:42:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:42:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:42:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:42:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:42:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:42:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:42:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:42:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:42:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:42:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:42:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:42:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:42:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:42:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:42:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:42:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:42:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:42:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:42:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:42:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:42:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:42:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:42:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:42:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:42:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:42:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:42:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:42:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:42:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:42:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:42:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:42:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:42:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:42:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:42:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:42:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:42:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:42:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:42:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:42:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:42:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:42:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:42:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:42:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:42:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:42:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:42:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:42:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:42:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:42:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:42:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:42:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:42:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:42:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:42:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:42:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:42:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:42:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:42:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:42:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:42:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:42:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:42:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:42:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:42:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:42:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:42:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:42:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:42:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:42:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:42:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:42:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:42:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:42:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:42:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:42:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:42:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:42:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:42:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:42:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:42:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:42:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:42:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:42:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:42:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:42:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:42:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:42:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:42:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:42:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:42:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:42:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:42:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:42:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:42:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:42:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:42:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:42:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:42:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:42:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:42:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:42:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:42:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:42:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:42:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:42:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:42:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:42:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:42:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:42:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:42:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:42:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:42:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:42:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:42:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:42:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:42:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:42:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:42:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:42:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:42:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:42:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:42:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:42:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:42:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:42:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:42:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:42:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:42:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:42:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:42:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:42:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:42:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:42:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:42:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:42:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:42:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:42:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:42:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:42:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:42:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:42:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:42:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:42:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:42:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:42:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:42:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:42:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:42:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:42:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:42:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:43:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:43:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:43:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:43:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:43:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:43:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:43:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:43:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:43:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:43:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:43:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:43:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:43:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:43:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:43:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:43:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:43:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:43:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:43:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:43:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:43:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:43:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:43:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:43:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:43:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:43:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:43:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:43:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:43:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:43:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:43:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:43:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:43:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:43:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:43:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:43:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:43:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:43:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:43:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:43:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:43:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:43:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:43:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:43:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:43:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:43:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:43:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:43:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:43:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:43:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:43:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:43:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:43:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:43:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:43:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:43:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:43:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:43:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:43:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:43:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:43:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:43:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:43:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:43:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:43:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:43:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:43:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:43:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:43:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:43:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:43:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:43:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:43:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:43:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:43:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:43:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:43:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:43:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:43:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:43:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:43:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:43:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:43:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:43:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:43:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:43:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:43:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:43:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:43:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:43:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:43:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:43:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:43:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:43:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:43:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:43:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:43:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:43:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:43:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:43:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:43:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:43:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:43:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:43:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:43:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:43:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:43:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:43:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:43:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:43:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:43:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:43:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:43:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:43:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:43:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:43:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:43:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:43:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:43:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:43:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:43:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:43:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:43:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:43:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:43:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:43:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:43:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:43:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:43:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:43:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:43:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:43:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:43:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:43:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:43:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:43:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:43:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:43:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:43:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:43:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:43:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:43:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:43:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:43:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:43:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:43:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:43:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:43:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:43:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:43:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:43:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:43:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:43:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:43:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:43:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:43:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:43:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:43:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:43:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:43:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:43:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:43:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:43:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:43:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:43:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:43:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:43:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:43:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:43:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:43:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:43:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:43:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:43:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:43:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:43:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:43:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:43:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:43:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:43:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:43:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:43:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:43:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:43:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:43:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:43:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:43:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:43:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:43:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:43:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:43:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:43:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:43:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:43:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:43:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:43:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:43:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:43:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:43:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:43:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:43:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:43:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:43:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:43:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:43:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:43:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:43:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:43:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:43:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:43:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:43:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:43:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:43:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:43:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:43:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:43:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:43:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:44:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:44:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:44:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:44:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:44:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:44:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:44:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:44:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:44:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:44:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:44:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:44:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:44:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:44:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:44:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:44:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:44:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:44:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:44:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:44:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:44:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:44:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:44:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:44:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:44:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:44:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:44:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:44:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:44:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:44:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:44:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:44:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:44:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:44:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:44:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:44:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:44:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:44:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:44:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:44:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:44:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:44:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:44:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:44:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:44:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:44:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:44:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:44:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:44:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:44:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:44:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:44:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:44:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:44:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:44:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:44:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:44:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:44:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:44:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:44:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:44:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:44:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:44:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:44:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:44:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:44:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:44:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:44:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:44:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:44:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:44:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:44:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:44:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:44:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:44:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:44:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:44:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:44:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:44:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:44:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:44:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:44:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:44:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:44:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:44:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:44:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:44:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:44:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:44:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:44:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:44:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:44:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:44:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:44:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:44:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:44:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:44:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:44:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:44:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:44:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:44:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:44:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:44:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:44:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:44:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:44:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:44:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:44:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:44:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:44:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:44:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:44:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:44:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:44:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:44:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:44:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:44:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:44:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:44:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:44:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:44:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:44:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:44:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:44:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:44:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:44:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:44:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:44:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:44:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:44:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:44:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:44:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:44:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:44:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:44:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:44:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:44:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:44:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:44:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:44:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:44:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:44:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:44:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:44:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:44:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:44:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:44:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:44:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:44:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:44:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:44:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:44:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:44:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:44:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:44:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:44:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:44:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:44:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:44:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:44:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:44:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:44:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:44:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:44:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:44:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:44:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:44:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:44:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:44:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:44:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:44:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:44:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:44:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:44:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:44:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:44:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:44:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:44:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:44:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:44:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:44:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:44:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:44:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:44:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:44:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:44:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:44:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:44:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:44:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:44:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:44:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:44:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:44:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:44:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:44:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:44:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:44:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:44:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:44:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:44:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:44:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:44:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:44:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:44:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:44:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:44:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:44:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:44:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:44:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:44:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:44:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:44:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:44:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:44:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:44:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:44:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:45:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:45:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:45:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:45:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:45:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:45:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:45:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:45:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:45:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:45:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:45:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:45:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:45:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:45:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:45:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:45:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:45:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:45:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:45:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:45:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:45:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:45:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:45:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:45:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:45:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:45:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:45:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:45:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:45:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:45:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:45:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:45:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:45:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:45:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:45:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:45:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:45:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:45:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:45:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:45:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:45:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:45:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:45:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:45:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:45:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:45:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:45:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:45:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:45:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:45:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:45:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:45:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:45:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:45:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:45:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:45:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:45:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:45:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:45:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:45:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:45:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:45:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:45:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:45:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:45:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:45:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:45:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:45:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:45:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:45:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:45:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:45:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:45:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:45:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:45:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:45:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:45:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:45:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:45:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:45:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:45:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:45:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:45:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:45:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:45:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:45:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:45:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:45:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:45:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:45:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:45:23 rdma06 systemd-journald[790]: Data hash table of /run/log/journal/95379d2f723847b68d9e976dee8967a9/system.journal has a fill level at 75.0 (163750 of 218332 items, 75497472 file size, 461 bytes per hash table item), suggesting rotation.
Apr 18 19:45:23 rdma06 systemd-journald[790]: /run/log/journal/95379d2f723847b68d9e976dee8967a9/system.journal: Journal header limits reached or header out-of-date, rotating.
Apr 18 19:45:23 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 18 19:45:24 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 18 19:45:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:45:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:45:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:45:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:45:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:45:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:45:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:45:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:45:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:45:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:45:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:45:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:45:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:45:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:45:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:45:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:45:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:45:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:45:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:45:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:45:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:45:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:45:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:45:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:45:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:45:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:45:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:45:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:45:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:45:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:45:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:45:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:45:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:45:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:45:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:45:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:45:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:45:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:45:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:45:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:45:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:45:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:45:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:45:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:45:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:45:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:45:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:45:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:45:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:45:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:45:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:45:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:45:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:45:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:45:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:45:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:45:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:45:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:45:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:45:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:45:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:45:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:45:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:45:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:45:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:45:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:45:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:45:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:45:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:45:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:45:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:45:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:45:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:45:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:45:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:45:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:45:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:45:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:45:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:45:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:45:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:45:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:45:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:45:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:45:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:45:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:45:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:45:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:45:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:45:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:45:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:45:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:45:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:45:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:45:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:45:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:45:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:45:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:45:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:45:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:45:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:45:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:45:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:45:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:45:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:45:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:45:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:45:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:45:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:45:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:45:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:45:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:45:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:45:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:45:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:45:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:45:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:45:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:45:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:45:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:45:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:45:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:45:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:45:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:45:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:45:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:46:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:46:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:46:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:46:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:46:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:46:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:46:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:46:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:46:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:46:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:46:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:46:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:46:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:46:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:46:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:46:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:46:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:46:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:46:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:46:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:46:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:46:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:46:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:46:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:46:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:46:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:46:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:46:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:46:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:46:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:46:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:46:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:46:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:46:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:46:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:46:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:48:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:48:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:48:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:48:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:48:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:48:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:48:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:48:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:48:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:48:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:48:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:48:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:48:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:48:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:48:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:48:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:48:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:48:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:48:19 rdma06 rsyslogd[1041]: imjournal: 5295 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 19:48:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:48:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:48:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:48:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:48:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:48:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:48:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:48:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:48:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:48:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:48:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:48:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:48:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:48:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:48:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:48:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:48:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:48:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:48:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:48:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:48:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:48:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:48:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:48:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:48:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:48:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:48:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:48:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:48:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:48:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:48:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:48:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:48:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:48:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:48:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:48:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:48:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:48:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:48:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:48:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:48:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:48:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:48:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:48:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:48:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:48:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:48:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:48:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:48:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:48:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:48:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:48:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:48:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:48:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:48:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:48:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:48:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:48:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:48:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:48:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:48:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:48:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:48:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:48:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:48:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:48:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:48:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:48:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:48:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:48:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:48:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:48:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:48:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:48:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:48:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:48:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:48:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:48:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:48:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:48:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:48:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:48:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:48:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:48:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:48:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:48:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:48:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:48:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:48:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:48:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:48:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:48:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:48:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:48:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:48:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:48:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:48:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:48:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:48:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:48:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:48:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:48:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:48:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:48:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:48:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:48:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:48:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:48:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:48:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:48:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:48:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:48:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:48:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:48:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:48:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:48:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:48:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:48:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:48:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:48:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:48:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:48:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:48:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:48:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:48:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:48:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:48:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:48:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:48:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:48:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:48:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:48:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:48:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:48:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:48:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:48:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:48:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:48:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:48:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:48:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:48:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:48:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:48:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:48:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:49:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:49:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:49:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:49:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:49:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:49:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:49:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:49:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:49:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:49:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:49:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:49:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:49:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:49:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:49:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:49:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:49:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:49:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:49:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:49:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:49:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:49:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:49:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:49:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:49:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:49:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:49:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:49:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:49:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:49:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:49:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:49:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:49:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:49:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:49:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:49:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:49:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:49:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:49:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:49:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:49:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:49:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:49:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:49:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:49:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:49:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:49:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:49:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:49:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:49:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:49:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:49:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:49:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:49:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:49:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:49:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:49:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:49:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:49:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:49:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:49:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:49:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:49:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:49:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:49:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:49:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:49:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:49:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:49:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:49:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:49:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:49:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:49:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:49:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:49:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:49:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:49:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:49:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:49:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:49:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:49:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:49:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:49:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:49:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:49:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:49:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:49:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:49:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:49:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:49:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:49:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:49:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:49:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:49:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:49:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:49:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:49:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:49:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:49:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:49:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:49:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:49:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:49:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:49:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:49:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:49:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:49:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:49:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:49:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:49:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:49:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:49:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:49:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:49:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:49:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:49:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:49:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:49:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:49:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:49:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:49:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:49:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:49:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:49:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:49:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:49:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:49:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:49:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:49:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:49:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:49:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:49:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:49:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:49:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:49:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:49:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:49:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:49:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:49:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:49:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:49:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:49:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:49:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:49:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:49:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:49:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:49:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:49:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:49:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:49:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:49:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:49:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:49:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:49:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:49:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:49:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:49:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:49:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:49:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:49:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:49:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:49:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:49:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:49:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:49:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:49:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:49:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:49:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:49:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:49:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:49:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:49:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:49:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:49:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:49:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:49:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:49:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:49:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:49:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:49:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:49:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:49:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:49:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:49:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:49:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:49:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:49:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:49:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:49:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:49:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:49:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:49:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:49:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:49:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:49:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:49:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:49:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:49:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:49:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:49:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:49:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:49:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:49:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:49:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:49:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:49:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:50:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:50:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:50:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:50:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:50:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:50:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:50:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:50:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:50:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:50:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:50:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:50:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:50:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:50:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:50:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:50:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:50:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:50:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:50:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:50:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:50:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:50:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:50:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:50:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:50:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:50:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:50:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:50:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:50:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:50:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:50:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:50:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:50:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:50:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:50:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:50:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:50:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:50:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:50:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:50:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:50:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:50:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:50:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:50:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:50:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:50:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:50:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:50:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:50:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:50:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:50:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:50:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:50:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:50:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:50:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:50:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:50:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:50:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:50:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:50:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:50:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:50:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:50:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:50:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:50:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:50:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:50:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:50:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:50:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:50:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:50:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:50:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:50:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:50:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:50:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:50:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:50:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:50:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:50:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:50:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:50:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:50:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:50:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:50:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:50:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:50:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:50:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:50:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:50:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:50:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:50:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:50:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:50:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:50:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:50:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:50:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:50:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:50:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:50:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:50:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:50:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:50:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:50:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:50:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:50:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:50:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:50:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:50:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:50:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:50:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:50:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:50:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:50:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:50:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:50:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:50:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:50:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:50:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:50:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:50:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:50:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:50:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:50:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:50:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:50:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:50:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:50:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:50:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:50:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:50:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:50:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:50:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:50:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:50:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:50:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:50:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:50:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:50:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:50:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:50:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:50:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:50:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:50:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:50:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:50:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:50:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:50:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:50:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:50:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:50:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:50:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:50:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:50:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:50:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:50:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:50:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:50:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:50:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:50:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:50:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:50:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:50:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:50:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:50:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:50:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:50:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:50:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:50:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:50:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:50:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:50:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:50:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:50:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:50:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:50:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:50:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:50:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:50:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:50:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:50:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:50:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:50:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:50:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:50:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:50:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:50:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:50:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:50:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:50:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:50:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:50:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:50:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:50:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:50:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:50:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:50:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:50:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:50:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:50:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:50:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:50:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:50:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:50:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:50:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:50:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:50:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:50:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:50:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:50:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:50:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:50:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:50:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:50:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:50:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:50:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:50:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:51:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:51:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:51:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:51:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:51:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:51:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:51:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:51:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:51:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:51:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:51:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:51:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:51:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:51:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:51:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:51:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:51:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:51:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:51:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:51:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:51:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:51:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:51:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:51:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:51:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:51:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:51:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:51:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:51:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:51:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:51:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:51:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:51:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:51:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:51:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:51:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:51:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:51:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:51:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:51:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:51:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:51:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:51:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:51:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:51:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:51:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:51:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:51:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:51:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:51:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:51:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:51:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:51:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:51:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:51:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:51:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:51:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:51:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:51:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:51:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:51:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:51:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:51:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:51:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:51:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:51:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:51:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:51:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:51:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:51:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:51:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:51:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:51:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:51:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:51:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:51:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:51:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:51:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:51:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:51:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:51:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:51:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:51:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:51:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:51:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:51:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:51:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:51:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:51:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:51:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:51:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:51:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:51:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:51:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:51:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:51:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:51:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:51:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:51:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:51:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:51:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:51:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:51:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:51:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:51:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:51:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:51:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:51:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:51:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:51:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:51:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:51:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:51:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:51:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:51:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:51:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:51:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:51:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:51:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:51:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:51:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:51:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:51:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:51:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:51:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:51:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:51:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:51:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:51:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:51:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:51:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:51:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:51:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:51:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:51:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:51:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:51:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:51:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:51:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:51:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:51:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:51:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:51:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:51:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:51:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:51:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:51:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:51:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:51:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:51:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:51:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:51:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:51:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:51:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:51:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:51:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:51:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:51:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:51:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:51:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:51:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:51:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:51:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:51:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:51:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:51:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:51:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:51:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:51:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:51:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:51:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:51:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:51:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:51:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:51:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:51:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:51:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:51:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:51:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:51:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:51:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:51:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:51:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:51:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:51:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:51:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:51:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:51:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:51:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:51:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:51:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:51:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:51:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:51:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:51:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:51:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:51:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:51:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:51:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:51:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:51:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:51:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:51:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:51:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:51:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:51:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:51:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:51:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:51:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:51:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:51:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:51:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:51:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:51:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:51:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:51:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:52:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:52:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:52:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:52:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:52:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:52:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:52:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:52:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:52:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:52:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:52:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:52:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:52:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:52:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:52:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:52:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:52:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:52:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:52:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:52:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:52:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:52:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:52:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:52:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:52:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:52:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:52:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:52:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:52:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:52:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:52:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:52:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:52:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:52:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:52:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:52:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:52:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:52:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:52:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:52:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:52:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:52:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:52:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:52:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:52:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:52:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:52:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:52:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:52:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:52:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:52:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:52:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:52:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:52:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:52:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:52:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:52:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:52:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:52:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:52:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:52:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:52:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:52:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:52:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:52:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:52:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:52:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:52:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:52:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:52:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:52:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:52:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:52:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:52:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:52:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:52:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:52:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:52:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:52:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:52:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:52:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:52:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:52:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:52:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:52:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:52:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:52:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:52:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:52:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:52:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:52:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:52:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:52:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:52:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:52:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:52:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:52:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:52:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:52:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:52:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:52:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:52:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:52:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:52:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:52:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:52:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:52:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:52:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:52:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:52:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:52:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:52:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:52:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:52:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:52:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:52:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:52:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:52:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:52:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:52:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:52:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:52:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:52:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:52:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:52:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:52:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:52:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:52:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:52:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:52:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:52:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:52:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:52:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:52:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:52:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:52:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:52:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:52:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:52:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:52:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:52:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:52:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:52:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:52:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:52:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:52:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:52:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:52:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:52:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:52:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:52:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:52:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:52:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:52:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:52:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:52:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:52:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:52:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:52:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:52:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:52:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:52:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:52:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:52:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:52:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:52:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:52:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:52:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:52:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:52:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:52:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:52:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:52:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:52:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:52:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:52:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:52:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:52:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:52:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:52:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:52:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:52:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:52:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:52:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:52:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:52:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:52:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:52:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:52:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:52:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:52:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:52:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:52:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:52:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:52:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:52:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:52:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:52:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:52:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:52:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:52:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:52:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:52:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:52:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:52:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:52:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:52:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:52:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:52:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:52:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:52:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:52:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:52:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:52:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:52:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:52:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:53:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:53:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:53:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:53:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:53:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:53:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:53:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:53:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:53:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:53:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:53:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:53:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:53:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:53:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:53:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:53:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:53:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:53:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:53:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:53:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:53:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:53:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:53:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:53:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:53:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:53:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:53:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:53:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:53:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:53:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:53:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:53:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:53:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:53:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:53:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:53:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:53:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:53:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:53:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:53:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:53:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:53:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:53:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:53:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:53:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:53:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:53:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:53:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:53:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:53:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:53:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:53:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:53:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:53:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:53:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:53:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:53:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:53:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:53:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:53:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:53:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:53:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:53:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:53:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:53:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:53:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:53:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:53:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:53:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:53:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:53:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:53:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:53:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:53:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:53:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:53:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:53:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:53:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:53:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:53:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:53:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:53:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:53:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:53:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:53:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:53:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:53:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:53:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:53:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:53:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:53:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:53:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:53:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:53:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:53:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:53:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:53:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:53:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:53:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:53:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:53:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:53:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:53:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:53:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:53:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:53:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:53:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:53:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:53:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:53:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:53:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:53:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:53:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:53:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:53:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:53:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:53:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:53:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:53:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:53:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:53:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:53:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:53:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:53:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:53:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:53:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:53:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:53:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:53:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:53:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:53:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:53:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:53:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:53:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:53:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:53:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:54:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:54:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:54:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:54:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:54:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:54:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:54:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:54:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:54:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:54:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:54:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:54:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:54:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:54:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:54:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:54:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:54:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:54:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:54:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:54:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:54:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:54:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:54:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:54:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:54:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:54:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:54:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:54:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:54:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:54:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:54:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:54:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:54:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:54:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:54:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:54:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:54:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:54:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:54:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:54:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:54:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:54:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:54:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:54:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:54:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:54:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:54:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:54:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:54:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:54:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:54:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:54:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:54:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:54:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:54:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:54:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:54:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:54:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:54:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:54:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:54:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:54:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:54:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:54:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:54:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:54:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:54:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:54:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:54:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:54:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:54:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:54:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:54:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:54:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:54:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:54:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:54:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:54:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:54:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:54:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:54:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:54:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:54:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:54:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:54:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:54:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:54:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:54:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:54:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:54:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:54:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:54:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:54:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:54:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:54:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:54:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:54:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:54:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:54:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:54:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:54:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:54:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:54:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:54:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:54:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:54:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:54:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:54:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:54:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:54:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:54:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:54:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:54:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:54:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:54:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:54:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:54:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:54:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:54:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:54:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:54:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:54:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:54:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:54:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:54:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:54:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:54:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:54:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:54:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:54:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:54:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:54:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:54:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:54:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:54:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:54:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:54:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:54:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:54:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:54:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:54:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:54:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:54:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:54:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:54:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:54:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:54:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:54:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:54:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:54:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:54:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:54:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:54:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:54:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:54:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:54:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:54:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:54:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:54:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:54:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:54:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:54:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:54:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:54:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:54:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:54:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:54:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:54:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:54:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:54:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:54:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:54:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:54:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:54:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:54:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:54:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:55:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:55:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:55:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:55:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:55:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:55:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:55:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:55:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:55:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:55:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:55:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:55:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:55:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:55:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:55:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:55:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:55:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:55:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:55:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:55:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:55:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:55:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:55:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:55:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:55:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:55:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:55:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:55:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:55:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:55:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:55:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:55:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:55:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:55:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:55:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:55:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:55:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:55:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:55:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:55:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:55:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:55:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:55:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:55:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:55:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:55:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:55:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:55:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:55:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:55:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:55:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:55:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:55:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:55:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:55:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:55:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:55:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:55:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:55:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:55:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:55:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:55:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:55:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:55:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:55:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:55:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:55:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:55:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:55:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:55:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:55:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:55:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:55:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:55:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:55:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:55:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:55:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:55:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:55:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:55:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:55:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:55:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:55:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:55:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:55:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:55:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:55:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:55:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:55:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:55:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:55:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:55:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:55:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:55:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:55:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:55:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:55:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:55:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:55:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:55:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:55:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:55:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:55:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:55:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:55:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:55:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:55:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:55:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:55:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:55:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:55:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:55:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:55:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:55:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:55:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:55:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:55:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:55:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:55:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:55:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:55:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:55:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:55:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:55:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:55:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:55:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:55:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:55:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:55:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:55:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:55:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:55:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:55:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:55:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:55:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:55:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:55:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:55:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:55:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:55:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:55:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:55:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:55:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:55:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:55:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:55:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:55:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:55:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:55:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:55:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:55:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:55:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:55:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:55:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:55:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:55:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:55:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:55:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:55:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:55:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:55:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:55:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:55:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:55:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:55:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:55:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:55:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:55:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:55:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:55:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:55:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:55:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:55:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:55:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:55:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:55:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:55:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:55:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:55:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:55:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:55:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:55:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:55:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:55:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:55:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:55:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:55:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:55:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:55:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:55:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:55:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:55:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:55:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:55:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:55:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:55:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:55:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:55:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:55:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:55:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:55:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:55:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:55:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:55:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:55:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:55:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:55:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:55:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:55:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:55:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:55:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:55:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:55:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:55:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:55:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:55:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:56:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:56:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:56:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:56:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:56:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:56:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:56:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:56:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:56:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:56:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:56:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:56:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:56:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:56:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:56:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:56:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:56:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:56:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:56:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:56:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:56:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:56:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:56:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:56:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:56:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:56:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:56:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:56:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:56:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:56:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:56:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:56:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:56:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:56:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:56:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:56:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:56:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:56:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:56:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:56:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:56:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:56:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:56:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:56:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:56:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:56:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:56:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:56:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:56:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:56:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:56:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:56:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:56:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:56:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:58:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:58:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:58:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:58:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:58:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:58:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:58:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:58:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:58:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:58:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:58:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:58:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:58:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:58:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:58:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:58:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:58:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:58:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:58:24 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 19:58:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:58:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:58:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:58:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:58:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:58:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:58:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:58:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:58:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:58:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:58:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:58:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:58:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:58:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:58:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:58:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:58:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:58:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:58:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:58:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:58:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:58:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:58:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:58:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:58:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:58:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:58:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:58:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:58:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:58:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:58:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:58:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:58:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:58:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:58:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:58:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:58:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:58:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:58:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:58:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:58:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:58:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:58:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:58:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:58:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:58:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:58:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:58:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:58:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:58:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:58:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:58:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:58:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:58:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:58:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:58:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:58:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:58:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:58:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:58:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:58:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:58:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:58:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:58:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:58:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:58:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:58:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:58:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:58:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:58:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:58:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:58:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:58:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:58:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:58:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:58:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:58:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:58:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:58:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:58:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:58:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:58:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:58:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:58:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:58:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:58:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:58:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:58:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:58:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:58:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:58:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:58:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:58:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:58:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:58:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:58:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:58:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:58:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:58:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:58:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:58:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:58:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:58:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:58:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:58:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:58:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:58:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:58:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:58:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:58:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:58:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:58:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:58:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:58:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:58:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:58:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:58:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:58:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:58:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:58:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:58:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:58:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:58:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:58:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:58:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:58:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:59:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:59:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:59:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:59:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:59:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:59:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:59:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:59:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:59:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:59:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:59:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:59:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:59:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:59:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:59:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:59:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:59:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:59:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:59:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:59:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:59:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:59:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:59:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:59:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:59:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:59:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:59:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:59:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:59:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:59:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:59:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:59:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:59:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:59:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:59:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:59:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:59:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:59:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:59:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:59:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:59:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:59:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:59:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:59:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:59:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:59:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:59:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:59:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:59:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:59:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:59:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:59:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:59:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:59:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:59:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:59:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:59:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:59:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:59:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:59:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:59:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:59:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:59:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:59:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:59:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:59:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:59:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:59:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:59:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:59:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:59:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:59:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:59:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:59:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:59:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:59:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:59:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:59:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:59:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:59:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:59:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:59:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:59:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:59:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:59:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:59:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:59:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:59:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:59:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:59:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:59:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:59:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:59:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:59:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:59:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:59:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:59:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:59:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:59:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:59:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:59:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:59:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:59:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:59:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:59:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:59:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:59:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:59:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:59:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:59:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:59:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:59:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:59:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:59:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:59:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:59:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:59:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:59:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:59:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:59:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:59:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:59:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:59:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:59:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:59:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:59:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:59:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:59:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:59:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:59:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:59:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:59:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:59:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:59:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:59:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:59:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:59:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:59:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 19:59:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 19:59:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:59:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:59:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:59:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 19:59:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 19:59:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 19:59:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 19:59:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 19:59:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 19:59:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 19:59:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 19:59:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 19:59:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 19:59:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 19:59:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 19:59:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 19:59:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:00:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:00:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:00:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:00:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:00:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:00:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:00:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:00:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:00:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:00:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:00:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:00:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:00:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:00:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:00:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:00:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:00:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:00:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:00:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:00:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:00:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:00:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:00:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:00:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:00:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:00:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:00:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:00:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:00:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:00:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:00:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:00:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:00:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:00:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:00:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:00:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:00:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:00:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:00:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:00:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:00:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:00:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:00:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:00:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:00:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:00:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:00:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:00:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:00:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:00:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:00:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:00:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:00:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:00:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:00:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:00:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:00:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:00:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:00:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:00:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:00:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:00:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:00:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:00:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:00:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:00:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:00:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:00:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:00:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:00:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:00:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:00:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:00:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:00:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:00:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:00:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:00:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:00:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:00:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:00:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:00:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:00:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:00:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:00:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:00:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:00:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:00:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:00:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:00:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:00:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:00:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:00:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:00:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:00:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:00:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:00:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:00:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:00:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:00:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:00:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:00:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:00:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:00:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:00:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:00:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:00:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:00:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:00:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:00:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:00:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:00:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:00:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:00:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:00:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:00:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:00:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:00:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:00:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:00:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:00:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:00:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:00:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:00:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:00:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:00:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:00:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:00:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:00:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:00:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:00:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:00:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:00:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:00:44 rdma06 systemd[1]: Starting dnf makecache...
Apr 18 20:00:44 rdma06 dnf[1201727]: Updating Subscription Management repositories.
Apr 18 20:00:44 rdma06 dnf[1201727]: Unable to read consumer identity
Apr 18 20:00:44 rdma06 dnf[1201727]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 18 20:00:44 rdma06 dnf[1201727]: Metadata cache refreshed recently.
Apr 18 20:00:45 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 18 20:00:45 rdma06 systemd[1]: Finished dnf makecache.
Apr 18 20:00:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:00:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:00:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:00:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:00:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:00:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:00:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:00:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:00:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:00:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:00:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:00:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:00:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:00:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:00:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:00:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:00:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:00:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:00:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:00:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:00:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:00:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:00:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:00:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:00:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:00:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:00:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:00:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:00:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:00:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:00:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:00:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:00:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:00:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:00:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:00:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:00:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:00:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:00:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:00:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:00:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:00:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:00:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:00:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:00:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:00:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:00:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:00:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:00:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:00:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:00:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:00:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:00:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:00:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:01:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:01:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:01:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:01:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:01:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:01:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:01:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:01:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:01:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:01:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:01:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:01:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:01:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:01:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:01:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:01:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:01:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:01:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:01:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:01:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:01:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:01:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:01:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:01:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:01:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:01:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:01:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:01:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:01:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:01:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:01:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:01:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:01:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:01:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:01:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:01:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:01:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:01:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:01:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:01:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:01:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:01:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:01:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:01:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:01:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:01:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:01:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:01:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:01:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:01:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:01:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:01:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:01:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:01:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:01:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:01:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:01:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:01:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:01:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:01:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:01:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:01:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:01:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:01:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:01:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:01:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:01:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:01:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:01:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:01:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:01:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:01:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:01:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:01:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:01:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:01:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:01:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:01:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:01:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:01:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:01:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:01:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:01:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:01:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:01:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:01:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:01:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:01:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:01:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:01:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:01:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:01:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:01:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:01:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:01:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:01:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:01:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:01:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:01:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:01:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:01:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:01:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:01:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:01:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:01:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:01:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:01:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:01:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:01:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:01:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:01:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:01:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:01:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:01:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:01:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:01:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:01:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:01:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:01:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:01:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:01:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:01:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:01:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:01:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:01:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:01:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:01:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:01:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:01:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:01:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:01:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:01:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:01:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:01:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:01:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:01:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:01:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:01:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:01:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:01:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:01:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:01:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:01:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:01:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:01:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:01:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:01:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:01:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:01:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:01:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:01:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:01:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:01:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:01:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:01:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:01:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:01:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:01:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:01:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:01:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:01:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:01:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:01:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:01:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:01:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:01:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:01:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:01:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:01:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:01:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:01:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:01:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:01:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:01:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:01:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:01:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:01:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:01:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:01:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:01:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:01:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:01:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:01:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:01:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:01:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:01:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:01:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:01:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:01:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:01:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:01:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:01:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:01:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:01:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:01:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:01:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:01:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:01:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:01:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:01:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:01:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:01:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:01:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:01:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:01:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:01:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:01:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:01:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:01:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:01:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:01:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:01:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:01:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:01:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:01:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:01:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:02:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:02:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:02:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:02:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:02:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:02:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:02:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:02:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:02:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:02:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:02:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:02:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:02:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:02:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:02:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:02:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:02:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:02:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:02:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:02:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:02:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:02:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:02:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:02:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:02:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:02:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:02:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:02:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:02:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:02:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:02:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:02:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:02:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:02:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:02:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:02:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:02:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:02:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:02:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:02:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:02:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:02:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:02:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:02:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:02:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:02:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:02:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:02:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:02:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:02:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:02:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:02:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:02:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:02:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:02:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:02:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:02:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:02:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:02:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:02:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:02:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:02:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:02:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:02:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:02:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:02:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:02:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:02:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:02:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:02:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:02:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:02:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:02:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:02:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:02:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:02:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:02:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:02:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:02:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:02:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:02:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:02:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:02:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:02:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:02:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:02:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:02:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:02:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:02:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:02:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:02:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:02:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:02:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:02:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:02:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:02:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:02:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:02:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:02:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:02:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:02:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:02:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:02:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:02:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:02:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:02:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:02:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:02:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:02:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:02:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:02:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:02:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:02:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:02:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:02:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:02:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:02:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:02:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:02:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:02:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:02:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:02:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:02:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:02:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:02:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:02:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:02:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:02:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:02:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:02:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:02:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:02:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:02:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:02:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:02:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:02:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:02:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:02:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:02:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:02:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:02:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:02:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:02:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:02:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:02:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:02:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:02:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:02:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:02:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:02:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:02:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:02:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:02:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:02:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:02:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:02:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:02:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:02:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:02:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:02:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:02:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:02:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:02:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:02:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:02:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:02:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:02:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:02:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:02:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:02:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:02:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:02:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:02:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:02:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:02:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:02:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:02:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:02:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:02:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:02:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:02:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:02:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:02:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:02:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:02:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:02:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:02:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:02:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:02:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:02:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:02:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:02:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:02:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:02:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:02:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:02:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:02:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:02:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:02:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:02:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:02:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:02:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:02:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:02:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:02:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:02:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:02:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:02:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:02:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:02:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:02:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:02:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:02:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:02:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:02:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:02:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:03:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:03:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:03:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:03:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:03:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:03:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:03:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:03:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:03:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:03:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:03:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:03:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:03:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:03:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:03:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:03:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:03:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:03:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:03:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:03:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:03:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:03:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:03:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:03:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:03:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:03:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:03:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:03:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:03:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:03:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:03:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:03:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:03:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:03:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:03:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:03:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:03:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:03:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:03:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:03:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:03:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:03:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:03:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:03:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:03:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:03:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:03:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:03:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:03:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:03:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:03:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:03:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:03:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:03:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:03:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:03:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:03:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:03:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:03:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:03:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:03:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:03:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:03:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:03:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:03:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:03:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:03:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:03:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:03:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:03:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:03:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:03:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:03:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:03:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:03:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:03:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:03:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:03:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:03:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:03:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:03:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:03:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:03:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:03:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:03:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:03:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:03:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:03:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:03:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:03:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:03:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:03:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:03:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:03:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:03:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:03:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:03:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:03:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:03:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:03:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:03:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:03:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:03:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:03:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:03:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:03:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:03:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:03:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:03:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:03:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:03:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:03:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:03:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:03:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:03:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:03:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:03:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:03:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:03:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:03:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:03:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:03:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:03:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:03:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:03:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:03:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:03:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:03:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:03:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:03:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:03:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:03:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:03:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:03:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:03:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:03:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:03:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:03:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:03:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:03:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:03:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:03:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:03:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:03:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:03:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:03:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:03:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:03:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:03:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:03:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:03:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:03:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:03:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:03:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:03:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:03:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:03:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:03:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:03:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:03:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:03:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:03:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:03:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:03:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:03:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:03:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:03:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:03:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:03:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:03:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:03:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:03:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:03:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:03:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:03:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:03:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:03:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:03:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:03:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:03:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:03:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:03:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:03:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:03:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:03:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:03:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:03:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:03:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:03:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:03:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:03:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:03:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:03:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:03:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:03:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:03:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:03:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:03:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:03:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:03:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:03:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:03:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:03:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:03:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:03:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:03:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:03:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:03:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:03:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:03:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:03:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:03:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:03:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:03:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:03:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:03:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:04:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:04:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:04:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:04:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:04:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:04:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:04:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:04:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:04:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:04:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:04:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:04:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:04:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:04:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:04:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:04:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:04:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:04:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:04:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:04:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:04:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:04:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:04:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:04:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:04:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:04:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:04:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:04:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:04:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:04:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:04:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:04:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:04:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:04:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:04:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:04:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:04:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:04:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:04:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:04:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:04:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:04:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:04:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:04:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:04:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:04:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:04:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:04:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:04:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:04:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:04:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:04:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:04:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:04:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:04:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:04:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:04:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:04:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:04:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:04:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:04:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:04:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:04:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:04:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:04:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:04:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:04:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:04:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:04:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:04:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:04:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:04:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:04:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:04:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:04:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:04:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:04:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:04:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:04:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:04:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:04:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:04:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:04:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:04:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:04:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:04:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:04:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:04:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:04:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:04:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:04:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:04:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:04:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:04:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:04:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:04:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:04:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:04:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:04:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:04:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:04:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:04:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:04:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:04:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:04:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:04:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:04:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:04:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:04:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:04:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:04:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:04:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:04:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:04:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:04:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:04:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:04:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:04:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:04:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:04:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:04:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:04:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:04:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:04:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:04:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:04:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:04:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:04:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:04:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:04:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:04:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:04:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:04:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:04:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:04:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:04:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:04:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:04:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:04:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:04:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:04:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:04:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:04:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:04:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:04:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:04:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:04:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:04:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:04:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:04:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:04:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:04:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:04:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:04:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:04:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:04:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:04:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:04:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:04:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:04:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:04:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:04:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:04:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:04:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:04:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:04:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:04:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:04:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:04:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:04:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:04:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:04:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:04:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:04:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:04:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:04:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:04:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:04:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:04:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:04:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:04:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:04:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:04:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:04:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:04:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:04:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:04:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:04:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:04:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:04:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:04:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:04:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:04:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:04:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:04:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:04:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:04:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:04:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:04:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:04:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:04:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:04:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:04:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:04:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:04:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:04:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:04:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:04:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:04:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:04:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:04:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:04:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:04:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:04:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:04:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:04:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:05:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:05:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:05:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:05:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:05:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:05:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:05:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:05:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:05:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:05:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:05:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:05:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:05:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:05:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:05:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:05:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:05:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:05:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:05:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:05:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:05:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:05:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:05:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:05:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:05:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:05:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:05:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:05:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:05:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:05:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:05:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:05:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:05:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:05:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:05:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:05:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:05:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:05:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:05:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:05:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:05:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:05:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:05:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:05:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:05:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:05:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:05:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:05:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:05:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:05:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:05:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:05:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:05:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:05:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:05:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:05:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:05:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:05:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:05:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:05:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:05:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:05:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:05:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:05:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:05:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:05:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:05:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:05:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:05:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:05:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:05:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:05:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:05:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:05:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:05:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:05:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:05:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:05:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:05:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:05:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:05:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:05:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:05:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:05:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:05:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:05:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:05:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:05:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:05:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:05:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:05:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:05:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:05:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:05:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:05:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:05:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:05:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:05:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:05:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:05:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:05:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:05:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:05:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:05:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:05:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:05:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:05:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:05:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:05:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:05:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:05:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:05:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:05:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:05:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:05:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:05:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:05:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:05:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:05:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:05:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:05:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:05:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:05:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:05:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:05:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:05:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:05:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:05:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:05:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:05:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:05:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:05:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:05:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:05:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:05:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:05:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:05:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:05:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:05:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:05:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:05:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:05:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:05:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:05:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:05:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:05:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:05:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:05:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:05:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:05:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:05:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:05:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:05:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:05:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:05:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:05:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:05:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:05:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:05:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:05:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:05:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:05:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:05:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:05:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:05:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:05:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:05:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:05:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:05:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:05:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:05:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:05:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:05:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:05:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:05:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:05:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:05:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:05:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:05:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:05:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:05:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:05:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:05:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:05:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:05:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:05:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:05:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:05:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:05:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:05:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:05:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:05:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:05:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:05:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:05:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:05:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:05:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:05:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:05:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:05:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:05:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:05:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:05:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:05:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:05:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:05:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:05:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:05:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:05:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:05:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:05:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:05:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:05:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:05:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:05:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:05:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:06:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:06:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:06:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:06:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:06:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:06:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:06:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:06:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:06:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:06:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:06:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:06:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:06:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:06:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:06:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:06:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:06:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:06:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:06:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:06:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:06:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:06:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:06:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:06:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:06:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:06:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:06:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:06:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:06:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:06:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:06:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:06:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:06:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:06:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:06:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:06:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:06:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:06:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:06:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:06:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:06:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:06:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:06:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:06:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:06:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:06:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:06:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:06:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:06:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:06:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:06:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:06:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:06:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:06:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:06:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:06:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:06:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:06:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:06:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:06:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:06:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:06:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:06:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:06:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:06:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:06:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:06:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:06:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:06:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:06:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:06:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:06:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:08:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:08:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:08:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:08:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:08:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:08:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:08:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:08:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:08:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:08:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:08:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:08:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:08:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:08:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:08:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:08:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:08:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:08:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:08:25 rdma06 rsyslogd[1041]: imjournal: 5093 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 20:08:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:08:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:08:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:08:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:08:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:08:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:08:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:08:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:08:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:08:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:08:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:08:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:08:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:08:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:08:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:08:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:08:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:08:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:08:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:08:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:08:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:08:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:08:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:08:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:08:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:08:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:08:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:08:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:08:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:08:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:08:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:08:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:08:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:08:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:08:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:08:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:08:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:08:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:08:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:08:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:08:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:08:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:08:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:08:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:08:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:08:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:08:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:08:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:08:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:08:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:08:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:08:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:08:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:08:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:08:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:08:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:08:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:08:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:08:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:08:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:08:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:08:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:08:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:08:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:08:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:08:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:08:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:08:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:08:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:08:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:08:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:08:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:08:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:08:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:08:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:08:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:08:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:08:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:08:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:08:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:08:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:08:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:08:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:08:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:08:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:08:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:08:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:08:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:08:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:08:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:08:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:08:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:08:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:08:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:08:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:08:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:08:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:08:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:08:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:08:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:08:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:08:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:08:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:08:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:08:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:08:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:08:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:08:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:08:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:08:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:08:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:08:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:08:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:08:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:08:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:08:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:08:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:08:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:08:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:08:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:08:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:08:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:08:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:08:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:08:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:08:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:09:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:09:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:09:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:09:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:09:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:09:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:09:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:09:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:09:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:09:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:09:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:09:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:09:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:09:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:09:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:09:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:09:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:09:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:09:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:09:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:09:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:09:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:09:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:09:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:09:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:09:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:09:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:09:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:09:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:09:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:09:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:09:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:09:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:09:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:09:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:09:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:09:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:09:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:09:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:09:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:09:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:09:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:09:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:09:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:09:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:09:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:09:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:09:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:09:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:09:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:09:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:09:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:09:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:09:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:09:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:09:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:09:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:09:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:09:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:09:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:09:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:09:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:09:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:09:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:09:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:09:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:09:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:09:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:09:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:09:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:09:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:09:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:09:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:09:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:09:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:09:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:09:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:09:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:09:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:09:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:09:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:09:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:09:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:09:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:09:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:09:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:09:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:09:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:09:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:09:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:09:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:09:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:09:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:09:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:09:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:09:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:09:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:09:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:09:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:09:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:09:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:09:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:09:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:09:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:09:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:09:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:09:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:09:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:09:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:09:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:09:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:09:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:09:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:09:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:09:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:09:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:09:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:09:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:09:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:09:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:09:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:09:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:09:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:09:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:09:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:09:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:09:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:09:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:09:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:09:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:09:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:09:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:09:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:09:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:09:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:09:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:09:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:09:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:09:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:09:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:09:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:09:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:09:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:09:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:09:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:09:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:09:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:09:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:09:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:09:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:09:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:09:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:09:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:09:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:09:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:09:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:09:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:09:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:09:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:09:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:09:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:09:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:09:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:09:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:09:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:09:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:09:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:09:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:09:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:09:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:09:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:09:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:09:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:09:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:09:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:09:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:09:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:09:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:09:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:09:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:09:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:09:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:09:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:09:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:09:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:09:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:09:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:09:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:09:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:09:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:09:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:09:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:09:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:09:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:09:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:09:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:09:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:09:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:09:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:09:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:09:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:09:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:09:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:09:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:09:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:09:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:09:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:09:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:09:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:09:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:09:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:09:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:09:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:09:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:09:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:09:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:10:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:10:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:10:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:10:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:10:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:10:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:10:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:10:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:10:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:10:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:10:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:10:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:10:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:10:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:10:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:10:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:10:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:10:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:10:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:10:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:10:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:10:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:10:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:10:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:10:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:10:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:10:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:10:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:10:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:10:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:10:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:10:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:10:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:10:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:10:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:10:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:10:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:10:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:10:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:10:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:10:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:10:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:10:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:10:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:10:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:10:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:10:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:10:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:10:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:10:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:10:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:10:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:10:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:10:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:10:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:10:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:10:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:10:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:10:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:10:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:10:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:10:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:10:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:10:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:10:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:10:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:10:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:10:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:10:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:10:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:10:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:10:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:10:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:10:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:10:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:10:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:10:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:10:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:10:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:10:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:10:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:10:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:10:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:10:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:10:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:10:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:10:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:10:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:10:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:10:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:10:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:10:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:10:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:10:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:10:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:10:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:10:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:10:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:10:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:10:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:10:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:10:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:10:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:10:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:10:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:10:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:10:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:10:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:10:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:10:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:10:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:10:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:10:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:10:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:10:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:10:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:10:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:10:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:10:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:10:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:10:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:10:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:10:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:10:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:10:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:10:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:10:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:10:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:10:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:10:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:10:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:10:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:10:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:10:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:10:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:10:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:10:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:10:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:10:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:10:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:10:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:10:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:10:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:10:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:10:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:10:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:10:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:10:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:10:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:10:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:10:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:10:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:10:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:10:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:10:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:10:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:10:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:10:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:10:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:10:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:10:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:10:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:10:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:10:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:10:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:10:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:10:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:10:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:10:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:10:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:10:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:10:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:10:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:10:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:10:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:10:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:10:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:10:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:10:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:10:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:10:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:10:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:10:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:10:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:10:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:10:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:10:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:10:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:10:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:10:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:10:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:10:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:10:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:10:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:10:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:10:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:10:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:10:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:10:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:10:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:10:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:10:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:10:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:10:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:10:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:10:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:10:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:10:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:10:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:10:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:10:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:10:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:10:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:10:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:10:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:10:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:11:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:11:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:11:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:11:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:11:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:11:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:11:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:11:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:11:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:11:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:11:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:11:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:11:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:11:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:11:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:11:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:11:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:11:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:11:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:11:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:11:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:11:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:11:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:11:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:11:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:11:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:11:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:11:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:11:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:11:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:11:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:11:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:11:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:11:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:11:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:11:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:11:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:11:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:11:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:11:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:11:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:11:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:11:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:11:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:11:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:11:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:11:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:11:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:11:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:11:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:11:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:11:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:11:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:11:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:11:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:11:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:11:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:11:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:11:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:11:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:11:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:11:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:11:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:11:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:11:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:11:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:11:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:11:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:11:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:11:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:11:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:11:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:11:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:11:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:11:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:11:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:11:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:11:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:11:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:11:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:11:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:11:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:11:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:11:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:11:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:11:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:11:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:11:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:11:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:11:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:11:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:11:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:11:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:11:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:11:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:11:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:11:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:11:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:11:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:11:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:11:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:11:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:11:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:11:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:11:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:11:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:11:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:11:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:11:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:11:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:11:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:11:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:11:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:11:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:11:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:11:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:11:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:11:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:11:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:11:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:11:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:11:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:11:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:11:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:11:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:11:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:11:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:11:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:11:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:11:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:11:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:11:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:11:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:11:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:11:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:11:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:11:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:11:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:11:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:11:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:11:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:11:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:11:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:11:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:11:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:11:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:11:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:11:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:11:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:11:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:11:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:11:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:11:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:11:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:11:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:11:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:11:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:11:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:11:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:11:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:11:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:11:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:11:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:11:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:11:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:11:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:11:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:11:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:11:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:11:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:11:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:11:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:11:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:11:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:11:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:11:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:11:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:11:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:11:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:11:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:11:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:11:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:11:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:11:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:11:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:11:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:11:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:11:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:11:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:11:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:11:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:11:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:11:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:11:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:11:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:11:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:11:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:11:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:11:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:11:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:11:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:11:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:11:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:11:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:11:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:11:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:11:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:11:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:11:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:11:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:11:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:11:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:11:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:11:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:11:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:11:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:12:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:12:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:12:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:12:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:12:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:12:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:12:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:12:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:12:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:12:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:12:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:12:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:12:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:12:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:12:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:12:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:12:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:12:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:12:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:12:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:12:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:12:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:12:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:12:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:12:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:12:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:12:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:12:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:12:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:12:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:12:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:12:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:12:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:12:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:12:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:12:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:12:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:12:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:12:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:12:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:12:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:12:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:12:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:12:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:12:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:12:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:12:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:12:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:12:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:12:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:12:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:12:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:12:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:12:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:12:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:12:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:12:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:12:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:12:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:12:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:12:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:12:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:12:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:12:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:12:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:12:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:12:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:12:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:12:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:12:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:12:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:12:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:12:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:12:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:12:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:12:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:12:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:12:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:12:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:12:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:12:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:12:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:12:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:12:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:12:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:12:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:12:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:12:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:12:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:12:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:12:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:12:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:12:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:12:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:12:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:12:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:12:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:12:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:12:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:12:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:12:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:12:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:12:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:12:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:12:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:12:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:12:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:12:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:12:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:12:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:12:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:12:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:12:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:12:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:12:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:12:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:12:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:12:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:12:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:12:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:12:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:12:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:12:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:12:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:12:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:12:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:12:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:12:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:12:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:12:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:12:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:12:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:12:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:12:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:12:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:12:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:12:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:12:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:12:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:12:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:12:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:12:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:12:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:12:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:12:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:12:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:12:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:12:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:12:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:12:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:12:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:12:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:12:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:12:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:12:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:12:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:12:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:12:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:12:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:12:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:12:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:12:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:12:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:12:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:12:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:12:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:12:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:12:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:12:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:12:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:12:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:12:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:12:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:12:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:12:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:12:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:12:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:12:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:12:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:12:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:12:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:12:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:12:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:12:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:12:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:12:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:12:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:12:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:12:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:12:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:12:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:12:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:12:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:12:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:12:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:12:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:12:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:12:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:12:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:12:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:12:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:12:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:12:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:12:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:12:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:12:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:12:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:12:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:12:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:12:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:12:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:12:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:12:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:12:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:12:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:12:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:13:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:13:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:13:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:13:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:13:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:13:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:13:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:13:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:13:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:13:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:13:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:13:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:13:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:13:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:13:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:13:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:13:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:13:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:13:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:13:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:13:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:13:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:13:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:13:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:13:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:13:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:13:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:13:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:13:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:13:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:13:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:13:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:13:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:13:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:13:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:13:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:13:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:13:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:13:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:13:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:13:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:13:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:13:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:13:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:13:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:13:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:13:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:13:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:13:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:13:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:13:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:13:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:13:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:13:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:13:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:13:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:13:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:13:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:13:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:13:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:13:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:13:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:13:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:13:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:13:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:13:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:13:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:13:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:13:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:13:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:13:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:13:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:13:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:13:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:13:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:13:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:13:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:13:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:13:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:13:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:13:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:13:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:13:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:13:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:13:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:13:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:13:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:13:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:13:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:13:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:13:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:13:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:13:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:13:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:13:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:13:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:13:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:13:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:13:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:13:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:13:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:13:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:13:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:13:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:13:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:13:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:13:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:13:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:13:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:13:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:13:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:13:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:13:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:13:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:13:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:13:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:13:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:13:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:13:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:13:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:13:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:13:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:13:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:13:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:13:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:13:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:13:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:13:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:13:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:13:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:13:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:13:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:13:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:13:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:13:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:13:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:13:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:13:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:13:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:13:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:13:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:13:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:13:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:13:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:13:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:13:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:13:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:13:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:13:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:13:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:13:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:13:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:13:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:13:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:13:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:13:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:13:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:13:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:13:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:13:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:13:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:13:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:13:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:13:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:13:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:13:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:13:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:13:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:13:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:13:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:13:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:13:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:13:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:13:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:13:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:13:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:13:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:13:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:13:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:13:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:13:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:13:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:13:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:13:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:13:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:13:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:13:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:13:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:13:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:13:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:13:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:13:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:13:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:13:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:13:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:13:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:13:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:13:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:13:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:13:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:13:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:13:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:13:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:13:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:13:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:13:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:13:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:13:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:13:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:13:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:13:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:13:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:13:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:13:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:13:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:13:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:14:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:14:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:14:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:14:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:14:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:14:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:14:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:14:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:14:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:14:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:14:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:14:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:14:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:14:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:14:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:14:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:14:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:14:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:14:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:14:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:14:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:14:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:14:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:14:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:14:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:14:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:14:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:14:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:14:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:14:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:14:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:14:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:14:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:14:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:14:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:14:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:14:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:14:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:14:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:14:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:14:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:14:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:14:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:14:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:14:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:14:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:14:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:14:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:14:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:14:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:14:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:14:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:14:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:14:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:14:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:14:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:14:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:14:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:14:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:14:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:14:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:14:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:14:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:14:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:14:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:14:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:14:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:14:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:14:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:14:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:14:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:14:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:14:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:14:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:14:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:14:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:14:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:14:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:14:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:14:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:14:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:14:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:14:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:14:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:14:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:14:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:14:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:14:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:14:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:14:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:14:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:14:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:14:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:14:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:14:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:14:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:14:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:14:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:14:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:14:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:14:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:14:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:14:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:14:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:14:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:14:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:14:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:14:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:14:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:14:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:14:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:14:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:14:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:14:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:14:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:14:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:14:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:14:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:14:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:14:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:14:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:14:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:14:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:14:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:14:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:14:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:14:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:14:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:14:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:14:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:14:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:14:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:14:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:14:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:14:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:14:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:14:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:14:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:14:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:14:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:14:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:14:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:14:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:14:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:14:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:14:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:14:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:14:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:14:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:14:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:14:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:14:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:14:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:14:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:14:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:14:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:14:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:14:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:14:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:14:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:14:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:14:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:14:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:14:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:14:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:14:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:14:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:14:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:14:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:14:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:14:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:14:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:14:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:14:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:14:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:14:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:14:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:14:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:14:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:14:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:14:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:14:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:14:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:14:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:14:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:14:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:14:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:14:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:14:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:14:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:14:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:14:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:14:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:14:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:14:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:14:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:14:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:14:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:14:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:14:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:14:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:14:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:14:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:14:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:14:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:14:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:14:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:14:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:14:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:14:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:14:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:14:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:14:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:14:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:14:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:14:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:15:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:15:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:15:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:15:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:15:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:15:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:15:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:15:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:15:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:15:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:15:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:15:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:15:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:15:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:15:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:15:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:15:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:15:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:15:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:15:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:15:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:15:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:15:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:15:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:15:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:15:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:15:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:15:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:15:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:15:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:15:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:15:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:15:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:15:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:15:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:15:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:15:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:15:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:15:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:15:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:15:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:15:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:15:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:15:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:15:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:15:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:15:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:15:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:15:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:15:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:15:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:15:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:15:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:15:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:15:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:15:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:15:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:15:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:15:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:15:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:15:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:15:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:15:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:15:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:15:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:15:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:15:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:15:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:15:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:15:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:15:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:15:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:15:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:15:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:15:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:15:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:15:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:15:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:15:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:15:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:15:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:15:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:15:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:15:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:15:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:15:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:15:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:15:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:15:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:15:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:15:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:15:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:15:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:15:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:15:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:15:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:15:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:15:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:15:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:15:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:15:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:15:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:15:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:15:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:15:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:15:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:15:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:15:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:15:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:15:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:15:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:15:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:15:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:15:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:15:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:15:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:15:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:15:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:15:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:15:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:15:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:15:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:15:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:15:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:15:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:15:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:15:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:15:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:15:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:15:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:15:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:15:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:15:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:15:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:15:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:15:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:15:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:15:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:15:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:15:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:15:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:15:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:15:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:15:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:15:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:15:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:15:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:15:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:15:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:15:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:15:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:15:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:15:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:15:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:15:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:15:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:15:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:15:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:15:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:15:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:15:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:15:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:15:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:15:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:15:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:15:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:15:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:15:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:15:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:15:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:15:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:15:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:15:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:15:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:15:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:15:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:15:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:15:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:15:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:15:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:15:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:15:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:15:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:15:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:15:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:15:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:15:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:15:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:15:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:15:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:15:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:15:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:15:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:15:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:15:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:15:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:15:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:15:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:15:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:15:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:15:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:15:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:15:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:15:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:15:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:15:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:15:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:15:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:15:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:15:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:15:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:15:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:15:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:15:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:15:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:15:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:16:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:16:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:16:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:16:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:16:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:16:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:16:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:16:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:16:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:16:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:16:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:16:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:16:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:16:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:16:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:16:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:16:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:16:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:16:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:16:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:16:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:16:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:16:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:16:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:16:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:16:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:16:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:16:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:16:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:16:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:16:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:16:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:16:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:16:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:16:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:16:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:16:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:16:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:16:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:16:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:16:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:16:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:16:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:16:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:16:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:16:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:16:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:16:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:16:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:16:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:16:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:16:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:16:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:16:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:16:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:16:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:16:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:16:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:16:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:16:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:16:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:16:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:16:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:16:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:16:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:16:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:16:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:16:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:16:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:16:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:16:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:16:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:18:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:18:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:18:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:18:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:18:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:18:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:18:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:18:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:18:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:18:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:18:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:18:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:18:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:18:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:18:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:18:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:18:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:18:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:18:30 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 20:18:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:18:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:18:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:18:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:18:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:18:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:18:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:18:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:18:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:18:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:18:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:18:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:18:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:18:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:18:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:18:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:18:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:18:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:18:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:18:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:18:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:18:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:18:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:18:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:18:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:18:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:18:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:18:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:18:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:18:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:18:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:18:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:18:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:18:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:18:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:18:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:18:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:18:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:18:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:18:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:18:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:18:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:18:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:18:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:18:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:18:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:18:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:18:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:18:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:18:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:18:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:18:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:18:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:18:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:18:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:18:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:18:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:18:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:18:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:18:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:18:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:18:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:18:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:18:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:18:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:18:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:18:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:18:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:18:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:18:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:18:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:18:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:18:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:18:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:18:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:18:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:18:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:18:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:18:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:18:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:18:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:18:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:18:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:18:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:18:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:18:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:18:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:18:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:18:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:18:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:19:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:19:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:19:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:19:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:19:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:19:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:19:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:19:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:19:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:19:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:19:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:19:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:19:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:19:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:19:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:19:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:19:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:19:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:19:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:19:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:19:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:19:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:19:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:19:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:19:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:19:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:19:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:19:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:19:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:19:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:19:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:19:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:19:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:19:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:19:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:19:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:19:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:19:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:19:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:19:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:19:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:19:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:19:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:19:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:19:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:19:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:19:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:19:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:19:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:19:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:19:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:19:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:19:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:19:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:19:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:19:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:19:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:19:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:19:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:19:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:19:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:19:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:19:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:19:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:19:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:19:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:19:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:19:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:19:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:19:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:19:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:19:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:19:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:19:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:19:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:19:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:19:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:19:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:19:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:19:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:19:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:19:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:19:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:19:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:19:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:19:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:19:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:19:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:19:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:19:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:19:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:19:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:19:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:19:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:19:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:19:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:19:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:19:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:19:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:19:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:19:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:19:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:19:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:19:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:19:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:19:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:19:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:19:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:19:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:19:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:19:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:19:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:19:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:19:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:19:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:19:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:19:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:19:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:19:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:19:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:19:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:19:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:19:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:19:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:19:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:19:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:19:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:19:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:19:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:19:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:19:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:19:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:19:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:19:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:19:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:19:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:19:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:19:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:19:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:19:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:19:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:19:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:19:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:19:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:19:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:19:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:19:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:19:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:19:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:19:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:19:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:19:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:19:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:19:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:19:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:19:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:19:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:19:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:19:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:19:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:19:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:19:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:19:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:19:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:19:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:19:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:19:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:19:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:19:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:19:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:19:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:19:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:19:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:19:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:19:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:19:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:19:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:19:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:19:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:19:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:19:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:19:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:19:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:19:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:19:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:19:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:19:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:19:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:19:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:19:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:19:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:19:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:19:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:19:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:19:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:19:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:19:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:19:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:19:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:19:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:19:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:19:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:19:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:19:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:19:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:19:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:19:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:19:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:19:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:19:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:19:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:19:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:19:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:19:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:19:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:19:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:20:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:20:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:20:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:20:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:20:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:20:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:20:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:20:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:20:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:20:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:20:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:20:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:20:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:20:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:20:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:20:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:20:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:20:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:20:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:20:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:20:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:20:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:20:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:20:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:20:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:20:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:20:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:20:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:20:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:20:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:20:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:20:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:20:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:20:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:20:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:20:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:20:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:20:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:20:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:20:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:20:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:20:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:20:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:20:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:20:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:20:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:20:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:20:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:20:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:20:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:20:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:20:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:20:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:20:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:20:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:20:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:20:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:20:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:20:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:20:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:20:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:20:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:20:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:20:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:20:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:20:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:20:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:20:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:20:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:20:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:20:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:20:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:20:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:20:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:20:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:20:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:20:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:20:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:20:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:20:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:20:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:20:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:20:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:20:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:20:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:20:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:20:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:20:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:20:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:20:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:20:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:20:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:20:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:20:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:20:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:20:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:20:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:20:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:20:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:20:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:20:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:20:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:20:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:20:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:20:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:20:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:20:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:20:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:20:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:20:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:20:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:20:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:20:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:20:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:20:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:20:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:20:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:20:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:20:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:20:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:20:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:20:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:20:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:20:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:20:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:20:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:20:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:20:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:20:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:20:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:20:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:20:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:20:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:20:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:20:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:20:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:20:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:20:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:20:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:20:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:20:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:20:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:20:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:20:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:20:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:20:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:20:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:20:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:20:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:20:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:20:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:20:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:20:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:20:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:20:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:20:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:20:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:20:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:20:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:20:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:20:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:20:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:20:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:20:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:20:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:20:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:20:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:20:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:20:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:20:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:20:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:20:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:20:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:20:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:20:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:20:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:20:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:20:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:20:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:20:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:20:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:20:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:20:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:20:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:20:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:20:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:20:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:20:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:20:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:20:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:20:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:20:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:20:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:20:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:20:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:20:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:20:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:20:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:20:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:20:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:20:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:20:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:20:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:20:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:20:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:20:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:20:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:20:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:20:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:20:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:20:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:20:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:20:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:20:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:20:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:20:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:21:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:21:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:21:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:21:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:21:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:21:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:21:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:21:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:21:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:21:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:21:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:21:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:21:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:21:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:21:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:21:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:21:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:21:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:21:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:21:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:21:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:21:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:21:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:21:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:21:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:21:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:21:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:21:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:21:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:21:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:21:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:21:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:21:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:21:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:21:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:21:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:21:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:21:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:21:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:21:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:21:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:21:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:21:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:21:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:21:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:21:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:21:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:21:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:21:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:21:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:21:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:21:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:21:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:21:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:21:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:21:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:21:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:21:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:21:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:21:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:21:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:21:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:21:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:21:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:21:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:21:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:21:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:21:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:21:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:21:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:21:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:21:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:21:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:21:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:21:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:21:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:21:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:21:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:21:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:21:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:21:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:21:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:21:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:21:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:21:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:21:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:21:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:21:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:21:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:21:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:21:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:21:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:21:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:21:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:21:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:21:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:21:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:21:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:21:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:21:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:21:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:21:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:21:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:21:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:21:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:21:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:21:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:21:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:21:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:21:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:21:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:21:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:21:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:21:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:21:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:21:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:21:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:21:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:21:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:21:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:21:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:21:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:21:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:21:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:21:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:21:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:21:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:21:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:21:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:21:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:21:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:21:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:21:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:21:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:21:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:21:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:21:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:21:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:21:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:21:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:21:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:21:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:21:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:21:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:21:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:21:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:21:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:21:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:21:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:21:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:21:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:21:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:21:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:21:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:21:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:21:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:21:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:21:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:21:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:21:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:21:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:21:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:21:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:21:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:21:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:21:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:21:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:21:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:21:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:21:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:21:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:21:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:21:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:21:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:21:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:21:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:21:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:21:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:21:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:21:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:21:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:21:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:21:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:21:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:21:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:21:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:21:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:21:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:21:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:21:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:21:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:21:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:21:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:21:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:21:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:21:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:21:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:21:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:21:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:21:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:21:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:21:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:21:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:21:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:21:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:21:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:21:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:21:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:21:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:21:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:21:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:21:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:21:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:21:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:21:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:21:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:22:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:22:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:22:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:22:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:22:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:22:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:22:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:22:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:22:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:22:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:22:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:22:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:22:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:22:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:22:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:22:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:22:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:22:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:22:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:22:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:22:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:22:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:22:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:22:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:22:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:22:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:22:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:22:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:22:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:22:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:22:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:22:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:22:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:22:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:22:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:22:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:22:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:22:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:22:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:22:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:22:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:22:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:22:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:22:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:22:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:22:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:22:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:22:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:22:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:22:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:22:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:22:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:22:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:22:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:22:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:22:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:22:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:22:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:22:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:22:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:22:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:22:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:22:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:22:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:22:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:22:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:22:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:22:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:22:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:22:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:22:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:22:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:22:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:22:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:22:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:22:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:22:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:22:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:22:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:22:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:22:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:22:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:22:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:22:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:22:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:22:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:22:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:22:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:22:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:22:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:22:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:22:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:22:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:22:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:22:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:22:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:22:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:22:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:22:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:22:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:22:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:22:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:22:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:22:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:22:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:22:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:22:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:22:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:22:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:22:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:22:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:22:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:22:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:22:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:22:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:22:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:22:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:22:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:22:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:22:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:22:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:22:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:22:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:22:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:22:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:22:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:22:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:22:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:22:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:22:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:22:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:22:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:22:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:22:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:22:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:22:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:22:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:22:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:22:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:22:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:22:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:22:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:22:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:22:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:22:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:22:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:22:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:22:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:22:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:22:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:22:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:22:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:22:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:22:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:22:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:22:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:22:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:22:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:22:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:22:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:22:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:22:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:22:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:22:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:22:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:22:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:22:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:22:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:22:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:22:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:22:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:22:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:22:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:22:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:22:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:22:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:22:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:22:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:22:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:22:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:22:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:22:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:22:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:22:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:22:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:22:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:22:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:22:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:22:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:22:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:22:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:22:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:22:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:22:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:22:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:22:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:22:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:22:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:22:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:22:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:22:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:22:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:22:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:22:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:22:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:22:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:22:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:22:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:22:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:22:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:22:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:22:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:22:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:22:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:22:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:22:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:23:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:23:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:23:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:23:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:23:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:23:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:23:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:23:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:23:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:23:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:23:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:23:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:23:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:23:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:23:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:23:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:23:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:23:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:23:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:23:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:23:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:23:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:23:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:23:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:23:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:23:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:23:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:23:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:23:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:23:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:23:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:23:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:23:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:23:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:23:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:23:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:23:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:23:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:23:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:23:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:23:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:23:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:23:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:23:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:23:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:23:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:23:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:23:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:23:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:23:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:23:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:23:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:23:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:23:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:23:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:23:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:23:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:23:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:23:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:23:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:23:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:23:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:23:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:23:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:23:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:23:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:23:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:23:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:23:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:23:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:23:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:23:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:23:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:23:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:23:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:23:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:23:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:23:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:23:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:23:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:23:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:23:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:23:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:23:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:23:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:23:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:23:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:23:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:23:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:23:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:23:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:23:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:23:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:23:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:23:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:23:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:23:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:23:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:23:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:23:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:23:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:23:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:23:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:23:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:23:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:23:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:23:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:23:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:23:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:23:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:23:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:23:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:23:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:23:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:23:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:23:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:23:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:23:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:23:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:23:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:23:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:23:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:23:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:23:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:23:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:23:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:23:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:23:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:23:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:23:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:23:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:23:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:23:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:23:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:23:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:23:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:23:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:23:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:23:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:23:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:23:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:23:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:23:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:23:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:23:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:23:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:23:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:23:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:23:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:23:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:23:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:23:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:23:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:23:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:23:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:23:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:23:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:23:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:23:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:23:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:23:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:23:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:23:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:23:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:23:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:23:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:23:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:23:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:23:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:23:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:23:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:23:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:23:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:23:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:23:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:23:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:23:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:23:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:23:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:23:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:23:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:23:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:23:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:23:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:23:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:23:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:23:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:23:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:23:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:23:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:23:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:23:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:23:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:23:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:23:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:23:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:23:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:23:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:23:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:23:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:23:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:23:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:23:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:23:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:23:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:23:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:23:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:23:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:23:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:23:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:23:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:23:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:23:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:23:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:23:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:23:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:24:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:24:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:24:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:24:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:24:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:24:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:24:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:24:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:24:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:24:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:24:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:24:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:24:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:24:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:24:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:24:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:24:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:24:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:24:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:24:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:24:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:24:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:24:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:24:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:24:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:24:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:24:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:24:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:24:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:24:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:24:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:24:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:24:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:24:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:24:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:24:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:24:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:24:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:24:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:24:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:24:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:24:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:24:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:24:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:24:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:24:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:24:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:24:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:24:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:24:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:24:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:24:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:24:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:24:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:24:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:24:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:24:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:24:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:24:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:24:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:24:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:24:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:24:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:24:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:24:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:24:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:24:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:24:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:24:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:24:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:24:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:24:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:24:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:24:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:24:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:24:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:24:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:24:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:24:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:24:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:24:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:24:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:24:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:24:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:24:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:24:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:24:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:24:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:24:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:24:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:24:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:24:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:24:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:24:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:24:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:24:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:24:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:24:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:24:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:24:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:24:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:24:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:24:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:24:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:24:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:24:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:24:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:24:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:24:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:24:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:24:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:24:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:24:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:24:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:24:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:24:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:24:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:24:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:24:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:24:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:24:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:24:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:24:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:24:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:24:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:24:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:24:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:24:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:24:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:24:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:24:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:24:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:24:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:24:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:24:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:24:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:24:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:24:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:24:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:24:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:24:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:24:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:24:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:24:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:24:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:24:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:24:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:24:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:24:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:24:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:24:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:24:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:24:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:24:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:24:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:24:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:24:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:24:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:24:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:24:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:24:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:24:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:24:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:24:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:24:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:24:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:24:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:24:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:24:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:24:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:24:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:24:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:24:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:24:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:24:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:24:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:24:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:24:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:24:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:24:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:24:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:24:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:24:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:24:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:24:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:24:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:24:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:24:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:24:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:24:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:24:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:24:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:24:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:24:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:24:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:24:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:24:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:24:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:24:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:24:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:24:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:24:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:24:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:24:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:24:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:24:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:24:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:24:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:24:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:24:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:24:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:24:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:24:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:24:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:24:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:24:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:25:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:25:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:25:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:25:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:25:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:25:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:25:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:25:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:25:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:25:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:25:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:25:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:25:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:25:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:25:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:25:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:25:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:25:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:25:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:25:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:25:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:25:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:25:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:25:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:25:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:25:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:25:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:25:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:25:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:25:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:25:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:25:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:25:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:25:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:25:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:25:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:25:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:25:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:25:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:25:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:25:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:25:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:25:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:25:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:25:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:25:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:25:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:25:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:25:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:25:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:25:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:25:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:25:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:25:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:25:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:25:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:25:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:25:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:25:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:25:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:25:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:25:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:25:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:25:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:25:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:25:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:25:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:25:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:25:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:25:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:25:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:25:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:25:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:25:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:25:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:25:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:25:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:25:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:25:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:25:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:25:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:25:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:25:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:25:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:25:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:25:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:25:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:25:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:25:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:25:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:25:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:25:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:25:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:25:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:25:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:25:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:25:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:25:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:25:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:25:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:25:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:25:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:25:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:25:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:25:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:25:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:25:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:25:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:25:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:25:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:25:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:25:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:25:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:25:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:25:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:25:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:25:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:25:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:25:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:25:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:25:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:25:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:25:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:25:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:25:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:25:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:25:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:25:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:25:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:25:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:25:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:25:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:25:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:25:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:25:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:25:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:25:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:25:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:25:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:25:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:25:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:25:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:25:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:25:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:25:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:25:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:25:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:25:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:25:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:25:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:25:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:25:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:25:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:25:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:25:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:25:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:25:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:25:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:25:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:25:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:25:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:25:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:25:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:25:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:25:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:25:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:25:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:25:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:25:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:25:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:25:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:25:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:25:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:25:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:25:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:25:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:25:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:25:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:25:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:25:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:25:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:25:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:25:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:25:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:25:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:25:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:25:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:25:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:25:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:25:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:25:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:25:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:25:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:25:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:25:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:25:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:25:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:25:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:25:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:25:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:25:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:25:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:25:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:25:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:25:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:25:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:25:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:25:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:25:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:25:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:25:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:25:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:25:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:25:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:25:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:25:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:26:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:26:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:26:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:26:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:26:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:26:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:26:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:26:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:26:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:26:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:26:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:26:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:26:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:26:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:26:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:26:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:26:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:26:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:26:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:26:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:26:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:26:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:26:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:26:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:26:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:26:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:26:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:26:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:26:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:26:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:26:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:26:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:26:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:26:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:26:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:26:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:26:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:26:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:26:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:26:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:26:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:26:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:26:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:26:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:26:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:26:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:26:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:26:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:26:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:26:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:26:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:26:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:26:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:26:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:26:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:26:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:26:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:26:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:26:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:26:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:26:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:26:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:26:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:26:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:26:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:26:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:26:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:26:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:26:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:26:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:26:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:26:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:26:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:26:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:26:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:26:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:26:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:26:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:26:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:26:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:26:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:26:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:26:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:26:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:26:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:26:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:26:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:26:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:26:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:26:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:26:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:26:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:26:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:26:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:26:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:26:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:26:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:26:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:26:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:26:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:26:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:26:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:26:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:26:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:26:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:26:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:26:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:26:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:28:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:28:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:28:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:28:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:28:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:28:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:28:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:28:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:28:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:28:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:28:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:28:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:28:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:28:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:28:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:28:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:28:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:28:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:28:35 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 20:28:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:28:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:28:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:28:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:28:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:28:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:28:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:28:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:28:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:28:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:28:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:28:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:28:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:28:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:28:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:28:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:28:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:28:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:28:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:28:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:28:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:28:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:28:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:28:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:28:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:28:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:28:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:28:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:28:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:28:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:28:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:28:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:28:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:28:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:28:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:28:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:28:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:28:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:28:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:28:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:28:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:28:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:28:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:28:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:28:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:28:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:28:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:28:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:28:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:28:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:28:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:28:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:28:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:28:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:28:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:28:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:28:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:28:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:28:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:28:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:28:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:28:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:28:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:28:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:28:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:28:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:28:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:28:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:28:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:28:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:28:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:28:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:29:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:29:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:29:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:29:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:29:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:29:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:29:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:29:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:29:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:29:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:29:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:29:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:29:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:29:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:29:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:29:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:29:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:29:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:29:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:29:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:29:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:29:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:29:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:29:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:29:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:29:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:29:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:29:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:29:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:29:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:29:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:29:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:29:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:29:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:29:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:29:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:29:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:29:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:29:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:29:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:29:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:29:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:29:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:29:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:29:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:29:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:29:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:29:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:29:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:29:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:29:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:29:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:29:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:29:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:29:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:29:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:29:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:29:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:29:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:29:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:29:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:29:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:29:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:29:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:29:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:29:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:29:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:29:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:29:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:29:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:29:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:29:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:29:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:29:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:29:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:29:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:29:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:29:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:29:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:29:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:29:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:29:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:29:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:29:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:29:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:29:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:29:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:29:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:29:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:29:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:29:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:29:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:29:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:29:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:29:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:29:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:29:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:29:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:29:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:29:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:29:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:29:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:29:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:29:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:29:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:29:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:29:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:29:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:29:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:29:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:29:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:29:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:29:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:29:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:29:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:29:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:29:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:29:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:29:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:29:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:29:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:29:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:29:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:29:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:29:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:29:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:29:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:29:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:29:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:29:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:29:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:29:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:29:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:29:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:29:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:29:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:29:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:29:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:29:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:29:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:29:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:29:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:29:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:29:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:29:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:29:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:29:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:29:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:29:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:29:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:29:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:29:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:29:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:29:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:29:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:29:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:29:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:29:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:29:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:29:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:29:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:29:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:29:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:29:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:29:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:29:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:29:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:29:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:29:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:29:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:29:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:29:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:29:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:29:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:29:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:29:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:29:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:29:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:29:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:29:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:29:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:29:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:29:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:29:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:29:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:29:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:29:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:29:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:29:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:29:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:29:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:29:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:29:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:29:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:29:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:29:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:29:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:29:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:29:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:29:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:29:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:29:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:29:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:29:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:29:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:29:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:29:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:29:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:29:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:29:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:29:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:29:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:29:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:29:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:29:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:29:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:30:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:30:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:30:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:30:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:30:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:30:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:30:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:30:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:30:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:30:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:30:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:30:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:30:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:30:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:30:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:30:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:30:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:30:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:30:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:30:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:30:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:30:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:30:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:30:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:30:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:30:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:30:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:30:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:30:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:30:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:30:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:30:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:30:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:30:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:30:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:30:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:30:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:30:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:30:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:30:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:30:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:30:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:30:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:30:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:30:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:30:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:30:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:30:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:30:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:30:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:30:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:30:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:30:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:30:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:30:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:30:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:30:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:30:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:30:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:30:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:30:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:30:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:30:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:30:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:30:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:30:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:30:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:30:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:30:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:30:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:30:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:30:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:30:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:30:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:30:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:30:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:30:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:30:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:30:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:30:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:30:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:30:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:30:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:30:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:30:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:30:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:30:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:30:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:30:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:30:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:30:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:30:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:30:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:30:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:30:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:30:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:30:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:30:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:30:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:30:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:30:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:30:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:30:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:30:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:30:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:30:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:30:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:30:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:30:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:30:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:30:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:30:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:30:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:30:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:30:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:30:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:30:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:30:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:30:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:30:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:30:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:30:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:30:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:30:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:30:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:30:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:30:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:30:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:30:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:30:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:30:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:30:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:30:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:30:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:30:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:30:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:30:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:30:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:30:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:30:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:30:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:30:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:30:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:30:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:30:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:30:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:30:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:30:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:30:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:30:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:30:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:30:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:30:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:30:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:30:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:30:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:30:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:30:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:30:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:30:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:30:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:30:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:30:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:30:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:30:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:30:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:30:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:30:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:30:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:30:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:30:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:30:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:30:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:30:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:30:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:30:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:30:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:30:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:30:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:30:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:30:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:30:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:30:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:30:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:30:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:30:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:30:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:30:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:30:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:30:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:30:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:30:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:30:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:30:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:30:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:30:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:30:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:30:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:30:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:30:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:30:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:30:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:30:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:30:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:30:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:30:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:30:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:30:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:30:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:30:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:30:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:30:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:30:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:30:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:30:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:30:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:31:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:31:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:31:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:31:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:31:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:31:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:31:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:31:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:31:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:31:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:31:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:31:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:31:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:31:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:31:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:31:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:31:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:31:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:31:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:31:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:31:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:31:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:31:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:31:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:31:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:31:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:31:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:31:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:31:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:31:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:31:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:31:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:31:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:31:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:31:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:31:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:31:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:31:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:31:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:31:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:31:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:31:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:31:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:31:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:31:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:31:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:31:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:31:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:31:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:31:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:31:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:31:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:31:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:31:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:31:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:31:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:31:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:31:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:31:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:31:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:31:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:31:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:31:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:31:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:31:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:31:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:31:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:31:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:31:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:31:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:31:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:31:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:31:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:31:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:31:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:31:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:31:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:31:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:31:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:31:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:31:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:31:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:31:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:31:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:31:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:31:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:31:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:31:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:31:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:31:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:31:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:31:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:31:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:31:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:31:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:31:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:31:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:31:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:31:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:31:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:31:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:31:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:31:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:31:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:31:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:31:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:31:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:31:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:31:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:31:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:31:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:31:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:31:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:31:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:31:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:31:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:31:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:31:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:31:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:31:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:31:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:31:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:31:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:31:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:31:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:31:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:31:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:31:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:31:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:31:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:31:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:31:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:31:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:31:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:31:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:31:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:31:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:31:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:31:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:31:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:31:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:31:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:31:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:31:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:31:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:31:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:31:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:31:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:31:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:31:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:31:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:31:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:31:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:31:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:31:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:31:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:31:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:31:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:31:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:31:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:31:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:31:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:31:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:31:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:31:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:31:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:31:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:31:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:31:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:31:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:31:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:31:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:31:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:31:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:31:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:31:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:31:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:31:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:31:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:31:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:31:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:31:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:31:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:31:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:31:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:31:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:31:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:31:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:31:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:31:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:31:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:31:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:31:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:31:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:31:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:31:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:31:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:31:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:31:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:31:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:31:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:31:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:31:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:31:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:31:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:31:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:31:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:31:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:31:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:31:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:31:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:31:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:31:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:31:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:31:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:31:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:32:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:32:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:32:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:32:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:32:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:32:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:32:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:32:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:32:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:32:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:32:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:32:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:32:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:32:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:32:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:32:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:32:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:32:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:32:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:32:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:32:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:32:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:32:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:32:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:32:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:32:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:32:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:32:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:32:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:32:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:32:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:32:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:32:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:32:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:32:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:32:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:32:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:32:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:32:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:32:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:32:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:32:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:32:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:32:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:32:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:32:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:32:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:32:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:32:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:32:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:32:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:32:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:32:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:32:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:32:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:32:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:32:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:32:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:32:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:32:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:32:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:32:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:32:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:32:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:32:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:32:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:32:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:32:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:32:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:32:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:32:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:32:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:32:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:32:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:32:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:32:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:32:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:32:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:32:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:32:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:32:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:32:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:32:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:32:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:32:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:32:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:32:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:32:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:32:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:32:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:32:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:32:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:32:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:32:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:32:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:32:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:32:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:32:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:32:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:32:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:32:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:32:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:32:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:32:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:32:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:32:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:32:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:32:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:32:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:32:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:32:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:32:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:32:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:32:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:32:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:32:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:32:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:32:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:32:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:32:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:32:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:32:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:32:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:32:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:32:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:32:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:32:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:32:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:32:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:32:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:32:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:32:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:32:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:32:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:32:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:32:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:32:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:32:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:32:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:32:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:32:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:32:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:32:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:32:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:32:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:32:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:32:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:32:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:32:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:32:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:32:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:32:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:32:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:32:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:32:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:32:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:32:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:32:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:32:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:32:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:32:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:32:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:32:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:32:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:32:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:32:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:32:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:32:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:32:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:32:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:32:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:32:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:32:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:32:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:32:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:32:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:32:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:32:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:32:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:32:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:32:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:32:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:32:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:32:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:32:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:32:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:32:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:32:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:32:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:32:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:32:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:32:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:32:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:32:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:32:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:32:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:32:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:32:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:32:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:32:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:32:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:32:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:32:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:32:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:32:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:32:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:32:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:32:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:32:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:32:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:32:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:32:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:32:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:32:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:32:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:32:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:33:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:33:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:33:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:33:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:33:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:33:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:33:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:33:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:33:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:33:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:33:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:33:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:33:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:33:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:33:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:33:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:33:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:33:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:33:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:33:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:33:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:33:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:33:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:33:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:33:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:33:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:33:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:33:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:33:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:33:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:33:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:33:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:33:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:33:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:33:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:33:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:33:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:33:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:33:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:33:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:33:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:33:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:33:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:33:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:33:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:33:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:33:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:33:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:33:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:33:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:33:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:33:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:33:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:33:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:33:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:33:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:33:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:33:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:33:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:33:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:33:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:33:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:33:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:33:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:33:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:33:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:33:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:33:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:33:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:33:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:33:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:33:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:33:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:33:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:33:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:33:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:33:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:33:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:33:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:33:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:33:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:33:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:33:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:33:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:33:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:33:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:33:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:33:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:33:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:33:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:33:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:33:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:33:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:33:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:33:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:33:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:33:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:33:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:33:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:33:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:33:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:33:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:33:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:33:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:33:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:33:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:33:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:33:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:33:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:33:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:33:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:33:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:33:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:33:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:33:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:33:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:33:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:33:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:33:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:33:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:33:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:33:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:33:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:33:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:33:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:33:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:33:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:33:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:33:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:33:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:33:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:33:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:33:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:33:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:33:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:33:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:33:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:33:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:33:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:33:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:33:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:33:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:33:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:33:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:33:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:33:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:33:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:33:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:33:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:33:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:33:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:33:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:33:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:33:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:33:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:33:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:33:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:33:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:33:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:33:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:33:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:33:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:33:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:33:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:33:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:33:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:33:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:33:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:33:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:33:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:33:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:33:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:33:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:33:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:33:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:33:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:33:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:33:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:33:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:33:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:33:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:33:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:33:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:33:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:33:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:33:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:33:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:33:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:33:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:33:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:33:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:33:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:33:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:33:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:33:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:33:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:33:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:33:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:33:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:33:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:33:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:33:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:33:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:33:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:33:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:33:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:33:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:33:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:33:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:33:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:33:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:33:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:33:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:33:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:33:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:33:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:34:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:34:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:34:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:34:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:34:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:34:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:34:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:34:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:34:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:34:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:34:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:34:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:34:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:34:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:34:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:34:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:34:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:34:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:34:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:34:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:34:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:34:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:34:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:34:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:34:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:34:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:34:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:34:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:34:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:34:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:34:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:34:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:34:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:34:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:34:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:34:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:34:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:34:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:34:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:34:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:34:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:34:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:34:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:34:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:34:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:34:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:34:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:34:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:34:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:34:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:34:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:34:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:34:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:34:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:34:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:34:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:34:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:34:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:34:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:34:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:34:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:34:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:34:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:34:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:34:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:34:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:34:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:34:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:34:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:34:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:34:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:34:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:34:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:34:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:34:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:34:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:34:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:34:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:34:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:34:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:34:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:34:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:34:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:34:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:34:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:34:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:34:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:34:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:34:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:34:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:34:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:34:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:34:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:34:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:34:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:34:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:34:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:34:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:34:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:34:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:34:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:34:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:34:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:34:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:34:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:34:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:34:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:34:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:34:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:34:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:34:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:34:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:34:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:34:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:34:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:34:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:34:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:34:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:34:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:34:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:34:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:34:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:34:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:34:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:34:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:34:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:34:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:34:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:34:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:34:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:34:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:34:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:34:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:34:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:34:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:34:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:34:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:34:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:34:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:34:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:34:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:34:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:34:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:34:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:34:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:34:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:34:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:34:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:34:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:34:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:34:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:34:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:34:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:34:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:34:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:34:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:34:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:34:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:34:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:34:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:34:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:34:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:34:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:34:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:34:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:34:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:34:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:34:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:34:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:34:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:34:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:34:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:34:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:34:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:34:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:34:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:34:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:34:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:34:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:34:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:34:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:34:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:34:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:34:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:34:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:34:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:34:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:34:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:34:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:34:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:34:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:34:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:34:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:34:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:34:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:34:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:34:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:34:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:34:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:34:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:34:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:34:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:34:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:34:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:34:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:34:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:34:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:34:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:34:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:34:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:34:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:34:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:34:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:34:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:34:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:34:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:35:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:35:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:35:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:35:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:35:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:35:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:35:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:35:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:35:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:35:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:35:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:35:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:35:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:35:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:35:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:35:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:35:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:35:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:35:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:35:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:35:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:35:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:35:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:35:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:35:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:35:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:35:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:35:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:35:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:35:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:35:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:35:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:35:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:35:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:35:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:35:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:35:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:35:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:35:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:35:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:35:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:35:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:35:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:35:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:35:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:35:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:35:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:35:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:35:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:35:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:35:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:35:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:35:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:35:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:35:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:35:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:35:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:35:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:35:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:35:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:35:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:35:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:35:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:35:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:35:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:35:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:35:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:35:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:35:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:35:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:35:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:35:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:35:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:35:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:35:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:35:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:35:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:35:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:35:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:35:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:35:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:35:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:35:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:35:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:35:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:35:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:35:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:35:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:35:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:35:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:35:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:35:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:35:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:35:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:35:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:35:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:35:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:35:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:35:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:35:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:35:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:35:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:35:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:35:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:35:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:35:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:35:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:35:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:35:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:35:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:35:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:35:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:35:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:35:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:35:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:35:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:35:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:35:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:35:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:35:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:35:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:35:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:35:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:35:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:35:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:35:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:35:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:35:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:35:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:35:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:35:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:35:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:35:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:35:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:35:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:35:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:35:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:35:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:35:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:35:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:35:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:35:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:35:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:35:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:35:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:35:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:35:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:35:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:35:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:35:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:35:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:35:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:35:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:35:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:35:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:35:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:35:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:35:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:35:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:35:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:35:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:35:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:35:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:35:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:35:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:35:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:35:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:35:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:35:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:35:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:35:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:35:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:35:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:35:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:35:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:35:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:35:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:35:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:35:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:35:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:35:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:35:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:35:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:35:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:35:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:35:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:35:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:35:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:35:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:35:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:35:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:35:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:35:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:35:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:35:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:35:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:35:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:35:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:35:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:35:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:35:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:35:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:35:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:35:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:35:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:35:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:35:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:35:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:35:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:35:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:35:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:35:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:35:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:35:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:35:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:35:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:36:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:36:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:36:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:36:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:36:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:36:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:36:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:36:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:36:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:36:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:36:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:36:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:36:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:36:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:36:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:36:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:36:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:36:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:36:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:36:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:36:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:36:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:36:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:36:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:36:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:36:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:36:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:36:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:36:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:36:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:36:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:36:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:36:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:36:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:36:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:36:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:36:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:36:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:36:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:36:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:36:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:36:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:36:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:36:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:36:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:36:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:36:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:36:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:36:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:36:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:36:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:36:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:36:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:36:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:36:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:36:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:36:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:36:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:36:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:36:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:36:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:36:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:36:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:36:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:36:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:36:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:36:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:36:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:36:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:36:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:36:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:36:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:36:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:36:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:36:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:36:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:36:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:36:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:36:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:36:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:36:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:36:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:36:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:36:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:36:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:36:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:36:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:36:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:36:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:36:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:36:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:36:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:36:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:36:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:36:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:36:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:36:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:36:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:36:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:36:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:36:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:36:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:36:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:36:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:36:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:36:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:36:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:36:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:36:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:36:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:36:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:36:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:36:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:36:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:36:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:36:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:36:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:36:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:36:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:36:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:36:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:36:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:36:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:36:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:36:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:36:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:38:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:38:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:38:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:38:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:38:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:38:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:38:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:38:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:38:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:38:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:38:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:38:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:38:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:38:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:38:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:38:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:38:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:38:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:38:36 rdma06 rsyslogd[1041]: imjournal: 5082 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 20:38:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:38:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:38:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:38:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:38:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:38:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:38:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:38:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:38:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:38:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:38:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:38:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:38:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:38:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:38:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:38:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:38:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:38:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:38:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:38:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:38:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:38:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:38:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:38:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:38:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:38:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:38:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:38:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:38:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:38:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:38:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:38:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:38:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:38:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:38:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:38:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:38:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:38:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:38:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:38:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:38:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:38:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:38:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:38:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:38:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:38:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:38:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:38:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:38:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:38:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:38:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:38:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:38:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:38:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:38:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:38:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:38:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:38:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:38:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:38:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:38:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:38:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:38:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:38:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:38:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:38:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:38:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:38:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:38:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:38:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:38:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:38:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:39:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:39:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:39:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:39:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:39:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:39:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:39:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:39:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:39:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:39:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:39:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:39:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:39:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:39:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:39:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:39:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:39:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:39:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:39:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:39:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:39:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:39:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:39:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:39:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:39:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:39:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:39:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:39:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:39:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:39:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:39:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:39:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:39:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:39:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:39:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:39:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:39:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:39:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:39:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:39:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:39:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:39:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:39:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:39:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:39:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:39:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:39:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:39:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:39:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:39:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:39:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:39:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:39:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:39:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:39:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:39:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:39:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:39:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:39:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:39:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:39:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:39:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:39:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:39:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:39:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:39:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:39:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:39:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:39:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:39:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:39:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:39:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:39:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:39:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:39:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:39:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:39:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:39:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:39:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:39:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:39:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:39:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:39:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:39:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:39:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:39:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:39:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:39:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:39:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:39:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:39:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:39:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:39:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:39:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:39:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:39:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:39:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:39:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:39:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:39:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:39:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:39:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:39:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:39:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:39:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:39:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:39:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:39:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:39:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:39:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:39:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:39:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:39:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:39:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:39:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:39:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:39:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:39:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:39:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:39:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:39:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:39:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:39:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:39:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:39:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:39:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:39:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:39:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:39:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:39:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:39:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:39:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:39:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:39:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:39:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:39:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:39:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:39:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:39:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:39:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:39:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:39:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:39:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:39:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:39:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:39:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:39:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:39:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:39:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:39:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:39:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:39:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:39:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:39:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:39:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:39:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:39:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:39:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:39:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:39:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:39:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:39:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:39:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:39:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:39:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:39:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:39:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:39:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:39:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:39:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:39:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:39:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:39:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:39:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:39:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:39:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:39:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:39:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:39:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:39:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:39:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:39:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:39:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:39:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:39:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:39:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:39:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:39:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:39:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:39:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:39:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:39:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:39:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:39:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:39:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:39:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:39:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:39:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:39:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:39:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:39:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:39:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:39:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:39:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:39:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:39:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:39:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:39:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:39:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:39:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:39:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:39:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:39:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:39:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:39:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:39:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:40:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:40:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:40:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:40:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:40:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:40:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:40:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:40:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:40:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:40:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:40:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:40:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:40:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:40:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:40:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:40:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:40:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:40:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:40:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:40:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:40:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:40:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:40:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:40:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:40:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:40:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:40:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:40:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:40:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:40:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:40:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:40:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:40:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:40:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:40:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:40:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:40:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:40:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:40:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:40:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:40:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:40:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:40:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:40:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:40:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:40:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:40:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:40:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:40:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:40:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:40:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:40:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:40:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:40:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:40:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:40:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:40:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:40:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:40:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:40:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:40:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:40:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:40:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:40:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:40:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:40:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:40:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:40:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:40:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:40:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:40:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:40:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:40:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:40:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:40:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:40:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:40:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:40:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:40:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:40:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:40:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:40:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:40:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:40:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:40:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:40:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:40:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:40:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:40:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:40:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:40:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:40:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:40:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:40:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:40:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:40:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:40:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:40:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:40:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:40:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:40:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:40:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:40:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:40:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:40:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:40:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:40:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:40:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:40:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:40:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:40:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:40:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:40:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:40:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:40:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:40:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:40:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:40:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:40:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:40:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:40:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:40:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:40:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:40:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:40:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:40:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:40:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:40:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:40:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:40:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:40:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:40:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:40:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:40:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:40:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:40:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:40:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:40:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:40:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:40:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:40:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:40:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:40:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:40:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:40:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:40:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:40:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:40:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:40:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:40:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:40:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:40:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:40:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:40:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:40:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:40:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:40:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:40:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:40:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:40:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:40:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:40:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:40:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:40:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:40:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:40:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:40:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:40:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:40:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:40:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:40:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:40:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:40:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:40:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:40:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:40:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:40:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:40:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:40:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:40:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:40:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:40:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:40:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:40:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:40:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:40:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:40:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:40:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:40:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:40:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:40:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:40:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:40:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:40:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:40:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:40:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:40:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:40:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:40:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:40:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:40:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:40:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:40:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:40:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:40:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:40:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:40:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:40:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:40:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:40:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:40:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:40:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:40:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:40:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:40:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:40:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:41:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:41:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:41:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:41:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:41:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:41:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:41:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:41:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:41:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:41:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:41:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:41:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:41:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:41:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:41:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:41:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:41:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:41:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:41:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:41:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:41:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:41:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:41:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:41:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:41:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:41:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:41:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:41:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:41:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:41:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:41:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:41:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:41:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:41:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:41:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:41:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:41:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:41:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:41:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:41:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:41:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:41:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:41:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:41:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:41:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:41:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:41:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:41:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:41:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:41:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:41:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:41:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:41:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:41:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:41:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:41:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:41:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:41:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:41:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:41:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:41:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:41:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:41:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:41:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:41:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:41:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:41:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:41:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:41:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:41:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:41:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:41:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:41:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:41:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:41:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:41:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:41:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:41:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:41:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:41:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:41:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:41:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:41:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:41:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:41:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:41:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:41:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:41:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:41:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:41:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:41:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:41:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:41:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:41:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:41:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:41:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:41:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:41:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:41:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:41:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:41:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:41:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:41:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:41:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:41:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:41:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:41:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:41:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:41:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:41:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:41:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:41:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:41:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:41:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:41:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:41:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:41:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:41:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:41:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:41:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:41:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:41:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:41:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:41:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:41:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:41:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:41:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:41:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:41:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:41:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:41:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:41:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:41:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:41:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:41:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:41:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:41:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:41:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:41:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:41:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:41:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:41:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:41:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:41:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:41:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:41:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:41:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:41:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:41:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:41:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:41:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:41:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:41:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:41:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:41:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:41:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:41:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:41:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:41:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:41:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:41:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:41:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:41:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:41:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:41:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:41:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:41:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:41:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:41:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:41:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:41:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:41:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:41:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:41:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:41:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:41:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:41:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:41:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:41:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:41:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:41:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:41:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:41:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:41:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:41:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:41:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:41:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:41:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:41:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:41:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:41:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:41:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:41:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:41:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:41:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:41:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:41:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:41:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:41:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:41:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:41:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:41:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:41:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:41:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:41:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:41:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:41:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:41:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:41:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:41:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:41:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:41:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:41:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:41:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:41:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:41:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:42:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:42:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:42:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:42:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:42:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:42:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:42:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:42:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:42:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:42:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:42:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:42:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:42:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:42:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:42:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:42:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:42:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:42:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:42:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:42:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:42:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:42:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:42:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:42:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:42:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:42:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:42:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:42:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:42:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:42:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:42:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:42:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:42:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:42:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:42:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:42:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:42:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:42:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:42:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:42:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:42:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:42:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:42:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:42:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:42:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:42:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:42:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:42:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:42:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:42:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:42:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:42:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:42:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:42:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:42:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:42:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:42:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:42:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:42:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:42:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:42:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:42:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:42:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:42:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:42:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:42:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:42:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:42:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:42:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:42:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:42:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:42:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:42:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:42:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:42:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:42:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:42:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:42:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:42:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:42:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:42:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:42:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:42:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:42:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:42:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:42:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:42:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:42:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:42:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:42:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:42:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:42:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:42:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:42:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:42:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:42:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:42:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:42:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:42:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:42:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:42:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:42:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:42:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:42:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:42:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:42:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:42:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:42:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:42:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:42:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:42:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:42:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:42:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:42:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:42:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:42:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:42:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:42:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:42:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:42:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:42:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:42:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:42:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:42:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:42:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:42:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:42:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:42:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:42:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:42:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:42:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:42:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:42:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:42:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:42:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:42:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:42:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:42:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:42:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:42:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:42:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:42:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:42:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:42:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:42:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:42:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:42:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:42:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:42:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:42:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:42:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:42:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:42:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:42:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:42:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:42:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:42:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:42:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:42:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:42:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:42:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:42:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:42:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:42:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:42:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:42:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:42:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:42:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:42:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:42:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:42:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:42:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:42:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:42:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:42:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:42:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:42:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:42:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:42:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:42:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:42:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:42:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:42:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:42:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:42:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:42:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:42:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:42:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:42:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:42:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:42:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:42:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:42:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:42:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:42:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:42:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:42:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:42:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:42:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:42:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:42:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:42:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:42:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:42:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:42:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:42:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:42:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:42:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:42:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:42:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:42:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:42:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:42:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:42:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:42:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:42:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:43:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:43:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:43:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:43:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:43:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:43:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:43:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:43:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:43:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:43:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:43:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:43:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:43:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:43:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:43:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:43:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:43:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:43:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:43:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:43:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:43:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:43:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:43:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:43:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:43:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:43:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:43:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:43:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:43:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:43:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:43:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:43:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:43:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:43:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:43:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:43:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:43:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:43:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:43:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:43:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:43:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:43:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:43:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:43:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:43:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:43:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:43:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:43:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:43:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:43:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:43:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:43:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:43:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:43:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:43:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:43:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:43:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:43:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:43:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:43:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:43:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:43:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:43:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:43:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:43:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:43:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:43:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:43:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:43:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:43:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:43:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:43:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:43:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:43:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:43:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:43:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:43:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:43:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:43:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:43:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:43:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:43:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:43:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:43:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:43:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:43:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:43:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:43:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:43:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:43:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:43:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:43:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:43:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:43:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:43:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:43:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:43:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:43:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:43:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:43:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:43:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:43:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:43:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:43:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:43:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:43:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:43:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:43:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:43:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:43:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:43:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:43:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:43:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:43:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:43:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:43:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:43:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:43:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:43:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:43:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:43:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:43:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:43:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:43:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:43:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:43:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:43:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:43:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:43:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:43:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:43:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:43:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:43:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:43:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:43:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:43:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:43:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:43:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:43:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:43:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:43:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:43:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:43:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:43:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:43:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:43:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:43:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:43:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:43:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:43:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:43:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:43:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:43:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:43:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:43:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:43:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:43:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:43:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:43:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:43:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:43:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:43:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:43:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:43:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:43:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:43:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:43:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:43:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:43:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:43:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:43:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:43:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:43:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:43:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:43:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:43:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:43:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:43:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:43:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:43:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:43:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:43:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:43:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:43:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:43:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:43:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:43:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:43:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:43:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:43:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:43:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:43:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:43:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:43:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:43:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:43:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:43:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:43:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:43:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:43:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:43:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:43:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:43:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:43:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:43:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:43:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:43:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:43:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:43:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:43:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:43:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:43:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:43:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:43:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:43:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:43:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:44:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:44:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:44:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:44:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:44:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:44:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:44:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:44:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:44:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:44:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:44:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:44:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:44:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:44:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:44:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:44:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:44:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:44:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:44:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:44:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:44:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:44:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:44:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:44:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:44:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:44:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:44:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:44:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:44:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:44:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:44:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:44:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:44:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:44:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:44:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:44:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:44:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:44:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:44:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:44:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:44:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:44:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:44:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:44:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:44:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:44:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:44:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:44:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:44:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:44:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:44:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:44:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:44:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:44:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:44:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:44:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:44:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:44:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:44:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:44:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:44:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:44:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:44:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:44:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:44:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:44:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:44:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:44:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:44:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:44:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:44:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:44:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:44:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:44:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:44:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:44:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:44:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:44:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:44:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:44:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:44:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:44:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:44:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:44:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:44:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:44:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:44:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:44:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:44:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:44:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:44:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:44:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:44:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:44:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:44:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:44:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:44:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:44:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:44:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:44:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:44:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:44:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:44:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:44:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:44:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:44:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:44:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:44:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:44:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:44:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:44:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:44:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:44:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:44:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:44:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:44:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:44:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:44:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:44:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:44:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:44:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:44:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:44:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:44:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:44:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:44:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:44:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:44:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:44:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:44:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:44:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:44:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:44:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:44:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:44:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:44:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:44:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:44:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:44:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:44:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:44:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:44:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:44:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:44:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:44:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:44:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:44:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:44:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:44:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:44:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:44:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:44:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:44:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:44:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:44:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:44:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:44:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:44:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:44:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:44:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:44:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:44:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:44:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:44:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:44:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:44:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:44:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:44:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:44:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:44:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:44:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:44:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:44:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:44:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:44:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:44:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:44:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:44:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:44:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:44:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:44:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:44:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:44:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:44:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:44:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:44:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:44:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:44:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:44:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:44:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:44:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:44:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:44:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:44:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:44:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:44:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:44:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:44:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:44:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:44:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:44:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:44:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:44:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:44:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:44:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:44:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:44:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:44:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:44:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:44:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:44:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:44:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:44:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:44:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:44:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:44:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:45:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:45:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:45:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:45:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:45:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:45:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:45:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:45:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:45:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:45:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:45:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:45:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:45:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:45:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:45:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:45:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:45:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:45:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:45:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:45:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:45:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:45:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:45:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:45:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:45:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:45:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:45:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:45:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:45:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:45:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:45:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:45:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:45:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:45:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:45:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:45:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:45:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:45:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:45:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:45:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:45:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:45:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:45:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:45:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:45:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:45:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:45:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:45:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:45:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:45:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:45:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:45:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:45:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:45:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:45:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:45:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:45:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:45:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:45:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:45:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:45:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:45:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:45:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:45:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:45:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:45:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:45:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:45:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:45:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:45:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:45:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:45:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:45:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:45:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:45:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:45:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:45:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:45:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:45:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:45:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:45:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:45:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:45:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:45:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:45:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:45:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:45:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:45:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:45:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:45:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:45:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:45:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:45:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:45:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:45:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:45:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:45:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:45:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:45:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:45:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:45:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:45:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:45:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:45:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:45:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:45:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:45:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:45:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:45:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:45:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:45:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:45:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:45:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:45:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:45:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:45:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:45:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:45:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:45:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:45:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:45:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:45:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:45:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:45:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:45:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:45:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:45:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:45:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:45:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:45:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:45:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:45:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:45:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:45:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:45:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:45:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:45:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:45:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:45:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:45:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:45:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:45:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:45:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:45:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:45:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:45:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:45:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:45:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:45:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:45:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:45:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:45:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:45:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:45:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:45:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:45:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:45:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:45:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:45:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:45:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:45:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:45:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:45:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:45:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:45:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:45:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:45:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:45:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:45:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:45:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:45:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:45:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:45:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:45:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:45:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:45:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:45:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:45:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:45:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:45:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:45:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:45:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:45:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:45:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:45:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:45:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:45:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:45:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:45:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:45:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:45:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:45:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:45:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:45:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:45:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:45:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:45:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:45:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:45:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:45:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:45:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:45:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:45:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:45:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:45:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:45:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:45:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:45:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:45:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:45:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:45:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:45:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:45:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:45:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:45:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:45:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:46:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:46:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:46:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:46:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:46:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:46:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:46:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:46:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:46:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:46:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:46:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:46:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:46:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:46:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:46:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:46:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:46:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:46:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:46:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:46:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:46:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:46:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:46:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:46:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:46:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:46:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:46:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:46:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:46:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:46:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:46:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:46:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:46:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:46:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:46:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:46:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:46:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:46:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:46:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:46:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:46:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:46:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:46:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:46:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:46:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:46:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:46:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:46:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:46:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:46:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:46:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:46:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:46:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:46:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:46:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:46:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:46:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:46:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:46:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:46:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:46:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:46:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:46:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:46:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:46:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:46:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:46:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:46:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:46:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:46:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:46:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:46:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:46:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:46:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:46:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:46:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:46:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:46:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:46:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:46:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:46:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:46:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:46:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:46:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:46:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:46:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:46:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:46:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:46:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:46:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:46:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:46:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:46:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:46:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:46:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:46:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:46:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:46:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:46:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:46:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:46:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:46:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:46:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:46:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:46:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:46:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:46:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:46:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:46:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:46:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:46:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:46:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:46:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:46:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:46:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:46:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:46:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:46:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:46:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:46:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:46:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:46:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:46:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:46:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:46:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:46:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:48:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:48:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:48:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:48:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:48:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:48:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:48:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:48:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:48:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:48:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:48:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:48:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:48:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:48:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:48:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:48:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:48:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:48:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:48:41 rdma06 rsyslogd[1041]: imjournal: 5295 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 20:48:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:48:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:48:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:48:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:48:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:48:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:48:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:48:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:48:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:48:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:48:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:48:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:48:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:48:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:48:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:48:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:48:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:48:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:48:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:48:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:48:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:48:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:48:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:48:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:48:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:48:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:48:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:48:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:48:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:48:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:48:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:48:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:48:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:48:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:48:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:48:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:48:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:48:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:48:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:48:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:48:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:48:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:48:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:48:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:48:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:48:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:48:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:48:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:48:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:48:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:48:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:48:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:48:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:48:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:49:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:49:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:49:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:49:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:49:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:49:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:49:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:49:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:49:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:49:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:49:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:49:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:49:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:49:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:49:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:49:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:49:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:49:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:49:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:49:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:49:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:49:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:49:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:49:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:49:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:49:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:49:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:49:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:49:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:49:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:49:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:49:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:49:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:49:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:49:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:49:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:49:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:49:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:49:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:49:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:49:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:49:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:49:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:49:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:49:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:49:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:49:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:49:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:49:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:49:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:49:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:49:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:49:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:49:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:49:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:49:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:49:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:49:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:49:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:49:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:49:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:49:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:49:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:49:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:49:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:49:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:49:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:49:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:49:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:49:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:49:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:49:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:49:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:49:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:49:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:49:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:49:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:49:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:49:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:49:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:49:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:49:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:49:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:49:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:49:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:49:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:49:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:49:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:49:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:49:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:49:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:49:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:49:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:49:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:49:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:49:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:49:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:49:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:49:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:49:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:49:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:49:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:49:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:49:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:49:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:49:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:49:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:49:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:49:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:49:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:49:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:49:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:49:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:49:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:49:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:49:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:49:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:49:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:49:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:49:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:49:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:49:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:49:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:49:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:49:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:49:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:49:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:49:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:49:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:49:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:49:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:49:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:49:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:49:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:49:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:49:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:49:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:49:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:49:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:49:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:49:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:49:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:49:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:49:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:49:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:49:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:49:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:49:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:49:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:49:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:49:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:49:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:49:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:49:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:49:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:49:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:49:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:49:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:49:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:49:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:49:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:49:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:49:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:49:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:49:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:49:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:49:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:49:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:49:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:49:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:49:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:49:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:49:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:49:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:49:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:49:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:49:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:49:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:49:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:49:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:49:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:49:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:49:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:49:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:49:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:49:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:49:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:49:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:49:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:49:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:49:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:49:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:49:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:49:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:49:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:49:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:49:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:49:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:49:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:49:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:49:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:49:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:49:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:49:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:49:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:49:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:49:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:49:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:49:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:49:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:49:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:49:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:49:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:49:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:49:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:49:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:50:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:50:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:50:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:50:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:50:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:50:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:50:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:50:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:50:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:50:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:50:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:50:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:50:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:50:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:50:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:50:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:50:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:50:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:50:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:50:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:50:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:50:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:50:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:50:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:50:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:50:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:50:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:50:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:50:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:50:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:50:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:50:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:50:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:50:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:50:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:50:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:50:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:50:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:50:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:50:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:50:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:50:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:50:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:50:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:50:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:50:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:50:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:50:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:50:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:50:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:50:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:50:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:50:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:50:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:50:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:50:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:50:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:50:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:50:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:50:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:50:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:50:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:50:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:50:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:50:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:50:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:50:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:50:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:50:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:50:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:50:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:50:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:50:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:50:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:50:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:50:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:50:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:50:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:50:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:50:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:50:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:50:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:50:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:50:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:50:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:50:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:50:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:50:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:50:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:50:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:50:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:50:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:50:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:50:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:50:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:50:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:50:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:50:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:50:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:50:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:50:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:50:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:50:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:50:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:50:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:50:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:50:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:50:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:50:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:50:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:50:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:50:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:50:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:50:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:50:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:50:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:50:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:50:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:50:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:50:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:50:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:50:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:50:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:50:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:50:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:50:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:50:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:50:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:50:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:50:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:50:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:50:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:50:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:50:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:50:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:50:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:50:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:50:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:50:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:50:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:50:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:50:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:50:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:50:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:50:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:50:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:50:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:50:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:50:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:50:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:50:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:50:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:50:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:50:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:50:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:50:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:50:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:50:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:50:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:50:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:50:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:50:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:50:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:50:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:50:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:50:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:50:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:50:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:50:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:50:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:50:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:50:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:50:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:50:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:50:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:50:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:50:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:50:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:50:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:50:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:50:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:50:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:50:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:50:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:50:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:50:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:50:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:50:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:50:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:50:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:50:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:50:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:50:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:50:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:50:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:50:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:50:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:50:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:50:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:50:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:50:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:50:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:50:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:50:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:50:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:50:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:50:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:50:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:50:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:50:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:50:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:50:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:50:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:50:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:50:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:50:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:51:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:51:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:51:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:51:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:51:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:51:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:51:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:51:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:51:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:51:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:51:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:51:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:51:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:51:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:51:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:51:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:51:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:51:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:51:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:51:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:51:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:51:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:51:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:51:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:51:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:51:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:51:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:51:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:51:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:51:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:51:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:51:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:51:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:51:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:51:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:51:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:51:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:51:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:51:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:51:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:51:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:51:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:51:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:51:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:51:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:51:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:51:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:51:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:51:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:51:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:51:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:51:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:51:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:51:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:51:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:51:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:51:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:51:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:51:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:51:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:51:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:51:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:51:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:51:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:51:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:51:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:51:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:51:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:51:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:51:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:51:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:51:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:51:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:51:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:51:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:51:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:51:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:51:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:51:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:51:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:51:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:51:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:51:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:51:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:51:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:51:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:51:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:51:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:51:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:51:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:51:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:51:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:51:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:51:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:51:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:51:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:51:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:51:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:51:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:51:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:51:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:51:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:51:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:51:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:51:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:51:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:51:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:51:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:51:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:51:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:51:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:51:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:51:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:51:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:51:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:51:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:51:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:51:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:51:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:51:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:51:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:51:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:51:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:51:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:51:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:51:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:51:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:51:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:51:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:51:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:51:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:51:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:51:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:51:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:51:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:51:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:51:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:51:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:51:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:51:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:51:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:51:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:51:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:51:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:51:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:51:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:51:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:51:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:51:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:51:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:51:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:51:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:51:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:51:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:51:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:51:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:51:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:51:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:51:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:51:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:51:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:51:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:51:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:51:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:51:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:51:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:51:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:51:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:51:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:51:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:51:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:51:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:51:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:51:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:51:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:51:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:51:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:51:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:51:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:51:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:51:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:51:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:51:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:51:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:51:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:51:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:51:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:51:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:51:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:51:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:51:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:51:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:51:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:51:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:51:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:51:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:51:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:51:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:51:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:51:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:51:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:51:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:51:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:51:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:51:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:51:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:51:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:51:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:51:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:51:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:51:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:51:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:51:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:51:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:51:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:51:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:52:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:52:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:52:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:52:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:52:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:52:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:52:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:52:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:52:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:52:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:52:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:52:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:52:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:52:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:52:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:52:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:52:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:52:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:52:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:52:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:52:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:52:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:52:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:52:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:52:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:52:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:52:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:52:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:52:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:52:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:52:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:52:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:52:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:52:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:52:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:52:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:52:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:52:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:52:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:52:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:52:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:52:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:52:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:52:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:52:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:52:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:52:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:52:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:52:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:52:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:52:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:52:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:52:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:52:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:52:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:52:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:52:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:52:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:52:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:52:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:52:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:52:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:52:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:52:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:52:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:52:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:52:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:52:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:52:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:52:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:52:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:52:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:52:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:52:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:52:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:52:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:52:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:52:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:52:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:52:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:52:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:52:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:52:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:52:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:52:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:52:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:52:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:52:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:52:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:52:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:52:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:52:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:52:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:52:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:52:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:52:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:52:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:52:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:52:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:52:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:52:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:52:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:52:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:52:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:52:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:52:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:52:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:52:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:52:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:52:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:52:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:52:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:52:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:52:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:52:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:52:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:52:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:52:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:52:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:52:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:52:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:52:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:52:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:52:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:52:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:52:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:52:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:52:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:52:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:52:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:52:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:52:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:52:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:52:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:52:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:52:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:52:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:52:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:52:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:52:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:52:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:52:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:52:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:52:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:52:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:52:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:52:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:52:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:52:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:52:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:52:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:52:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:52:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:52:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:52:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:52:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:52:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:52:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:52:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:52:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:52:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:52:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:52:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:52:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:52:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:52:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:52:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:52:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:52:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:52:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:52:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:52:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:52:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:52:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:52:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:52:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:52:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:52:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:52:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:52:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:52:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:52:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:52:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:52:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:52:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:52:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:52:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:52:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:52:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:52:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:52:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:52:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:52:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:52:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:52:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:52:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:52:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:52:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:52:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:52:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:52:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:52:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:52:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:52:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:52:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:52:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:52:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:52:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:52:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:52:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:52:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:52:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:52:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:52:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:52:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:52:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:53:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:53:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:53:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:53:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:53:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:53:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:53:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:53:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:53:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:53:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:53:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:53:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:53:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:53:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:53:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:53:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:53:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:53:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:53:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:53:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:53:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:53:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:53:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:53:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:53:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:53:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:53:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:53:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:53:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:53:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:53:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:53:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:53:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:53:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:53:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:53:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:53:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:53:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:53:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:53:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:53:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:53:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:53:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:53:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:53:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:53:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:53:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:53:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:53:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:53:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:53:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:53:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:53:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:53:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:53:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:53:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:53:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:53:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:53:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:53:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:53:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:53:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:53:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:53:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:53:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:53:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:53:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:53:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:53:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:53:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:53:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:53:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:53:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:53:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:53:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:53:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:53:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:53:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:53:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:53:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:53:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:53:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:53:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:53:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:53:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:53:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:53:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:53:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:53:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:53:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:53:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:53:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:53:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:53:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:53:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:53:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:53:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:53:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:53:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:53:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:53:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:53:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:53:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:53:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:53:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:53:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:53:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:53:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:53:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:53:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:53:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:53:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:53:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:53:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:53:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:53:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:53:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:53:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:53:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:53:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:53:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:53:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:53:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:53:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:53:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:53:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:53:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:53:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:53:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:53:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:53:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:53:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:53:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:53:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:53:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:53:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:53:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:53:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:53:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:53:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:53:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:53:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:53:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:53:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:53:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:53:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:53:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:53:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:53:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:53:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:53:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:53:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:53:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:53:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:53:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:53:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:53:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:53:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:53:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:53:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:53:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:53:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:53:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:53:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:53:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:53:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:53:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:53:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:53:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:53:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:53:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:53:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:53:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:53:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:53:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:53:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:53:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:53:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:53:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:53:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:53:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:53:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:53:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:53:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:53:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:53:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:53:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:53:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:53:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:53:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:53:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:53:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:53:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:53:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:53:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:53:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:53:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:53:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:53:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:53:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:53:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:53:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:53:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:53:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:53:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:53:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:53:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:53:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:53:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:53:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:53:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:53:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:53:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:53:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:53:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:53:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:54:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:54:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:54:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:54:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:54:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:54:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:54:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:54:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:54:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:54:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:54:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:54:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:54:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:54:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:54:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:54:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:54:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:54:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:54:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:54:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:54:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:54:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:54:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:54:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:54:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:54:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:54:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:54:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:54:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:54:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:54:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:54:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:54:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:54:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:54:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:54:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:54:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:54:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:54:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:54:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:54:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:54:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:54:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:54:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:54:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:54:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:54:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:54:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:54:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:54:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:54:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:54:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:54:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:54:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:54:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:54:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:54:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:54:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:54:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:54:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:54:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:54:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:54:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:54:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:54:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:54:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:54:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:54:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:54:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:54:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:54:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:54:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:54:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:54:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:54:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:54:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:54:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:54:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:54:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:54:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:54:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:54:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:54:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:54:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:54:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:54:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:54:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:54:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:54:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:54:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:54:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:54:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:54:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:54:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:54:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:54:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:54:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:54:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:54:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:54:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:54:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:54:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:54:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:54:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:54:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:54:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:54:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:54:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:54:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:54:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:54:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:54:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:54:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:54:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:54:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:54:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:54:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:54:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:54:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:54:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:54:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:54:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:54:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:54:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:54:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:54:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:54:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:54:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:54:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:54:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:54:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:54:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:54:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:54:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:54:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:54:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:54:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:54:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:54:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:54:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:54:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:54:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:54:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:54:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:54:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:54:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:54:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:54:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:54:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:54:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:54:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:54:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:54:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:54:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:54:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:54:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:54:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:54:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:54:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:54:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:54:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:54:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:54:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:54:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:54:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:54:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:54:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:54:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:54:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:54:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:54:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:54:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:54:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:54:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:54:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:54:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:54:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:54:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:54:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:54:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:54:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:54:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:54:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:54:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:54:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:54:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:54:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:54:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:54:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:54:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:54:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:54:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:54:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:54:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:54:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:54:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:54:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:54:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:54:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:54:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:54:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:54:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:54:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:54:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:54:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:54:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:54:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:54:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:54:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:54:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:54:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:54:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:54:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:54:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:54:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:54:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:55:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:55:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:55:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:55:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:55:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:55:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:55:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:55:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:55:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:55:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:55:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:55:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:55:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:55:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:55:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:55:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:55:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:55:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:55:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:55:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:55:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:55:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:55:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:55:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:55:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:55:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:55:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:55:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:55:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:55:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:55:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:55:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:55:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:55:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:55:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:55:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:55:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:55:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:55:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:55:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:55:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:55:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:55:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:55:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:55:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:55:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:55:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:55:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:55:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:55:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:55:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:55:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:55:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:55:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:55:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:55:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:55:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:55:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:55:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:55:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:55:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:55:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:55:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:55:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:55:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:55:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:55:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:55:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:55:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:55:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:55:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:55:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:55:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:55:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:55:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:55:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:55:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:55:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:55:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:55:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:55:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:55:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:55:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:55:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:55:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:55:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:55:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:55:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:55:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:55:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:55:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:55:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:55:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:55:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:55:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:55:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:55:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:55:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:55:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:55:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:55:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:55:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:55:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:55:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:55:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:55:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:55:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:55:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:55:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:55:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:55:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:55:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:55:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:55:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:55:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:55:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:55:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:55:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:55:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:55:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:55:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:55:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:55:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:55:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:55:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:55:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:55:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:55:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:55:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:55:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:55:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:55:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:55:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:55:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:55:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:55:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:55:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:55:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:55:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:55:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:55:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:55:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:55:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:55:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:55:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:55:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:55:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:55:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:55:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:55:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:55:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:55:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:55:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:55:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:55:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:55:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:55:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:55:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:55:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:55:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:55:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:55:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:55:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:55:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:55:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:55:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:55:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:55:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:55:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:55:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:55:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:55:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:55:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:55:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:55:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:55:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:55:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:55:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:55:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:55:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:55:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:55:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:55:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:55:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:55:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:55:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:55:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:55:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:55:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:55:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:55:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:55:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:55:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:55:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:55:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:55:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:55:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:55:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:55:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:55:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:55:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:55:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:55:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:55:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:55:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:55:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:55:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:55:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:55:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:55:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:55:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:55:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:55:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:55:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:55:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:55:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:56:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:56:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:56:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:56:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:56:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:56:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:56:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:56:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:56:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:56:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:56:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:56:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:56:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:56:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:56:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:56:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:56:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:56:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:56:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:56:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:56:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:56:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:56:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:56:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:56:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:56:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:56:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:56:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:56:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:56:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:56:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:56:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:56:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:56:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:56:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:56:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:56:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:56:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:56:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:56:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:56:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:56:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:56:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:56:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:56:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:56:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:56:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:56:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:56:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:56:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:56:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:56:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:56:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:56:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:56:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:56:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:56:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:56:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:56:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:56:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:56:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:56:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:56:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:56:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:56:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:56:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:56:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:56:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:56:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:56:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:56:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:56:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:56:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:56:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:56:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:56:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:56:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:56:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:56:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:56:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:56:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:56:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:56:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:56:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:56:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:56:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:56:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:56:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:56:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:56:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:56:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:56:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:56:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:56:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:56:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:56:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:56:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:56:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:56:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:56:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:56:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:56:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:56:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:56:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:56:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:56:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:56:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:56:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:56:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:56:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:56:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:56:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:56:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:56:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:56:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:56:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:56:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:56:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:56:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:56:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:56:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:56:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:56:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:56:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:56:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:56:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:56:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:56:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:56:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:56:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:56:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:56:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:56:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:56:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:56:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:56:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:56:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:56:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:56:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:56:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:56:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:56:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:56:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:56:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:58:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:58:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:58:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:58:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:58:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:58:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:58:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:58:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:58:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:58:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:58:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:58:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:58:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:58:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:58:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:58:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:58:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:58:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:58:46 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 20:58:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:58:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:58:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:58:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:58:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:58:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:58:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:58:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:58:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:58:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:58:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:58:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:58:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:58:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:58:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:58:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:58:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:58:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:58:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:58:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:58:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:58:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:58:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:58:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:58:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:58:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:58:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:58:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:58:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:58:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:58:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:58:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:58:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:58:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:58:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:58:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:59:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:59:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:59:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:59:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:59:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:59:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:59:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:59:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:59:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:59:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:59:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:59:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:59:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:59:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:59:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:59:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:59:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:59:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:59:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:59:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:59:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:59:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:59:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:59:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:59:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:59:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:59:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:59:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:59:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:59:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:59:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:59:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:59:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:59:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:59:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:59:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:59:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:59:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:59:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:59:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:59:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:59:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:59:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:59:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:59:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:59:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:59:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:59:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:59:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:59:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:59:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:59:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:59:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:59:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:59:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:59:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:59:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:59:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:59:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:59:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:59:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:59:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:59:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:59:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:59:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:59:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:59:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:59:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:59:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:59:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:59:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:59:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:59:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:59:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:59:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:59:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:59:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:59:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:59:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:59:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:59:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:59:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:59:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:59:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:59:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:59:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:59:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:59:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:59:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:59:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:59:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:59:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:59:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:59:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:59:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:59:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:59:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:59:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:59:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:59:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:59:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:59:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:59:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:59:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:59:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:59:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:59:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:59:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:59:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:59:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:59:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:59:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:59:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:59:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:59:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:59:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:59:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:59:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:59:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:59:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:59:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:59:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:59:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:59:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:59:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:59:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:59:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:59:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:59:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:59:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:59:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:59:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:59:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:59:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:59:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:59:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:59:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:59:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:59:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:59:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:59:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:59:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:59:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:59:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:59:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:59:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:59:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:59:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:59:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:59:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:59:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:59:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:59:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:59:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:59:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:59:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:59:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:59:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:59:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:59:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:59:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:59:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:59:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:59:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:59:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:59:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:59:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:59:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:59:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:59:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:59:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:59:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:59:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:59:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:59:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:59:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:59:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:59:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:59:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:59:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:59:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:59:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:59:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:59:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:59:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:59:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:59:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:59:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:59:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:59:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:59:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:59:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:59:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:59:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:59:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:59:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:59:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:59:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 20:59:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 20:59:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:59:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:59:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:59:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 20:59:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 20:59:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 20:59:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 20:59:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 20:59:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 20:59:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 20:59:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 20:59:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 20:59:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 20:59:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 20:59:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 20:59:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 20:59:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:00:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:00:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:00:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:00:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:00:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:00:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:00:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:00:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:00:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:00:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:00:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:00:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:00:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:00:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:00:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:00:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:00:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:00:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:00:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:00:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:00:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:00:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:00:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:00:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:00:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:00:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:00:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:00:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:00:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:00:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:00:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:00:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:00:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:00:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:00:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:00:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:00:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:00:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:00:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:00:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:00:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:00:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:00:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:00:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:00:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:00:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:00:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:00:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:00:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:00:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:00:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:00:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:00:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:00:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:00:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:00:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:00:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:00:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:00:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:00:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:00:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:00:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:00:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:00:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:00:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:00:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:00:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:00:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:00:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:00:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:00:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:00:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:00:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:00:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:00:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:00:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:00:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:00:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:00:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:00:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:00:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:00:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:00:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:00:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:00:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:00:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:00:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:00:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:00:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:00:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:00:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:00:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:00:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:00:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:00:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:00:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:00:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:00:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:00:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:00:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:00:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:00:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:00:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:00:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:00:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:00:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:00:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:00:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:00:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:00:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:00:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:00:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:00:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:00:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:00:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:00:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:00:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:00:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:00:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:00:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:00:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:00:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:00:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:00:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:00:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:00:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:00:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:00:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:00:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:00:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:00:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:00:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:00:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:00:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:00:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:00:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:00:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:00:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:00:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:00:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:00:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:00:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:00:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:00:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:00:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:00:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:00:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:00:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:00:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:00:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:00:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:00:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:00:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:00:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:00:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:00:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:00:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:00:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:00:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:00:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:00:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:00:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:00:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:00:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:00:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:00:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:00:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:00:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:00:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:00:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:00:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:00:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:00:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:00:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:00:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:00:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:00:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:00:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:00:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:00:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:00:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:00:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:00:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:00:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:00:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:00:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:00:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:00:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:00:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:00:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:00:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:00:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:00:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:00:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:00:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:00:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:00:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:00:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:00:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:00:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:00:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:00:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:00:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:00:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:00:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:00:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:00:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:00:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:00:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:00:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:00:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:00:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:00:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:00:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:00:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:00:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:01:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:01:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:01:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:01:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:01:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:01:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:01:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:01:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:01:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:01:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:01:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:01:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:01:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:01:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:01:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:01:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:01:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:01:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:01:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:01:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:01:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:01:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:01:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:01:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:01:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:01:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:01:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:01:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:01:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:01:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:01:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:01:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:01:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:01:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:01:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:01:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:01:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:01:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:01:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:01:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:01:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:01:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:01:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:01:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:01:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:01:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:01:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:01:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:01:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:01:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:01:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:01:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:01:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:01:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:01:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:01:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:01:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:01:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:01:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:01:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:01:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:01:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:01:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:01:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:01:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:01:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:01:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:01:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:01:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:01:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:01:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:01:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:01:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:01:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:01:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:01:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:01:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:01:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:01:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:01:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:01:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:01:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:01:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:01:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:01:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:01:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:01:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:01:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:01:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:01:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:01:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:01:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:01:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:01:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:01:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:01:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:01:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:01:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:01:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:01:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:01:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:01:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:01:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:01:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:01:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:01:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:01:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:01:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:01:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:01:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:01:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:01:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:01:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:01:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:01:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:01:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:01:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:01:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:01:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:01:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:01:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:01:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:01:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:01:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:01:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:01:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:01:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:01:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:01:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:01:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:01:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:01:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:01:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:01:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:01:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:01:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:01:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:01:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:01:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:01:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:01:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:01:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:01:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:01:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:01:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:01:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:01:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:01:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:01:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:01:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:01:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:01:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:01:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:01:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:01:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:01:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:01:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:01:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:01:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:01:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:01:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:01:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:01:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:01:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:01:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:01:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:01:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:01:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:01:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:01:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:01:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:01:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:01:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:01:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:01:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:01:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:01:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:01:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:01:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:01:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:01:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:01:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:01:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:01:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:01:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:01:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:01:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:01:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:01:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:01:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:01:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:01:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:01:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:01:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:01:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:01:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:01:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:01:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:01:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:01:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:01:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:01:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:01:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:01:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:01:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:01:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:01:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:01:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:01:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:01:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:01:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:01:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:01:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:01:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:01:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:01:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:02:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:02:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:02:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:02:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:02:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:02:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:02:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:02:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:02:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:02:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:02:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:02:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:02:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:02:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:02:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:02:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:02:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:02:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:02:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:02:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:02:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:02:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:02:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:02:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:02:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:02:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:02:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:02:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:02:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:02:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:02:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:02:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:02:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:02:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:02:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:02:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:02:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:02:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:02:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:02:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:02:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:02:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:02:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:02:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:02:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:02:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:02:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:02:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:02:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:02:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:02:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:02:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:02:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:02:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:02:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:02:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:02:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:02:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:02:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:02:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:02:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:02:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:02:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:02:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:02:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:02:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:02:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:02:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:02:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:02:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:02:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:02:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:02:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:02:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:02:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:02:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:02:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:02:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:02:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:02:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:02:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:02:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:02:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:02:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:02:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:02:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:02:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:02:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:02:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:02:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:02:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:02:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:02:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:02:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:02:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:02:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:02:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:02:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:02:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:02:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:02:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:02:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:02:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:02:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:02:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:02:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:02:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:02:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:02:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:02:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:02:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:02:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:02:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:02:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:02:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:02:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:02:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:02:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:02:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:02:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:02:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:02:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:02:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:02:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:02:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:02:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:02:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:02:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:02:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:02:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:02:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:02:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:02:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:02:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:02:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:02:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:02:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:02:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:02:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:02:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:02:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:02:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:02:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:02:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:02:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:02:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:02:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:02:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:02:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:02:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:02:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:02:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:02:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:02:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:02:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:02:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:02:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:02:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:02:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:02:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:02:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:02:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:02:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:02:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:02:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:02:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:02:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:02:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:02:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:02:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:02:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:02:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:02:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:02:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:02:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:02:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:02:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:02:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:02:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:02:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:02:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:02:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:02:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:02:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:02:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:02:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:02:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:02:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:02:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:02:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:02:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:02:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:02:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:02:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:02:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:02:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:02:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:02:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:02:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:02:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:02:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:02:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:02:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:02:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:02:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:02:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:02:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:02:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:02:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:02:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:02:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:02:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:02:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:02:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:02:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:02:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:03:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:03:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:03:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:03:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:03:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:03:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:03:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:03:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:03:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:03:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:03:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:03:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:03:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:03:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:03:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:03:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:03:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:03:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:03:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:03:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:03:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:03:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:03:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:03:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:03:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:03:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:03:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:03:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:03:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:03:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:03:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:03:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:03:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:03:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:03:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:03:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:03:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:03:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:03:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:03:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:03:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:03:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:03:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:03:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:03:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:03:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:03:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:03:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:03:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:03:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:03:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:03:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:03:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:03:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:03:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:03:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:03:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:03:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:03:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:03:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:03:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:03:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:03:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:03:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:03:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:03:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:03:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:03:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:03:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:03:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:03:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:03:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:03:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:03:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:03:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:03:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:03:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:03:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:03:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:03:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:03:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:03:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:03:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:03:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:03:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:03:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:03:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:03:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:03:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:03:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:03:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:03:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:03:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:03:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:03:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:03:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:03:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:03:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:03:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:03:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:03:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:03:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:03:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:03:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:03:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:03:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:03:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:03:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:03:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:03:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:03:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:03:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:03:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:03:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:03:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:03:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:03:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:03:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:03:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:03:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:03:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:03:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:03:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:03:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:03:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:03:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:03:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:03:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:03:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:03:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:03:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:03:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:03:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:03:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:03:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:03:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:03:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:03:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:03:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:03:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:03:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:03:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:03:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:03:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:03:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:03:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:03:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:03:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:03:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:03:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:03:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:03:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:03:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:03:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:03:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:03:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:03:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:03:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:03:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:03:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:03:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:03:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:03:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:03:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:03:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:03:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:03:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:03:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:03:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:03:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:03:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:03:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:03:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:03:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:03:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:03:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:03:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:03:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:03:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:03:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:03:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:03:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:03:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:03:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:03:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:03:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:03:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:03:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:03:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:03:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:03:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:03:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:03:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:03:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:03:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:03:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:03:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:03:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:03:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:03:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:03:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:03:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:03:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:03:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:03:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:03:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:03:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:03:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:03:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:03:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:03:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:03:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:03:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:03:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:03:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:03:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:04:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:04:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:04:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:04:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:04:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:04:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:04:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:04:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:04:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:04:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:04:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:04:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:04:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:04:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:04:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:04:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:04:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:04:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:04:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:04:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:04:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:04:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:04:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:04:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:04:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:04:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:04:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:04:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:04:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:04:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:04:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:04:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:04:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:04:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:04:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:04:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:04:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:04:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:04:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:04:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:04:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:04:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:04:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:04:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:04:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:04:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:04:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:04:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:04:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:04:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:04:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:04:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:04:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:04:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:04:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:04:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:04:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:04:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:04:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:04:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:04:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:04:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:04:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:04:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:04:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:04:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:04:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:04:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:04:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:04:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:04:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:04:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:04:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:04:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:04:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:04:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:04:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:04:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:04:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:04:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:04:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:04:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:04:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:04:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:04:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:04:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:04:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:04:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:04:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:04:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:04:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:04:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:04:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:04:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:04:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:04:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:04:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:04:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:04:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:04:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:04:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:04:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:04:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:04:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:04:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:04:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:04:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:04:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:04:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:04:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:04:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:04:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:04:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:04:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:04:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:04:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:04:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:04:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:04:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:04:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:04:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:04:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:04:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:04:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:04:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:04:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:04:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:04:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:04:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:04:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:04:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:04:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:04:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:04:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:04:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:04:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:04:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:04:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:04:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:04:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:04:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:04:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:04:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:04:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:04:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:04:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:04:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:04:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:04:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:04:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:04:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:04:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:04:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:04:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:04:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:04:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:04:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:04:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:04:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:04:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:04:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:04:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:04:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:04:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:04:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:04:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:04:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:04:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:04:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:04:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:04:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:04:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:04:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:04:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:04:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:04:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:04:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:04:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:04:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:04:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:04:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:04:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:04:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:04:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:04:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:04:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:04:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:04:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:04:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:04:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:04:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:04:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:04:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:04:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:04:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:04:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:04:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:04:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:04:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:04:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:04:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:04:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:04:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:04:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:04:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:04:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:04:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:04:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:04:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:04:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:04:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:04:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:04:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:04:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:04:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:04:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:05:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:05:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:05:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:05:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:05:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:05:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:05:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:05:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:05:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:05:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:05:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:05:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:05:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:05:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:05:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:05:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:05:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:05:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:05:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:05:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:05:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:05:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:05:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:05:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:05:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:05:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:05:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:05:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:05:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:05:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:05:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:05:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:05:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:05:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:05:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:05:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:05:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:05:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:05:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:05:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:05:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:05:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:05:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:05:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:05:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:05:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:05:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:05:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:05:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:05:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:05:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:05:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:05:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:05:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:05:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:05:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:05:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:05:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:05:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:05:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:05:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:05:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:05:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:05:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:05:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:05:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:05:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:05:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:05:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:05:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:05:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:05:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:05:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:05:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:05:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:05:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:05:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:05:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:05:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:05:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:05:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:05:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:05:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:05:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:05:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:05:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:05:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:05:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:05:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:05:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:05:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:05:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:05:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:05:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:05:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:05:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:05:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:05:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:05:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:05:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:05:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:05:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:05:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:05:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:05:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:05:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:05:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:05:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:05:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:05:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:05:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:05:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:05:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:05:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:05:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:05:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:05:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:05:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:05:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:05:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:05:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:05:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:05:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:05:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:05:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:05:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:05:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:05:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:05:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:05:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:05:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:05:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:05:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:05:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:05:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:05:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:05:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:05:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:05:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:05:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:05:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:05:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:05:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:05:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:05:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:05:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:05:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:05:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:05:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:05:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:05:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:05:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:05:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:05:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:05:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:05:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:05:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:05:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:05:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:05:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:05:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:05:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:05:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:05:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:05:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:05:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:05:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:05:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:05:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:05:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:05:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:05:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:05:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:05:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:05:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:05:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:05:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:05:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:05:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:05:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:05:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:05:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:05:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:05:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:05:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:05:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:05:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:05:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:05:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:05:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:05:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:05:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:05:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:05:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:05:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:05:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:05:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:05:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:05:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:05:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:05:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:05:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:05:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:05:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:05:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:05:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:05:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:05:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:05:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:05:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:05:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:05:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:05:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:05:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:05:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:05:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:06:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:06:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:06:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:06:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:06:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:06:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:06:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:06:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:06:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:06:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:06:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:06:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:06:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:06:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:06:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:06:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:06:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:06:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:06:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:06:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:06:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:06:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:06:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:06:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:06:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:06:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:06:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:06:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:06:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:06:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:06:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:06:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:06:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:06:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:06:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:06:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:06:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:06:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:06:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:06:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:06:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:06:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:06:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:06:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:06:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:06:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:06:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:06:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:06:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:06:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:06:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:06:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:06:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:06:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:06:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:06:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:06:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:06:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:06:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:06:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:06:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:06:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:06:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:06:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:06:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:06:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:06:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:06:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:06:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:06:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:06:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:06:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:06:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:06:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:06:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:06:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:06:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:06:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:06:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:06:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:06:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:06:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:06:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:06:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:06:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:06:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:06:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:06:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:06:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:06:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:06:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:06:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:06:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:06:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:06:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:06:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:06:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:06:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:06:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:06:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:06:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:06:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:06:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:06:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:06:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:06:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:06:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:06:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:06:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:06:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:06:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:06:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:06:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:06:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:06:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:06:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:06:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:06:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:06:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:06:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:06:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:06:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:06:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:06:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:06:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:06:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:06:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:06:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:06:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:06:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:06:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:06:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:06:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:06:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:06:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:06:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:06:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:06:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:06:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:06:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:06:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:06:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:06:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:06:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:06:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:06:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:06:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:06:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:06:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:06:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:06:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:06:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:06:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:06:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:06:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:06:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:06:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:06:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:06:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:06:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:06:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:06:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:08:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:08:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:08:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:08:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:08:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:08:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:08:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:08:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:08:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:08:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:08:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:08:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:08:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:08:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:08:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:08:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:08:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:08:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:08:47 rdma06 rsyslogd[1041]: imjournal: 5093 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 21:08:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:08:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:08:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:08:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:08:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:08:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:08:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:08:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:08:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:08:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:08:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:08:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:08:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:08:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:08:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:08:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:08:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:08:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:08:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:08:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:08:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:08:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:08:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:08:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:08:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:08:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:08:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:08:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:08:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:08:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:08:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:08:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:08:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:08:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:08:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:08:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:09:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:09:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:09:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:09:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:09:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:09:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:09:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:09:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:09:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:09:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:09:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:09:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:09:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:09:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:09:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:09:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:09:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:09:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:09:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:09:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:09:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:09:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:09:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:09:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:09:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:09:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:09:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:09:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:09:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:09:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:09:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:09:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:09:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:09:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:09:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:09:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:09:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:09:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:09:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:09:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:09:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:09:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:09:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:09:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:09:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:09:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:09:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:09:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:09:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:09:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:09:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:09:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:09:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:09:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:09:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:09:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:09:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:09:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:09:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:09:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:09:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:09:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:09:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:09:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:09:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:09:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:09:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:09:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:09:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:09:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:09:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:09:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:09:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:09:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:09:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:09:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:09:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:09:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:09:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:09:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:09:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:09:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:09:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:09:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:09:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:09:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:09:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:09:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:09:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:09:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:09:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:09:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:09:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:09:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:09:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:09:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:09:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:09:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:09:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:09:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:09:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:09:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:09:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:09:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:09:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:09:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:09:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:09:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:09:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:09:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:09:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:09:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:09:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:09:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:09:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:09:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:09:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:09:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:09:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:09:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:09:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:09:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:09:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:09:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:09:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:09:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:09:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:09:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:09:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:09:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:09:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:09:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:09:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:09:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:09:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:09:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:09:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:09:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:09:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:09:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:09:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:09:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:09:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:09:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:09:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:09:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:09:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:09:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:09:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:09:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:09:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:09:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:09:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:09:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:09:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:09:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:09:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:09:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:09:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:09:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:09:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:09:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:09:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:09:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:09:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:09:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:09:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:09:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:09:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:09:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:09:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:09:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:09:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:09:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:09:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:09:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:09:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:09:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:09:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:09:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:09:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:09:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:09:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:09:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:09:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:09:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:09:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:09:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:09:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:09:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:09:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:09:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:09:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:09:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:09:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:09:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:09:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:09:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:09:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:09:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:09:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:09:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:09:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:09:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:09:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:09:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:09:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:09:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:09:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:09:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:09:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:09:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:09:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:09:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:09:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:09:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:10:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:10:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:10:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:10:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:10:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:10:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:10:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:10:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:10:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:10:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:10:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:10:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:10:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:10:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:10:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:10:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:10:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:10:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:10:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:10:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:10:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:10:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:10:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:10:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:10:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:10:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:10:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:10:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:10:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:10:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:10:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:10:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:10:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:10:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:10:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:10:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:10:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:10:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:10:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:10:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:10:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:10:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:10:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:10:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:10:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:10:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:10:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:10:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:10:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:10:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:10:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:10:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:10:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:10:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:10:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:10:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:10:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:10:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:10:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:10:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:10:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:10:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:10:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:10:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:10:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:10:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:10:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:10:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:10:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:10:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:10:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:10:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:10:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:10:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:10:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:10:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:10:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:10:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:10:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:10:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:10:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:10:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:10:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:10:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:10:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:10:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:10:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:10:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:10:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:10:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:10:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:10:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:10:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:10:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:10:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:10:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:10:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:10:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:10:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:10:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:10:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:10:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:10:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:10:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:10:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:10:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:10:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:10:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:10:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:10:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:10:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:10:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:10:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:10:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:10:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:10:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:10:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:10:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:10:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:10:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:10:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:10:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:10:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:10:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:10:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:10:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:10:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:10:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:10:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:10:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:10:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:10:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:10:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:10:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:10:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:10:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:10:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:10:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:10:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:10:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:10:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:10:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:10:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:10:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:10:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:10:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:10:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:10:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:10:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:10:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:10:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:10:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:10:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:10:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:10:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:10:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:10:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:10:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:10:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:10:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:10:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:10:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:10:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:10:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:10:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:10:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:10:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:10:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:10:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:10:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:10:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:10:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:10:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:10:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:10:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:10:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:10:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:10:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:10:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:10:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:10:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:10:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:10:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:10:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:10:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:10:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:10:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:10:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:10:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:10:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:10:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:10:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:10:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:10:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:10:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:10:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:10:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:10:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:10:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:10:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:10:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:10:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:10:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:10:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:10:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:10:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:10:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:10:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:10:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:10:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:10:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:10:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:10:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:10:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:10:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:10:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:11:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:11:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:11:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:11:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:11:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:11:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:11:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:11:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:11:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:11:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:11:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:11:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:11:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:11:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:11:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:11:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:11:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:11:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:11:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:11:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:11:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:11:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:11:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:11:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:11:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:11:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:11:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:11:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:11:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:11:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:11:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:11:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:11:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:11:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:11:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:11:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:11:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:11:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:11:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:11:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:11:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:11:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:11:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:11:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:11:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:11:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:11:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:11:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:11:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:11:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:11:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:11:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:11:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:11:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:11:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:11:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:11:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:11:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:11:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:11:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:11:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:11:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:11:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:11:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:11:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:11:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:11:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:11:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:11:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:11:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:11:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:11:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:11:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:11:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:11:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:11:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:11:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:11:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:11:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:11:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:11:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:11:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:11:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:11:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:11:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:11:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:11:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:11:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:11:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:11:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:11:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:11:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:11:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:11:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:11:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:11:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:11:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:11:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:11:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:11:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:11:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:11:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:11:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:11:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:11:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:11:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:11:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:11:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:11:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:11:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:11:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:11:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:11:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:11:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:11:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:11:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:11:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:11:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:11:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:11:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:11:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:11:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:11:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:11:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:11:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:11:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:11:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:11:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:11:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:11:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:11:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:11:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:11:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:11:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:11:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:11:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:11:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:11:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:11:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:11:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:11:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:11:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:11:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:11:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:11:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:11:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:11:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:11:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:11:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:11:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:11:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:11:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:11:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:11:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:11:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:11:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:11:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:11:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:11:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:11:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:11:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:11:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:11:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:11:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:11:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:11:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:11:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:11:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:11:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:11:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:11:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:11:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:11:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:11:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:11:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:11:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:11:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:11:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:11:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:11:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:11:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:11:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:11:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:11:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:11:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:11:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:11:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:11:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:11:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:11:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:11:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:11:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:11:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:11:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:11:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:11:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:11:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:11:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:11:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:11:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:11:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:11:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:11:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:11:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:11:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:11:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:11:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:11:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:11:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:11:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:11:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:11:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:11:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:11:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:11:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:11:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:12:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:12:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:12:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:12:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:12:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:12:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:12:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:12:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:12:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:12:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:12:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:12:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:12:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:12:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:12:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:12:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:12:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:12:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:12:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:12:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:12:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:12:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:12:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:12:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:12:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:12:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:12:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:12:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:12:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:12:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:12:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:12:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:12:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:12:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:12:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:12:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:12:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:12:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:12:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:12:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:12:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:12:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:12:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:12:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:12:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:12:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:12:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:12:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:12:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:12:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:12:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:12:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:12:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:12:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:12:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:12:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:12:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:12:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:12:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:12:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:12:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:12:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:12:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:12:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:12:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:12:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:12:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:12:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:12:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:12:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:12:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:12:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:12:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:12:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:12:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:12:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:12:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:12:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:12:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:12:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:12:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:12:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:12:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:12:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:12:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:12:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:12:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:12:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:12:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:12:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:12:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:12:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:12:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:12:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:12:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:12:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:12:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:12:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:12:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:12:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:12:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:12:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:12:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:12:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:12:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:12:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:12:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:12:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:12:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:12:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:12:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:12:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:12:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:12:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:12:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:12:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:12:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:12:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:12:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:12:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:12:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:12:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:12:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:12:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:12:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:12:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:12:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:12:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:12:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:12:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:12:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:12:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:12:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:12:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:12:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:12:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:12:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:12:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:12:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:12:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:12:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:12:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:12:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:12:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:12:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:12:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:12:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:12:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:12:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:12:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:12:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:12:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:12:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:12:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:12:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:12:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:12:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:12:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:12:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:12:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:12:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:12:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:12:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:12:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:12:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:12:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:12:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:12:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:12:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:12:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:12:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:12:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:12:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:12:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:12:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:12:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:12:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:12:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:12:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:12:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:12:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:12:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:12:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:12:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:12:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:12:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:12:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:12:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:12:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:12:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:12:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:12:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:12:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:12:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:12:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:12:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:12:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:12:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:12:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:12:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:12:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:12:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:12:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:12:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:12:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:12:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:12:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:12:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:12:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:12:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:12:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:12:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:12:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:12:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:12:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:12:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:13:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:13:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:13:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:13:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:13:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:13:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:13:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:13:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:13:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:13:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:13:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:13:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:13:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:13:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:13:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:13:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:13:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:13:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:13:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:13:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:13:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:13:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:13:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:13:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:13:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:13:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:13:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:13:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:13:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:13:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:13:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:13:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:13:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:13:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:13:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:13:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:13:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:13:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:13:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:13:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:13:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:13:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:13:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:13:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:13:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:13:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:13:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:13:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:13:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:13:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:13:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:13:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:13:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:13:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:13:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:13:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:13:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:13:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:13:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:13:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:13:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:13:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:13:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:13:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:13:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:13:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:13:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:13:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:13:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:13:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:13:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:13:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:13:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:13:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:13:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:13:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:13:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:13:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:13:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:13:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:13:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:13:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:13:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:13:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:13:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:13:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:13:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:13:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:13:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:13:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:13:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:13:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:13:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:13:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:13:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:13:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:13:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:13:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:13:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:13:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:13:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:13:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:13:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:13:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:13:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:13:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:13:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:13:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:13:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:13:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:13:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:13:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:13:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:13:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:13:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:13:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:13:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:13:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:13:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:13:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:13:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:13:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:13:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:13:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:13:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:13:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:13:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:13:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:13:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:13:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:13:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:13:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:13:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:13:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:13:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:13:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:13:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:13:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:13:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:13:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:13:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:13:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:13:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:13:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:13:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:13:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:13:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:13:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:13:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:13:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:13:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:13:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:13:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:13:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:13:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:13:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:13:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:13:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:13:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:13:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:13:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:13:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:13:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:13:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:13:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:13:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:13:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:13:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:13:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:13:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:13:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:13:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:13:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:13:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:13:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:13:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:13:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:13:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:13:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:13:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:13:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:13:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:13:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:13:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:13:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:13:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:13:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:13:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:13:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:13:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:13:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:13:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:13:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:13:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:13:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:13:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:13:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:13:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:13:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:13:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:13:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:13:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:13:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:13:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:13:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:13:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:13:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:13:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:13:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:13:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:13:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:13:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:13:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:13:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:13:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:13:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:14:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:14:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:14:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:14:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:14:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:14:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:14:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:14:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:14:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:14:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:14:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:14:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:14:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:14:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:14:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:14:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:14:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:14:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:14:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:14:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:14:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:14:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:14:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:14:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:14:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:14:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:14:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:14:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:14:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:14:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:14:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:14:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:14:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:14:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:14:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:14:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:14:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:14:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:14:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:14:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:14:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:14:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:14:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:14:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:14:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:14:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:14:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:14:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:14:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:14:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:14:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:14:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:14:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:14:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:14:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:14:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:14:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:14:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:14:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:14:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:14:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:14:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:14:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:14:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:14:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:14:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:14:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:14:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:14:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:14:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:14:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:14:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:14:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:14:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:14:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:14:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:14:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:14:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:14:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:14:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:14:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:14:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:14:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:14:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:14:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:14:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:14:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:14:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:14:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:14:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:14:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:14:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:14:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:14:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:14:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:14:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:14:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:14:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:14:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:14:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:14:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:14:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:14:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:14:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:14:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:14:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:14:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:14:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:14:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:14:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:14:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:14:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:14:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:14:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:14:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:14:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:14:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:14:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:14:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:14:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:14:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:14:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:14:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:14:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:14:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:14:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:14:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:14:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:14:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:14:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:14:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:14:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:14:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:14:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:14:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:14:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:14:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:14:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:14:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:14:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:14:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:14:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:14:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:14:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:14:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:14:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:14:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:14:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:14:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:14:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:14:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:14:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:14:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:14:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:14:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:14:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:14:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:14:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:14:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:14:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:14:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:14:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:14:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:14:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:14:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:14:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:14:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:14:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:14:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:14:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:14:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:14:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:14:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:14:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:14:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:14:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:14:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:14:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:14:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:14:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:14:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:14:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:14:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:14:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:14:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:14:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:14:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:14:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:14:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:14:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:14:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:14:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:14:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:14:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:14:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:14:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:14:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:14:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:14:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:14:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:14:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:14:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:14:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:14:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:14:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:14:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:14:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:14:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:14:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:14:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:14:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:14:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:14:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:14:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:14:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:14:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:15:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:15:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:15:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:15:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:15:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:15:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:15:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:15:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:15:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:15:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:15:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:15:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:15:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:15:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:15:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:15:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:15:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:15:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:15:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:15:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:15:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:15:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:15:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:15:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:15:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:15:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:15:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:15:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:15:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:15:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:15:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:15:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:15:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:15:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:15:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:15:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:15:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:15:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:15:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:15:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:15:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:15:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:15:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:15:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:15:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:15:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:15:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:15:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:15:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:15:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:15:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:15:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:15:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:15:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:15:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:15:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:15:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:15:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:15:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:15:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:15:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:15:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:15:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:15:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:15:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:15:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:15:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:15:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:15:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:15:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:15:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:15:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:15:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:15:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:15:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:15:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:15:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:15:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:15:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:15:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:15:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:15:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:15:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:15:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:15:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:15:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:15:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:15:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:15:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:15:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:15:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:15:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:15:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:15:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:15:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:15:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:15:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:15:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:15:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:15:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:15:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:15:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:15:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:15:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:15:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:15:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:15:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:15:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:15:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:15:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:15:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:15:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:15:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:15:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:15:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:15:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:15:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:15:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:15:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:15:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:15:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:15:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:15:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:15:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:15:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:15:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:15:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:15:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:15:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:15:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:15:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:15:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:15:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:15:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:15:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:15:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:15:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:15:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:15:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:15:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:15:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:15:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:15:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:15:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:15:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:15:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:15:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:15:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:15:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:15:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:15:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:15:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:15:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:15:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:15:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:15:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:15:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:15:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:15:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:15:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:15:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:15:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:15:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:15:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:15:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:15:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:15:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:15:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:15:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:15:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:15:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:15:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:15:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:15:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:15:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:15:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:15:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:15:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:15:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:15:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:15:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:15:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:15:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:15:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:15:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:15:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:15:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:15:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:15:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:15:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:15:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:15:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:15:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:15:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:15:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:15:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:15:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:15:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:15:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:15:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:15:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:15:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:15:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:15:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:15:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:15:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:15:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:15:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:15:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:15:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:15:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:15:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:15:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:15:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:15:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:15:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:16:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:16:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:16:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:16:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:16:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:16:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:16:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:16:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:16:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:16:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:16:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:16:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:16:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:16:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:16:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:16:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:16:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:16:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:16:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:16:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:16:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:16:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:16:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:16:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:16:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:16:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:16:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:16:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:16:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:16:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:16:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:16:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:16:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:16:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:16:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:16:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:16:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:16:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:16:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:16:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:16:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:16:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:16:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:16:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:16:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:16:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:16:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:16:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:16:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:16:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:16:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:16:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:16:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:16:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:16:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:16:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:16:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:16:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:16:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:16:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:16:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:16:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:16:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:16:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:16:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:16:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:16:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:16:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:16:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:16:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:16:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:16:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:16:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:16:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:16:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:16:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:16:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:16:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:16:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:16:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:16:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:16:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:16:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:16:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:16:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:16:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:16:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:16:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:16:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:16:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:16:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:16:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:16:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:16:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:16:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:16:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:16:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:16:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:16:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:16:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:16:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:16:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:16:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:16:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:16:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:16:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:16:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:16:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:16:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:16:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:16:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:16:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:16:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:16:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:16:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:16:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:16:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:16:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:16:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:16:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:16:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:16:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:16:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:16:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:16:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:16:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:16:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:16:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:16:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:16:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:16:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:16:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:16:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:16:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:16:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:16:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:16:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:16:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:16:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:16:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:16:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:16:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:16:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:16:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:16:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:16:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:16:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:16:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:16:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:16:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:16:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:16:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:16:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:16:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:16:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:16:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:16:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:16:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:16:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:16:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:16:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:16:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:18:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:18:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:18:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:18:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:18:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:18:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:18:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:18:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:18:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:18:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:18:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:18:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:18:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:18:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:18:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:18:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:18:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:18:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:18:52 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 21:18:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:18:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:18:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:18:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:18:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:18:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:18:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:18:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:18:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:18:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:18:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:18:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:18:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:18:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:18:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:18:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:18:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:18:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:19:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:19:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:19:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:19:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:19:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:19:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:19:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:19:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:19:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:19:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:19:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:19:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:19:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:19:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:19:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:19:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:19:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:19:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:19:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:19:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:19:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:19:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:19:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:19:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:19:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:19:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:19:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:19:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:19:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:19:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:19:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:19:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:19:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:19:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:19:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:19:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:19:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:19:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:19:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:19:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:19:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:19:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:19:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:19:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:19:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:19:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:19:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:19:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:19:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:19:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:19:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:19:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:19:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:19:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:19:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:19:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:19:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:19:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:19:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:19:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:19:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:19:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:19:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:19:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:19:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:19:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:19:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:19:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:19:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:19:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:19:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:19:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:19:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:19:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:19:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:19:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:19:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:19:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:19:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:19:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:19:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:19:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:19:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:19:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:19:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:19:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:19:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:19:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:19:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:19:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:19:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:19:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:19:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:19:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:19:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:19:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:19:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:19:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:19:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:19:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:19:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:19:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:19:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:19:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:19:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:19:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:19:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:19:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:19:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:19:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:19:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:19:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:19:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:19:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:19:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:19:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:19:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:19:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:19:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:19:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:19:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:19:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:19:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:19:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:19:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:19:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:19:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:19:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:19:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:19:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:19:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:19:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:19:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:19:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:19:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:19:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:19:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:19:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:19:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:19:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:19:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:19:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:19:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:19:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:19:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:19:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:19:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:19:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:19:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:19:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:19:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:19:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:19:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:19:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:19:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:19:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:19:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:19:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:19:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:19:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:19:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:19:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:19:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:19:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:19:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:19:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:19:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:19:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:19:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:19:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:19:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:19:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:19:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:19:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:19:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:19:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:19:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:19:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:19:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:19:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:19:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:19:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:19:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:19:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:19:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:19:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:19:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:19:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:19:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:19:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:19:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:19:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:19:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:19:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:19:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:19:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:19:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:19:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:19:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:19:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:19:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:19:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:19:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:19:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:19:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:19:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:19:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:19:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:19:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:19:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:19:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:19:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:19:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:19:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:19:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:19:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:20:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:20:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:20:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:20:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:20:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:20:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:20:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:20:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:20:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:20:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:20:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:20:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:20:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:20:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:20:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:20:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:20:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:20:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:20:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:20:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:20:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:20:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:20:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:20:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:20:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:20:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:20:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:20:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:20:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:20:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:20:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:20:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:20:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:20:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:20:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:20:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:20:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:20:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:20:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:20:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:20:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:20:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:20:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:20:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:20:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:20:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:20:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:20:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:20:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:20:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:20:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:20:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:20:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:20:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:20:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:20:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:20:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:20:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:20:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:20:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:20:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:20:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:20:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:20:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:20:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:20:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:20:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:20:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:20:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:20:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:20:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:20:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:20:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:20:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:20:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:20:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:20:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:20:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:20:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:20:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:20:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:20:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:20:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:20:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:20:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:20:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:20:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:20:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:20:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:20:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:20:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:20:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:20:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:20:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:20:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:20:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:20:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:20:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:20:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:20:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:20:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:20:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:20:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:20:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:20:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:20:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:20:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:20:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:20:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:20:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:20:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:20:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:20:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:20:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:20:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:20:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:20:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:20:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:20:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:20:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:20:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:20:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:20:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:20:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:20:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:20:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:20:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:20:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:20:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:20:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:20:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:20:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:20:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:20:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:20:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:20:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:20:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:20:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:20:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:20:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:20:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:20:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:20:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:20:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:20:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:20:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:20:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:20:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:20:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:20:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:20:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:20:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:20:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:20:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:20:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:20:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:20:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:20:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:20:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:20:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:20:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:20:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:20:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:20:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:20:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:20:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:20:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:20:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:20:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:20:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:20:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:20:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:20:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:20:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:20:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:20:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:20:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:20:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:20:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:20:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:20:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:20:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:20:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:20:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:20:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:20:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:20:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:20:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:20:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:20:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:20:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:20:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:20:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:20:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:20:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:20:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:20:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:20:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:20:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:20:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:20:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:20:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:20:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:20:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:20:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:20:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:20:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:20:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:20:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:20:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:20:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:20:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:20:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:20:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:20:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:20:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:21:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:21:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:21:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:21:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:21:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:21:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:21:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:21:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:21:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:21:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:21:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:21:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:21:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:21:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:21:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:21:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:21:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:21:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:21:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:21:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:21:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:21:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:21:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:21:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:21:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:21:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:21:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:21:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:21:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:21:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:21:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:21:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:21:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:21:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:21:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:21:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:21:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:21:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:21:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:21:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:21:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:21:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:21:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:21:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:21:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:21:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:21:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:21:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:21:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:21:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:21:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:21:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:21:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:21:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:21:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:21:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:21:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:21:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:21:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:21:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:21:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:21:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:21:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:21:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:21:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:21:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:21:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:21:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:21:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:21:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:21:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:21:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:21:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:21:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:21:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:21:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:21:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:21:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:21:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:21:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:21:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:21:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:21:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:21:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:21:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:21:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:21:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:21:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:21:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:21:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:21:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:21:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:21:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:21:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:21:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:21:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:21:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:21:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:21:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:21:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:21:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:21:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:21:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:21:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:21:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:21:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:21:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:21:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:21:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:21:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:21:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:21:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:21:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:21:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:21:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:21:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:21:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:21:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:21:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:21:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:21:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:21:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:21:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:21:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:21:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:21:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:21:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:21:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:21:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:21:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:21:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:21:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:21:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:21:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:21:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:21:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:21:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:21:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:21:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:21:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:21:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:21:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:21:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:21:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:21:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:21:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:21:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:21:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:21:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:21:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:21:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:21:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:21:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:21:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:21:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:21:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:21:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:21:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:21:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:21:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:21:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:21:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:21:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:21:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:21:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:21:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:21:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:21:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:21:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:21:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:21:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:21:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:21:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:21:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:21:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:21:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:21:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:21:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:21:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:21:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:21:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:21:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:21:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:21:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:21:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:21:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:21:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:21:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:21:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:21:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:21:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:21:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:21:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:21:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:21:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:21:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:21:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:21:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:21:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:21:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:21:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:21:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:21:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:21:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:21:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:21:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:22:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:22:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:22:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:22:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:22:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:22:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:22:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:22:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:22:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:22:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:22:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:22:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:22:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:22:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:22:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:22:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:22:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:22:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:22:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:22:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:22:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:22:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:22:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:22:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:22:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:22:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:22:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:22:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:22:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:22:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:22:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:22:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:22:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:22:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:22:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:22:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:22:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:22:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:22:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:22:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:22:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:22:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:22:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:22:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:22:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:22:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:22:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:22:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:22:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:22:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:22:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:22:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:22:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:22:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:22:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:22:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:22:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:22:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:22:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:22:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:22:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:22:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:22:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:22:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:22:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:22:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:22:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:22:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:22:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:22:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:22:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:22:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:22:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:22:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:22:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:22:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:22:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:22:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:22:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:22:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:22:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:22:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:22:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:22:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:22:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:22:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:22:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:22:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:22:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:22:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:22:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:22:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:22:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:22:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:22:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:22:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:22:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:22:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:22:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:22:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:22:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:22:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:22:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:22:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:22:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:22:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:22:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:22:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:22:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:22:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:22:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:22:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:22:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:22:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:22:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:22:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:22:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:22:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:22:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:22:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:22:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:22:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:22:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:22:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:22:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:22:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:22:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:22:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:22:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:22:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:22:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:22:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:22:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:22:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:22:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:22:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:22:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:22:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:22:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:22:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:22:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:22:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:22:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:22:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:22:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:22:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:22:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:22:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:22:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:22:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:22:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:22:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:22:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:22:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:22:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:22:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:22:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:22:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:22:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:22:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:22:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:22:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:22:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:22:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:22:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:22:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:22:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:22:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:22:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:22:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:22:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:22:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:22:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:22:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:22:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:22:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:22:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:22:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:22:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:22:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:22:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:22:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:22:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:22:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:22:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:22:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:22:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:22:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:22:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:22:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:22:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:22:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:22:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:22:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:22:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:22:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:23:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:23:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:23:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:23:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:23:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:23:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:23:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:23:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:23:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:23:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:23:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:23:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:23:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:23:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:23:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:23:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:23:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:23:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:23:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:23:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:23:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:23:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:23:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:23:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:23:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:23:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:23:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:23:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:23:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:23:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:23:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:23:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:23:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:23:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:23:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:23:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:23:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:23:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:23:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:23:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:23:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:23:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:23:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:23:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:23:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:23:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:23:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:23:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:23:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:23:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:23:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:23:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:23:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:23:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:23:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:23:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:23:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:23:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:23:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:23:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:23:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:23:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:23:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:23:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:23:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:23:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:23:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:23:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:23:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:23:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:23:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:23:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:23:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:23:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:23:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:23:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:23:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:23:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:23:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:23:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:23:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:23:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:23:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:23:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:23:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:23:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:23:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:23:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:23:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:23:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:23:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:23:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:23:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:23:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:23:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:23:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:23:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:23:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:23:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:23:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:23:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:23:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:23:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:23:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:23:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:23:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:23:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:23:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:23:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:23:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:23:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:23:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:23:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:23:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:23:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:23:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:23:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:23:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:23:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:23:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:23:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:23:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:23:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:23:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:23:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:23:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:23:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:23:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:23:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:23:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:23:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:23:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:23:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:23:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:23:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:23:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:23:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:23:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:23:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:23:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:23:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:23:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:23:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:23:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:23:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:23:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:23:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:23:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:23:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:23:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:23:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:23:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:23:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:23:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:23:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:23:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:23:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:23:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:23:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:23:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:23:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:23:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:23:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:23:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:23:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:23:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:23:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:23:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:23:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:23:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:23:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:23:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:23:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:23:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:23:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:23:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:23:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:23:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:23:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:23:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:23:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:23:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:23:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:23:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:23:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:23:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:23:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:23:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:23:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:23:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:23:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:23:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:23:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:23:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:23:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:23:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:23:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:23:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:23:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:23:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:23:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:23:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:23:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:23:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:23:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:23:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:23:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:23:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:23:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:23:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:23:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:23:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:23:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:23:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:23:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:23:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:24:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:24:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:24:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:24:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:24:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:24:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:24:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:24:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:24:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:24:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:24:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:24:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:24:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:24:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:24:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:24:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:24:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:24:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:24:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:24:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:24:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:24:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:24:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:24:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:24:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:24:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:24:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:24:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:24:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:24:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:24:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:24:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:24:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:24:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:24:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:24:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:24:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:24:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:24:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:24:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:24:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:24:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:24:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:24:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:24:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:24:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:24:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:24:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:24:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:24:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:24:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:24:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:24:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:24:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:24:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:24:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:24:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:24:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:24:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:24:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:24:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:24:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:24:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:24:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:24:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:24:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:24:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:24:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:24:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:24:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:24:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:24:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:24:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:24:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:24:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:24:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:24:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:24:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:24:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:24:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:24:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:24:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:24:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:24:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:24:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:24:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:24:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:24:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:24:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:24:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:24:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:24:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:24:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:24:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:24:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:24:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:24:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:24:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:24:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:24:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:24:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:24:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:24:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:24:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:24:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:24:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:24:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:24:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:24:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:24:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:24:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:24:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:24:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:24:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:24:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:24:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:24:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:24:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:24:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:24:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:24:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:24:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:24:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:24:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:24:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:24:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:24:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:24:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:24:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:24:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:24:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:24:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:24:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:24:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:24:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:24:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:24:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:24:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:24:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:24:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:24:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:24:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:24:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:24:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:24:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:24:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:24:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:24:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:24:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:24:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:24:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:24:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:24:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:24:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:24:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:24:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:24:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:24:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:24:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:24:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:24:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:24:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:24:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:24:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:24:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:24:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:24:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:24:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:24:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:24:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:24:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:24:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:24:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:24:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:24:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:24:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:24:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:24:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:24:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:24:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:24:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:24:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:24:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:24:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:24:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:24:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:24:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:24:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:24:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:24:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:24:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:24:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:24:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:24:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:24:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:24:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:24:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:24:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:24:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:24:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:24:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:24:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:24:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:24:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:24:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:24:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:24:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:24:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:24:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:24:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:24:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:24:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:24:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:24:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:24:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:24:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:25:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:25:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:25:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:25:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:25:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:25:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:25:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:25:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:25:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:25:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:25:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:25:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:25:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:25:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:25:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:25:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:25:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:25:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:25:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:25:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:25:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:25:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:25:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:25:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:25:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:25:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:25:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:25:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:25:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:25:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:25:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:25:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:25:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:25:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:25:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:25:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:25:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:25:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:25:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:25:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:25:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:25:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:25:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:25:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:25:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:25:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:25:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:25:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:25:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:25:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:25:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:25:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:25:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:25:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:25:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:25:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:25:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:25:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:25:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:25:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:25:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:25:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:25:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:25:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:25:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:25:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:25:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:25:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:25:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:25:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:25:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:25:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:25:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:25:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:25:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:25:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:25:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:25:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:25:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:25:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:25:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:25:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:25:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:25:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:25:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:25:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:25:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:25:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:25:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:25:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:25:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:25:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:25:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:25:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:25:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:25:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:25:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:25:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:25:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:25:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:25:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:25:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:25:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:25:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:25:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:25:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:25:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:25:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:25:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:25:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:25:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:25:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:25:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:25:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:25:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:25:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:25:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:25:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:25:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:25:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:25:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:25:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:25:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:25:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:25:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:25:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:25:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:25:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:25:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:25:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:25:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:25:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:25:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:25:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:25:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:25:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:25:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:25:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:25:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:25:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:25:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:25:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:25:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:25:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:25:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:25:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:25:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:25:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:25:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:25:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:25:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:25:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:25:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:25:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:25:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:25:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:25:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:25:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:25:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:25:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:25:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:25:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:25:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:25:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:25:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:25:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:25:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:25:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:25:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:25:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:25:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:25:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:25:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:25:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:25:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:25:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:25:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:25:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:25:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:25:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:25:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:25:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:25:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:25:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:25:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:25:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:25:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:25:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:25:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:25:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:25:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:25:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:25:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:25:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:25:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:25:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:25:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:25:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:25:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:25:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:25:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:25:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:25:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:25:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:25:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:25:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:25:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:25:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:25:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:25:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:25:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:25:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:25:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:25:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:25:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:25:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:26:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:26:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:26:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:26:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:26:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:26:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:26:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:26:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:26:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:26:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:26:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:26:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:26:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:26:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:26:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:26:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:26:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:26:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:26:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:26:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:26:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:26:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:26:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:26:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:26:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:26:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:26:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:26:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:26:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:26:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:26:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:26:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:26:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:26:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:26:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:26:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:26:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:26:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:26:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:26:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:26:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:26:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:26:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:26:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:26:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:26:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:26:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:26:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:26:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:26:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:26:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:26:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:26:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:26:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:26:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:26:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:26:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:26:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:26:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:26:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:26:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:26:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:26:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:26:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:26:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:26:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:26:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:26:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:26:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:26:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:26:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:26:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:26:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:26:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:26:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:26:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:26:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:26:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:26:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:26:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:26:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:26:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:26:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:26:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:26:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:26:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:26:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:26:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:26:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:26:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:26:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:26:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:26:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:26:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:26:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:26:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:26:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:26:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:26:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:26:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:26:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:26:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:26:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:26:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:26:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:26:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:26:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:26:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:26:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:26:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:26:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:26:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:26:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:26:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:26:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:26:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:26:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:26:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:26:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:26:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:26:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:26:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:26:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:26:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:26:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:26:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:26:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:26:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:26:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:26:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:26:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:26:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:26:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:26:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:26:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:26:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:26:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:26:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:26:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:26:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:26:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:26:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:26:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:26:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:26:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:26:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:26:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:26:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:26:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:26:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:26:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:26:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:26:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:26:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:26:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:26:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:26:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:26:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:26:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:26:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:26:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:26:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:26:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:26:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:26:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:26:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:26:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:26:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:26:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:26:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:26:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:26:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:26:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:26:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:26:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:26:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:26:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:26:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:26:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:26:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:28:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:28:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:28:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:28:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:28:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:28:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:28:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:28:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:28:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:28:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:28:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:28:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:28:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:28:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:28:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:28:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:28:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:28:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:28:57 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 21:29:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:29:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:29:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:29:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:29:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:29:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:29:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:29:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:29:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:29:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:29:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:29:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:29:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:29:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:29:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:29:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:29:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:29:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:29:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:29:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:29:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:29:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:29:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:29:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:29:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:29:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:29:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:29:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:29:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:29:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:29:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:29:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:29:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:29:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:29:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:29:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:29:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:29:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:29:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:29:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:29:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:29:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:29:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:29:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:29:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:29:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:29:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:29:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:29:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:29:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:29:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:29:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:29:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:29:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:29:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:29:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:29:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:29:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:29:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:29:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:29:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:29:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:29:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:29:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:29:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:29:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:29:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:29:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:29:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:29:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:29:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:29:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:29:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:29:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:29:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:29:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:29:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:29:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:29:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:29:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:29:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:29:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:29:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:29:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:29:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:29:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:29:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:29:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:29:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:29:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:29:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:29:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:29:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:29:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:29:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:29:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:29:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:29:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:29:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:29:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:29:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:29:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:29:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:29:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:29:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:29:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:29:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:29:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:29:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:29:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:29:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:29:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:29:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:29:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:29:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:29:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:29:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:29:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:29:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:29:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:29:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:29:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:29:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:29:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:29:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:29:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:29:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:29:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:29:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:29:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:29:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:29:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:29:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:29:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:29:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:29:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:29:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:29:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:29:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:29:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:29:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:29:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:29:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:29:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:29:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:29:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:29:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:29:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:29:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:29:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:29:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:29:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:29:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:29:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:29:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:29:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:29:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:29:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:29:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:29:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:29:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:29:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:29:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:29:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:29:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:29:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:29:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:29:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:29:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:29:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:29:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:29:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:29:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:29:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:29:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:29:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:29:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:29:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:29:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:29:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:29:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:29:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:29:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:29:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:29:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:29:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:29:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:29:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:29:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:29:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:29:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:29:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:29:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:29:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:29:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:29:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:29:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:29:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:29:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:29:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:29:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:29:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:29:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:29:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:29:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:29:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:29:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:29:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:29:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:29:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:29:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:29:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:29:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:29:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:29:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:29:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:30:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:30:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:30:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:30:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:30:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:30:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:30:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:30:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:30:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:30:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:30:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:30:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:30:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:30:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:30:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:30:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:30:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:30:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:30:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:30:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:30:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:30:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:30:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:30:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:30:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:30:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:30:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:30:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:30:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:30:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:30:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:30:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:30:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:30:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:30:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:30:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:30:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:30:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:30:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:30:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:30:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:30:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:30:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:30:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:30:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:30:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:30:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:30:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:30:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:30:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:30:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:30:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:30:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:30:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:30:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:30:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:30:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:30:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:30:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:30:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:30:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:30:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:30:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:30:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:30:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:30:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:30:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:30:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:30:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:30:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:30:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:30:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:30:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:30:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:30:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:30:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:30:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:30:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:30:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:30:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:30:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:30:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:30:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:30:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:30:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:30:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:30:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:30:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:30:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:30:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:30:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:30:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:30:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:30:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:30:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:30:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:30:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:30:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:30:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:30:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:30:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:30:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:30:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:30:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:30:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:30:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:30:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:30:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:30:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:30:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:30:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:30:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:30:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:30:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:30:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:30:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:30:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:30:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:30:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:30:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:30:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:30:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:30:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:30:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:30:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:30:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:30:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:30:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:30:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:30:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:30:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:30:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:30:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:30:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:30:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:30:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:30:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:30:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:30:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:30:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:30:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:30:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:30:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:30:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:30:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:30:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:30:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:30:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:30:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:30:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:30:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:30:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:30:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:30:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:30:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:30:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:30:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:30:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:30:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:30:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:30:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:30:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:30:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:30:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:30:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:30:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:30:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:30:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:30:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:30:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:30:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:30:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:30:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:30:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:30:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:30:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:30:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:30:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:30:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:30:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:30:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:30:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:30:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:30:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:30:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:30:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:30:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:30:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:30:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:30:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:30:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:30:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:30:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:30:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:30:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:30:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:30:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:30:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:30:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:30:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:30:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:30:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:30:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:30:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:30:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:30:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:30:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:30:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:30:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:30:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:30:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:30:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:30:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:30:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:30:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:30:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:31:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:31:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:31:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:31:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:31:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:31:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:31:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:31:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:31:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:31:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:31:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:31:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:31:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:31:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:31:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:31:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:31:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:31:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:31:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:31:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:31:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:31:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:31:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:31:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:31:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:31:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:31:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:31:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:31:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:31:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:31:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:31:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:31:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:31:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:31:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:31:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:31:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:31:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:31:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:31:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:31:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:31:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:31:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:31:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:31:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:31:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:31:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:31:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:31:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:31:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:31:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:31:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:31:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:31:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:31:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:31:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:31:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:31:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:31:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:31:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:31:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:31:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:31:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:31:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:31:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:31:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:31:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:31:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:31:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:31:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:31:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:31:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:31:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:31:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:31:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:31:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:31:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:31:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:31:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:31:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:31:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:31:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:31:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:31:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:31:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:31:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:31:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:31:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:31:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:31:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:31:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:31:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:31:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:31:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:31:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:31:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:31:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:31:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:31:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:31:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:31:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:31:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:31:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:31:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:31:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:31:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:31:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:31:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:31:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:31:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:31:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:31:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:31:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:31:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:31:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:31:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:31:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:31:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:31:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:31:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:31:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:31:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:31:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:31:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:31:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:31:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:31:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:31:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:31:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:31:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:31:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:31:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:31:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:31:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:31:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:31:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:31:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:31:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:31:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:31:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:31:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:31:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:31:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:31:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:31:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:31:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:31:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:31:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:31:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:31:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:31:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:31:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:31:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:31:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:31:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:31:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:31:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:31:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:31:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:31:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:31:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:31:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:31:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:31:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:31:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:31:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:31:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:31:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:31:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:31:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:31:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:31:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:31:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:31:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:31:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:31:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:31:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:31:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:31:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:31:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:31:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:31:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:31:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:31:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:31:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:31:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:31:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:31:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:31:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:31:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:31:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:31:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:31:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:31:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:31:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:31:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:31:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:31:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:31:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:31:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:31:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:31:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:31:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:31:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:31:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:31:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:31:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:31:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:31:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:31:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:31:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:31:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:31:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:31:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:31:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:31:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:32:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:32:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:32:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:32:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:32:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:32:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:32:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:32:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:32:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:32:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:32:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:32:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:32:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:32:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:32:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:32:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:32:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:32:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:32:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:32:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:32:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:32:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:32:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:32:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:32:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:32:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:32:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:32:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:32:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:32:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:32:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:32:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:32:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:32:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:32:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:32:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:32:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:32:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:32:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:32:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:32:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:32:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:32:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:32:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:32:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:32:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:32:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:32:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:32:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:32:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:32:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:32:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:32:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:32:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:32:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:32:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:32:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:32:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:32:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:32:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:32:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:32:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:32:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:32:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:32:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:32:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:32:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:32:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:32:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:32:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:32:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:32:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:32:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:32:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:32:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:32:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:32:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:32:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:32:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:32:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:32:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:32:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:32:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:32:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:32:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:32:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:32:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:32:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:32:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:32:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:32:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:32:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:32:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:32:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:32:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:32:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:32:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:32:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:32:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:32:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:32:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:32:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:32:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:32:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:32:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:32:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:32:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:32:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:32:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:32:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:32:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:32:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:32:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:32:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:32:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:32:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:32:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:32:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:32:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:32:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:32:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:32:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:32:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:32:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:32:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:32:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:32:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:32:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:32:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:32:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:32:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:32:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:32:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:32:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:32:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:32:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:32:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:32:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:32:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:32:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:32:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:32:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:32:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:32:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:32:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:32:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:32:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:32:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:32:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:32:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:32:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:32:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:32:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:32:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:32:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:32:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:32:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:32:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:32:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:32:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:32:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:32:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:32:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:32:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:32:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:32:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:32:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:32:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:32:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:32:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:32:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:32:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:32:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:32:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:32:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:32:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:32:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:32:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:32:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:32:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:32:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:32:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:32:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:32:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:32:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:32:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:32:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:32:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:32:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:32:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:32:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:32:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:32:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:32:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:32:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:32:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:32:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:32:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:32:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:32:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:32:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:32:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:32:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:32:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:32:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:32:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:32:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:32:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:32:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:32:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:32:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:32:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:32:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:32:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:32:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:32:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:33:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:33:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:33:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:33:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:33:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:33:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:33:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:33:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:33:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:33:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:33:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:33:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:33:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:33:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:33:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:33:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:33:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:33:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:33:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:33:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:33:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:33:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:33:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:33:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:33:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:33:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:33:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:33:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:33:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:33:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:33:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:33:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:33:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:33:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:33:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:33:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:33:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:33:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:33:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:33:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:33:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:33:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:33:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:33:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:33:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:33:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:33:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:33:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:33:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:33:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:33:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:33:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:33:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:33:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:33:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:33:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:33:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:33:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:33:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:33:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:33:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:33:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:33:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:33:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:33:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:33:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:33:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:33:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:33:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:33:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:33:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:33:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:33:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:33:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:33:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:33:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:33:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:33:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:33:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:33:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:33:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:33:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:33:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:33:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:33:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:33:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:33:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:33:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:33:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:33:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:33:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:33:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:33:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:33:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:33:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:33:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:33:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:33:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:33:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:33:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:33:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:33:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:33:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:33:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:33:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:33:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:33:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:33:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:33:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:33:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:33:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:33:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:33:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:33:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:33:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:33:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:33:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:33:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:33:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:33:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:33:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:33:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:33:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:33:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:33:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:33:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:33:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:33:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:33:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:33:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:33:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:33:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:33:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:33:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:33:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:33:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:33:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:33:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:33:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:33:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:33:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:33:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:33:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:33:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:33:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:33:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:33:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:33:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:33:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:33:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:33:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:33:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:33:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:33:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:33:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:33:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:33:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:33:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:33:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:33:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:33:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:33:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:33:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:33:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:33:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:33:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:33:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:33:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:33:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:33:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:33:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:33:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:33:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:33:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:33:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:33:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:33:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:33:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:33:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:33:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:33:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:33:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:33:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:33:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:33:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:33:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:33:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:33:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:33:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:33:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:33:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:33:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:33:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:33:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:33:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:33:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:33:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:33:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:33:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:33:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:33:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:33:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:33:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:33:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:33:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:33:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:33:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:33:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:33:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:33:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:33:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:33:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:33:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:33:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:33:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:33:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:34:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:34:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:34:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:34:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:34:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:34:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:34:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:34:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:34:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:34:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:34:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:34:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:34:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:34:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:34:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:34:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:34:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:34:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:34:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:34:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:34:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:34:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:34:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:34:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:34:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:34:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:34:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:34:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:34:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:34:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:34:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:34:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:34:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:34:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:34:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:34:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:34:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:34:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:34:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:34:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:34:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:34:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:34:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:34:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:34:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:34:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:34:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:34:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:34:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:34:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:34:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:34:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:34:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:34:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:34:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:34:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:34:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:34:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:34:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:34:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:34:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:34:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:34:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:34:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:34:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:34:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:34:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:34:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:34:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:34:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:34:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:34:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:34:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:34:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:34:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:34:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:34:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:34:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:34:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:34:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:34:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:34:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:34:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:34:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:34:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:34:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:34:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:34:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:34:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:34:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:34:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:34:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:34:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:34:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:34:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:34:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:34:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:34:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:34:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:34:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:34:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:34:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:34:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:34:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:34:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:34:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:34:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:34:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:34:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:34:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:34:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:34:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:34:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:34:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:34:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:34:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:34:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:34:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:34:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:34:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:34:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:34:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:34:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:34:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:34:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:34:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:34:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:34:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:34:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:34:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:34:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:34:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:34:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:34:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:34:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:34:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:34:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:34:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:34:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:34:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:34:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:34:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:34:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:34:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:34:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:34:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:34:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:34:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:34:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:34:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:34:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:34:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:34:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:34:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:34:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:34:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:34:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:34:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:34:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:34:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:34:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:34:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:34:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:34:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:34:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:34:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:34:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:34:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:34:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:34:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:34:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:34:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:34:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:34:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:34:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:34:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:34:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:34:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:34:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:34:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:34:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:34:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:34:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:34:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:34:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:34:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:34:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:34:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:34:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:34:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:34:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:34:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:34:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:34:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:34:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:34:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:34:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:34:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:34:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:34:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:34:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:34:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:34:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:34:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:34:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:34:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:34:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:34:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:34:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:34:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:34:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:34:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:34:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:34:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:34:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:34:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:35:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:35:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:35:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:35:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:35:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:35:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:35:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:35:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:35:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:35:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:35:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:35:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:35:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:35:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:35:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:35:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:35:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:35:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:35:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:35:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:35:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:35:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:35:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:35:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:35:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:35:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:35:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:35:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:35:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:35:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:35:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:35:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:35:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:35:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:35:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:35:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:35:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:35:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:35:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:35:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:35:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:35:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:35:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:35:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:35:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:35:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:35:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:35:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:35:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:35:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:35:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:35:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:35:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:35:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:35:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:35:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:35:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:35:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:35:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:35:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:35:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:35:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:35:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:35:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:35:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:35:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:35:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:35:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:35:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:35:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:35:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:35:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:35:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:35:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:35:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:35:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:35:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:35:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:35:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:35:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:35:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:35:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:35:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:35:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:35:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:35:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:35:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:35:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:35:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:35:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:35:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:35:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:35:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:35:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:35:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:35:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:35:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:35:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:35:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:35:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:35:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:35:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:35:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:35:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:35:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:35:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:35:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:35:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:35:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:35:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:35:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:35:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:35:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:35:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:35:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:35:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:35:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:35:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:35:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:35:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:35:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:35:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:35:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:35:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:35:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:35:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:35:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:35:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:35:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:35:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:35:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:35:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:35:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:35:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:35:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:35:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:35:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:35:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:35:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:35:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:35:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:35:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:35:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:35:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:35:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:35:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:35:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:35:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:35:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:35:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:35:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:35:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:35:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:35:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:35:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:35:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:35:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:35:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:35:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:35:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:35:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:35:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:35:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:35:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:35:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:35:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:35:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:35:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:35:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:35:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:35:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:35:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:35:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:35:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:35:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:35:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:35:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:35:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:35:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:35:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:35:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:35:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:35:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:35:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:35:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:35:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:35:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:35:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:35:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:35:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:35:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:35:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:35:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:35:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:35:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:35:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:35:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:35:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:35:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:35:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:35:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:35:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:35:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:35:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:35:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:35:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:35:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:35:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:35:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:35:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:35:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:35:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:35:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:35:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:35:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:35:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:36:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:36:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:36:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:36:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:36:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:36:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:36:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:36:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:36:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:36:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:36:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:36:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:36:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:36:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:36:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:36:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:36:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:36:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:36:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:36:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:36:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:36:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:36:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:36:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:36:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:36:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:36:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:36:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:36:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:36:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:36:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:36:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:36:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:36:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:36:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:36:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:36:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:36:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:36:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:36:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:36:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:36:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:36:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:36:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:36:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:36:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:36:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:36:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:36:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:36:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:36:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:36:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:36:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:36:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:36:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:36:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:36:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:36:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:36:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:36:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:36:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:36:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:36:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:36:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:36:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:36:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:36:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:36:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:36:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:36:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:36:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:36:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:36:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:36:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:36:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:36:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:36:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:36:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:36:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:36:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:36:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:36:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:36:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:36:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:36:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:36:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:36:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:36:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:36:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:36:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:36:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:36:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:36:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:36:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:36:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:36:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:36:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:36:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:36:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:36:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:36:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:36:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:36:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:36:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:36:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:36:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:36:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:36:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:36:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:36:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:36:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:36:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:36:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:36:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:36:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:36:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:36:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:36:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:36:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:36:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:36:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:36:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:36:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:36:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:36:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:36:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:36:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:36:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:36:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:36:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:36:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:36:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:36:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:36:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:36:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:36:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:36:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:36:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:36:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:36:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:36:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:36:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:36:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:36:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:36:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:36:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:36:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:36:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:36:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:36:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:36:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:36:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:36:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:36:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:36:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:36:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:36:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:36:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:36:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:36:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:36:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:36:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:36:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:36:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:36:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:36:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:36:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:36:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:36:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:36:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:36:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:36:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:36:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:36:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:36:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:36:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:36:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:36:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:36:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:36:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:36:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:36:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:36:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:36:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:36:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:36:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:36:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:36:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:36:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:36:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:36:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:36:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:36:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:36:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:36:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:36:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:36:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:36:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:38:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:38:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:38:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:38:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:38:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:38:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:38:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:38:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:38:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:38:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:38:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:38:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:38:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:38:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:38:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:38:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:38:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:38:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:38:58 rdma06 rsyslogd[1041]: imjournal: 5082 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 21:39:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:39:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:39:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:39:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:39:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:39:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:39:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:39:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:39:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:39:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:39:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:39:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:39:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:39:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:39:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:39:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:39:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:39:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:39:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:39:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:39:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:39:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:39:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:39:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:39:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:39:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:39:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:39:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:39:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:39:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:39:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:39:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:39:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:39:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:39:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:39:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:39:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:39:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:39:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:39:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:39:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:39:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:39:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:39:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:39:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:39:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:39:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:39:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:39:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:39:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:39:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:39:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:39:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:39:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:39:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:39:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:39:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:39:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:39:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:39:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:39:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:39:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:39:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:39:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:39:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:39:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:39:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:39:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:39:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:39:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:39:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:39:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:39:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:39:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:39:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:39:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:39:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:39:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:39:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:39:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:39:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:39:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:39:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:39:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:39:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:39:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:39:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:39:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:39:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:39:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:39:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:39:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:39:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:39:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:39:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:39:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:39:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:39:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:39:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:39:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:39:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:39:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:39:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:39:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:39:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:39:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:39:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:39:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:39:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:39:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:39:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:39:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:39:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:39:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:39:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:39:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:39:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:39:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:39:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:39:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:39:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:39:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:39:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:39:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:39:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:39:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:39:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:39:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:39:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:39:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:39:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:39:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:39:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:39:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:39:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:39:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:39:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:39:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:39:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:39:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:39:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:39:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:39:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:39:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:39:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:39:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:39:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:39:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:39:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:39:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:39:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:39:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:39:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:39:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:39:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:39:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:39:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:39:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:39:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:39:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:39:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:39:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:39:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:39:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:39:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:39:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:39:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:39:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:39:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:39:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:39:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:39:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:39:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:39:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:39:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:39:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:39:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:39:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:39:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:39:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:39:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:39:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:39:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:39:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:39:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:39:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:39:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:39:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:39:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:39:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:39:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:39:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:39:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:39:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:39:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:39:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:39:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:39:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:39:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:39:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:39:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:39:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:39:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:39:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:39:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:39:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:39:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:39:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:39:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:39:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:39:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:39:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:39:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:39:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:39:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:39:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:40:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:40:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:40:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:40:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:40:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:40:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:40:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:40:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:40:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:40:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:40:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:40:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:40:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:40:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:40:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:40:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:40:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:40:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:40:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:40:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:40:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:40:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:40:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:40:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:40:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:40:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:40:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:40:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:40:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:40:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:40:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:40:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:40:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:40:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:40:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:40:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:40:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:40:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:40:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:40:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:40:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:40:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:40:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:40:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:40:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:40:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:40:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:40:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:40:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:40:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:40:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:40:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:40:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:40:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:40:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:40:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:40:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:40:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:40:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:40:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:40:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:40:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:40:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:40:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:40:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:40:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:40:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:40:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:40:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:40:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:40:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:40:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:40:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:40:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:40:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:40:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:40:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:40:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:40:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:40:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:40:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:40:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:40:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:40:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:40:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:40:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:40:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:40:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:40:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:40:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:40:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:40:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:40:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:40:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:40:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:40:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:40:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:40:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:40:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:40:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:40:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:40:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:40:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:40:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:40:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:40:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:40:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:40:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:40:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:40:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:40:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:40:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:40:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:40:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:40:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:40:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:40:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:40:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:40:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:40:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:40:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:40:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:40:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:40:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:40:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:40:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:40:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:40:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:40:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:40:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:40:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:40:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:40:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:40:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:40:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:40:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:40:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:40:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:40:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:40:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:40:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:40:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:40:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:40:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:40:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:40:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:40:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:40:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:40:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:40:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:40:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:40:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:40:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:40:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:40:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:40:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:40:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:40:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:40:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:40:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:40:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:40:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:40:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:40:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:40:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:40:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:40:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:40:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:40:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:40:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:40:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:40:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:40:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:40:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:40:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:40:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:40:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:40:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:40:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:40:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:40:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:40:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:40:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:40:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:40:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:40:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:40:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:40:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:40:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:40:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:40:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:40:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:40:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:40:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:40:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:40:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:40:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:40:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:40:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:40:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:40:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:40:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:40:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:40:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:40:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:40:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:40:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:40:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:40:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:40:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:40:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:40:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:40:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:40:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:40:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:40:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:41:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:41:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:41:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:41:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:41:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:41:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:41:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:41:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:41:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:41:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:41:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:41:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:41:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:41:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:41:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:41:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:41:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:41:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:41:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:41:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:41:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:41:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:41:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:41:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:41:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:41:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:41:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:41:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:41:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:41:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:41:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:41:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:41:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:41:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:41:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:41:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:41:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:41:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:41:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:41:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:41:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:41:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:41:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:41:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:41:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:41:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:41:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:41:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:41:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:41:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:41:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:41:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:41:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:41:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:41:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:41:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:41:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:41:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:41:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:41:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:41:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:41:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:41:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:41:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:41:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:41:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:41:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:41:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:41:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:41:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:41:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:41:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:41:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:41:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:41:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:41:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:41:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:41:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:41:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:41:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:41:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:41:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:41:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:41:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:41:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:41:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:41:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:41:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:41:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:41:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:41:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:41:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:41:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:41:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:41:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:41:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:41:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:41:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:41:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:41:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:41:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:41:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:41:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:41:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:41:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:41:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:41:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:41:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:41:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:41:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:41:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:41:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:41:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:41:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:41:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:41:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:41:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:41:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:41:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:41:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:41:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:41:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:41:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:41:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:41:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:41:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:41:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:41:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:41:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:41:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:41:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:41:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:41:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:41:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:41:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:41:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:41:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:41:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:41:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:41:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:41:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:41:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:41:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:41:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:41:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:41:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:41:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:41:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:41:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:41:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:41:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:41:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:41:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:41:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:41:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:41:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:41:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:41:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:41:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:41:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:41:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:41:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:41:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:41:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:41:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:41:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:41:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:41:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:41:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:41:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:41:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:41:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:41:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:41:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:41:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:41:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:41:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:41:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:41:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:41:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:41:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:41:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:41:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:41:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:41:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:41:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:41:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:41:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:41:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:41:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:41:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:41:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:41:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:41:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:41:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:41:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:41:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:41:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:41:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:41:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:41:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:41:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:41:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:41:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:41:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:41:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:41:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:41:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:41:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:41:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:41:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:41:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:41:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:41:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:41:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:41:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:42:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:42:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:42:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:42:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:42:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:42:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:42:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:42:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:42:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:42:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:42:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:42:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:42:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:42:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:42:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:42:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:42:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:42:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:42:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:42:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:42:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:42:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:42:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:42:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:42:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:42:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:42:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:42:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:42:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:42:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:42:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:42:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:42:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:42:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:42:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:42:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:42:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:42:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:42:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:42:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:42:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:42:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:42:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:42:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:42:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:42:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:42:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:42:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:42:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:42:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:42:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:42:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:42:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:42:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:42:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:42:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:42:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:42:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:42:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:42:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:42:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:42:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:42:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:42:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:42:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:42:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:42:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:42:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:42:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:42:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:42:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:42:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:42:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:42:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:42:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:42:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:42:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:42:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:42:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:42:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:42:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:42:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:42:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:42:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:42:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:42:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:42:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:42:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:42:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:42:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:42:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:42:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:42:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:42:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:42:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:42:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:42:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:42:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:42:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:42:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:42:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:42:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:42:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:42:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:42:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:42:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:42:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:42:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:42:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:42:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:42:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:42:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:42:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:42:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:42:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:42:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:42:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:42:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:42:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:42:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:42:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:42:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:42:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:42:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:42:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:42:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:42:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:42:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:42:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:42:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:42:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:42:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:42:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:42:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:42:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:42:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:42:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:42:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:42:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:42:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:42:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:42:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:42:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:42:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:42:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:42:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:42:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:42:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:42:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:42:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:42:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:42:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:42:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:42:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:42:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:42:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:42:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:42:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:42:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:42:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:42:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:42:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:42:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:42:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:42:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:42:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:42:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:42:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:42:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:42:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:42:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:42:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:42:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:42:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:42:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:42:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:42:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:42:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:42:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:42:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:42:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:42:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:42:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:42:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:42:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:42:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:42:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:42:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:42:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:42:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:42:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:42:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:42:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:42:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:42:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:42:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:42:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:42:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:42:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:42:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:42:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:42:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:42:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:42:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:42:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:42:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:42:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:42:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:42:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:42:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:42:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:42:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:42:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:42:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:42:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:42:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:43:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:43:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:43:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:43:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:43:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:43:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:43:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:43:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:43:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:43:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:43:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:43:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:43:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:43:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:43:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:43:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:43:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:43:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:43:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:43:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:43:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:43:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:43:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:43:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:43:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:43:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:43:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:43:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:43:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:43:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:43:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:43:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:43:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:43:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:43:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:43:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:43:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:43:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:43:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:43:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:43:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:43:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:43:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:43:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:43:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:43:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:43:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:43:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:43:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:43:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:43:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:43:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:43:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:43:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:43:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:43:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:43:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:43:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:43:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:43:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:43:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:43:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:43:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:43:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:43:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:43:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:43:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:43:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:43:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:43:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:43:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:43:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:43:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:43:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:43:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:43:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:43:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:43:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:43:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:43:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:43:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:43:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:43:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:43:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:43:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:43:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:43:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:43:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:43:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:43:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:43:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:43:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:43:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:43:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:43:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:43:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:43:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:43:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:43:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:43:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:43:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:43:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:43:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:43:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:43:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:43:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:43:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:43:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:43:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:43:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:43:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:43:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:43:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:43:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:43:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:43:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:43:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:43:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:43:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:43:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:43:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:43:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:43:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:43:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:43:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:43:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:43:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:43:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:43:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:43:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:43:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:43:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:43:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:43:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:43:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:43:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:43:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:43:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:43:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:43:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:43:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:43:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:43:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:43:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:43:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:43:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:43:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:43:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:43:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:43:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:43:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:43:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:43:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:43:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:43:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:43:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:43:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:43:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:43:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:43:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:43:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:43:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:43:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:43:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:43:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:43:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:43:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:43:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:43:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:43:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:43:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:43:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:43:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:43:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:43:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:43:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:43:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:43:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:43:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:43:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:43:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:43:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:43:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:43:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:43:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:43:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:43:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:43:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:43:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:43:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:43:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:43:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:43:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:43:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:43:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:43:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:43:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:43:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:43:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:43:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:43:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:43:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:43:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:43:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:43:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:43:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:43:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:43:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:43:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:43:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:43:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:43:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:43:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:43:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:43:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:43:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:44:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:44:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:44:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:44:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:44:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:44:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:44:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:44:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:44:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:44:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:44:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:44:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:44:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:44:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:44:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:44:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:44:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:44:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:44:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:44:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:44:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:44:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:44:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:44:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:44:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:44:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:44:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:44:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:44:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:44:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:44:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:44:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:44:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:44:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:44:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:44:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:44:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:44:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:44:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:44:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:44:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:44:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:44:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:44:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:44:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:44:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:44:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:44:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:44:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:44:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:44:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:44:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:44:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:44:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:44:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:44:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:44:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:44:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:44:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:44:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:44:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:44:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:44:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:44:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:44:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:44:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:44:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:44:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:44:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:44:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:44:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:44:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:44:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:44:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:44:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:44:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:44:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:44:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:44:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:44:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:44:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:44:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:44:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:44:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:44:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:44:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:44:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:44:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:44:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:44:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:44:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:44:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:44:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:44:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:44:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:44:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:44:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:44:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:44:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:44:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:44:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:44:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:44:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:44:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:44:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:44:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:44:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:44:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:44:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:44:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:44:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:44:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:44:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:44:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:44:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:44:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:44:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:44:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:44:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:44:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:44:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:44:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:44:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:44:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:44:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:44:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:44:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:44:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:44:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:44:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:44:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:44:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:44:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:44:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:44:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:44:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:44:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:44:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:44:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:44:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:44:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:44:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:44:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:44:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:44:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:44:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:44:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:44:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:44:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:44:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:44:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:44:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:44:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:44:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:44:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:44:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:44:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:44:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:44:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:44:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:44:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:44:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:44:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:44:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:44:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:44:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:44:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:44:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:44:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:44:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:44:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:44:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:44:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:44:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:44:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:44:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:44:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:44:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:44:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:44:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:44:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:44:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:44:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:44:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:44:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:44:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:44:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:44:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:44:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:44:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:44:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:44:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:44:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:44:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:44:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:44:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:44:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:44:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:44:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:44:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:44:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:44:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:44:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:44:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:44:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:44:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:44:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:44:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:44:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:44:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:44:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:44:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:44:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:44:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:44:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:44:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:45:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:45:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:45:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:45:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:45:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:45:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:45:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:45:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:45:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:45:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:45:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:45:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:45:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:45:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:45:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:45:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:45:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:45:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:45:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:45:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:45:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:45:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:45:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:45:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:45:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:45:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:45:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:45:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:45:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:45:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:45:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:45:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:45:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:45:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:45:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:45:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:45:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:45:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:45:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:45:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:45:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:45:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:45:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:45:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:45:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:45:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:45:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:45:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:45:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:45:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:45:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:45:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:45:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:45:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:45:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:45:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:45:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:45:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:45:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:45:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:45:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:45:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:45:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:45:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:45:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:45:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:45:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:45:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:45:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:45:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:45:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:45:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:45:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:45:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:45:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:45:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:45:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:45:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:45:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:45:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:45:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:45:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:45:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:45:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:45:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:45:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:45:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:45:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:45:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:45:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:45:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:45:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:45:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:45:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:45:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:45:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:45:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:45:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:45:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:45:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:45:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:45:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:45:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:45:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:45:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:45:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:45:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:45:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:45:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:45:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:45:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:45:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:45:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:45:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:45:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:45:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:45:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:45:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:45:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:45:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:45:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:45:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:45:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:45:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:45:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:45:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:45:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:45:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:45:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:45:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:45:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:45:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:45:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:45:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:45:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:45:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:45:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:45:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:45:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:45:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:45:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:45:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:45:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:45:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:45:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:45:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:45:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:45:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:45:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:45:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:45:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:45:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:45:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:45:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:45:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:45:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:45:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:45:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:45:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:45:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:45:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:45:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:45:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:45:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:45:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:45:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:45:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:45:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:45:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:45:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:45:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:45:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:45:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:45:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:45:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:45:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:45:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:45:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:45:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:45:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:45:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:45:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:45:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:45:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:45:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:45:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:45:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:45:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:45:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:45:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:45:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:45:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:45:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:45:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:45:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:45:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:45:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:45:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:45:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:45:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:45:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:45:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:45:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:45:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:45:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:45:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:45:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:45:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:45:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:45:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:45:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:45:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:45:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:45:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:45:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:45:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:46:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:46:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:46:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:46:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:46:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:46:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:46:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:46:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:46:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:46:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:46:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:46:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:46:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:46:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:46:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:46:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:46:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:46:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:46:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:46:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:46:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:46:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:46:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:46:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:46:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:46:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:46:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:46:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:46:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:46:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:46:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:46:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:46:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:46:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:46:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:46:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:46:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:46:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:46:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:46:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:46:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:46:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:46:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:46:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:46:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:46:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:46:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:46:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:46:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:46:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:46:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:46:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:46:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:46:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:46:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:46:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:46:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:46:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:46:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:46:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:46:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:46:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:46:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:46:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:46:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:46:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:46:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:46:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:46:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:46:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:46:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:46:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:46:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:46:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:46:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:46:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:46:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:46:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:46:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:46:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:46:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:46:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:46:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:46:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:46:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:46:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:46:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:46:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:46:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:46:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:46:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:46:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:46:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:46:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:46:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:46:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:46:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:46:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:46:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:46:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:46:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:46:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:46:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:46:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:46:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:46:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:46:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:46:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:46:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:46:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:46:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:46:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:46:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:46:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:46:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:46:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:46:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:46:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:46:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:46:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:46:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:46:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:46:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:46:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:46:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:46:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:46:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:46:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:46:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:46:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:46:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:46:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:46:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:46:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:46:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:46:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:46:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:46:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:46:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:46:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:46:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:46:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:46:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:46:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:46:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:46:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:46:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:46:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:46:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:46:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:46:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:46:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:46:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:46:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:46:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:46:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:46:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:46:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:46:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:46:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:46:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:46:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:46:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:46:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:46:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:46:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:46:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:46:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:46:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:46:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:46:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:46:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:46:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:46:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:46:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:46:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:46:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:46:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:46:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:46:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:46:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:46:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:46:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:46:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:46:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:46:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:46:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:46:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:46:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:46:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:46:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:46:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:46:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:46:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:46:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:46:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:46:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:46:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:49:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:49:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:49:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:49:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:49:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:49:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:49:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:49:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:49:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:49:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:49:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:49:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:49:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:49:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:49:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:49:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:49:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:49:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:49:03 rdma06 rsyslogd[1041]: imjournal: 5295 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 21:49:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:49:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:49:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:49:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:49:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:49:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:49:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:49:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:49:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:49:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:49:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:49:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:49:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:49:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:49:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:49:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:49:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:49:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:49:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:49:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:49:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:49:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:49:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:49:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:49:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:49:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:49:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:49:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:49:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:49:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:49:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:49:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:49:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:49:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:49:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:49:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:49:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:49:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:49:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:49:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:49:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:49:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:49:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:49:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:49:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:49:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:49:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:49:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:49:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:49:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:49:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:49:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:49:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:49:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:49:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:49:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:49:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:49:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:49:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:49:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:49:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:49:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:49:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:49:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:49:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:49:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:49:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:49:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:49:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:49:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:49:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:49:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:49:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:49:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:49:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:49:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:49:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:49:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:49:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:49:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:49:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:49:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:49:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:49:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:49:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:49:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:49:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:49:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:49:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:49:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:49:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:49:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:49:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:49:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:49:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:49:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:49:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:49:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:49:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:49:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:49:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:49:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:49:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:49:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:49:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:49:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:49:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:49:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:49:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:49:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:49:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:49:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:49:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:49:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:49:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:49:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:49:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:49:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:49:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:49:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:49:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:49:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:49:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:49:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:49:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:49:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:49:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:49:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:49:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:49:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:49:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:49:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:49:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:49:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:49:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:49:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:49:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:49:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:49:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:49:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:49:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:49:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:49:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:49:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:49:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:49:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:49:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:49:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:49:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:49:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:49:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:49:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:49:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:49:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:49:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:49:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:49:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:49:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:49:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:49:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:49:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:49:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:49:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:49:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:49:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:49:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:49:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:49:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:49:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:49:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:49:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:49:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:49:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:49:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:49:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:49:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:49:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:49:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:49:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:49:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:49:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:49:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:49:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:49:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:49:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:49:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:49:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:49:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:49:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:49:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:49:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:49:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:49:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:49:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:49:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:49:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:49:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:49:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:50:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:50:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:50:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:50:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:50:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:50:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:50:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:50:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:50:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:50:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:50:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:50:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:50:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:50:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:50:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:50:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:50:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:50:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:50:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:50:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:50:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:50:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:50:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:50:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:50:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:50:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:50:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:50:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:50:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:50:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:50:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:50:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:50:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:50:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:50:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:50:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:50:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:50:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:50:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:50:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:50:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:50:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:50:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:50:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:50:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:50:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:50:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:50:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:50:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:50:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:50:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:50:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:50:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:50:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:50:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:50:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:50:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:50:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:50:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:50:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:50:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:50:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:50:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:50:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:50:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:50:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:50:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:50:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:50:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:50:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:50:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:50:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:50:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:50:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:50:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:50:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:50:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:50:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:50:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:50:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:50:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:50:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:50:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:50:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:50:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:50:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:50:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:50:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:50:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:50:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:50:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:50:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:50:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:50:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:50:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:50:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:50:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:50:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:50:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:50:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:50:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:50:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:50:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:50:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:50:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:50:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:50:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:50:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:50:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:50:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:50:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:50:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:50:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:50:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:50:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:50:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:50:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:50:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:50:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:50:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:50:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:50:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:50:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:50:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:50:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:50:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:50:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:50:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:50:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:50:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:50:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:50:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:50:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:50:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:50:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:50:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:50:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:50:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:50:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:50:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:50:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:50:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:50:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:50:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:50:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:50:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:50:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:50:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:50:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:50:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:50:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:50:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:50:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:50:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:50:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:50:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:50:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:50:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:50:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:50:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:50:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:50:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:50:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:50:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:50:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:50:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:50:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:50:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:50:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:50:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:50:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:50:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:50:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:50:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:50:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:50:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:50:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:50:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:50:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:50:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:50:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:50:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:50:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:50:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:50:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:50:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:50:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:50:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:50:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:50:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:50:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:50:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:50:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:50:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:50:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:50:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:50:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:50:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:50:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:50:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:50:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:50:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:50:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:50:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:50:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:50:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:50:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:50:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:50:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:50:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:50:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:50:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:50:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:50:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:50:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:50:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:51:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:51:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:51:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:51:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:51:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:51:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:51:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:51:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:51:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:51:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:51:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:51:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:51:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:51:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:51:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:51:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:51:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:51:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:51:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:51:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:51:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:51:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:51:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:51:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:51:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:51:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:51:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:51:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:51:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:51:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:51:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:51:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:51:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:51:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:51:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:51:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:51:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:51:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:51:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:51:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:51:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:51:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:51:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:51:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:51:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:51:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:51:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:51:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:51:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:51:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:51:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:51:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:51:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:51:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:51:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:51:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:51:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:51:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:51:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:51:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:51:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:51:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:51:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:51:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:51:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:51:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:51:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:51:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:51:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:51:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:51:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:51:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:51:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:51:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:51:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:51:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:51:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:51:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:51:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:51:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:51:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:51:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:51:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:51:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:51:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:51:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:51:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:51:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:51:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:51:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:51:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:51:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:51:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:51:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:51:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:51:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:51:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:51:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:51:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:51:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:51:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:51:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:51:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:51:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:51:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:51:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:51:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:51:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:51:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:51:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:51:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:51:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:51:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:51:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:51:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:51:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:51:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:51:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:51:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:51:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:51:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:51:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:51:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:51:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:51:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:51:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:51:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:51:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:51:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:51:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:51:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:51:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:51:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:51:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:51:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:51:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:51:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:51:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:51:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:51:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:51:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:51:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:51:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:51:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:51:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:51:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:52:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:52:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:52:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:52:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:52:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:52:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:52:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:52:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:52:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:52:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:52:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:52:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:52:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:52:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:52:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:52:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:52:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:52:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:52:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:52:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:52:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:52:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:52:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:52:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:52:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:52:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:52:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:52:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:52:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:52:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:52:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:52:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:52:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:52:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:52:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:52:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:52:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:52:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:52:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:52:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:52:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:52:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:52:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:52:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:52:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:52:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:52:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:52:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:52:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:52:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:52:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:52:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:52:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:52:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:52:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:52:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:52:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:52:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:52:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:52:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:52:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:52:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:52:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:52:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:52:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:52:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:52:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:52:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:52:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:52:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:52:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:52:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:52:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:52:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:52:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:52:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:52:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:52:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:52:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:52:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:52:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:52:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:52:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:52:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:52:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:52:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:52:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:52:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:52:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:52:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:52:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:52:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:52:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:52:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:52:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:52:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:52:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:52:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:52:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:52:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:52:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:52:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:52:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:52:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:52:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:52:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:52:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:52:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:52:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:52:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:52:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:52:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:52:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:52:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:52:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:52:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:52:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:52:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:52:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:52:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:52:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:52:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:52:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:52:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:52:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:52:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:52:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:52:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:52:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:52:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:52:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:52:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:52:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:52:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:52:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:52:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:52:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:52:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:52:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:52:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:52:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:52:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:52:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:52:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:52:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:52:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:52:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:52:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:52:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:52:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:52:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:52:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:52:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:52:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:52:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:52:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:52:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:52:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:52:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:52:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:52:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:52:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:52:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:52:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:52:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:52:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:52:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:52:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:52:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:52:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:52:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:52:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:52:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:52:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:52:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:52:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:53:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:53:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:53:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:53:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:53:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:53:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:53:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:53:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:53:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:53:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:53:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:53:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:53:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:53:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:53:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:53:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:53:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:53:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:53:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:53:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:53:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:53:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:53:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:53:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:53:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:53:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:53:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:53:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:53:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:53:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:53:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:53:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:53:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:53:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:53:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:53:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:53:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:53:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:53:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:53:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:53:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:53:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:53:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:53:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:53:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:53:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:53:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:53:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:53:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:53:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:53:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:53:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:53:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:53:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:53:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:53:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:53:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:53:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:53:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:53:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:53:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:53:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:53:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:53:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:53:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:53:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:53:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:53:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:53:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:53:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:53:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:53:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:53:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:53:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:53:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:53:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:53:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:53:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:53:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:53:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:53:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:53:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:53:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:53:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:53:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:53:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:53:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:53:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:53:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:53:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:53:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:53:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:53:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:53:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:53:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:53:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:53:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:53:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:53:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:53:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:53:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:53:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:53:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:53:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:53:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:53:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:53:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:53:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:53:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:53:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:53:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:53:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:53:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:53:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:53:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:53:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:53:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:53:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:53:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:53:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:53:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:53:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:53:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:53:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:53:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:53:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:53:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:53:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:53:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:53:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:53:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:53:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:53:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:53:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:53:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:53:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:53:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:53:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:53:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:53:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:53:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:53:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:53:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:53:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:53:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:53:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:53:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:53:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:53:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:53:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:53:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:53:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:53:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:53:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:53:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:53:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:53:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:53:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:53:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:53:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:53:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:53:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:53:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:53:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:53:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:53:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:53:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:53:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:53:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:53:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:53:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:53:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:53:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:53:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:53:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:53:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:53:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:53:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:53:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:53:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:53:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:53:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:53:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:53:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:53:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:53:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:53:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:53:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:53:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:53:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:53:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:53:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:53:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:53:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:53:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:53:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:53:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:53:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:53:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:53:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:53:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:53:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:53:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:53:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:53:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:53:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:53:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:53:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:53:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:53:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:53:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:53:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:53:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:53:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:53:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:53:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:54:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:54:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:54:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:54:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:54:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:54:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:54:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:54:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:54:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:54:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:54:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:54:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:54:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:54:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:54:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:54:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:54:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:54:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:54:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:54:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:54:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:54:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:54:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:54:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:54:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:54:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:54:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:54:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:54:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:54:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:54:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:54:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:54:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:54:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:54:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:54:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:54:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:54:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:54:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:54:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:54:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:54:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:54:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:54:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:54:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:54:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:54:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:54:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:54:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:54:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:54:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:54:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:54:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:54:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:54:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:54:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:54:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:54:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:54:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:54:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:54:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:54:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:54:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:54:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:54:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:54:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:54:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:54:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:54:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:54:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:54:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:54:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:54:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:54:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:54:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:54:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:54:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:54:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:54:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:54:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:54:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:54:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:54:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:54:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:54:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:54:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:54:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:54:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:54:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:54:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:54:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:54:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:54:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:54:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:54:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:54:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:54:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:54:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:54:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:54:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:54:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:54:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:54:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:54:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:54:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:54:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:54:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:54:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:54:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:54:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:54:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:54:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:54:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:54:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:54:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:54:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:54:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:54:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:54:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:54:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:54:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:54:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:54:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:54:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:54:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:54:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:54:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:54:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:54:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:54:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:54:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:54:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:54:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:54:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:54:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:54:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:54:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:54:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:54:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:54:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:54:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:54:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:54:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:54:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:54:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:54:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:54:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:54:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:54:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:54:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:54:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:54:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:54:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:54:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:54:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:54:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:54:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:54:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:54:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:54:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:54:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:54:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:54:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:54:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:54:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:54:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:54:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:54:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:54:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:54:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:54:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:54:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:54:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:54:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:54:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:54:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:54:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:54:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:54:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:54:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:54:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:54:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:54:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:54:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:54:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:54:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:54:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:54:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:54:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:54:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:54:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:54:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:54:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:54:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:54:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:54:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:54:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:54:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:54:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:54:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:54:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:54:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:54:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:54:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:54:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:54:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:54:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:54:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:54:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:54:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:54:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:54:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:54:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:54:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:54:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:54:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:55:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:55:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:55:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:55:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:55:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:55:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:55:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:55:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:55:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:55:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:55:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:55:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:55:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:55:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:55:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:55:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:55:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:55:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:55:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:55:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:55:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:55:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:55:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:55:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:55:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:55:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:55:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:55:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:55:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:55:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:55:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:55:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:55:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:55:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:55:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:55:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:55:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:55:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:55:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:55:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:55:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:55:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:55:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:55:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:55:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:55:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:55:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:55:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:55:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:55:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:55:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:55:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:55:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:55:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:55:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:55:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:55:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:55:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:55:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:55:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:55:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:55:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:55:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:55:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:55:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:55:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:55:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:55:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:55:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:55:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:55:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:55:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:55:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:55:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:55:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:55:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:55:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:55:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:55:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:55:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:55:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:55:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:55:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:55:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:55:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:55:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:55:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:55:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:55:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:55:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:55:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:55:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:55:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:55:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:55:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:55:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:55:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:55:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:55:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:55:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:55:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:55:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:55:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:55:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:55:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:55:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:55:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:55:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:55:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:55:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:55:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:55:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:55:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:55:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:55:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:55:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:55:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:55:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:55:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:55:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:55:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:55:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:55:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:55:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:55:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:55:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:55:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:55:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:55:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:55:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:55:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:55:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:55:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:55:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:55:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:55:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:55:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:55:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:55:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:55:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:55:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:55:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:55:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:55:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:55:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:55:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:55:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:55:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:55:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:55:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:55:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:55:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:55:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:55:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:55:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:55:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:55:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:55:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:55:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:55:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:55:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:55:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:55:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:55:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:55:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:55:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:55:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:55:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:55:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:55:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:55:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:55:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:55:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:55:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:55:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:55:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:55:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:55:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:55:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:55:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:55:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:55:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:55:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:55:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:55:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:55:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:55:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:55:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:55:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:55:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:55:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:55:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:55:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:55:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:55:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:55:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:55:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:55:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:55:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:55:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:55:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:55:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:55:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:55:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:55:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:55:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:55:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:55:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:55:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:55:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:55:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:55:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:55:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:55:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:55:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:55:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:56:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:56:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:56:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:56:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:56:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:56:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:56:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:56:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:56:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:56:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:56:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:56:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:56:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:56:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:56:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:56:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:56:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:56:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:56:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:56:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:56:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:56:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:56:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:56:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:56:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:56:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:56:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:56:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:56:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:56:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:56:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:56:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:56:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:56:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:56:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:56:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:56:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:56:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:56:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:56:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:56:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:56:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:56:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:56:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:56:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:56:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:56:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:56:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:56:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:56:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:56:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:56:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:56:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:56:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:56:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:56:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:56:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:56:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:56:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:56:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:56:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:56:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:56:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:56:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:56:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:56:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:56:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:56:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:56:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:56:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:56:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:56:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:56:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:56:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:56:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:56:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:56:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:56:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:56:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:56:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:56:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:56:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:56:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:56:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:56:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:56:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:56:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:56:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:56:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:56:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:56:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:56:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:56:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:56:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:56:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:56:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:56:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:56:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:56:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:56:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:56:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:56:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:56:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:56:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:56:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:56:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:56:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:56:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:56:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:56:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:56:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:56:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:56:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:56:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:56:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:56:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:56:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:56:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:56:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:56:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:56:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:56:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:56:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:56:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:56:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:56:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:56:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:56:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:56:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:56:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:56:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:56:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:56:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:56:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:56:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:56:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:56:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:56:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:56:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:56:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:56:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:56:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:56:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:56:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:56:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:56:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:56:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:56:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:56:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:56:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:56:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:56:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:56:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:56:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:56:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:56:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:56:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:56:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:56:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:56:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:56:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:56:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:56:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:56:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:56:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:56:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:56:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:56:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:56:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:56:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:56:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:56:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:56:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:56:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:56:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:56:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:56:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:56:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:56:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:56:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:56:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:56:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:56:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:56:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:56:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:56:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:56:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:56:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:56:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:56:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:56:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:56:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:56:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:56:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:56:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:56:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:56:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:56:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:56:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:56:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:56:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:56:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:56:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:56:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:56:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:56:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:56:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:56:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:56:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:56:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:56:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:56:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:56:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:56:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:56:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:56:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:59:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:59:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:59:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:59:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:59:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:59:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:59:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:59:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:59:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:59:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:59:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:59:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:59:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:59:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:59:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:59:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:59:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:59:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:59:08 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 21:59:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:59:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:59:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:59:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:59:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:59:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:59:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:59:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:59:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:59:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:59:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:59:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:59:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:59:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:59:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:59:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:59:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:59:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:59:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:59:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:59:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:59:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:59:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:59:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:59:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:59:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:59:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:59:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:59:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:59:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:59:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:59:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:59:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:59:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:59:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:59:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:59:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:59:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:59:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:59:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:59:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:59:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:59:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:59:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:59:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:59:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:59:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:59:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:59:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:59:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:59:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:59:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:59:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:59:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:59:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:59:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:59:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:59:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:59:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:59:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:59:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:59:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:59:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:59:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:59:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:59:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:59:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:59:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:59:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:59:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:59:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:59:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:59:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:59:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:59:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:59:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:59:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:59:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:59:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:59:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:59:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:59:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:59:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:59:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:59:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:59:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:59:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:59:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:59:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:59:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:59:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:59:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:59:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:59:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:59:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:59:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:59:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:59:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:59:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:59:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:59:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:59:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:59:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:59:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:59:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:59:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:59:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:59:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:59:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:59:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:59:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:59:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:59:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:59:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:59:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:59:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:59:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:59:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:59:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:59:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:59:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:59:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:59:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:59:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:59:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:59:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:59:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:59:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:59:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:59:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:59:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:59:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:59:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:59:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:59:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:59:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:59:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:59:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:59:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:59:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:59:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:59:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:59:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:59:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:59:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:59:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:59:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:59:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:59:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:59:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:59:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:59:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:59:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:59:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:59:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:59:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:59:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:59:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:59:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:59:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:59:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:59:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 21:59:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 21:59:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:59:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:59:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:59:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 21:59:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 21:59:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 21:59:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 21:59:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 21:59:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 21:59:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 21:59:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 21:59:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 21:59:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 21:59:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 21:59:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 21:59:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 21:59:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:00:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:00:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:00:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:00:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:00:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:00:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:00:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:00:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:00:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:00:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:00:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:00:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:00:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:00:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:00:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:00:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:00:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:00:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:00:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:00:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:00:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:00:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:00:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:00:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:00:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:00:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:00:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:00:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:00:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:00:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:00:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:00:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:00:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:00:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:00:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:00:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:00:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:00:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:00:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:00:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:00:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:00:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:00:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:00:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:00:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:00:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:00:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:00:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:00:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:00:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:00:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:00:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:00:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:00:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:00:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:00:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:00:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:00:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:00:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:00:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:00:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:00:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:00:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:00:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:00:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:00:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:00:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:00:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:00:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:00:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:00:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:00:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:00:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:00:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:00:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:00:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:00:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:00:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:00:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:00:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:00:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:00:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:00:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:00:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:00:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:00:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:00:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:00:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:00:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:00:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:00:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:00:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:00:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:00:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:00:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:00:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:00:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:00:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:00:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:00:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:00:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:00:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:00:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:00:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:00:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:00:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:00:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:00:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:00:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:00:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:00:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:00:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:00:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:00:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:00:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:00:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:00:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:00:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:00:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:00:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:00:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:00:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:00:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:00:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:00:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:00:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:00:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:00:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:00:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:00:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:00:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:00:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:00:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:00:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:00:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:00:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:00:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:00:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:00:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:00:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:00:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:00:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:00:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:00:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:00:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:00:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:00:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:00:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:00:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:00:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:00:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:00:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:00:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:00:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:00:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:00:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:00:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:00:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:00:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:00:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:00:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:00:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:00:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:00:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:00:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:00:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:00:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:00:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:00:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:00:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:00:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:00:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:00:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:00:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:00:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:00:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:00:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:00:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:00:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:00:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:00:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:00:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:00:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:00:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:00:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:00:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:00:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:00:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:00:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:00:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:00:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:00:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:00:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:00:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:00:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:00:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:00:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:00:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:00:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:00:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:00:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:00:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:00:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:00:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:00:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:00:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:00:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:00:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:00:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:00:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:00:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:00:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:00:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:00:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:00:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:00:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:01:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:01:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:01:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:01:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:01:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:01:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:01:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:01:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:01:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:01:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:01:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:01:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:01:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:01:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:01:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:01:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:01:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:01:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:01:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:01:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:01:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:01:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:01:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:01:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:01:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:01:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:01:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:01:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:01:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:01:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:01:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:01:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:01:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:01:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:01:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:01:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:01:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:01:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:01:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:01:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:01:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:01:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:01:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:01:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:01:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:01:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:01:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:01:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:01:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:01:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:01:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:01:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:01:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:01:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:01:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:01:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:01:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:01:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:01:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:01:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:01:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:01:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:01:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:01:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:01:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:01:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:01:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:01:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:01:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:01:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:01:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:01:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:01:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:01:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:01:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:01:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:01:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:01:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:01:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:01:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:01:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:01:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:01:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:01:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:01:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:01:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:01:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:01:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:01:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:01:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:01:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:01:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:01:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:01:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:01:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:01:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:01:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:01:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:01:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:01:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:01:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:01:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:01:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:01:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:01:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:01:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:01:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:01:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:01:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:01:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:01:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:01:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:01:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:01:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:01:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:01:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:01:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:01:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:01:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:01:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:01:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:01:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:01:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:01:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:01:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:01:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:01:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:01:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:01:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:01:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:01:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:01:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:01:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:01:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:01:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:01:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:01:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:01:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:01:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:01:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:01:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:01:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:01:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:01:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:01:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:01:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:01:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:01:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:01:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:01:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:01:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:01:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:01:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:01:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:01:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:01:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:01:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:01:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:01:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:01:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:01:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:01:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:01:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:01:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:01:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:01:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:01:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:01:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:01:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:01:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:01:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:01:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:01:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:01:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:01:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:01:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:01:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:01:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:01:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:01:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:01:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:01:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:01:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:01:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:01:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:01:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:01:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:01:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:01:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:01:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:01:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:01:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:01:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:01:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:01:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:01:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:01:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:01:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:01:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:01:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:01:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:01:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:01:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:01:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:01:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:01:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:01:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:01:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:01:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:01:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:01:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:01:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:01:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:01:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:01:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:01:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:02:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:02:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:02:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:02:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:02:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:02:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:02:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:02:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:02:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:02:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:02:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:02:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:02:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:02:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:02:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:02:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:02:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:02:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:02:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:02:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:02:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:02:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:02:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:02:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:02:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:02:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:02:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:02:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:02:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:02:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:02:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:02:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:02:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:02:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:02:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:02:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:02:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:02:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:02:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:02:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:02:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:02:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:02:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:02:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:02:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:02:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:02:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:02:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:02:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:02:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:02:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:02:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:02:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:02:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:02:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:02:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:02:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:02:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:02:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:02:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:02:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:02:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:02:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:02:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:02:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:02:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:02:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:02:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:02:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:02:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:02:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:02:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:02:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:02:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:02:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:02:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:02:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:02:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:02:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:02:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:02:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:02:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:02:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:02:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:02:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:02:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:02:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:02:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:02:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:02:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:02:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:02:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:02:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:02:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:02:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:02:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:02:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:02:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:02:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:02:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:02:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:02:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:02:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:02:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:02:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:02:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:02:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:02:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:02:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:02:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:02:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:02:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:02:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:02:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:02:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:02:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:02:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:02:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:02:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:02:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:02:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:02:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:02:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:02:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:02:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:02:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:02:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:02:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:02:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:02:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:02:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:02:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:02:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:02:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:02:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:02:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:02:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:02:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:02:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:02:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:02:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:02:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:02:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:02:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:02:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:02:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:02:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:02:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:02:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:02:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:02:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:02:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:02:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:02:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:02:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:02:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:02:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:02:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:02:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:02:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:02:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:02:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:02:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:02:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:02:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:02:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:02:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:02:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:02:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:02:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:02:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:02:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:02:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:02:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:02:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:02:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:02:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:02:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:02:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:02:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:02:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:02:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:02:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:02:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:02:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:02:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:02:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:02:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:02:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:02:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:02:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:02:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:02:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:02:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:02:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:02:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:02:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:02:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:02:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:02:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:02:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:02:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:02:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:02:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:02:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:02:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:02:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:02:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:02:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:02:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:02:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:02:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:02:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:02:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:02:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:02:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:03:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:03:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:03:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:03:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:03:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:03:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:03:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:03:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:03:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:03:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:03:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:03:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:03:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:03:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:03:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:03:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:03:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:03:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:03:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:03:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:03:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:03:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:03:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:03:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:03:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:03:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:03:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:03:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:03:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:03:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:03:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:03:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:03:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:03:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:03:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:03:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:03:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:03:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:03:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:03:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:03:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:03:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:03:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:03:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:03:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:03:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:03:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:03:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:03:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:03:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:03:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:03:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:03:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:03:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:03:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:03:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:03:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:03:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:03:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:03:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:03:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:03:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:03:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:03:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:03:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:03:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:03:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:03:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:03:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:03:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:03:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:03:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:03:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:03:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:03:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:03:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:03:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:03:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:03:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:03:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:03:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:03:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:03:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:03:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:03:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:03:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:03:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:03:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:03:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:03:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:03:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:03:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:03:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:03:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:03:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:03:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:03:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:03:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:03:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:03:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:03:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:03:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:03:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:03:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:03:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:03:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:03:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:03:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:03:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:03:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:03:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:03:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:03:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:03:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:03:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:03:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:03:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:03:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:03:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:03:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:03:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:03:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:03:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:03:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:03:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:03:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:03:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:03:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:03:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:03:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:03:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:03:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:03:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:03:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:03:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:03:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:03:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:03:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:03:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:03:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:03:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:03:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:03:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:03:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:03:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:03:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:03:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:03:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:03:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:03:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:03:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:03:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:03:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:03:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:03:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:03:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:03:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:03:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:03:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:03:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:03:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:03:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:03:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:03:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:03:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:03:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:03:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:03:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:03:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:03:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:03:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:03:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:03:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:03:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:03:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:03:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:03:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:03:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:03:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:03:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:03:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:03:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:03:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:03:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:03:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:03:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:03:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:03:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:03:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:03:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:03:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:03:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:03:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:03:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:03:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:03:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:03:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:03:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:03:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:03:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:03:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:03:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:03:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:03:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:03:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:03:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:03:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:03:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:03:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:03:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:03:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:03:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:03:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:03:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:03:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:03:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:04:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:04:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:04:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:04:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:04:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:04:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:04:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:04:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:04:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:04:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:04:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:04:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:04:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:04:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:04:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:04:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:04:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:04:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:04:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:04:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:04:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:04:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:04:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:04:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:04:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:04:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:04:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:04:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:04:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:04:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:04:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:04:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:04:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:04:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:04:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:04:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:04:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:04:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:04:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:04:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:04:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:04:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:04:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:04:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:04:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:04:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:04:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:04:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:04:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:04:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:04:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:04:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:04:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:04:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:04:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:04:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:04:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:04:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:04:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:04:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:04:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:04:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:04:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:04:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:04:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:04:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:04:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:04:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:04:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:04:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:04:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:04:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:04:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:04:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:04:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:04:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:04:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:04:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:04:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:04:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:04:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:04:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:04:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:04:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:04:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:04:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:04:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:04:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:04:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:04:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:04:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:04:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:04:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:04:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:04:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:04:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:04:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:04:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:04:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:04:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:04:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:04:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:04:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:04:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:04:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:04:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:04:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:04:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:04:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:04:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:04:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:04:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:04:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:04:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:04:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:04:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:04:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:04:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:04:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:04:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:04:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:04:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:04:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:04:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:04:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:04:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:04:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:04:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:04:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:04:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:04:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:04:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:04:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:04:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:04:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:04:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:04:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:04:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:04:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:04:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:04:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:04:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:04:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:04:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:04:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:04:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:04:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:04:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:04:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:04:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:04:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:04:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:04:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:04:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:04:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:04:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:04:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:04:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:04:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:04:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:04:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:04:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:04:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:04:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:04:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:04:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:04:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:04:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:04:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:04:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:04:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:04:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:04:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:04:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:04:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:04:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:04:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:04:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:04:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:04:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:04:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:04:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:04:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:04:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:04:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:04:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:04:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:04:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:04:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:04:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:04:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:04:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:04:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:04:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:04:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:04:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:04:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:04:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:04:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:04:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:04:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:04:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:04:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:04:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:04:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:04:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:04:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:04:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:04:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:04:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:04:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:04:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:04:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:04:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:04:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:04:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:05:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:05:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:05:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:05:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:05:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:05:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:05:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:05:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:05:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:05:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:05:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:05:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:05:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:05:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:05:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:05:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:05:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:05:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:05:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:05:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:05:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:05:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:05:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:05:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:05:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:05:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:05:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:05:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:05:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:05:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:05:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:05:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:05:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:05:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:05:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:05:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:05:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:05:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:05:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:05:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:05:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:05:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:05:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:05:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:05:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:05:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:05:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:05:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:05:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:05:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:05:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:05:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:05:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:05:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:05:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:05:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:05:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:05:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:05:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:05:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:05:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:05:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:05:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:05:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:05:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:05:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:05:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:05:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:05:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:05:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:05:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:05:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:05:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:05:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:05:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:05:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:05:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:05:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:05:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:05:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:05:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:05:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:05:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:05:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:05:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:05:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:05:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:05:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:05:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:05:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:05:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:05:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:05:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:05:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:05:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:05:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:05:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:05:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:05:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:05:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:05:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:05:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:05:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:05:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:05:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:05:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:05:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:05:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:05:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:05:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:05:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:05:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:05:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:05:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:05:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:05:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:05:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:05:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:05:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:05:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:05:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:05:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:05:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:05:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:05:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:05:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:05:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:05:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:05:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:05:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:05:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:05:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:05:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:05:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:05:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:05:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:05:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:05:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:05:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:05:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:05:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:05:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:05:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:05:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:05:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:05:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:05:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:05:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:05:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:05:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:05:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:05:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:05:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:05:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:05:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:05:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:05:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:05:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:05:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:05:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:05:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:05:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:05:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:05:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:05:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:05:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:05:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:05:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:05:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:05:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:05:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:05:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:05:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:05:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:05:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:05:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:05:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:05:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:05:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:05:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:05:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:05:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:05:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:05:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:05:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:05:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:05:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:05:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:05:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:05:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:05:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:05:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:05:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:05:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:05:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:05:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:05:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:05:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:05:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:05:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:05:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:05:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:05:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:05:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:05:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:05:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:05:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:05:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:05:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:05:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:05:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:05:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:05:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:05:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:05:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:05:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:06:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:06:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:06:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:06:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:06:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:06:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:06:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:06:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:06:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:06:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:06:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:06:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:06:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:06:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:06:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:06:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:06:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:06:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:06:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:06:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:06:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:06:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:06:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:06:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:06:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:06:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:06:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:06:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:06:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:06:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:06:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:06:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:06:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:06:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:06:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:06:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:06:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:06:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:06:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:06:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:06:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:06:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:06:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:06:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:06:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:06:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:06:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:06:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:06:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:06:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:06:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:06:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:06:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:06:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:06:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:06:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:06:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:06:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:06:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:06:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:06:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:06:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:06:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:06:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:06:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:06:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:06:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:06:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:06:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:06:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:06:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:06:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:06:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:06:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:06:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:06:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:06:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:06:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:06:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:06:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:06:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:06:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:06:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:06:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:06:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:06:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:06:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:06:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:06:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:06:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:06:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:06:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:06:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:06:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:06:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:06:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:06:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:06:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:06:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:06:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:06:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:06:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:06:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:06:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:06:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:06:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:06:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:06:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:06:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:06:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:06:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:06:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:06:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:06:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:06:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:06:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:06:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:06:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:06:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:06:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:06:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:06:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:06:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:06:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:06:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:06:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:06:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:06:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:06:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:06:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:06:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:06:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:06:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:06:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:06:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:06:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:06:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:06:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:06:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:06:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:06:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:06:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:06:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:06:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:06:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:06:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:06:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:06:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:06:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:06:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:06:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:06:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:06:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:06:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:06:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:06:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:06:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:06:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:06:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:06:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:06:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:06:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:06:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:06:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:06:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:06:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:06:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:06:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:06:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:06:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:06:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:06:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:06:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:06:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:06:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:06:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:06:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:06:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:06:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:06:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:06:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:06:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:06:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:06:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:06:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:06:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:06:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:06:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:06:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:06:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:06:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:06:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:06:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:06:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:06:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:06:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:06:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:06:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:06:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:06:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:06:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:06:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:06:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:06:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:06:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:06:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:06:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:06:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:06:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:06:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:06:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:06:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:06:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:06:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:06:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:06:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:07:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:07:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:07:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:07:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:07:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:07:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:07:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:07:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:07:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:07:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:07:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:07:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:07:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:07:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:07:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:07:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:07:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:07:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:09:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:09:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:09:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:09:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:09:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:09:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:09:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:09:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:09:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:09:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:09:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:09:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:09:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:09:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:09:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:09:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:09:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:09:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:09:09 rdma06 rsyslogd[1041]: imjournal: 5086 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 22:09:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:09:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:09:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:09:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:09:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:09:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:09:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:09:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:09:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:09:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:09:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:09:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:09:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:09:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:09:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:09:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:09:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:09:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:09:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:09:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:09:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:09:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:09:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:09:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:09:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:09:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:09:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:09:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:09:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:09:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:09:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:09:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:09:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:09:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:09:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:09:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:09:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:09:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:09:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:09:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:09:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:09:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:09:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:09:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:09:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:09:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:09:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:09:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:09:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:09:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:09:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:09:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:09:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:09:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:09:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:09:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:09:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:09:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:09:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:09:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:09:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:09:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:09:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:09:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:09:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:09:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:09:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:09:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:09:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:09:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:09:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:09:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:09:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:09:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:09:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:09:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:09:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:09:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:09:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:09:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:09:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:09:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:09:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:09:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:09:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:09:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:09:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:09:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:09:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:09:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:09:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:09:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:09:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:09:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:09:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:09:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:09:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:09:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:09:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:09:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:09:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:09:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:09:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:09:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:09:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:09:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:09:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:09:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:09:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:09:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:09:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:09:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:09:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:09:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:09:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:09:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:09:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:09:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:09:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:09:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:09:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:09:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:09:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:09:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:09:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:09:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:09:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:09:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:09:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:09:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:09:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:09:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:09:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:09:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:09:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:09:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:09:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:09:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:09:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:09:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:09:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:09:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:09:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:09:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:09:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:09:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:09:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:09:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:09:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:09:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:09:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:09:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:09:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:09:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:09:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:09:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:09:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:09:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:09:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:09:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:09:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:09:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:09:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:09:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:09:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:09:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:09:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:09:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:09:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:09:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:09:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:09:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:09:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:09:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:09:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:09:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:09:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:09:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:09:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:09:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:10:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:10:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:10:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:10:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:10:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:10:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:10:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:10:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:10:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:10:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:10:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:10:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:10:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:10:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:10:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:10:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:10:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:10:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:10:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:10:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:10:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:10:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:10:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:10:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:10:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:10:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:10:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:10:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:10:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:10:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:10:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:10:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:10:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:10:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:10:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:10:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:10:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:10:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:10:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:10:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:10:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:10:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:10:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:10:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:10:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:10:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:10:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:10:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:10:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:10:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:10:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:10:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:10:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:10:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:10:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:10:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:10:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:10:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:10:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:10:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:10:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:10:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:10:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:10:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:10:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:10:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:10:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:10:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:10:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:10:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:10:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:10:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:10:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:10:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:10:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:10:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:10:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:10:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:10:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:10:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:10:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:10:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:10:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:10:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:10:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:10:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:10:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:10:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:10:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:10:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:10:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:10:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:10:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:10:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:10:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:10:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:10:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:10:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:10:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:10:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:10:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:10:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:10:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:10:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:10:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:10:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:10:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:10:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:10:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:10:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:10:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:10:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:10:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:10:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:10:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:10:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:10:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:10:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:10:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:10:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:10:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:10:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:10:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:10:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:10:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:10:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:10:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:10:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:10:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:10:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:10:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:10:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:10:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:10:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:10:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:10:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:10:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:10:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:10:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:10:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:10:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:10:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:10:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:10:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:10:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:10:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:10:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:10:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:10:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:10:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:10:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:10:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:10:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:10:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:10:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:10:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:10:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:10:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:10:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:10:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:10:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:10:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:10:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:10:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:10:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:10:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:10:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:10:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:10:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:10:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:10:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:10:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:10:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:10:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:10:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:10:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:10:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:10:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:10:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:10:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:10:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:10:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:10:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:10:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:10:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:10:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:10:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:10:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:10:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:10:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:10:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:10:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:10:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:10:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:10:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:10:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:10:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:10:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:10:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:10:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:10:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:10:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:10:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:10:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:10:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:10:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:10:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:10:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:10:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:10:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:10:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:10:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:10:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:10:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:10:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:10:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:11:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:11:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:11:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:11:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:11:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:11:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:11:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:11:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:11:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:11:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:11:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:11:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:11:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:11:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:11:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:11:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:11:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:11:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:11:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:11:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:11:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:11:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:11:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:11:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:11:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:11:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:11:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:11:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:11:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:11:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:11:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:11:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:11:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:11:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:11:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:11:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:11:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:11:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:11:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:11:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:11:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:11:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:11:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:11:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:11:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:11:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:11:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:11:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:11:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:11:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:11:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:11:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:11:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:11:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:11:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:11:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:11:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:11:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:11:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:11:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:11:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:11:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:11:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:11:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:11:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:11:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:11:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:11:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:11:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:11:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:11:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:11:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:11:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:11:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:11:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:11:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:11:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:11:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:11:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:11:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:11:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:11:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:11:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:11:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:11:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:11:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:11:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:11:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:11:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:11:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:11:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:11:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:11:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:11:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:11:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:11:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:11:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:11:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:11:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:11:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:11:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:11:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:11:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:11:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:11:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:11:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:11:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:11:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:11:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:11:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:11:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:11:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:11:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:11:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:11:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:11:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:11:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:11:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:11:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:11:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:11:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:11:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:11:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:11:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:11:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:11:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:11:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:11:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:11:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:11:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:11:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:11:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:11:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:11:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:11:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:11:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:11:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:11:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:11:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:11:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:11:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:11:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:11:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:11:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:11:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:11:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:11:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:11:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:11:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:11:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:11:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:11:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:11:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:11:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:11:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:11:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:11:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:11:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:11:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:11:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:11:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:11:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:11:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:11:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:11:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:11:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:11:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:11:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:11:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:11:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:11:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:11:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:11:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:11:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:11:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:11:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:11:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:11:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:11:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:11:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:11:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:11:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:11:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:11:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:11:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:11:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:11:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:11:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:11:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:11:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:11:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:11:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:11:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:11:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:11:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:11:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:11:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:11:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:11:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:11:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:11:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:11:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:11:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:11:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:11:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:11:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:11:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:11:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:11:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:11:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:11:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:11:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:11:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:11:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:11:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:11:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:12:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:12:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:12:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:12:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:12:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:12:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:12:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:12:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:12:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:12:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:12:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:12:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:12:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:12:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:12:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:12:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:12:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:12:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:12:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:12:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:12:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:12:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:12:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:12:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:12:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:12:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:12:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:12:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:12:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:12:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:12:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:12:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:12:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:12:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:12:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:12:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:12:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:12:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:12:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:12:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:12:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:12:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:12:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:12:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:12:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:12:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:12:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:12:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:12:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:12:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:12:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:12:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:12:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:12:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:12:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:12:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:12:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:12:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:12:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:12:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:12:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:12:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:12:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:12:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:12:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:12:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:12:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:12:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:12:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:12:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:12:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:12:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:12:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:12:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:12:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:12:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:12:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:12:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:12:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:12:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:12:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:12:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:12:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:12:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:12:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:12:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:12:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:12:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:12:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:12:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:12:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:12:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:12:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:12:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:12:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:12:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:12:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:12:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:12:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:12:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:12:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:12:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:12:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:12:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:12:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:12:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:12:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:12:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:12:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:12:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:12:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:12:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:12:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:12:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:12:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:12:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:12:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:12:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:12:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:12:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:12:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:12:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:12:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:12:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:12:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:12:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:12:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:12:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:12:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:12:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:12:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:12:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:12:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:12:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:12:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:12:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:12:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:12:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:12:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:12:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:12:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:12:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:12:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:12:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:12:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:12:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:12:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:12:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:12:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:12:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:12:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:12:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:12:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:12:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:12:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:12:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:12:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:12:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:12:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:12:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:12:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:12:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:12:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:12:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:12:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:12:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:12:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:12:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:12:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:12:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:12:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:12:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:12:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:12:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:12:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:12:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:12:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:12:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:12:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:12:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:12:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:12:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:12:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:12:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:12:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:12:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:12:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:12:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:12:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:12:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:12:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:12:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:12:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:12:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:12:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:12:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:12:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:12:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:12:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:12:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:12:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:12:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:12:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:12:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:12:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:12:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:12:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:12:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:12:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:12:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:12:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:12:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:12:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:12:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:12:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:12:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:13:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:13:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:13:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:13:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:13:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:13:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:13:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:13:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:13:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:13:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:13:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:13:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:13:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:13:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:13:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:13:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:13:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:13:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:13:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:13:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:13:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:13:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:13:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:13:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:13:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:13:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:13:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:13:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:13:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:13:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:13:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:13:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:13:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:13:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:13:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:13:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:13:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:13:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:13:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:13:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:13:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:13:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:13:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:13:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:13:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:13:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:13:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:13:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:13:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:13:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:13:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:13:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:13:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:13:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:13:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:13:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:13:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:13:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:13:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:13:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:13:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:13:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:13:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:13:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:13:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:13:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:13:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:13:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:13:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:13:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:13:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:13:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:13:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:13:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:13:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:13:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:13:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:13:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:13:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:13:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:13:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:13:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:13:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:13:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:13:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:13:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:13:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:13:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:13:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:13:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:13:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:13:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:13:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:13:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:13:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:13:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:13:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:13:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:13:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:13:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:13:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:13:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:13:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:13:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:13:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:13:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:13:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:13:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:13:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:13:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:13:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:13:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:13:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:13:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:13:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:13:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:13:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:13:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:13:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:13:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:13:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:13:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:13:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:13:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:13:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:13:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:13:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:13:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:13:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:13:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:13:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:13:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:13:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:13:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:13:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:13:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:13:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:13:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:13:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:13:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:13:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:13:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:13:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:13:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:13:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:13:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:13:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:13:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:13:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:13:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:13:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:13:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:13:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:13:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:13:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:13:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:13:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:13:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:13:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:13:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:13:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:13:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:13:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:13:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:13:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:13:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:13:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:13:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:13:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:13:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:13:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:13:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:13:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:13:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:13:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:13:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:13:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:13:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:13:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:13:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:13:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:13:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:13:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:13:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:13:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:13:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:13:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:13:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:13:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:13:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:13:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:13:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:13:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:13:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:13:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:13:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:13:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:13:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:13:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:13:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:13:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:13:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:13:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:13:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:13:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:13:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:13:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:13:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:13:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:13:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:13:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:13:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:13:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:13:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:13:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:13:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:14:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:14:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:14:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:14:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:14:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:14:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:14:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:14:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:14:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:14:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:14:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:14:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:14:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:14:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:14:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:14:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:14:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:14:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:14:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:14:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:14:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:14:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:14:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:14:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:14:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:14:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:14:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:14:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:14:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:14:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:14:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:14:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:14:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:14:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:14:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:14:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:14:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:14:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:14:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:14:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:14:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:14:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:14:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:14:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:14:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:14:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:14:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:14:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:14:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:14:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:14:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:14:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:14:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:14:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:14:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:14:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:14:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:14:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:14:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:14:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:14:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:14:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:14:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:14:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:14:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:14:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:14:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:14:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:14:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:14:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:14:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:14:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:14:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:14:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:14:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:14:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:14:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:14:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:14:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:14:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:14:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:14:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:14:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:14:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:14:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:14:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:14:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:14:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:14:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:14:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:14:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:14:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:14:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:14:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:14:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:14:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:14:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:14:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:14:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:14:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:14:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:14:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:14:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:14:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:14:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:14:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:14:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:14:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:14:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:14:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:14:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:14:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:14:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:14:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:14:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:14:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:14:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:14:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:14:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:14:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:14:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:14:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:14:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:14:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:14:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:14:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:14:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:14:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:14:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:14:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:14:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:14:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:14:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:14:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:14:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:14:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:14:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:14:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:14:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:14:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:14:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:14:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:14:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:14:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:14:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:14:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:14:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:14:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:14:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:14:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:14:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:14:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:14:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:14:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:14:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:14:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:14:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:14:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:14:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:14:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:14:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:14:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:14:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:14:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:14:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:14:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:14:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:14:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:14:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:14:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:14:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:14:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:14:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:14:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:14:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:14:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:14:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:14:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:14:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:14:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:14:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:14:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:14:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:14:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:14:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:14:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:14:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:14:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:14:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:14:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:14:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:14:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:14:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:14:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:14:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:14:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:14:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:14:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:14:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:14:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:14:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:14:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:14:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:14:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:14:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:14:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:14:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:14:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:14:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:14:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:14:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:14:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:14:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:14:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:14:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:14:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:15:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:15:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:15:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:15:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:15:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:15:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:15:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:15:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:15:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:15:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:15:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:15:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:15:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:15:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:15:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:15:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:15:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:15:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:15:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:15:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:15:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:15:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:15:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:15:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:15:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:15:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:15:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:15:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:15:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:15:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:15:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:15:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:15:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:15:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:15:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:15:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:15:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:15:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:15:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:15:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:15:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:15:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:15:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:15:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:15:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:15:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:15:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:15:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:15:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:15:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:15:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:15:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:15:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:15:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:15:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:15:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:15:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:15:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:15:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:15:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:15:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:15:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:15:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:15:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:15:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:15:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:15:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:15:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:15:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:15:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:15:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:15:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:15:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:15:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:15:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:15:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:15:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:15:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:15:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:15:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:15:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:15:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:15:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:15:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:15:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:15:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:15:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:15:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:15:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:15:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:15:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:15:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:15:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:15:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:15:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:15:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:15:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:15:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:15:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:15:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:15:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:15:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:15:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:15:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:15:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:15:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:15:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:15:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:15:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:15:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:15:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:15:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:15:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:15:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:15:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:15:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:15:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:15:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:15:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:15:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:15:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:15:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:15:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:15:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:15:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:15:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:15:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:15:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:15:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:15:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:15:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:15:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:15:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:15:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:15:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:15:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:15:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:15:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:15:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:15:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:15:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:15:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:15:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:15:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:15:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:15:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:15:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:15:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:15:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:15:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:15:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:15:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:15:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:15:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:15:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:15:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:15:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:15:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:15:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:15:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:15:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:15:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:15:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:15:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:15:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:15:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:15:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:15:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:15:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:15:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:15:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:15:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:15:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:15:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:15:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:15:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:15:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:15:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:15:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:15:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:15:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:15:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:15:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:15:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:15:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:15:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:15:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:15:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:15:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:15:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:15:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:15:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:15:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:15:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:15:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:15:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:15:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:15:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:15:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:15:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:15:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:15:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:15:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:15:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:15:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:15:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:15:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:15:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:15:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:15:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:15:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:15:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:15:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:15:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:15:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:15:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:16:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:16:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:16:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:16:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:16:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:16:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:16:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:16:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:16:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:16:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:16:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:16:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:16:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:16:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:16:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:16:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:16:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:16:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:16:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:16:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:16:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:16:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:16:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:16:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:16:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:16:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:16:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:16:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:16:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:16:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:16:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:16:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:16:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:16:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:16:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:16:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:16:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:16:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:16:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:16:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:16:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:16:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:16:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:16:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:16:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:16:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:16:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:16:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:16:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:16:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:16:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:16:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:16:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:16:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:16:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:16:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:16:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:16:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:16:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:16:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:16:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:16:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:16:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:16:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:16:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:16:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:16:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:16:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:16:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:16:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:16:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:16:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:16:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:16:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:16:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:16:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:16:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:16:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:16:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:16:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:16:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:16:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:16:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:16:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:16:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:16:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:16:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:16:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:16:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:16:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:16:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:16:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:16:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:16:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:16:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:16:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:16:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:16:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:16:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:16:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:16:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:16:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:16:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:16:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:16:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:16:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:16:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:16:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:16:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:16:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:16:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:16:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:16:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:16:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:16:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:16:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:16:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:16:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:16:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:16:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:16:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:16:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:16:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:16:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:16:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:16:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:16:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:16:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:16:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:16:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:16:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:16:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:16:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:16:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:16:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:16:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:16:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:16:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:16:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:16:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:16:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:16:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:16:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:16:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:16:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:16:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:16:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:16:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:16:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:16:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:16:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:16:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:16:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:16:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:16:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:16:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:16:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:16:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:16:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:16:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:16:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:16:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:16:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:16:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:16:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:16:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:16:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:16:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:16:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:16:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:16:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:16:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:16:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:16:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:16:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:16:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:16:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:16:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:16:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:16:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:16:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:16:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:16:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:16:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:16:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:16:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:16:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:16:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:16:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:16:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:16:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:16:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:16:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:16:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:16:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:16:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:16:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:16:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:16:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:16:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:16:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:16:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:16:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:16:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:16:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:16:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:16:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:16:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:16:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:16:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:16:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:16:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:16:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:16:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:16:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:16:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:17:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:17:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:17:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:17:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:17:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:17:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:17:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:17:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:17:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:17:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:17:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:17:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:17:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:17:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:17:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:17:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:17:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:17:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:19:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:19:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:19:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:19:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:19:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:19:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:19:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:19:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:19:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:19:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:19:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:19:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:19:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:19:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:19:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:19:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:19:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:19:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:19:14 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 22:19:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:19:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:19:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:19:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:19:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:19:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:19:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:19:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:19:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:19:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:19:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:19:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:19:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:19:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:19:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:19:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:19:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:19:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:19:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:19:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:19:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:19:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:19:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:19:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:19:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:19:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:19:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:19:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:19:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:19:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:19:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:19:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:19:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:19:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:19:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:19:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:19:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:19:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:19:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:19:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:19:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:19:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:19:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:19:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:19:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:19:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:19:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:19:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:19:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:19:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:19:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:19:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:19:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:19:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:19:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:19:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:19:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:19:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:19:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:19:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:19:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:19:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:19:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:19:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:19:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:19:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:19:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:19:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:19:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:19:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:19:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:19:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:19:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:19:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:19:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:19:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:19:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:19:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:19:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:19:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:19:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:19:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:19:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:19:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:19:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:19:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:19:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:19:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:19:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:19:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:19:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:19:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:19:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:19:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:19:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:19:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:19:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:19:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:19:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:19:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:19:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:19:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:19:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:19:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:19:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:19:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:19:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:19:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:19:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:19:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:19:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:19:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:19:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:19:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:19:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:19:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:19:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:19:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:19:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:19:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:19:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:19:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:19:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:19:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:19:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:19:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:19:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:19:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:19:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:19:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:19:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:19:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:19:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:19:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:19:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:19:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:19:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:19:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:19:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:19:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:19:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:19:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:19:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:19:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:19:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:19:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:19:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:19:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:19:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:19:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:19:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:19:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:19:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:19:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:19:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:19:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:19:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:19:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:19:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:19:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:19:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:19:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:20:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:20:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:20:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:20:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:20:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:20:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:20:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:20:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:20:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:20:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:20:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:20:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:20:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:20:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:20:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:20:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:20:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:20:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:20:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:20:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:20:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:20:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:20:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:20:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:20:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:20:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:20:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:20:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:20:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:20:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:20:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:20:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:20:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:20:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:20:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:20:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:20:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:20:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:20:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:20:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:20:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:20:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:20:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:20:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:20:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:20:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:20:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:20:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:20:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:20:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:20:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:20:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:20:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:20:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:20:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:20:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:20:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:20:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:20:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:20:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:20:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:20:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:20:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:20:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:20:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:20:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:20:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:20:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:20:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:20:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:20:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:20:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:20:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:20:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:20:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:20:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:20:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:20:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:20:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:20:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:20:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:20:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:20:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:20:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:20:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:20:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:20:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:20:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:20:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:20:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:20:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:20:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:20:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:20:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:20:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:20:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:20:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:20:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:20:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:20:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:20:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:20:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:20:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:20:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:20:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:20:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:20:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:20:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:20:32 rdma06 NetworkManager[1025]: <info>  [1681870832.8406] dhcp4 (eno1): state changed new lease, address=10.73.131.45
Apr 18 22:20:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:20:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:20:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:20:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:20:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:20:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:20:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:20:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:20:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:20:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:20:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:20:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:20:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:20:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:20:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:20:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:20:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:20:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:20:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:20:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:20:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:20:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:20:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:20:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:20:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:20:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:20:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:20:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:20:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:20:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:20:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:20:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:20:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:20:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:20:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:20:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:20:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:20:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:20:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:20:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:20:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:20:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:20:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:20:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:20:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:20:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:20:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:20:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:20:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:20:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:20:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:20:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:20:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:20:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:20:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:20:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:20:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:20:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:20:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:20:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:20:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:20:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:20:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:20:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:20:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:20:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:20:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:20:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:20:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:20:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:20:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:20:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:20:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:20:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:20:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:20:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:20:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:20:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:20:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:20:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:20:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:20:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:20:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:20:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:20:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:20:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:20:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:20:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:20:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:20:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:20:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:20:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:20:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:20:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:20:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:20:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:20:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:20:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:20:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:20:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:20:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:20:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:20:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:20:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:20:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:20:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:20:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:20:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:21:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:21:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:21:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:21:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:21:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:21:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:21:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:21:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:21:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:21:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:21:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:21:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:21:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:21:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:21:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:21:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:21:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:21:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:21:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:21:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:21:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:21:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:21:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:21:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:21:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:21:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:21:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:21:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:21:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:21:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:21:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:21:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:21:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:21:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:21:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:21:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:21:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:21:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:21:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:21:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:21:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:21:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:21:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:21:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:21:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:21:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:21:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:21:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:21:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:21:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:21:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:21:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:21:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:21:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:21:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:21:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:21:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:21:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:21:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:21:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:21:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:21:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:21:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:21:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:21:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:21:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:21:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:21:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:21:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:21:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:21:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:21:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:21:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:21:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:21:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:21:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:21:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:21:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:21:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:21:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:21:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:21:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:21:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:21:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:21:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:21:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:21:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:21:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:21:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:21:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:21:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:21:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:21:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:21:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:21:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:21:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:21:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:21:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:21:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:21:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:21:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:21:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:21:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:21:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:21:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:21:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:21:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:21:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:21:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:21:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:21:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:21:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:21:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:21:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:21:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:21:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:22:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:22:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:22:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:22:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:22:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:22:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:22:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:22:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:22:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:22:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:22:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:22:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:22:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:22:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:22:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:22:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:22:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:22:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:22:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:22:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:22:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:22:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:22:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:22:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:22:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:22:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:22:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:22:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:22:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:22:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:22:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:22:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:22:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:22:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:22:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:22:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:22:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:22:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:22:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:22:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:22:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:22:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:22:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:22:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:22:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:22:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:22:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:22:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:22:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:22:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:22:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:22:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:22:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:22:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:22:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:22:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:22:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:22:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:22:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:22:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:22:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:22:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:22:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:22:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:22:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:22:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:22:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:22:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:22:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:22:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:22:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:22:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:22:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:22:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:22:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:22:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:22:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:22:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:22:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:22:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:22:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:22:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:22:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:22:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:22:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:22:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:22:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:22:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:22:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:22:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:22:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:22:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:22:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:22:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:22:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:22:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:22:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:22:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:22:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:22:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:22:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:22:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:22:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:22:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:22:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:22:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:22:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:22:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:22:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:22:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:22:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:22:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:22:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:22:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:22:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:22:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:22:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:22:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:22:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:22:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:22:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:22:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:22:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:22:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:22:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:22:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:22:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:22:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:22:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:22:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:22:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:22:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:22:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:22:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:22:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:22:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:22:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:22:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:22:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:22:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:22:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:22:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:22:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:22:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:22:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:22:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:22:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:22:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:22:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:22:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:22:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:22:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:22:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:22:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:22:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:22:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:22:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:22:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:22:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:22:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:22:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:22:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:22:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:22:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:22:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:22:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:22:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:22:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:22:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:22:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:22:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:22:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:22:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:22:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:22:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:22:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:22:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:22:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:22:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:22:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:22:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:22:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:22:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:22:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:22:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:22:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:22:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:22:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:22:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:22:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:22:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:22:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:22:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:22:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:22:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:22:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:22:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:22:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:22:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:22:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:22:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:22:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:22:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:22:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:22:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:22:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:22:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:22:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:22:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:22:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:22:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:22:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:22:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:22:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:22:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:22:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:23:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:23:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:23:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:23:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:23:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:23:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:23:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:23:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:23:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:23:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:23:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:23:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:23:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:23:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:23:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:23:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:23:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:23:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:23:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:23:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:23:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:23:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:23:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:23:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:23:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:23:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:23:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:23:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:23:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:23:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:23:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:23:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:23:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:23:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:23:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:23:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:23:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:23:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:23:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:23:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:23:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:23:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:23:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:23:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:23:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:23:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:23:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:23:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:23:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:23:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:23:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:23:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:23:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:23:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:23:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:23:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:23:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:23:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:23:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:23:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:23:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:23:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:23:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:23:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:23:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:23:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:23:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:23:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:23:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:23:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:23:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:23:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:23:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:23:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:23:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:23:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:23:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:23:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:23:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:23:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:23:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:23:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:23:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:23:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:23:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:23:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:23:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:23:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:23:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:23:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:23:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:23:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:23:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:23:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:23:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:23:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:23:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:23:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:23:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:23:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:23:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:23:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:23:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:23:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:23:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:23:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:23:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:23:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:23:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:23:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:23:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:23:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:23:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:23:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:23:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:23:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:23:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:23:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:23:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:23:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:23:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:23:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:23:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:23:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:23:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:23:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:23:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:23:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:23:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:23:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:23:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:23:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:23:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:23:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:23:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:23:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:23:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:23:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:23:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:23:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:23:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:23:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:23:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:23:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:23:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:23:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:23:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:23:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:23:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:23:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:23:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:23:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:23:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:23:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:23:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:23:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:23:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:23:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:23:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:23:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:23:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:23:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:23:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:23:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:23:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:23:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:23:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:23:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:23:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:23:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:23:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:23:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:23:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:23:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:23:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:23:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:23:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:23:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:23:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:23:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:23:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:23:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:23:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:23:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:23:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:23:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:23:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:23:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:23:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:23:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:23:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:23:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:23:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:23:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:23:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:23:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:23:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:23:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:23:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:23:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:23:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:23:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:23:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:23:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:23:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:23:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:23:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:23:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:23:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:23:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:23:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:23:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:23:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:23:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:23:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:23:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:24:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:24:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:24:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:24:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:24:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:24:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:24:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:24:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:24:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:24:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:24:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:24:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:24:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:24:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:24:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:24:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:24:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:24:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:24:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:24:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:24:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:24:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:24:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:24:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:24:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:24:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:24:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:24:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:24:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:24:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:24:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:24:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:24:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:24:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:24:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:24:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:24:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:24:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:24:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:24:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:24:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:24:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:24:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:24:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:24:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:24:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:24:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:24:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:24:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:24:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:24:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:24:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:24:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:24:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:24:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:24:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:24:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:24:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:24:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:24:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:24:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:24:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:24:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:24:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:24:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:24:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:24:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:24:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:24:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:24:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:24:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:24:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:24:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:24:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:24:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:24:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:24:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:24:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:24:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:24:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:24:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:24:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:24:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:24:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:24:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:24:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:24:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:24:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:24:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:24:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:24:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:24:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:24:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:24:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:24:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:24:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:24:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:24:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:24:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:24:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:24:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:24:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:24:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:24:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:24:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:24:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:24:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:24:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:24:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:24:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:24:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:24:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:24:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:24:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:24:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:24:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:24:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:24:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:24:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:24:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:24:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:24:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:24:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:24:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:24:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:24:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:24:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:24:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:24:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:24:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:24:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:24:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:24:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:24:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:24:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:24:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:24:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:24:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:24:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:24:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:24:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:24:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:24:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:24:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:24:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:24:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:24:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:24:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:24:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:24:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:24:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:24:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:24:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:24:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:24:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:24:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:24:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:24:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:24:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:24:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:24:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:24:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:24:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:24:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:24:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:24:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:24:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:24:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:24:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:24:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:24:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:24:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:24:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:24:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:24:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:24:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:24:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:24:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:24:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:24:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:24:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:24:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:24:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:24:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:24:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:24:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:24:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:24:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:24:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:24:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:24:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:24:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:24:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:24:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:24:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:24:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:24:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:24:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:24:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:24:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:24:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:24:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:24:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:24:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:24:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:24:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:24:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:24:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:24:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:24:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:24:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:24:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:24:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:24:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:24:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:24:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:25:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:25:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:25:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:25:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:25:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:25:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:25:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:25:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:25:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:25:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:25:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:25:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:25:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:25:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:25:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:25:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:25:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:25:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:25:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:25:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:25:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:25:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:25:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:25:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:25:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:25:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:25:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:25:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:25:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:25:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:25:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:25:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:25:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:25:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:25:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:25:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:25:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:25:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:25:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:25:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:25:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:25:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:25:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:25:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:25:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:25:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:25:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:25:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:25:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:25:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:25:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:25:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:25:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:25:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:25:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:25:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:25:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:25:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:25:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:25:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:25:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:25:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:25:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:25:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:25:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:25:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:25:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:25:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:25:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:25:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:25:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:25:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:25:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:25:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:25:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:25:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:25:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:25:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:25:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:25:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:25:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:25:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:25:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:25:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:25:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:25:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:25:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:25:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:25:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:25:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:25:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:25:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:25:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:25:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:25:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:25:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:25:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:25:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:25:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:25:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:25:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:25:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:25:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:25:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:25:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:25:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:25:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:25:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:25:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:25:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:25:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:25:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:25:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:25:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:25:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:25:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:25:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:25:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:25:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:25:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:25:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:25:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:25:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:25:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:25:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:25:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:25:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:25:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:25:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:25:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:25:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:25:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:25:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:25:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:25:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:25:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:25:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:25:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:25:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:25:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:25:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:25:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:25:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:25:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:25:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:25:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:25:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:25:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:25:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:25:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:25:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:25:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:25:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:25:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:25:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:25:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:25:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:25:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:25:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:25:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:25:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:25:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:25:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:25:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:25:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:25:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:25:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:25:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:25:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:25:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:25:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:25:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:25:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:25:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:25:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:25:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:25:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:25:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:25:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:25:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:25:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:25:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:25:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:25:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:25:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:25:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:25:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:25:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:25:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:25:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:25:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:25:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:25:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:25:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:25:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:25:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:25:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:25:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:25:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:25:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:25:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:25:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:25:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:25:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:25:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:25:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:25:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:25:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:25:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:25:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:25:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:25:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:25:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:25:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:25:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:25:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:26:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:26:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:26:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:26:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:26:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:26:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:26:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:26:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:26:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:26:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:26:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:26:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:26:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:26:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:26:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:26:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:26:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:26:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:26:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:26:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:26:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:26:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:26:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:26:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:26:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:26:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:26:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:26:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:26:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:26:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:26:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:26:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:26:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:26:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:26:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:26:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:26:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:26:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:26:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:26:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:26:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:26:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:26:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:26:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:26:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:26:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:26:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:26:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:26:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:26:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:26:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:26:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:26:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:26:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:26:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:26:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:26:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:26:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:26:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:26:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:26:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:26:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:26:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:26:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:26:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:26:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:26:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:26:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:26:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:26:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:26:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:26:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:26:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:26:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:26:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:26:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:26:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:26:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:26:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:26:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:26:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:26:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:26:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:26:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:26:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:26:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:26:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:26:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:26:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:26:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:26:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:26:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:26:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:26:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:26:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:26:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:26:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:26:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:26:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:26:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:26:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:26:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:26:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:26:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:26:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:26:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:26:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:26:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:26:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:26:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:26:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:26:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:26:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:26:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:26:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:26:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:26:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:26:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:26:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:26:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:26:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:26:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:26:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:26:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:26:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:26:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:26:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:26:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:26:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:26:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:26:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:26:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:26:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:26:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:26:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:26:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:26:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:26:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:26:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:26:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:26:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:26:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:26:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:26:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:26:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:26:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:26:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:26:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:26:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:26:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:26:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:26:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:26:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:26:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:26:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:26:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:26:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:26:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:26:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:26:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:26:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:26:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:26:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:26:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:26:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:26:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:26:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:26:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:26:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:26:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:26:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:26:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:26:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:26:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:26:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:26:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:26:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:26:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:26:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:26:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:26:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:26:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:26:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:26:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:26:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:26:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:26:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:26:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:26:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:26:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:26:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:26:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:26:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:26:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:26:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:26:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:26:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:26:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:26:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:26:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:26:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:26:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:26:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:26:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:26:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:26:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:26:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:26:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:26:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:26:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:26:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:26:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:26:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:26:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:26:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:26:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:27:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:27:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:27:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:27:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:27:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:27:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:27:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:27:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:27:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:27:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:27:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:27:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:27:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:27:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:27:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:27:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:27:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:27:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:27:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:27:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:27:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:27:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:27:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:27:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:27:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:27:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:27:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:27:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:27:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:27:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:27:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:27:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:27:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:27:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:27:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:27:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:29:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:29:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:29:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:29:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:29:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:29:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:29:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:29:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:29:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:29:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:29:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:29:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:29:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:29:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:29:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:29:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:29:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:29:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:29:19 rdma06 rsyslogd[1041]: imjournal: 5292 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 22:29:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:29:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:29:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:29:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:29:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:29:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:29:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:29:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:29:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:29:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:29:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:29:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:29:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:29:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:29:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:29:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:29:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:29:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:29:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:29:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:29:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:29:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:29:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:29:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:29:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:29:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:29:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:29:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:29:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:29:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:29:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:29:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:29:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:29:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:29:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:29:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:29:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:29:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:29:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:29:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:29:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:29:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:29:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:29:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:29:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:29:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:29:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:29:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:29:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:29:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:29:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:29:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:29:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:29:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:29:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:29:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:29:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:29:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:29:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:29:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:29:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:29:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:29:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:29:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:29:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:29:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:29:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:29:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:29:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:29:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:29:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:29:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:29:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:29:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:29:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:29:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:29:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:29:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:29:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:29:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:29:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:29:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:29:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:29:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:29:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:29:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:29:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:29:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:29:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:29:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:29:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:29:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:29:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:29:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:29:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:29:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:29:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:29:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:29:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:29:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:29:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:29:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:29:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:29:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:29:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:29:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:29:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:29:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:29:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:29:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:29:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:29:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:29:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:29:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:29:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:29:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:29:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:29:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:29:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:29:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:29:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:29:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:29:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:29:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:29:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:29:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:29:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:29:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:29:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:29:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:29:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:29:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:29:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:29:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:29:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:29:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:29:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:29:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:29:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:29:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:29:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:29:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:29:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:29:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:30:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:30:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:30:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:30:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:30:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:30:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:30:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:30:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:30:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:30:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:30:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:30:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:30:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:30:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:30:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:30:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:30:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:30:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:30:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:30:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:30:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:30:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:30:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:30:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:30:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:30:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:30:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:30:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:30:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:30:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:30:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:30:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:30:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:30:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:30:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:30:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:30:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:30:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:30:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:30:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:30:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:30:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:30:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:30:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:30:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:30:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:30:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:30:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:30:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:30:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:30:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:30:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:30:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:30:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:30:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:30:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:30:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:30:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:30:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:30:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:30:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:30:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:30:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:30:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:30:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:30:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:30:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:30:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:30:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:30:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:30:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:30:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:30:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:30:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:30:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:30:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:30:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:30:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:30:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:30:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:30:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:30:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:30:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:30:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:30:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:30:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:30:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:30:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:30:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:30:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:30:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:30:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:30:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:30:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:30:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:30:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:30:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:30:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:30:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:30:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:30:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:30:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:30:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:30:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:30:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:30:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:30:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:30:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:30:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:30:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:30:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:30:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:30:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:30:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:30:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:30:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:30:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:30:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:30:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:30:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:30:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:30:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:30:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:30:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:30:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:30:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:30:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:30:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:30:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:30:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:30:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:30:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:30:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:30:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:30:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:30:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:30:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:30:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:30:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:30:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:30:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:30:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:30:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:30:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:30:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:30:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:30:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:30:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:30:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:30:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:30:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:30:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:30:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:30:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:30:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:30:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:30:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:30:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:30:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:30:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:30:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:30:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:30:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:30:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:30:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:30:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:30:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:30:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:30:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:30:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:30:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:30:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:30:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:30:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:30:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:30:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:30:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:30:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:30:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:30:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:30:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:30:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:30:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:30:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:30:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:30:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:30:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:30:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:30:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:30:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:30:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:30:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:30:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:30:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:30:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:30:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:30:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:30:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:30:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:30:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:30:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:30:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:30:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:30:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:30:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:30:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:30:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:30:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:30:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:30:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:30:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:30:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:30:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:30:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:30:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:30:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:31:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:31:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:31:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:31:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:31:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:31:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:31:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:31:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:31:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:31:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:31:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:31:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:31:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:31:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:31:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:31:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:31:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:31:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:31:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:31:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:31:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:31:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:31:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:31:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:31:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:31:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:31:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:31:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:31:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:31:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:31:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:31:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:31:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:31:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:31:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:31:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:31:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:31:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:31:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:31:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:31:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:31:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:31:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:31:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:31:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:31:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:31:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:31:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:31:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:31:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:31:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:31:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:31:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:31:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:31:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:31:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:31:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:31:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:31:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:31:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:31:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:31:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:31:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:31:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:31:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:31:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:31:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:31:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:31:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:31:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:31:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:31:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:31:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:31:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:31:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:31:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:31:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:31:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:31:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:31:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:31:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:31:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:31:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:31:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:31:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:31:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:31:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:31:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:31:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:31:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:31:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:31:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:31:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:31:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:31:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:31:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:31:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:31:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:31:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:31:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:31:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:31:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:31:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:31:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:31:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:31:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:31:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:31:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:31:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:31:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:31:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:31:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:31:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:31:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:31:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:31:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:31:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:31:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:31:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:31:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:31:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:31:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:31:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:31:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:31:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:31:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:31:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:31:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:31:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:31:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:31:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:31:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:31:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:31:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:31:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:31:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:31:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:31:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:31:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:31:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:31:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:31:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:31:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:31:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:31:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:31:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:31:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:31:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:31:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:31:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:31:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:31:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:31:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:31:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:31:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:31:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:31:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:31:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:31:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:31:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:31:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:31:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:31:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:31:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:31:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:31:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:31:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:31:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:31:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:31:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:31:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:31:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:31:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:31:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:31:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:31:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:31:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:31:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:31:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:31:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:31:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:31:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:31:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:31:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:31:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:31:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:31:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:31:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:31:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:31:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:31:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:31:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:31:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:31:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:31:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:31:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:31:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:31:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:31:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:31:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:31:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:31:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:31:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:31:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:31:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:31:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:32:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:32:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:32:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:32:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:32:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:32:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:32:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:32:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:32:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:32:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:32:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:32:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:32:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:32:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:32:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:32:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:32:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:32:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:32:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:32:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:32:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:32:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:32:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:32:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:32:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:32:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:32:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:32:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:32:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:32:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:32:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:32:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:32:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:32:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:32:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:32:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:32:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:32:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:32:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:32:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:32:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:32:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:32:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:32:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:32:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:32:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:32:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:32:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:32:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:32:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:32:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:32:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:32:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:32:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:32:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:32:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:32:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:32:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:32:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:32:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:32:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:32:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:32:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:32:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:32:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:32:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:32:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:32:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:32:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:32:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:32:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:32:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:32:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:32:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:32:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:32:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:32:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:32:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:32:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:32:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:32:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:32:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:32:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:32:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:32:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:32:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:32:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:32:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:32:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:32:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:32:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:32:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:32:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:32:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:32:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:32:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:32:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:32:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:32:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:32:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:32:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:32:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:32:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:32:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:32:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:32:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:32:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:32:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:32:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:32:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:32:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:32:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:32:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:32:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:32:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:32:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:32:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:32:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:32:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:32:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:32:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:32:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:32:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:32:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:32:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:32:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:32:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:32:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:32:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:32:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:32:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:32:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:32:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:32:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:32:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:32:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:32:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:32:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:32:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:32:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:32:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:32:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:32:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:32:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:32:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:32:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:32:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:32:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:32:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:32:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:32:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:32:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:32:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:32:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:32:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:32:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:32:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:32:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:32:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:32:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:32:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:32:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:32:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:32:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:32:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:32:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:32:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:32:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:32:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:32:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:32:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:32:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:32:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:32:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:32:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:32:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:32:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:32:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:32:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:32:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:32:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:32:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:32:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:32:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:32:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:32:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:32:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:32:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:32:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:32:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:32:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:32:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:32:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:32:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:32:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:32:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:32:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:32:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:32:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:32:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:32:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:32:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:32:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:32:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:32:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:32:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:32:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:32:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:32:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:32:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:32:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:32:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:32:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:32:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:32:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:32:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:33:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:33:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:33:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:33:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:33:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:33:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:33:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:33:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:33:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:33:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:33:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:33:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:33:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:33:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:33:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:33:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:33:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:33:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:33:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:33:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:33:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:33:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:33:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:33:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:33:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:33:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:33:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:33:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:33:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:33:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:33:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:33:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:33:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:33:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:33:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:33:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:33:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:33:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:33:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:33:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:33:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:33:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:33:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:33:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:33:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:33:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:33:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:33:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:33:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:33:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:33:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:33:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:33:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:33:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:33:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:33:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:33:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:33:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:33:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:33:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:33:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:33:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:33:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:33:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:33:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:33:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:33:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:33:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:33:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:33:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:33:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:33:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:33:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:33:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:33:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:33:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:33:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:33:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:33:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:33:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:33:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:33:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:33:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:33:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:33:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:33:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:33:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:33:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:33:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:33:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:33:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:33:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:33:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:33:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:33:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:33:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:33:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:33:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:33:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:33:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:33:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:33:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:33:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:33:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:33:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:33:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:33:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:33:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:33:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:33:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:33:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:33:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:33:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:33:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:33:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:33:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:33:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:33:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:33:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:33:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:33:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:33:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:33:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:33:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:33:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:33:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:33:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:33:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:33:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:33:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:33:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:33:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:33:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:33:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:33:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:33:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:33:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:33:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:33:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:33:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:33:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:33:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:33:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:33:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:33:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:33:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:33:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:33:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:33:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:33:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:33:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:33:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:33:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:33:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:33:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:33:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:33:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:33:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:33:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:33:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:33:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:33:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:33:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:33:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:33:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:33:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:33:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:33:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:33:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:33:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:33:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:33:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:33:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:33:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:33:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:33:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:33:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:33:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:33:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:33:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:33:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:33:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:33:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:33:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:33:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:33:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:33:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:33:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:33:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:33:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:33:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:33:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:33:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:33:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:33:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:33:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:33:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:33:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:33:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:33:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:33:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:33:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:33:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:33:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:33:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:33:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:33:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:33:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:33:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:33:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:33:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:33:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:33:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:33:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:33:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:33:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:34:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:34:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:34:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:34:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:34:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:34:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:34:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:34:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:34:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:34:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:34:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:34:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:34:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:34:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:34:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:34:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:34:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:34:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:34:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:34:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:34:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:34:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:34:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:34:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:34:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:34:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:34:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:34:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:34:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:34:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:34:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:34:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:34:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:34:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:34:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:34:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:34:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:34:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:34:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:34:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:34:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:34:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:34:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:34:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:34:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:34:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:34:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:34:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:34:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:34:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:34:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:34:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:34:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:34:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:34:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:34:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:34:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:34:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:34:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:34:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:34:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:34:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:34:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:34:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:34:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:34:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:34:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:34:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:34:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:34:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:34:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:34:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:34:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:34:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:34:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:34:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:34:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:34:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:34:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:34:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:34:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:34:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:34:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:34:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:34:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:34:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:34:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:34:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:34:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:34:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:34:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:34:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:34:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:34:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:34:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:34:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:34:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:34:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:34:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:34:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:34:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:34:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:34:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:34:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:34:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:34:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:34:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:34:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:34:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:34:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:34:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:34:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:34:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:34:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:34:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:34:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:34:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:34:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:34:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:34:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:34:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:34:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:34:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:34:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:34:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:34:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:34:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:34:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:34:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:34:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:34:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:34:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:34:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:34:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:34:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:34:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:34:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:34:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:34:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:34:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:34:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:34:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:34:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:34:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:34:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:34:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:34:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:34:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:34:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:34:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:34:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:34:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:34:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:34:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:34:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:34:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:34:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:34:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:34:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:34:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:34:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:34:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:34:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:34:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:34:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:34:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:34:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:34:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:34:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:34:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:34:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:34:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:34:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:34:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:34:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:34:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:34:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:34:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:34:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:34:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:34:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:34:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:34:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:34:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:34:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:34:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:34:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:34:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:34:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:34:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:34:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:34:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:34:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:34:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:34:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:34:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:34:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:34:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:34:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:34:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:34:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:34:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:34:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:34:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:34:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:34:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:34:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:34:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:34:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:34:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:34:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:34:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:34:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:34:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:34:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:34:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:35:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:35:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:35:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:35:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:35:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:35:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:35:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:35:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:35:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:35:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:35:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:35:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:35:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:35:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:35:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:35:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:35:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:35:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:35:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:35:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:35:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:35:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:35:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:35:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:35:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:35:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:35:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:35:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:35:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:35:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:35:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:35:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:35:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:35:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:35:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:35:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:35:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:35:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:35:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:35:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:35:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:35:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:35:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:35:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:35:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:35:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:35:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:35:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:35:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:35:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:35:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:35:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:35:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:35:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:35:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:35:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:35:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:35:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:35:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:35:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:35:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:35:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:35:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:35:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:35:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:35:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:35:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:35:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:35:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:35:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:35:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:35:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:35:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:35:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:35:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:35:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:35:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:35:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:35:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:35:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:35:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:35:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:35:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:35:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:35:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:35:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:35:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:35:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:35:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:35:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:35:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:35:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:35:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:35:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:35:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:35:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:35:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:35:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:35:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:35:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:35:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:35:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:35:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:35:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:35:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:35:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:35:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:35:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:35:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:35:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:35:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:35:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:35:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:35:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:35:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:35:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:35:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:35:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:35:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:35:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:35:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:35:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:35:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:35:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:35:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:35:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:35:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:35:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:35:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:35:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:35:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:35:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:35:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:35:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:35:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:35:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:35:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:35:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:35:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:35:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:35:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:35:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:35:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:35:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:35:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:35:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:35:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:35:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:35:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:35:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:35:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:35:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:35:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:35:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:35:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:35:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:35:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:35:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:35:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:35:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:35:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:35:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:35:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:35:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:35:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:35:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:35:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:35:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:35:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:35:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:35:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:35:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:35:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:35:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:35:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:35:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:35:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:35:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:35:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:35:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:35:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:35:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:35:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:35:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:35:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:35:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:35:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:35:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:35:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:35:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:35:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:35:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:35:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:35:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:35:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:35:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:35:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:35:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:35:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:35:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:35:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:35:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:35:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:35:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:35:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:35:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:35:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:35:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:35:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:35:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:35:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:35:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:35:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:35:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:35:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:35:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:36:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:36:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:36:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:36:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:36:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:36:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:36:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:36:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:36:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:36:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:36:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:36:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:36:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:36:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:36:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:36:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:36:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:36:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:36:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:36:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:36:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:36:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:36:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:36:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:36:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:36:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:36:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:36:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:36:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:36:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:36:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:36:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:36:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:36:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:36:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:36:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:36:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:36:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:36:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:36:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:36:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:36:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:36:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:36:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:36:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:36:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:36:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:36:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:36:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:36:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:36:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:36:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:36:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:36:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:36:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:36:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:36:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:36:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:36:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:36:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:36:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:36:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:36:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:36:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:36:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:36:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:36:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:36:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:36:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:36:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:36:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:36:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:36:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:36:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:36:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:36:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:36:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:36:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:36:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:36:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:36:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:36:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:36:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:36:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:36:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:36:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:36:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:36:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:36:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:36:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:36:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:36:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:36:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:36:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:36:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:36:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:36:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:36:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:36:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:36:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:36:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:36:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:36:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:36:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:36:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:36:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:36:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:36:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:36:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:36:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:36:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:36:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:36:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:36:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:36:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:36:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:36:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:36:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:36:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:36:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:36:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:36:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:36:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:36:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:36:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:36:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:36:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:36:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:36:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:36:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:36:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:36:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:36:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:36:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:36:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:36:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:36:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:36:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:36:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:36:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:36:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:36:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:36:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:36:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:36:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:36:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:36:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:36:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:36:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:36:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:36:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:36:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:36:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:36:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:36:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:36:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:36:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:36:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:36:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:36:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:36:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:36:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:36:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:36:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:36:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:36:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:36:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:36:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:36:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:36:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:36:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:36:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:36:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:36:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:36:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:36:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:36:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:36:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:36:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:36:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:36:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:36:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:36:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:36:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:36:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:36:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:36:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:36:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:36:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:36:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:36:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:36:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:36:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:36:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:36:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:36:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:36:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:36:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:36:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:36:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:36:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:36:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:36:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:36:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:36:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:36:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:36:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:36:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:36:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:36:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:36:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:36:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:36:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:36:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:36:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:36:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:37:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:37:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:37:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:37:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:37:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:37:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:37:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:37:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:37:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:37:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:37:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:37:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:37:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:37:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:37:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:37:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:37:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:37:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:37:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:37:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:37:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:37:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:37:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:37:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:37:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:37:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:37:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:37:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:37:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:37:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:37:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:37:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:37:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:37:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:37:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:37:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:37:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:37:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:37:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:37:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:37:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:37:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:37:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:37:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:37:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:37:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:37:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:37:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:37:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:37:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:37:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:37:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:37:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:37:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:39:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:39:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:39:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:39:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:39:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:39:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:39:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:39:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:39:20 rdma06 rsyslogd[1041]: imjournal: 5082 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 22:39:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:39:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:39:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:39:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:39:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:39:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:39:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:39:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:39:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:39:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:39:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:39:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:39:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:39:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:39:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:39:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:39:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:39:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:39:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:39:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:39:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:39:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:39:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:39:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:39:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:39:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:39:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:39:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:39:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:39:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:39:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:39:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:39:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:39:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:39:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:39:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:39:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:39:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:39:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:39:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:39:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:39:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:39:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:39:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:39:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:39:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:39:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:39:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:39:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:39:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:39:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:39:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:39:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:39:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:39:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:39:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:39:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:39:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:39:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:39:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:39:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:39:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:39:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:39:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:39:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:39:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:39:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:39:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:39:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:39:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:39:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:39:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:39:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:39:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:39:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:39:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:39:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:39:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:39:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:39:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:39:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:39:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:39:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:39:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:39:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:39:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:39:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:39:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:39:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:39:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:39:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:39:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:39:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:39:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:40:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:40:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:40:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:40:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:40:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:40:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:40:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:40:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:40:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:40:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:40:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:40:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:40:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:40:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:40:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:40:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:40:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:40:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:40:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:40:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:40:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:40:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:40:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:40:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:40:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:40:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:40:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:40:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:40:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:40:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:40:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:40:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:40:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:40:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:40:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:40:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:40:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:40:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:40:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:40:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:40:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:40:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:40:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:40:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:40:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:40:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:40:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:40:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:40:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:40:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:40:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:40:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:40:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:40:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:40:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:40:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:40:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:40:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:40:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:40:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:40:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:40:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:40:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:40:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:40:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:40:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:40:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:40:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:40:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:40:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:40:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:40:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:40:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:40:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:40:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:40:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:40:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:40:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:40:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:40:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:40:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:40:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:40:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:40:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:40:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:40:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:40:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:40:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:40:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:40:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:40:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:40:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:40:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:40:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:40:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:40:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:40:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:40:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:40:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:40:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:40:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:40:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:40:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:40:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:40:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:40:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:40:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:40:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:40:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:40:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:40:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:40:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:40:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:40:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:40:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:40:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:40:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:40:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:40:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:40:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:40:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:40:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:40:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:40:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:40:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:40:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:40:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:40:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:40:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:40:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:40:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:40:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:40:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:40:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:40:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:40:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:40:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:40:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:40:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:40:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:40:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:40:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:40:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:40:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:40:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:40:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:40:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:40:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:40:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:40:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:40:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:40:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:40:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:40:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:40:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:40:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:40:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:40:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:40:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:40:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:40:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:40:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:40:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:40:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:40:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:40:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:40:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:40:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:40:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:40:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:40:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:40:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:40:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:40:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:40:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:40:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:40:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:40:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:40:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:40:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:40:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:40:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:40:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:40:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:40:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:40:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:40:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:40:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:40:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:40:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:40:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:40:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:40:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:40:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:40:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:40:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:40:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:40:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:40:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:40:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:40:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:40:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:40:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:40:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:40:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:40:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:40:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:40:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:40:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:40:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:40:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:40:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:40:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:40:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:40:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:40:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:41:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:41:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:41:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:41:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:41:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:41:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:41:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:41:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:41:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:41:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:41:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:41:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:41:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:41:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:41:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:41:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:41:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:41:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:41:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:41:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:41:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:41:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:41:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:41:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:41:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:41:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:41:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:41:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:41:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:41:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:41:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:41:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:41:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:41:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:41:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:41:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:41:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:41:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:41:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:41:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:41:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:41:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:41:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:41:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:41:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:41:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:41:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:41:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:41:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:41:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:41:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:41:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:41:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:41:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:41:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:41:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:41:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:41:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:41:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:41:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:41:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:41:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:41:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:41:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:41:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:41:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:41:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:41:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:41:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:41:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:41:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:41:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:41:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:41:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:41:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:41:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:41:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:41:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:41:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:41:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:41:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:41:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:41:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:41:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:41:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:41:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:41:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:41:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:41:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:41:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:41:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:41:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:41:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:41:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:41:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:41:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:41:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:41:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:41:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:41:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:41:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:41:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:41:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:41:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:41:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:41:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:41:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:41:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:41:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:41:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:41:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:41:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:41:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:41:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:41:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:41:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:41:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:41:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:41:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:41:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:41:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:41:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:41:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:41:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:41:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:41:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:41:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:41:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:41:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:41:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:41:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:41:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:41:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:41:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:41:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:41:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:41:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:41:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:41:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:41:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:41:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:41:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:41:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:41:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:41:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:41:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:41:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:41:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:41:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:41:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:41:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:41:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:41:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:41:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:41:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:41:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:41:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:41:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:41:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:41:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:41:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:41:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:41:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:41:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:41:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:41:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:41:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:41:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:41:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:41:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:41:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:41:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:41:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:41:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:41:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:41:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:41:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:41:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:41:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:41:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:41:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:41:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:41:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:41:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:41:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:41:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:41:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:41:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:41:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:41:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:41:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:41:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:41:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:41:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:41:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:41:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:41:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:41:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:41:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:41:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:41:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:41:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:41:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:41:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:41:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:41:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:41:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:41:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:41:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:41:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:41:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:41:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:41:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:41:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:41:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:41:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:42:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:42:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:42:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:42:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:42:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:42:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:42:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:42:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:42:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:42:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:42:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:42:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:42:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:42:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:42:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:42:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:42:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:42:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:42:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:42:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:42:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:42:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:42:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:42:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:42:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:42:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:42:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:42:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:42:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:42:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:42:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:42:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:42:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:42:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:42:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:42:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:42:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:42:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:42:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:42:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:42:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:42:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:42:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:42:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:42:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:42:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:42:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:42:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:42:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:42:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:42:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:42:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:42:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:42:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:42:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:42:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:42:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:42:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:42:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:42:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:42:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:42:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:42:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:42:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:42:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:42:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:42:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:42:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:42:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:42:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:42:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:42:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:42:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:42:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:42:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:42:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:42:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:42:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:42:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:42:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:42:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:42:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:42:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:42:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:42:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:42:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:42:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:42:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:42:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:42:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:42:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:42:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:42:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:42:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:42:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:42:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:42:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:42:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:42:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:42:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:42:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:42:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:42:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:42:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:42:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:42:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:42:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:42:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:42:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:42:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:42:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:42:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:42:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:42:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:42:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:42:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:42:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:42:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:42:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:42:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:42:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:42:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:42:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:42:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:42:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:42:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:42:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:42:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:42:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:42:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:42:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:42:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:42:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:42:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:42:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:42:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:42:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:42:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:42:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:42:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:42:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:42:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:42:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:42:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:42:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:42:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:42:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:42:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:42:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:42:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:42:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:42:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:42:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:42:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:42:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:42:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:42:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:42:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:42:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:42:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:42:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:42:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:42:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:42:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:42:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:42:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:42:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:42:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:42:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:42:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:42:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:42:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:42:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:42:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:42:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:42:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:42:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:42:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:42:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:42:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:42:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:42:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:42:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:42:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:42:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:42:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:42:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:42:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:42:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:42:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:42:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:42:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:42:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:42:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:42:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:42:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:42:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:42:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:42:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:42:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:42:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:42:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:42:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:42:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:42:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:42:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:42:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:42:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:42:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:42:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:42:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:42:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:42:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:42:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:42:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:42:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:43:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:43:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:43:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:43:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:43:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:43:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:43:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:43:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:43:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:43:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:43:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:43:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:43:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:43:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:43:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:43:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:43:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:43:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:43:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:43:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:43:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:43:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:43:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:43:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:43:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:43:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:43:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:43:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:43:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:43:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:43:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:43:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:43:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:43:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:43:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:43:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:43:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:43:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:43:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:43:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:43:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:43:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:43:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:43:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:43:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:43:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:43:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:43:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:43:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:43:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:43:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:43:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:43:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:43:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:43:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:43:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:43:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:43:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:43:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:43:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:43:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:43:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:43:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:43:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:43:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:43:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:43:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:43:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:43:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:43:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:43:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:43:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:43:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:43:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:43:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:43:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:43:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:43:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:43:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:43:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:43:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:43:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:43:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:43:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:43:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:43:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:43:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:43:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:43:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:43:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:43:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:43:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:43:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:43:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:43:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:43:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:43:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:43:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:43:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:43:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:43:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:43:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:43:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:43:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:43:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:43:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:43:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:43:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:43:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:43:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:43:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:43:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:43:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:43:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:43:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:43:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:43:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:43:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:43:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:43:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:43:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:43:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:43:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:43:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:43:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:43:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:43:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:43:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:43:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:43:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:43:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:43:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:43:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:43:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:43:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:43:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:43:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:43:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:43:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:43:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:43:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:43:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:43:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:43:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:43:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:43:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:43:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:43:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:43:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:43:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:43:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:43:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:43:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:43:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:43:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:43:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:43:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:43:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:43:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:43:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:43:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:43:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:43:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:43:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:43:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:43:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:43:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:43:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:43:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:43:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:43:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:43:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:43:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:43:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:43:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:43:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:43:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:43:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:43:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:43:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:43:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:43:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:43:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:43:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:43:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:43:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:43:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:43:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:43:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:43:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:43:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:43:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:43:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:43:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:43:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:43:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:43:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:43:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:43:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:43:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:43:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:43:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:43:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:43:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:43:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:43:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:43:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:43:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:43:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:43:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:43:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:43:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:43:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:43:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:43:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:43:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:44:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:44:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:44:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:44:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:44:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:44:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:44:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:44:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:44:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:44:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:44:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:44:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:44:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:44:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:44:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:44:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:44:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:44:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:44:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:44:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:44:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:44:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:44:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:44:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:44:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:44:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:44:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:44:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:44:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:44:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:44:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:44:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:44:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:44:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:44:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:44:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:44:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:44:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:44:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:44:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:44:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:44:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:44:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:44:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:44:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:44:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:44:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:44:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:44:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:44:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:44:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:44:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:44:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:44:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:44:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:44:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:44:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:44:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:44:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:44:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:44:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:44:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:44:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:44:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:44:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:44:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:44:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:44:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:44:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:44:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:44:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:44:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:44:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:44:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:44:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:44:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:44:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:44:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:44:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:44:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:44:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:44:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:44:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:44:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:44:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:44:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:44:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:44:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:44:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:44:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:44:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:44:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:44:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:44:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:44:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:44:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:44:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:44:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:44:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:44:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:44:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:44:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:44:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:44:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:44:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:44:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:44:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:44:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:44:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:44:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:44:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:44:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:44:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:44:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:44:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:44:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:44:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:44:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:44:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:44:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:44:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:44:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:44:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:44:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:44:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:44:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:44:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:44:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:44:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:44:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:44:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:44:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:44:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:44:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:44:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:44:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:44:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:44:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:44:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:44:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:44:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:44:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:44:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:44:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:44:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:44:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:44:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:44:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:44:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:44:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:44:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:44:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:44:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:44:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:44:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:44:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:44:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:44:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:44:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:44:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:44:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:44:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:44:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:44:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:44:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:44:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:44:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:44:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:44:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:44:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:44:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:44:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:44:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:44:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:44:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:44:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:44:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:44:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:44:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:44:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:44:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:44:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:44:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:44:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:44:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:44:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:44:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:44:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:44:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:44:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:44:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:44:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:44:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:44:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:44:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:44:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:44:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:44:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:45:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:45:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:45:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:45:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:45:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:45:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:45:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:45:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:45:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:45:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:45:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:45:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:45:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:45:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:45:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:45:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:45:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:45:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:45:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:45:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:45:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:45:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:45:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:45:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:45:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:45:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:45:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:45:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:45:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:45:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:45:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:45:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:45:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:45:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:45:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:45:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:45:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:45:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:45:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:45:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:45:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:45:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:45:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:45:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:45:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:45:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:45:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:45:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:45:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:45:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:45:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:45:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:45:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:45:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:45:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:45:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:45:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:45:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:45:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:45:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:45:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:45:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:45:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:45:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:45:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:45:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:45:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:45:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:45:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:45:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:45:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:45:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:45:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:45:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:45:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:45:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:45:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:45:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:45:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:45:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:45:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:45:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:45:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:45:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:45:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:45:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:45:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:45:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:45:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:45:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:45:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:45:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:45:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:45:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:45:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:45:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:45:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:45:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:45:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:45:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:45:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:45:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:45:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:45:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:45:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:45:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:45:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:45:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:45:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:45:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:45:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:45:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:45:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:45:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:45:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:45:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:45:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:45:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:45:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:45:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:45:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:45:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:45:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:45:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:45:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:45:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:45:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:45:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:45:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:45:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:45:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:45:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:45:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:45:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:45:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:45:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:45:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:45:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:45:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:45:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:45:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:45:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:45:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:45:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:45:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:45:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:45:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:45:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:45:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:45:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:45:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:45:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:45:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:45:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:45:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:45:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:45:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:45:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:45:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:45:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:45:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:45:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:45:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:45:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:45:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:45:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:45:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:45:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:45:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:45:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:45:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:45:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:45:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:45:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:45:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:45:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:45:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:45:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:45:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:45:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:45:45 rdma06 systemd[1]: Starting dnf makecache...
Apr 18 22:45:45 rdma06 dnf[1205765]: Updating Subscription Management repositories.
Apr 18 22:45:45 rdma06 dnf[1205765]: Unable to read consumer identity
Apr 18 22:45:45 rdma06 dnf[1205765]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 18 22:45:46 rdma06 dnf[1205765]: Copr repo for fmf owned by psss                 2.7 kB/s | 3.3 kB     00:01
Apr 18 22:45:46 rdma06 dnf[1205765]: beaker-AppStream                                 48 kB/s | 168  B     00:00
Apr 18 22:45:46 rdma06 dnf[1205765]: Errors during downloading metadata for repository 'beaker-AppStream':
Apr 18 22:45:46 rdma06 dnf[1205765]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 22:45:46 rdma06 dnf[1205765]: Error: Failed to download metadata for repo 'beaker-AppStream': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 22:45:46 rdma06 dnf[1205765]: beaker-AppStream-debuginfo                       58 kB/s | 168  B     00:00
Apr 18 22:45:46 rdma06 dnf[1205765]: Errors during downloading metadata for repository 'beaker-AppStream-debuginfo':
Apr 18 22:45:46 rdma06 dnf[1205765]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 22:45:46 rdma06 dnf[1205765]: Error: Failed to download metadata for repo 'beaker-AppStream-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 22:45:46 rdma06 dnf[1205765]: beaker-BaseOS                                    57 kB/s | 168  B     00:00
Apr 18 22:45:46 rdma06 dnf[1205765]: Errors during downloading metadata for repository 'beaker-BaseOS':
Apr 18 22:45:46 rdma06 dnf[1205765]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 22:45:46 rdma06 dnf[1205765]: Error: Failed to download metadata for repo 'beaker-BaseOS': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 22:45:46 rdma06 dnf[1205765]: beaker-BaseOS-debuginfo                          65 kB/s | 168  B     00:00
Apr 18 22:45:46 rdma06 dnf[1205765]: Errors during downloading metadata for repository 'beaker-BaseOS-debuginfo':
Apr 18 22:45:46 rdma06 dnf[1205765]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 22:45:46 rdma06 dnf[1205765]: Error: Failed to download metadata for repo 'beaker-BaseOS-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 22:45:46 rdma06 dnf[1205765]: beaker-CRB                                       64 kB/s | 168  B     00:00
Apr 18 22:45:46 rdma06 dnf[1205765]: Errors during downloading metadata for repository 'beaker-CRB':
Apr 18 22:45:46 rdma06 dnf[1205765]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 22:45:46 rdma06 dnf[1205765]: Error: Failed to download metadata for repo 'beaker-CRB': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 22:45:46 rdma06 dnf[1205765]: beaker-CRB-debuginfo                             59 kB/s | 168  B     00:00
Apr 18 22:45:46 rdma06 dnf[1205765]: Errors during downloading metadata for repository 'beaker-CRB-debuginfo':
Apr 18 22:45:46 rdma06 dnf[1205765]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 22:45:46 rdma06 dnf[1205765]: Error: Failed to download metadata for repo 'beaker-CRB-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 22:45:46 rdma06 dnf[1205765]: beaker-HighAvailability                          53 kB/s | 168  B     00:00
Apr 18 22:45:46 rdma06 dnf[1205765]: Errors during downloading metadata for repository 'beaker-HighAvailability':
Apr 18 22:45:46 rdma06 dnf[1205765]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 22:45:46 rdma06 dnf[1205765]: Error: Failed to download metadata for repo 'beaker-HighAvailability': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 22:45:46 rdma06 dnf[1205765]: beaker-HighAvailability-debuginfo               4.6 kB/s | 168  B     00:00
Apr 18 22:45:46 rdma06 dnf[1205765]: Errors during downloading metadata for repository 'beaker-HighAvailability-debuginfo':
Apr 18 22:45:46 rdma06 dnf[1205765]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 22:45:46 rdma06 dnf[1205765]: Error: Failed to download metadata for repo 'beaker-HighAvailability-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 22:45:46 rdma06 dnf[1205765]: beaker-NFV                                       62 kB/s | 168  B     00:00
Apr 18 22:45:46 rdma06 dnf[1205765]: Errors during downloading metadata for repository 'beaker-NFV':
Apr 18 22:45:46 rdma06 dnf[1205765]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 22:45:46 rdma06 dnf[1205765]: Error: Failed to download metadata for repo 'beaker-NFV': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 22:45:46 rdma06 dnf[1205765]: beaker-NFV-debuginfo                             63 kB/s | 168  B     00:00
Apr 18 22:45:46 rdma06 dnf[1205765]: Errors during downloading metadata for repository 'beaker-NFV-debuginfo':
Apr 18 22:45:46 rdma06 dnf[1205765]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 22:45:46 rdma06 dnf[1205765]: Error: Failed to download metadata for repo 'beaker-NFV-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 22:45:46 rdma06 dnf[1205765]: beaker-RT                                        63 kB/s | 168  B     00:00
Apr 18 22:45:46 rdma06 dnf[1205765]: Errors during downloading metadata for repository 'beaker-RT':
Apr 18 22:45:46 rdma06 dnf[1205765]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 22:45:46 rdma06 dnf[1205765]: Error: Failed to download metadata for repo 'beaker-RT': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 22:45:46 rdma06 dnf[1205765]: beaker-RT-debuginfo                              62 kB/s | 168  B     00:00
Apr 18 22:45:46 rdma06 dnf[1205765]: Errors during downloading metadata for repository 'beaker-RT-debuginfo':
Apr 18 22:45:46 rdma06 dnf[1205765]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 22:45:46 rdma06 dnf[1205765]: Error: Failed to download metadata for repo 'beaker-RT-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 22:45:46 rdma06 dnf[1205765]: beaker-ResilientStorage                          64 kB/s | 168  B     00:00
Apr 18 22:45:46 rdma06 dnf[1205765]: Errors during downloading metadata for repository 'beaker-ResilientStorage':
Apr 18 22:45:46 rdma06 dnf[1205765]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 22:45:46 rdma06 dnf[1205765]: Error: Failed to download metadata for repo 'beaker-ResilientStorage': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 22:45:46 rdma06 dnf[1205765]: beaker-ResilientStorage-debuginfo                64 kB/s | 168  B     00:00
Apr 18 22:45:46 rdma06 dnf[1205765]: Errors during downloading metadata for repository 'beaker-ResilientStorage-debuginfo':
Apr 18 22:45:46 rdma06 dnf[1205765]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 22:45:46 rdma06 dnf[1205765]: Error: Failed to download metadata for repo 'beaker-ResilientStorage-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 22:45:46 rdma06 dnf[1205765]: beaker-SAP                                       65 kB/s | 168  B     00:00
Apr 18 22:45:46 rdma06 dnf[1205765]: Errors during downloading metadata for repository 'beaker-SAP':
Apr 18 22:45:46 rdma06 dnf[1205765]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 22:45:46 rdma06 dnf[1205765]: Error: Failed to download metadata for repo 'beaker-SAP': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 22:45:46 rdma06 dnf[1205765]: beaker-SAP-debuginfo                             65 kB/s | 168  B     00:00
Apr 18 22:45:46 rdma06 dnf[1205765]: Errors during downloading metadata for repository 'beaker-SAP-debuginfo':
Apr 18 22:45:46 rdma06 dnf[1205765]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 22:45:46 rdma06 dnf[1205765]: Error: Failed to download metadata for repo 'beaker-SAP-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 22:45:46 rdma06 dnf[1205765]: beaker-SAPHANA                                   66 kB/s | 168  B     00:00
Apr 18 22:45:46 rdma06 dnf[1205765]: Errors during downloading metadata for repository 'beaker-SAPHANA':
Apr 18 22:45:46 rdma06 dnf[1205765]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 22:45:46 rdma06 dnf[1205765]: Error: Failed to download metadata for repo 'beaker-SAPHANA': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 22:45:46 rdma06 dnf[1205765]: beaker-SAPHANA-debuginfo                         62 kB/s | 168  B     00:00
Apr 18 22:45:46 rdma06 dnf[1205765]: Errors during downloading metadata for repository 'beaker-SAPHANA-debuginfo':
Apr 18 22:45:46 rdma06 dnf[1205765]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 22:45:46 rdma06 dnf[1205765]: Error: Failed to download metadata for repo 'beaker-SAPHANA-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 18 22:45:47 rdma06 dnf[1205765]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr 18 22:45:48 rdma06 dnf[1205765]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr 18 22:45:48 rdma06 dnf[1205765]: Ignoring repositories: beaker-AppStream, beaker-AppStream-debuginfo, beaker-BaseOS, beaker-BaseOS-debuginfo, beaker-CRB, beaker-CRB-debuginfo, beaker-HighAvailability, beaker-HighAvailability-debuginfo, beaker-NFV, beaker-NFV-debuginfo, beaker-RT, beaker-RT-debuginfo, beaker-ResilientStorage, beaker-ResilientStorage-debuginfo, beaker-SAP, beaker-SAP-debuginfo, beaker-SAPHANA, beaker-SAPHANA-debuginfo
Apr 18 22:45:48 rdma06 dnf[1205765]: Metadata cache created.
Apr 18 22:45:48 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 18 22:45:48 rdma06 systemd[1]: Finished dnf makecache.
Apr 18 22:45:48 rdma06 systemd[1]: dnf-makecache.service: Consumed 1.004s CPU time.
Apr 18 22:45:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:45:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:45:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:45:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:45:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:45:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:45:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:45:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:45:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:45:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:45:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:45:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:45:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:45:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:45:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:45:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:45:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:45:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:45:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:45:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:45:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:45:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:45:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:45:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:45:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:45:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:45:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:45:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:45:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:45:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:45:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:45:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:45:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:45:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:45:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:45:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:46:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:46:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:46:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:46:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:46:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:46:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:46:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:46:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:46:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:46:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:46:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:46:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:46:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:46:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:46:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:46:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:46:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:46:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:46:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:46:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:46:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:46:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:46:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:46:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:46:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:46:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:46:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:46:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:46:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:46:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:46:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:46:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:46:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:46:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:46:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:46:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:46:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:46:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:46:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:46:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:46:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:46:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:46:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:46:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:46:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:46:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:46:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:46:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:46:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:46:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:46:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:46:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:46:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:46:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:46:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:46:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:46:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:46:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:46:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:46:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:46:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:46:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:46:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:46:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:46:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:46:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:46:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:46:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:46:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:46:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:46:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:46:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:46:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:46:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:46:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:46:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:46:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:46:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:46:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:46:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:46:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:46:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:46:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:46:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:46:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:46:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:46:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:46:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:46:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:46:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:46:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:46:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:46:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:46:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:46:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:46:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:46:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:46:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:46:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:46:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:46:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:46:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:46:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:46:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:46:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:46:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:46:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:46:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:46:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:46:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:46:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:46:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:46:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:46:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:46:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:46:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:46:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:46:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:46:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:46:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:46:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:46:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:46:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:46:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:46:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:46:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:46:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:46:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:46:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:46:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:46:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:46:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:46:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:46:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:46:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:46:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:46:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:46:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:46:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:46:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:46:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:46:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:46:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:46:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:46:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:46:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:46:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:46:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:46:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:46:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:46:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:46:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:46:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:46:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:46:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:46:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:46:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:46:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:46:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:46:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:46:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:46:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:46:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:46:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:46:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:46:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:46:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:46:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:46:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:46:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:46:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:46:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:46:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:46:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:46:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:46:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:46:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:46:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:46:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:46:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:46:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:46:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:46:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:46:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:46:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:46:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:46:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:46:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:46:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:46:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:46:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:46:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:46:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:46:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:46:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:46:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:46:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:46:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:46:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:46:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:46:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:46:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:46:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:46:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:46:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:46:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:46:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:46:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:46:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:46:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:46:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:46:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:46:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:46:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:46:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:46:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:47:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:47:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:47:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:47:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:47:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:47:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:47:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:47:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:47:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:47:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:47:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:47:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:47:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:47:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:47:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:47:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:47:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:47:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:47:00 rdma06 systemd[1]: Starting Cleanup of Temporary Directories...
Apr 18 22:47:00 rdma06 systemd[5685]: Starting Cleanup of User's Temporary Files and Directories...
Apr 18 22:47:00 rdma06 systemd[5685]: Finished Cleanup of User's Temporary Files and Directories.
Apr 18 22:47:00 rdma06 systemd[1]: systemd-tmpfiles-clean.service: Deactivated successfully.
Apr 18 22:47:00 rdma06 systemd[1]: Finished Cleanup of Temporary Directories.
Apr 18 22:47:00 rdma06 systemd[1]: run-credentials-systemd\x2dtmpfiles\x2dclean.service.mount: Deactivated successfully.
Apr 18 22:47:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:47:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:47:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:47:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:47:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:47:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:47:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:47:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:47:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:47:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:47:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:47:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:47:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:47:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:47:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:47:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:47:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:47:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:47:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:47:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:47:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:47:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:47:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:47:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:47:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:47:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:47:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:47:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:47:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:47:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:47:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:47:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:47:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:47:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:47:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:47:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:47:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:47:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:47:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:47:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:47:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:47:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:47:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:47:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:47:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:47:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:47:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:47:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:47:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:47:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:47:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:49:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:49:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:49:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:49:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:49:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:49:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:49:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:49:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:49:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:49:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:49:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:49:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:49:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:49:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:49:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:49:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:49:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:49:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:49:25 rdma06 rsyslogd[1041]: imjournal: 5387 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 22:49:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:49:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:49:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:49:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:49:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:49:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:49:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:49:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:49:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:49:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:49:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:49:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:49:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:49:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:49:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:49:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:49:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:49:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:49:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:49:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:49:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:49:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:49:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:49:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:49:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:49:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:49:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:49:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:49:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:49:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:49:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:49:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:49:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:49:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:49:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:49:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:49:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:49:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:49:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:49:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:49:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:49:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:49:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:49:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:49:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:49:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:49:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:49:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:49:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:49:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:49:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:49:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:49:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:49:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:49:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:49:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:49:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:49:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:49:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:49:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:49:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:49:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:49:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:49:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:49:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:49:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:49:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:49:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:49:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:49:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:49:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:49:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:49:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:49:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:49:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:49:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:49:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:49:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:49:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:49:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:49:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:49:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:49:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:49:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:49:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:49:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:49:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:49:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:49:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:49:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:49:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:49:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:49:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:49:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:49:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:49:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:49:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:49:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:49:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:49:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:49:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:49:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:49:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:49:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:49:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:49:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:49:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:49:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:50:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:50:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:50:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:50:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:50:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:50:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:50:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:50:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:50:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:50:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:50:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:50:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:50:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:50:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:50:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:50:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:50:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:50:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:50:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:50:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:50:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:50:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:50:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:50:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:50:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:50:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:50:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:50:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:50:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:50:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:50:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:50:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:50:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:50:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:50:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:50:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:50:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:50:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:50:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:50:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:50:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:50:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:50:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:50:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:50:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:50:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:50:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:50:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:50:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:50:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:50:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:50:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:50:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:50:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:50:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:50:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:50:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:50:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:50:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:50:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:50:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:50:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:50:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:50:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:50:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:50:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:50:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:50:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:50:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:50:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:50:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:50:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:50:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:50:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:50:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:50:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:50:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:50:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:50:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:50:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:50:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:50:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:50:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:50:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:50:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:50:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:50:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:50:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:50:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:50:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:50:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:50:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:50:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:50:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:50:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:50:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:50:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:50:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:50:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:50:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:50:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:50:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:50:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:50:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:50:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:50:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:50:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:50:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:50:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:50:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:50:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:50:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:50:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:50:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:50:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:50:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:50:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:50:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:50:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:50:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:50:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:50:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:50:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:50:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:50:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:50:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:50:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:50:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:50:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:50:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:50:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:50:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:50:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:50:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:50:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:50:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:50:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:50:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:50:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:50:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:50:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:50:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:50:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:50:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:50:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:50:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:50:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:50:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:50:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:50:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:50:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:50:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:50:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:50:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:50:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:50:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:50:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:50:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:50:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:50:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:50:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:50:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:50:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:50:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:50:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:50:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:50:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:50:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:50:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:50:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:50:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:50:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:50:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:50:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:50:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:50:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:50:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:50:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:50:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:50:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:50:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:50:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:50:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:50:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:50:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:50:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:50:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:50:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:50:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:50:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:50:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:50:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:50:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:50:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:50:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:50:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:50:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:50:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:50:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:50:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:50:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:50:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:50:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:50:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:50:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:50:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:50:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:50:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:50:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:50:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:50:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:50:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:50:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:50:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:50:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:50:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:51:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:51:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:51:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:51:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:51:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:51:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:51:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:51:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:51:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:51:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:51:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:51:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:51:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:51:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:51:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:51:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:51:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:51:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:51:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:51:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:51:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:51:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:51:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:51:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:51:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:51:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:51:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:51:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:51:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:51:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:51:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:51:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:51:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:51:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:51:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:51:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:51:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:51:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:51:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:51:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:51:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:51:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:51:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:51:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:51:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:51:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:51:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:51:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:51:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:51:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:51:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:51:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:51:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:51:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:51:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:51:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:51:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:51:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:51:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:51:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:51:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:51:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:51:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:51:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:51:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:51:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:51:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:51:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:51:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:51:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:51:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:51:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:51:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:51:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:51:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:51:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:51:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:51:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:51:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:51:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:51:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:51:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:51:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:51:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:51:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:51:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:51:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:51:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:51:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:51:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:51:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:51:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:51:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:51:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:51:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:51:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:51:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:51:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:51:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:51:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:51:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:51:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:51:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:51:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:51:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:51:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:51:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:51:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:51:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:51:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:51:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:51:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:51:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:51:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:51:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:51:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:51:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:51:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:51:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:51:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:51:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:51:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:51:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:51:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:51:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:51:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:51:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:51:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:51:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:51:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:51:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:51:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:51:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:51:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:51:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:51:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:51:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:51:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:51:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:51:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:51:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:51:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:51:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:51:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:51:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:51:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:51:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:51:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:51:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:51:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:51:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:51:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:51:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:51:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:51:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:51:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:51:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:51:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:51:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:51:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:51:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:51:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:51:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:51:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:51:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:51:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:51:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:51:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:51:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:51:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:51:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:51:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:51:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:51:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:51:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:51:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:51:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:51:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:51:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:51:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:51:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:51:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:51:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:51:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:51:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:51:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:52:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:52:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:52:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:52:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:52:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:52:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:52:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:52:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:52:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:52:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:52:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:52:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:52:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:52:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:52:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:52:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:52:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:52:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:52:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:52:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:52:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:52:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:52:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:52:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:52:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:52:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:52:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:52:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:52:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:52:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:52:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:52:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:52:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:52:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:52:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:52:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:52:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:52:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:52:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:52:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:52:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:52:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:52:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:52:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:52:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:52:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:52:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:52:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:52:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:52:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:52:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:52:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:52:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:52:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:52:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:52:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:52:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:52:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:52:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:52:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:52:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:52:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:52:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:52:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:52:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:52:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:52:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:52:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:52:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:52:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:52:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:52:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:52:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:52:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:52:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:52:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:52:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:52:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:52:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:52:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:52:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:52:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:52:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:52:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:52:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:52:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:52:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:52:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:52:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:52:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:52:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:52:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:52:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:52:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:52:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:52:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:52:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:52:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:52:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:52:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:52:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:52:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:52:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:52:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:52:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:52:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:52:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:52:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:52:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:52:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:52:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:52:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:52:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:52:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:52:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:52:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:52:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:52:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:52:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:52:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:52:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:52:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:52:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:52:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:52:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:52:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:52:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:52:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:52:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:52:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:52:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:52:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:52:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:52:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:52:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:52:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:52:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:52:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:52:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:52:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:52:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:52:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:52:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:52:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:52:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:52:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:52:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:52:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:52:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:52:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:52:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:52:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:52:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:52:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:52:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:52:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:52:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:52:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:52:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:52:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:52:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:52:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:52:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:52:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:52:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:52:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:52:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:52:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:52:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:52:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:52:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:52:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:52:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:52:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:52:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:52:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:52:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:52:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:52:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:52:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:52:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:52:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:52:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:52:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:52:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:52:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:52:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:52:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:52:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:52:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:52:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:52:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:52:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:52:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:52:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:52:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:52:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:52:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:52:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:52:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:52:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:52:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:52:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:52:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:52:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:52:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:52:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:52:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:52:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:52:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:52:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:52:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:52:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:52:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:52:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:52:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:53:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:53:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:53:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:53:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:53:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:53:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:53:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:53:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:53:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:53:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:53:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:53:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:53:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:53:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:53:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:53:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:53:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:53:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:53:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:53:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:53:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:53:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:53:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:53:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:53:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:53:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:53:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:53:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:53:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:53:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:53:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:53:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:53:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:53:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:53:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:53:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:53:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:53:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:53:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:53:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:53:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:53:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:53:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:53:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:53:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:53:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:53:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:53:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:53:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:53:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:53:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:53:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:53:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:53:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:53:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:53:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:53:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:53:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:53:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:53:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:53:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:53:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:53:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:53:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:53:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:53:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:53:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:53:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:53:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:53:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:53:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:53:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:53:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:53:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:53:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:53:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:53:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:53:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:53:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:53:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:53:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:53:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:53:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:53:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:53:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:53:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:53:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:53:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:53:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:53:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:53:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:53:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:53:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:53:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:53:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:53:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:53:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:53:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:53:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:53:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:53:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:53:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:53:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:53:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:53:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:53:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:53:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:53:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:53:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:53:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:53:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:53:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:53:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:53:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:53:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:53:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:53:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:53:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:53:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:53:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:53:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:53:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:53:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:53:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:53:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:53:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:53:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:53:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:53:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:53:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:53:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:53:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:53:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:53:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:53:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:53:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:53:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:53:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:53:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:53:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:53:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:53:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:53:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:53:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:53:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:53:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:53:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:53:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:53:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:53:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:53:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:53:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:53:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:53:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:53:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:53:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:53:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:53:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:53:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:53:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:53:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:53:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:53:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:53:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:53:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:53:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:53:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:53:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:53:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:53:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:53:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:53:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:53:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:53:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:53:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:53:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:53:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:53:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:53:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:53:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:53:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:53:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:53:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:53:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:53:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:53:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:53:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:53:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:53:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:53:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:53:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:53:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:53:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:53:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:53:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:53:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:53:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:53:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:53:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:53:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:53:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:53:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:53:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:53:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:53:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:53:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:53:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:53:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:53:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:53:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:53:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:53:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:53:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:53:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:53:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:53:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:54:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:54:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:54:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:54:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:54:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:54:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:54:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:54:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:54:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:54:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:54:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:54:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:54:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:54:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:54:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:54:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:54:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:54:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:54:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:54:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:54:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:54:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:54:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:54:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:54:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:54:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:54:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:54:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:54:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:54:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:54:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:54:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:54:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:54:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:54:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:54:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:54:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:54:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:54:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:54:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:54:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:54:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:54:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:54:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:54:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:54:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:54:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:54:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:54:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:54:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:54:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:54:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:54:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:54:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:54:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:54:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:54:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:54:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:54:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:54:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:54:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:54:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:54:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:54:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:54:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:54:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:54:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:54:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:54:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:54:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:54:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:54:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:54:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:54:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:54:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:54:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:54:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:54:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:54:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:54:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:54:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:54:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:54:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:54:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:54:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:54:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:54:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:54:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:54:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:54:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:54:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:54:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:54:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:54:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:54:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:54:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:54:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:54:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:54:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:54:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:54:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:54:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:54:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:54:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:54:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:54:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:54:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:54:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:54:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:54:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:54:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:54:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:54:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:54:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:54:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:54:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:54:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:54:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:54:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:54:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:54:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:54:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:54:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:54:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:54:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:54:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:54:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:54:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:54:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:54:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:54:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:54:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:54:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:54:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:54:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:54:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:54:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:54:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:54:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:54:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:54:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:54:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:54:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:54:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:54:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:54:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:54:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:54:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:54:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:54:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:54:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:54:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:54:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:54:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:54:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:54:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:54:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:54:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:54:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:54:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:54:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:54:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:54:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:54:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:54:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:54:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:54:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:54:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:54:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:54:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:54:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:54:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:54:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:54:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:54:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:54:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:54:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:54:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:54:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:54:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:54:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:54:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:54:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:54:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:54:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:54:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:54:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:54:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:54:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:54:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:54:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:54:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:54:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:54:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:54:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:54:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:54:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:54:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:54:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:54:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:54:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:54:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:54:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:54:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:54:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:54:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:54:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:54:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:54:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:54:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:54:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:54:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:54:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:54:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:54:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:54:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:55:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:55:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:55:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:55:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:55:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:55:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:55:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:55:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:55:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:55:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:55:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:55:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:55:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:55:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:55:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:55:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:55:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:55:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:55:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:55:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:55:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:55:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:55:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:55:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:55:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:55:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:55:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:55:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:55:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:55:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:55:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:55:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:55:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:55:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:55:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:55:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:55:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:55:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:55:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:55:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:55:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:55:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:55:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:55:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:55:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:55:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:55:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:55:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:55:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:55:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:55:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:55:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:55:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:55:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:55:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:55:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:55:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:55:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:55:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:55:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:55:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:55:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:55:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:55:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:55:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:55:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:55:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:55:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:55:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:55:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:55:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:55:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:55:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:55:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:55:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:55:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:55:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:55:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:55:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:55:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:55:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:55:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:55:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:55:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:55:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:55:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:55:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:55:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:55:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:55:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:55:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:55:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:55:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:55:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:55:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:55:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:55:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:55:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:55:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:55:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:55:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:55:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:55:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:55:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:55:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:55:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:55:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:55:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:55:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:55:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:55:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:55:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:55:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:55:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:55:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:55:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:55:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:55:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:55:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:55:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:55:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:55:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:55:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:55:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:55:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:55:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:55:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:55:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:55:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:55:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:55:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:55:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:55:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:55:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:55:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:55:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:55:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:55:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:55:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:55:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:55:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:55:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:55:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:55:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:55:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:55:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:55:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:55:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:55:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:55:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:55:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:55:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:55:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:55:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:55:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:55:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:55:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:55:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:55:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:55:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:55:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:55:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:55:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:55:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:55:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:55:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:55:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:55:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:55:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:55:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:55:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:55:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:55:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:55:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:55:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:55:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:55:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:55:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:55:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:55:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:55:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:55:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:55:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:55:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:55:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:55:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:55:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:55:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:55:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:55:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:55:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:55:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:55:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:55:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:55:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:55:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:55:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:55:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:55:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:55:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:55:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:55:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:55:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:55:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:55:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:55:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:55:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:55:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:55:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:55:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:55:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:55:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:55:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:55:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:55:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:55:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:56:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:56:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:56:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:56:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:56:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:56:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:56:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:56:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:56:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:56:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:56:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:56:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:56:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:56:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:56:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:56:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:56:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:56:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:56:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:56:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:56:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:56:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:56:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:56:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:56:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:56:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:56:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:56:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:56:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:56:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:56:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:56:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:56:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:56:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:56:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:56:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:56:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:56:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:56:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:56:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:56:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:56:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:56:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:56:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:56:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:56:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:56:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:56:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:56:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:56:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:56:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:56:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:56:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:56:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:56:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:56:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:56:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:56:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:56:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:56:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:56:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:56:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:56:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:56:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:56:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:56:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:56:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:56:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:56:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:56:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:56:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:56:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:56:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:56:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:56:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:56:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:56:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:56:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:56:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:56:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:56:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:56:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:56:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:56:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:56:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:56:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:56:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:56:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:56:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:56:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:56:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:56:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:56:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:56:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:56:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:56:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:56:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:56:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:56:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:56:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:56:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:56:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:56:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:56:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:56:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:56:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:56:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:56:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:56:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:56:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:56:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:56:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:56:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:56:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:56:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:56:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:56:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:56:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:56:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:56:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:56:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:56:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:56:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:56:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:56:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:56:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:56:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:56:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:56:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:56:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:56:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:56:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:56:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:56:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:56:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:56:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:56:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:56:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:56:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:56:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:56:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:56:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:56:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:56:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:56:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:56:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:56:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:56:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:56:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:56:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:56:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:56:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:56:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:56:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:56:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:56:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:56:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:56:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:56:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:56:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:56:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:56:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:56:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:56:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:56:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:56:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:56:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:56:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:56:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:56:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:56:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:56:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:56:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:56:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:56:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:56:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:56:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:56:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:56:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:56:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:56:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:56:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:56:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:56:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:56:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:56:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:56:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:56:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:56:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:56:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:56:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:56:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:56:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:56:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:56:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:56:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:56:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:56:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:56:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:56:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:56:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:56:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:56:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:56:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:56:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:56:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:56:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:56:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:56:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:56:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:56:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:56:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:56:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:56:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:56:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:56:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:57:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:57:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:57:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:57:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:57:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:57:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:57:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:57:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:57:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:57:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:57:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:57:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:57:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:57:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:57:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:57:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:57:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:57:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:57:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:57:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:57:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:57:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:57:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:57:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:57:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:57:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:57:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:57:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:57:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:57:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:57:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:57:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:57:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:57:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:57:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:57:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:57:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:57:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:57:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:57:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:57:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:57:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:57:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:57:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:57:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:57:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:57:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:57:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:57:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:57:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:57:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:57:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:57:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:57:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:57:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:57:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:57:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:57:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:57:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:57:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:57:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:57:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:57:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:57:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:57:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:57:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:57:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:57:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:57:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:57:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:57:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:57:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:57:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:57:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:57:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:57:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:57:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:57:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:57:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:57:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:57:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:57:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:57:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:57:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:57:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:57:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:57:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:57:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:57:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:57:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:59:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:59:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:59:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:59:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:59:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:59:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:59:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:59:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:59:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:59:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:59:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:59:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:59:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:59:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:59:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:59:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:59:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:59:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:59:30 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 22:59:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:59:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:59:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:59:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:59:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:59:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:59:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:59:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:59:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:59:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:59:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:59:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:59:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:59:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:59:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:59:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:59:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:59:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:59:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:59:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:59:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:59:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:59:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:59:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:59:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:59:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:59:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:59:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:59:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:59:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:59:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:59:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:59:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:59:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:59:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:59:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:59:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:59:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:59:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:59:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:59:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:59:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:59:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:59:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:59:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:59:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:59:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:59:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:59:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:59:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:59:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:59:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:59:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:59:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:59:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:59:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:59:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:59:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:59:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:59:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:59:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:59:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:59:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:59:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:59:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:59:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:59:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:59:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:59:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:59:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:59:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:59:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 22:59:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 22:59:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:59:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:59:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:59:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 22:59:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 22:59:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 22:59:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 22:59:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 22:59:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 22:59:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 22:59:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 22:59:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 22:59:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 22:59:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 22:59:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 22:59:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 22:59:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:00:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:00:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:00:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:00:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:00:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:00:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:00:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:00:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:00:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:00:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:00:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:00:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:00:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:00:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:00:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:00:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:00:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:00:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:00:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:00:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:00:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:00:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:00:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:00:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:00:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:00:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:00:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:00:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:00:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:00:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:00:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:00:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:00:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:00:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:00:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:00:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:00:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:00:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:00:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:00:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:00:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:00:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:00:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:00:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:00:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:00:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:00:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:00:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:00:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:00:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:00:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:00:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:00:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:00:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:00:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:00:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:00:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:00:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:00:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:00:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:00:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:00:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:00:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:00:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:00:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:00:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:00:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:00:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:00:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:00:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:00:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:00:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:00:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:00:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:00:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:00:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:00:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:00:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:00:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:00:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:00:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:00:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:00:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:00:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:00:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:00:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:00:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:00:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:00:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:00:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:00:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:00:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:00:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:00:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:00:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:00:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:00:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:00:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:00:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:00:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:00:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:00:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:00:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:00:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:00:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:00:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:00:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:00:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:00:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:00:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:00:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:00:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:00:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:00:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:00:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:00:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:00:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:00:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:00:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:00:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:00:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:00:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:00:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:00:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:00:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:00:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:00:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:00:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:00:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:00:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:00:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:00:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:00:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:00:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:00:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:00:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:00:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:00:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:00:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:00:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:00:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:00:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:00:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:00:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:00:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:00:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:00:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:00:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:00:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:00:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:00:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:00:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:00:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:00:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:00:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:00:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:00:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:00:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:00:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:00:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:00:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:00:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:00:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:00:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:00:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:00:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:00:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:00:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:00:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:00:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:00:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:00:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:00:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:00:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:00:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:00:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:00:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:00:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:00:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:00:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:00:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:00:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:00:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:00:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:00:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:00:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:00:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:00:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:00:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:00:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:00:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:00:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:00:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:00:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:00:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:00:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:00:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:00:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:00:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:00:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:00:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:00:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:00:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:00:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:00:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:00:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:00:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:00:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:00:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:00:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:00:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:00:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:00:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:00:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:00:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:00:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:01:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:01:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:01:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:01:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:01:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:01:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:01:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:01:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:01:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:01:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:01:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:01:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:01:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:01:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:01:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:01:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:01:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:01:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:01:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:01:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:01:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:01:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:01:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:01:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:01:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:01:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:01:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:01:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:01:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:01:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:01:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:01:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:01:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:01:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:01:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:01:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:01:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:01:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:01:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:01:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:01:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:01:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:01:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:01:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:01:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:01:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:01:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:01:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:01:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:01:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:01:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:01:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:01:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:01:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:01:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:01:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:01:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:01:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:01:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:01:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:01:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:01:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:01:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:01:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:01:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:01:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:01:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:01:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:01:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:01:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:01:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:01:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:01:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:01:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:01:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:01:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:01:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:01:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:01:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:01:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:01:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:01:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:01:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:01:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:01:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:01:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:01:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:01:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:01:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:01:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:01:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:01:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:01:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:01:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:01:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:01:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:01:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:01:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:01:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:01:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:01:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:01:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:01:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:01:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:01:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:01:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:01:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:01:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:01:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:01:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:01:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:01:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:01:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:01:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:01:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:01:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:01:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:01:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:01:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:01:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:01:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:01:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:01:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:01:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:01:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:01:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:01:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:01:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:01:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:01:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:01:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:01:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:01:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:01:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:01:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:01:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:01:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:01:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:01:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:01:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:01:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:01:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:01:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:01:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:01:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:01:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:01:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:01:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:01:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:01:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:01:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:01:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:01:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:01:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:01:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:01:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:02:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:02:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:02:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:02:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:02:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:02:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:02:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:02:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:02:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:02:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:02:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:02:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:02:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:02:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:02:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:02:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:02:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:02:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:02:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:02:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:02:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:02:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:02:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:02:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:02:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:02:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:02:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:02:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:02:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:02:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:02:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:02:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:02:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:02:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:02:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:02:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:02:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:02:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:02:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:02:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:02:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:02:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:02:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:02:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:02:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:02:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:02:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:02:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:02:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:02:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:02:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:02:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:02:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:02:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:02:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:02:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:02:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:02:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:02:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:02:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:02:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:02:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:02:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:02:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:02:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:02:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:02:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:02:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:02:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:02:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:02:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:02:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:02:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:02:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:02:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:02:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:02:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:02:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:02:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:02:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:02:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:02:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:02:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:02:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:02:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:02:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:02:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:02:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:02:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:02:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:02:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:02:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:02:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:02:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:02:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:02:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:02:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:02:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:02:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:02:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:02:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:02:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:02:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:02:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:02:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:02:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:02:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:02:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:02:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:02:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:02:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:02:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:02:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:02:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:02:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:02:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:02:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:02:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:02:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:02:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:02:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:02:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:02:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:02:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:02:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:02:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:02:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:02:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:02:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:02:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:02:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:02:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:02:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:02:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:02:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:02:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:02:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:02:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:02:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:02:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:02:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:02:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:02:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:02:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:02:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:02:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:02:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:02:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:02:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:02:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:02:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:02:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:02:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:02:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:02:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:02:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:02:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:02:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:02:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:02:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:02:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:02:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:02:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:02:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:02:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:02:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:02:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:02:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:02:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:02:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:02:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:02:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:02:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:02:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:02:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:02:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:02:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:02:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:02:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:02:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:02:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:02:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:02:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:02:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:02:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:02:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:02:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:02:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:02:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:02:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:02:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:02:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:02:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:02:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:02:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:02:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:02:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:02:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:02:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:02:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:02:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:02:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:02:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:02:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:02:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:02:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:02:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:02:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:02:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:02:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:02:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:02:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:02:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:02:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:02:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:02:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:03:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:03:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:03:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:03:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:03:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:03:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:03:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:03:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:03:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:03:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:03:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:03:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:03:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:03:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:03:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:03:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:03:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:03:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:03:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:03:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:03:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:03:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:03:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:03:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:03:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:03:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:03:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:03:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:03:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:03:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:03:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:03:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:03:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:03:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:03:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:03:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:03:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:03:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:03:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:03:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:03:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:03:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:03:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:03:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:03:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:03:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:03:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:03:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:03:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:03:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:03:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:03:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:03:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:03:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:03:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:03:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:03:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:03:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:03:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:03:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:03:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:03:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:03:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:03:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:03:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:03:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:03:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:03:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:03:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:03:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:03:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:03:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:03:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:03:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:03:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:03:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:03:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:03:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:03:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:03:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:03:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:03:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:03:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:03:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:03:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:03:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:03:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:03:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:03:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:03:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:03:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:03:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:03:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:03:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:03:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:03:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:03:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:03:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:03:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:03:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:03:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:03:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:03:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:03:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:03:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:03:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:03:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:03:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:03:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:03:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:03:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:03:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:03:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:03:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:03:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:03:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:03:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:03:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:03:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:03:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:03:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:03:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:03:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:03:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:03:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:03:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:03:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:03:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:03:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:03:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:03:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:03:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:03:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:03:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:03:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:03:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:03:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:03:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:03:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:03:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:03:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:03:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:03:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:03:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:03:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:03:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:03:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:03:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:03:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:03:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:03:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:03:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:03:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:03:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:03:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:03:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:03:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:03:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:03:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:03:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:03:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:03:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:03:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:03:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:03:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:03:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:03:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:03:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:03:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:03:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:03:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:03:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:03:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:03:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:03:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:03:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:03:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:03:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:03:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:03:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:03:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:03:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:03:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:03:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:03:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:03:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:03:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:03:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:03:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:03:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:03:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:03:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:03:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:03:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:03:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:03:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:03:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:03:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:03:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:03:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:03:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:03:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:03:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:03:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:03:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:03:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:03:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:03:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:03:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:03:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:03:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:03:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:03:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:03:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:03:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:03:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:04:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:04:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:04:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:04:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:04:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:04:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:04:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:04:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:04:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:04:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:04:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:04:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:04:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:04:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:04:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:04:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:04:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:04:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:04:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:04:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:04:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:04:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:04:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:04:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:04:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:04:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:04:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:04:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:04:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:04:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:04:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:04:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:04:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:04:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:04:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:04:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:04:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:04:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:04:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:04:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:04:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:04:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:04:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:04:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:04:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:04:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:04:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:04:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:04:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:04:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:04:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:04:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:04:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:04:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:04:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:04:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:04:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:04:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:04:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:04:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:04:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:04:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:04:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:04:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:04:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:04:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:04:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:04:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:04:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:04:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:04:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:04:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:04:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:04:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:04:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:04:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:04:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:04:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:04:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:04:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:04:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:04:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:04:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:04:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:04:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:04:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:04:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:04:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:04:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:04:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:04:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:04:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:04:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:04:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:04:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:04:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:04:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:04:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:04:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:04:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:04:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:04:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:04:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:04:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:04:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:04:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:04:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:04:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:04:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:04:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:04:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:04:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:04:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:04:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:04:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:04:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:04:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:04:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:04:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:04:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:04:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:04:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:04:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:04:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:04:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:04:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:04:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:04:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:04:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:04:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:04:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:04:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:04:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:04:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:04:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:04:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:04:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:04:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:04:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:04:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:04:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:04:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:04:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:04:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:04:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:04:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:04:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:04:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:04:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:04:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:04:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:04:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:04:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:04:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:04:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:04:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:04:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:04:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:04:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:04:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:04:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:04:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:04:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:04:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:04:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:04:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:04:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:04:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:04:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:04:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:04:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:04:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:04:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:04:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:04:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:04:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:04:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:04:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:04:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:04:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:04:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:04:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:04:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:04:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:04:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:04:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:04:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:04:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:04:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:04:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:04:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:04:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:04:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:04:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:04:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:04:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:04:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:04:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:04:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:04:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:04:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:04:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:04:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:04:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:04:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:04:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:04:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:04:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:04:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:04:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:04:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:04:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:04:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:04:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:04:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:04:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:05:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:05:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:05:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:05:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:05:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:05:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:05:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:05:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:05:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:05:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:05:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:05:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:05:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:05:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:05:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:05:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:05:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:05:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:05:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:05:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:05:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:05:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:05:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:05:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:05:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:05:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:05:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:05:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:05:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:05:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:05:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:05:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:05:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:05:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:05:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:05:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:05:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:05:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:05:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:05:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:05:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:05:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:05:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:05:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:05:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:05:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:05:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:05:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:05:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:05:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:05:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:05:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:05:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:05:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:05:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:05:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:05:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:05:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:05:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:05:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:05:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:05:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:05:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:05:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:05:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:05:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:05:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:05:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:05:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:05:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:05:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:05:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:05:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:05:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:05:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:05:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:05:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:05:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:05:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:05:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:05:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:05:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:05:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:05:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:05:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:05:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:05:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:05:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:05:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:05:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:05:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:05:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:05:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:05:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:05:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:05:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:05:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:05:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:05:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:05:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:05:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:05:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:05:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:05:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:05:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:05:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:05:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:05:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:05:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:05:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:05:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:05:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:05:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:05:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:05:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:05:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:05:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:05:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:05:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:05:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:05:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:05:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:05:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:05:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:05:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:05:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:05:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:05:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:05:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:05:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:05:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:05:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:05:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:05:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:05:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:05:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:05:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:05:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:05:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:05:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:05:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:05:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:05:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:05:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:05:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:05:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:05:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:05:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:05:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:05:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:05:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:05:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:05:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:05:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:05:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:05:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:05:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:05:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:05:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:05:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:05:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:05:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:05:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:05:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:05:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:05:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:05:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:05:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:05:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:05:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:05:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:05:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:05:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:05:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:05:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:05:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:05:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:05:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:05:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:05:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:05:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:05:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:05:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:05:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:05:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:05:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:05:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:05:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:05:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:05:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:05:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:05:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:05:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:05:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:05:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:05:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:05:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:05:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:05:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:05:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:05:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:05:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:05:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:05:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:05:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:05:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:05:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:05:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:05:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:05:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:05:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:05:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:05:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:05:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:05:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:05:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:06:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:06:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:06:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:06:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:06:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:06:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:06:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:06:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:06:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:06:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:06:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:06:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:06:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:06:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:06:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:06:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:06:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:06:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:06:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:06:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:06:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:06:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:06:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:06:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:06:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:06:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:06:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:06:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:06:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:06:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:06:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:06:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:06:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:06:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:06:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:06:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:06:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:06:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:06:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:06:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:06:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:06:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:06:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:06:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:06:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:06:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:06:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:06:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:06:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:06:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:06:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:06:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:06:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:06:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:06:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:06:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:06:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:06:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:06:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:06:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:06:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:06:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:06:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:06:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:06:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:06:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:06:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:06:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:06:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:06:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:06:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:06:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:06:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:06:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:06:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:06:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:06:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:06:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:06:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:06:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:06:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:06:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:06:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:06:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:06:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:06:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:06:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:06:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:06:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:06:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:06:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:06:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:06:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:06:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:06:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:06:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:06:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:06:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:06:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:06:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:06:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:06:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:06:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:06:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:06:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:06:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:06:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:06:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:06:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:06:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:06:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:06:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:06:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:06:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:06:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:06:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:06:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:06:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:06:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:06:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:06:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:06:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:06:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:06:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:06:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:06:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:06:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:06:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:06:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:06:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:06:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:06:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:06:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:06:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:06:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:06:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:06:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:06:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:06:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:06:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:06:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:06:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:06:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:06:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:06:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:06:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:06:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:06:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:06:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:06:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:06:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:06:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:06:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:06:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:06:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:06:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:06:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:06:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:06:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:06:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:06:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:06:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:06:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:06:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:06:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:06:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:06:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:06:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:06:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:06:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:06:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:06:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:06:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:06:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:06:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:06:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:06:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:06:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:06:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:06:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:06:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:06:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:06:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:06:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:06:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:06:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:06:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:06:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:06:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:06:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:06:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:06:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:06:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:06:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:06:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:06:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:06:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:06:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:06:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:06:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:06:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:06:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:06:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:06:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:06:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:06:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:06:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:06:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:06:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:06:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:06:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:06:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:06:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:06:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:06:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:06:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:07:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:07:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:07:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:07:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:07:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:07:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:07:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:07:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:07:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:07:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:07:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:07:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:07:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:07:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:07:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:07:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:07:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:07:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:07:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:07:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:07:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:07:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:07:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:07:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:07:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:07:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:07:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:07:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:07:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:07:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:07:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:07:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:07:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:07:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:07:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:07:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:07:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:07:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:07:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:07:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:07:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:07:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:07:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:07:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:07:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:07:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:07:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:07:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:07:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:07:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:07:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:07:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:07:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:07:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:07:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:07:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:07:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:07:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:07:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:07:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:07:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:07:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:07:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:07:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:07:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:07:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:07:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:07:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:07:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:07:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:07:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:07:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:07:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:07:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:07:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:07:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:07:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:07:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:07:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:07:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:07:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:07:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:07:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:07:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:07:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:07:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:07:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:07:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:07:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:07:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:07:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:07:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:07:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:07:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:07:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:07:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:07:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:07:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:07:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:07:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:07:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:07:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:07:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:07:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:07:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:07:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:07:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:07:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:09:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:09:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:09:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:09:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:09:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:09:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:09:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:09:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:09:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:09:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:09:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:09:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:09:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:09:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:09:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:09:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:09:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:09:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:09:31 rdma06 rsyslogd[1041]: imjournal: 5086 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 23:09:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:09:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:09:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:09:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:09:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:09:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:09:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:09:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:09:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:09:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:09:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:09:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:09:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:09:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:09:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:09:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:09:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:09:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:09:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:09:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:09:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:09:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:09:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:09:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:09:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:09:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:09:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:09:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:09:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:09:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:09:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:09:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:09:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:09:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:09:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:09:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:09:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:09:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:09:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:09:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:09:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:09:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:09:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:09:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:09:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:09:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:09:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:09:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:09:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:09:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:09:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:09:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:09:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:09:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:09:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:09:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:09:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:09:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:09:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:09:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:09:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:09:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:09:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:09:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:09:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:09:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:09:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:09:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:09:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:09:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:09:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:09:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:09:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:09:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:09:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:09:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:09:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:09:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:09:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:09:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:09:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:09:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:09:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:09:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:09:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:09:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:09:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:09:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:09:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:09:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:10:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:10:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:10:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:10:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:10:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:10:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:10:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:10:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:10:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:10:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:10:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:10:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:10:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:10:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:10:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:10:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:10:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:10:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:10:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:10:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:10:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:10:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:10:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:10:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:10:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:10:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:10:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:10:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:10:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:10:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:10:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:10:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:10:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:10:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:10:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:10:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:10:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:10:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:10:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:10:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:10:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:10:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:10:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:10:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:10:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:10:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:10:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:10:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:10:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:10:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:10:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:10:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:10:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:10:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:10:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:10:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:10:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:10:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:10:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:10:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:10:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:10:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:10:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:10:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:10:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:10:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:10:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:10:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:10:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:10:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:10:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:10:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:10:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:10:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:10:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:10:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:10:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:10:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:10:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:10:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:10:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:10:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:10:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:10:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:10:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:10:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:10:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:10:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:10:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:10:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:10:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:10:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:10:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:10:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:10:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:10:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:10:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:10:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:10:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:10:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:10:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:10:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:10:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:10:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:10:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:10:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:10:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:10:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:10:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:10:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:10:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:10:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:10:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:10:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:10:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:10:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:10:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:10:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:10:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:10:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:10:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:10:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:10:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:10:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:10:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:10:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:10:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:10:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:10:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:10:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:10:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:10:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:10:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:10:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:10:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:10:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:10:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:10:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:10:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:10:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:10:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:10:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:10:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:10:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:10:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:10:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:10:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:10:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:10:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:10:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:10:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:10:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:10:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:10:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:10:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:10:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:10:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:10:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:10:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:10:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:10:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:10:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:10:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:10:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:10:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:10:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:10:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:10:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:10:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:10:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:10:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:10:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:10:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:10:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:10:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:10:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:10:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:10:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:10:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:10:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:10:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:10:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:10:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:10:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:10:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:10:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:10:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:10:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:10:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:10:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:10:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:10:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:10:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:10:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:10:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:10:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:10:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:10:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:10:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:10:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:10:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:10:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:10:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:10:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:10:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:10:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:10:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:10:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:10:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:10:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:10:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:10:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:10:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:10:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:10:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:10:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:11:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:11:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:11:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:11:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:11:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:11:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:11:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:11:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:11:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:11:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:11:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:11:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:11:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:11:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:11:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:11:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:11:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:11:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:11:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:11:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:11:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:11:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:11:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:11:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:11:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:11:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:11:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:11:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:11:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:11:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:11:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:11:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:11:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:11:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:11:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:11:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:11:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:11:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:11:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:11:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:11:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:11:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:11:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:11:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:11:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:11:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:11:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:11:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:11:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:11:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:11:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:11:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:11:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:11:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:11:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:11:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:11:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:11:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:11:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:11:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:11:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:11:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:11:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:11:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:11:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:11:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:11:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:11:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:11:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:11:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:11:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:11:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:11:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:11:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:11:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:11:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:11:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:11:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:11:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:11:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:11:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:11:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:11:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:11:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:11:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:11:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:11:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:11:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:11:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:11:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:11:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:11:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:11:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:11:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:11:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:11:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:11:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:11:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:11:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:11:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:11:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:11:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:11:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:11:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:11:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:11:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:11:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:11:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:11:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:11:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:11:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:11:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:11:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:11:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:11:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:11:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:11:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:11:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:11:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:11:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:11:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:11:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:11:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:11:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:11:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:11:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:11:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:11:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:11:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:11:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:11:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:11:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:11:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:11:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:11:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:11:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:11:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:11:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:11:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:11:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:11:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:11:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:11:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:11:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:11:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:11:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:11:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:11:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:11:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:11:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:11:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:11:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:11:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:11:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:11:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:11:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:11:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:11:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:11:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:11:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:11:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:11:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:11:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:11:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:11:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:11:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:11:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:11:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:11:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:11:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:11:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:11:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:11:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:11:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:11:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:11:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:11:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:11:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:11:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:11:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:11:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:11:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:11:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:11:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:11:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:11:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:12:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:12:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:12:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:12:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:12:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:12:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:12:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:12:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:12:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:12:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:12:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:12:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:12:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:12:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:12:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:12:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:12:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:12:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:12:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:12:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:12:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:12:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:12:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:12:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:12:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:12:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:12:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:12:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:12:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:12:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:12:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:12:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:12:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:12:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:12:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:12:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:12:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:12:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:12:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:12:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:12:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:12:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:12:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:12:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:12:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:12:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:12:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:12:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:12:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:12:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:12:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:12:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:12:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:12:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:12:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:12:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:12:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:12:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:12:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:12:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:12:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:12:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:12:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:12:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:12:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:12:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:12:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:12:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:12:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:12:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:12:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:12:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:12:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:12:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:12:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:12:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:12:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:12:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:12:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:12:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:12:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:12:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:12:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:12:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:12:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:12:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:12:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:12:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:12:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:12:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:12:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:12:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:12:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:12:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:12:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:12:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:12:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:12:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:12:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:12:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:12:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:12:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:12:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:12:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:12:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:12:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:12:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:12:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:12:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:12:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:12:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:12:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:12:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:12:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:12:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:12:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:12:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:12:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:12:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:12:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:12:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:12:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:12:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:12:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:12:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:12:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:12:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:12:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:12:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:12:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:12:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:12:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:12:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:12:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:12:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:12:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:12:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:12:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:12:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:12:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:12:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:12:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:12:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:12:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:12:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:12:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:12:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:12:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:12:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:12:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:12:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:12:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:12:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:12:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:12:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:12:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:12:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:12:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:12:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:12:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:12:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:12:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:12:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:12:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:12:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:12:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:12:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:12:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:12:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:12:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:12:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:12:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:12:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:12:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:12:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:12:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:12:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:12:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:12:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:12:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:12:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:12:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:12:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:12:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:12:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:12:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:12:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:12:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:12:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:12:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:12:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:12:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:12:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:12:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:12:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:12:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:12:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:12:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:12:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:12:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:12:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:12:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:12:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:12:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:12:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:12:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:12:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:12:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:12:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:12:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:12:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:12:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:12:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:12:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:12:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:12:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:13:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:13:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:13:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:13:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:13:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:13:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:13:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:13:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:13:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:13:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:13:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:13:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:13:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:13:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:13:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:13:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:13:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:13:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:13:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:13:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:13:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:13:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:13:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:13:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:13:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:13:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:13:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:13:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:13:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:13:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:13:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:13:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:13:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:13:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:13:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:13:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:13:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:13:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:13:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:13:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:13:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:13:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:13:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:13:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:13:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:13:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:13:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:13:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:13:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:13:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:13:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:13:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:13:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:13:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:13:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:13:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:13:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:13:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:13:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:13:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:13:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:13:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:13:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:13:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:13:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:13:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:13:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:13:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:13:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:13:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:13:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:13:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:13:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:13:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:13:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:13:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:13:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:13:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:13:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:13:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:13:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:13:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:13:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:13:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:13:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:13:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:13:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:13:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:13:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:13:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:13:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:13:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:13:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:13:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:13:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:13:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:13:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:13:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:13:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:13:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:13:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:13:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:13:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:13:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:13:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:13:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:13:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:13:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:13:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:13:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:13:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:13:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:13:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:13:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:13:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:13:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:13:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:13:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:13:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:13:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:13:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:13:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:13:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:13:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:13:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:13:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:13:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:13:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:13:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:13:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:13:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:13:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:13:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:13:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:13:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:13:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:13:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:13:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:13:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:13:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:13:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:13:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:13:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:13:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:13:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:13:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:13:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:13:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:13:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:13:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:13:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:13:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:13:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:13:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:13:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:13:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:13:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:13:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:13:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:13:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:13:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:13:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:13:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:13:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:13:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:13:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:13:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:13:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:13:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:13:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:13:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:13:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:13:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:13:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:13:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:13:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:13:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:13:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:13:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:13:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:13:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:13:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:13:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:13:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:13:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:13:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:13:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:13:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:13:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:13:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:13:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:13:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:13:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:13:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:13:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:13:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:13:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:13:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:13:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:13:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:13:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:13:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:13:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:13:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:13:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:13:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:13:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:13:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:13:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:13:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:13:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:13:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:13:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:13:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:13:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:13:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:14:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:14:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:14:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:14:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:14:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:14:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:14:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:14:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:14:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:14:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:14:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:14:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:14:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:14:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:14:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:14:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:14:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:14:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:14:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:14:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:14:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:14:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:14:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:14:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:14:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:14:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:14:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:14:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:14:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:14:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:14:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:14:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:14:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:14:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:14:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:14:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:14:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:14:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:14:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:14:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:14:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:14:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:14:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:14:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:14:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:14:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:14:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:14:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:14:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:14:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:14:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:14:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:14:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:14:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:14:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:14:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:14:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:14:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:14:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:14:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:14:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:14:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:14:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:14:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:14:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:14:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:14:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:14:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:14:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:14:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:14:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:14:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:14:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:14:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:14:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:14:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:14:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:14:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:14:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:14:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:14:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:14:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:14:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:14:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:14:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:14:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:14:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:14:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:14:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:14:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:14:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:14:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:14:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:14:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:14:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:14:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:14:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:14:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:14:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:14:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:14:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:14:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:14:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:14:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:14:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:14:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:14:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:14:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:14:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:14:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:14:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:14:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:14:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:14:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:14:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:14:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:14:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:14:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:14:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:14:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:14:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:14:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:14:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:14:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:14:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:14:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:14:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:14:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:14:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:14:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:14:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:14:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:14:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:14:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:14:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:14:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:14:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:14:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:14:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:14:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:14:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:14:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:14:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:14:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:14:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:14:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:14:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:14:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:14:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:14:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:14:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:14:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:14:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:14:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:14:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:14:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:14:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:14:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:14:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:14:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:14:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:14:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:14:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:14:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:14:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:14:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:14:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:14:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:14:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:14:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:14:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:14:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:14:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:14:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:14:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:14:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:14:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:14:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:14:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:14:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:14:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:14:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:14:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:14:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:14:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:14:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:14:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:14:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:14:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:14:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:14:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:14:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:14:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:14:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:14:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:14:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:14:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:14:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:14:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:14:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:14:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:14:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:14:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:14:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:14:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:14:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:14:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:14:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:14:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:14:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:14:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:14:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:14:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:14:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:14:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:14:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:15:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:15:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:15:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:15:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:15:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:15:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:15:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:15:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:15:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:15:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:15:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:15:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:15:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:15:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:15:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:15:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:15:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:15:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:15:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:15:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:15:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:15:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:15:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:15:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:15:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:15:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:15:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:15:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:15:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:15:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:15:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:15:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:15:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:15:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:15:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:15:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:15:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:15:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:15:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:15:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:15:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:15:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:15:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:15:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:15:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:15:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:15:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:15:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:15:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:15:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:15:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:15:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:15:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:15:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:15:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:15:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:15:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:15:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:15:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:15:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:15:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:15:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:15:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:15:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:15:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:15:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:15:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:15:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:15:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:15:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:15:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:15:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:15:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:15:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:15:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:15:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:15:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:15:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:15:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:15:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:15:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:15:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:15:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:15:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:15:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:15:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:15:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:15:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:15:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:15:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:15:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:15:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:15:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:15:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:15:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:15:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:15:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:15:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:15:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:15:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:15:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:15:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:15:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:15:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:15:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:15:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:15:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:15:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:15:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:15:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:15:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:15:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:15:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:15:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:15:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:15:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:15:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:15:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:15:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:15:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:15:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:15:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:15:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:15:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:15:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:15:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:15:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:15:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:15:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:15:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:15:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:15:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:15:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:15:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:15:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:15:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:15:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:15:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:15:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:15:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:15:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:15:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:15:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:15:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:15:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:15:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:15:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:15:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:15:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:15:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:15:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:15:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:15:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:15:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:15:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:15:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:15:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:15:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:15:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:15:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:15:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:15:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:15:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:15:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:15:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:15:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:15:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:15:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:15:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:15:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:15:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:15:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:15:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:15:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:15:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:15:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:15:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:15:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:15:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:15:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:15:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:15:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:15:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:15:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:15:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:15:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:15:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:15:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:15:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:15:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:15:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:15:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:15:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:15:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:15:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:15:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:15:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:15:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:15:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:15:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:15:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:15:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:15:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:15:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:15:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:15:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:15:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:15:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:15:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:15:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:15:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:15:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:15:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:15:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:15:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:15:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:16:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:16:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:16:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:16:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:16:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:16:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:16:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:16:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:16:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:16:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:16:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:16:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:16:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:16:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:16:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:16:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:16:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:16:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:16:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:16:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:16:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:16:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:16:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:16:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:16:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:16:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:16:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:16:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:16:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:16:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:16:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:16:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:16:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:16:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:16:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:16:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:16:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:16:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:16:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:16:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:16:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:16:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:16:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:16:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:16:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:16:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:16:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:16:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:16:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:16:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:16:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:16:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:16:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:16:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:16:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:16:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:16:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:16:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:16:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:16:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:16:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:16:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:16:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:16:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:16:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:16:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:16:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:16:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:16:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:16:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:16:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:16:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:16:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:16:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:16:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:16:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:16:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:16:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:16:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:16:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:16:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:16:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:16:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:16:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:16:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:16:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:16:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:16:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:16:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:16:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:16:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:16:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:16:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:16:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:16:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:16:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:16:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:16:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:16:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:16:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:16:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:16:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:16:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:16:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:16:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:16:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:16:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:16:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:16:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:16:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:16:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:16:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:16:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:16:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:16:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:16:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:16:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:16:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:16:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:16:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:16:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:16:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:16:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:16:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:16:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:16:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:16:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:16:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:16:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:16:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:16:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:16:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:16:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:16:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:16:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:16:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:16:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:16:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:16:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:16:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:16:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:16:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:16:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:16:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:16:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:16:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:16:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:16:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:16:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:16:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:16:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:16:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:16:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:16:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:16:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:16:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:16:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:16:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:16:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:16:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:16:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:16:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:16:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:16:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:16:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:16:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:16:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:16:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:16:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:16:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:16:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:16:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:16:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:16:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:16:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:16:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:16:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:16:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:16:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:16:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:16:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:16:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:16:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:16:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:16:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:16:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:16:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:16:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:16:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:16:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:16:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:16:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:16:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:16:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:16:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:16:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:16:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:16:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:16:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:16:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:16:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:16:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:16:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:16:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:16:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:16:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:16:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:16:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:16:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:16:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:16:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:16:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:16:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:16:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:16:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:16:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:17:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:17:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:17:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:17:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:17:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:17:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:17:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:17:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:17:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:17:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:17:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:17:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:17:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:17:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:17:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:17:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:17:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:17:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:17:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:17:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:17:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:17:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:17:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:17:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:17:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:17:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:17:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:17:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:17:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:17:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:17:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:17:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:17:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:17:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:17:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:17:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:17:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:17:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:17:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:17:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:17:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:17:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:17:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:17:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:17:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:17:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:17:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:17:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:17:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:17:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:17:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:17:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:17:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:17:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:17:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:17:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:17:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:17:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:17:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:17:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:17:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:17:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:17:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:17:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:17:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:17:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:17:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:17:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:17:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:17:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:17:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:17:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:17:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:17:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:17:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:17:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:17:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:17:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:17:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:17:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:17:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:17:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:17:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:17:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:17:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:17:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:17:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:17:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:17:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:17:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:17:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:17:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:17:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:17:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:17:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:17:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:17:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:17:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:17:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:17:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:17:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:17:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:17:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:17:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:17:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:17:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:17:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:17:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:19:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:19:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:19:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:19:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:19:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:19:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:19:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:19:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:19:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:19:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:19:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:19:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:19:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:19:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:19:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:19:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:19:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:19:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:19:36 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 23:19:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:19:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:19:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:19:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:19:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:19:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:19:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:19:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:19:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:19:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:19:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:19:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:19:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:19:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:19:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:19:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:19:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:19:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:19:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:19:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:19:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:19:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:19:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:19:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:19:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:19:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:19:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:19:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:19:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:19:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:19:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:19:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:19:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:19:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:19:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:19:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:19:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:19:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:19:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:19:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:19:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:19:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:19:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:19:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:19:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:19:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:19:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:19:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:19:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:19:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:19:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:19:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:19:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:19:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:19:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:19:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:19:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:19:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:19:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:19:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:19:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:19:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:19:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:19:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:19:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:19:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:19:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:19:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:19:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:19:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:19:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:19:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:20:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:20:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:20:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:20:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:20:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:20:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:20:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:20:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:20:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:20:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:20:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:20:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:20:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:20:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:20:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:20:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:20:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:20:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:20:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:20:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:20:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:20:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:20:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:20:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:20:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:20:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:20:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:20:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:20:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:20:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:20:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:20:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:20:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:20:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:20:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:20:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:20:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:20:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:20:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:20:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:20:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:20:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:20:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:20:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:20:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:20:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:20:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:20:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:20:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:20:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:20:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:20:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:20:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:20:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:20:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:20:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:20:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:20:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:20:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:20:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:20:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:20:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:20:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:20:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:20:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:20:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:20:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:20:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:20:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:20:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:20:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:20:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:20:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:20:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:20:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:20:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:20:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:20:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:20:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:20:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:20:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:20:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:20:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:20:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:20:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:20:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:20:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:20:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:20:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:20:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:20:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:20:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:20:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:20:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:20:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:20:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:20:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:20:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:20:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:20:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:20:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:20:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:20:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:20:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:20:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:20:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:20:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:20:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:20:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:20:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:20:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:20:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:20:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:20:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:20:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:20:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:20:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:20:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:20:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:20:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:20:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:20:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:20:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:20:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:20:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:20:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:20:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:20:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:20:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:20:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:20:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:20:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:20:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:20:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:20:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:20:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:20:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:20:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:20:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:20:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:20:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:20:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:20:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:20:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:20:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:20:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:20:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:20:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:20:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:20:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:20:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:20:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:20:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:20:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:20:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:20:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:20:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:20:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:20:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:20:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:20:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:20:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:20:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:20:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:20:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:20:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:20:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:20:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:20:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:20:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:20:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:20:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:20:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:20:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:20:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:20:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:20:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:20:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:20:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:20:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:20:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:20:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:20:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:20:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:20:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:20:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:20:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:20:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:20:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:20:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:20:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:20:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:20:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:20:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:20:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:20:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:20:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:20:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:20:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:20:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:20:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:20:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:20:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:20:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:20:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:20:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:20:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:20:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:20:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:20:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:20:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:20:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:20:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:20:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:20:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:20:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:21:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:21:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:21:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:21:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:21:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:21:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:21:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:21:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:21:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:21:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:21:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:21:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:21:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:21:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:21:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:21:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:21:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:21:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:21:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:21:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:21:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:21:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:21:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:21:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:21:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:21:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:21:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:21:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:21:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:21:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:21:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:21:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:21:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:21:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:21:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:21:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:21:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:21:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:21:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:21:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:21:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:21:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:21:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:21:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:21:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:21:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:21:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:21:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:21:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:21:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:21:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:21:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:21:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:21:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:21:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:21:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:21:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:21:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:21:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:21:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:21:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:21:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:21:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:21:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:21:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:21:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:21:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:21:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:21:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:21:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:21:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:21:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:21:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:21:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:21:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:21:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:21:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:21:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:21:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:21:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:21:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:21:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:21:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:21:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:21:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:21:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:21:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:21:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:21:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:21:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:21:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:21:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:21:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:21:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:21:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:21:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:21:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:21:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:21:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:21:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:21:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:21:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:21:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:21:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:21:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:21:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:21:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:21:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:21:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:21:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:21:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:21:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:21:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:21:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:21:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:21:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:21:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:21:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:21:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:21:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:21:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:21:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:21:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:21:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:21:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:21:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:21:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:21:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:21:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:21:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:21:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:21:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:21:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:21:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:21:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:21:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:21:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:21:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:21:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:21:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:21:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:21:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:21:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:21:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:21:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:21:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:21:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:21:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:21:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:21:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:21:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:21:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:21:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:21:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:21:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:21:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:21:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:21:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:21:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:21:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:21:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:21:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:21:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:21:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:21:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:21:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:21:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:21:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:21:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:21:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:21:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:21:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:21:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:21:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:21:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:21:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:21:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:21:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:21:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:21:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:21:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:21:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:21:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:21:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:21:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:21:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:21:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:21:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:21:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:21:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:21:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:21:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:21:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:21:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:21:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:21:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:21:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:21:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:21:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:21:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:21:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:21:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:21:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:21:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:21:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:21:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:21:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:21:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:21:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:21:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:21:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:21:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:21:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:21:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:21:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:21:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:22:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:22:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:22:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:22:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:22:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:22:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:22:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:22:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:22:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:22:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:22:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:22:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:22:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:22:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:22:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:22:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:22:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:22:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:22:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:22:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:22:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:22:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:22:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:22:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:22:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:22:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:22:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:22:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:22:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:22:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:22:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:22:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:22:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:22:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:22:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:22:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:22:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:22:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:22:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:22:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:22:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:22:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:22:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:22:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:22:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:22:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:22:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:22:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:22:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:22:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:22:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:22:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:22:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:22:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:22:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:22:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:22:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:22:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:22:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:22:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:22:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:22:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:22:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:22:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:22:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:22:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:22:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:22:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:22:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:22:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:22:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:22:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:22:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:22:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:22:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:22:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:22:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:22:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:22:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:22:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:22:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:22:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:22:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:22:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:22:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:22:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:22:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:22:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:22:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:22:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:22:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:22:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:22:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:22:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:22:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:22:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:22:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:22:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:22:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:22:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:22:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:22:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:22:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:22:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:22:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:22:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:22:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:22:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:22:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:22:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:22:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:22:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:22:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:22:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:22:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:22:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:22:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:22:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:22:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:22:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:22:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:22:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:22:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:22:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:22:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:22:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:22:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:22:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:22:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:22:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:22:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:22:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:22:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:22:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:22:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:22:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:22:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:22:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:22:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:22:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:22:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:22:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:22:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:22:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:22:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:22:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:22:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:22:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:22:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:22:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:22:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:22:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:22:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:22:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:22:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:22:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:22:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:22:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:22:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:22:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:22:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:22:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:22:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:22:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:22:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:22:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:22:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:22:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:22:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:22:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:22:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:22:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:22:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:22:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:22:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:22:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:22:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:22:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:22:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:22:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:22:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:22:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:22:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:22:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:22:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:22:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:22:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:22:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:22:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:22:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:22:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:22:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:22:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:22:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:22:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:22:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:22:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:22:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:22:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:22:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:22:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:22:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:22:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:22:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:22:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:22:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:22:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:22:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:22:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:22:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:22:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:22:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:22:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:22:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:22:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:22:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:23:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:23:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:23:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:23:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:23:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:23:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:23:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:23:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:23:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:23:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:23:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:23:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:23:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:23:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:23:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:23:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:23:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:23:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:23:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:23:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:23:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:23:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:23:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:23:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:23:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:23:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:23:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:23:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:23:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:23:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:23:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:23:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:23:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:23:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:23:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:23:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:23:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:23:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:23:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:23:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:23:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:23:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:23:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:23:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:23:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:23:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:23:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:23:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:23:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:23:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:23:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:23:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:23:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:23:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:23:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:23:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:23:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:23:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:23:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:23:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:23:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:23:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:23:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:23:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:23:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:23:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:23:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:23:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:23:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:23:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:23:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:23:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:23:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:23:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:23:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:23:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:23:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:23:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:23:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:23:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:23:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:23:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:23:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:23:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:23:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:23:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:23:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:23:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:23:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:23:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:23:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:23:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:23:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:23:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:23:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:23:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:23:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:23:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:23:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:23:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:23:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:23:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:23:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:23:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:23:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:23:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:23:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:23:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:23:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:23:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:23:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:23:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:23:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:23:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:23:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:23:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:23:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:23:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:23:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:23:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:23:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:23:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:23:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:23:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:23:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:23:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:23:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:23:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:23:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:23:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:23:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:23:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:23:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:23:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:23:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:23:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:23:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:23:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:23:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:23:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:23:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:23:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:23:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:23:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:23:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:23:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:23:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:23:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:23:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:23:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:23:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:23:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:23:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:23:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:23:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:23:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:23:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:23:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:23:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:23:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:23:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:23:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:23:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:23:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:23:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:23:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:23:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:23:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:23:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:23:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:23:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:23:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:23:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:23:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:23:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:23:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:23:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:23:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:23:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:23:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:23:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:23:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:23:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:23:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:23:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:23:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:23:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:23:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:23:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:23:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:23:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:23:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:23:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:23:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:23:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:23:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:23:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:23:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:23:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:23:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:23:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:23:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:23:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:23:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:23:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:23:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:23:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:23:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:23:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:23:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:23:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:23:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:23:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:23:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:23:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:23:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:24:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:24:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:24:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:24:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:24:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:24:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:24:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:24:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:24:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:24:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:24:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:24:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:24:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:24:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:24:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:24:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:24:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:24:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:24:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:24:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:24:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:24:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:24:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:24:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:24:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:24:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:24:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:24:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:24:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:24:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:24:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:24:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:24:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:24:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:24:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:24:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:24:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:24:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:24:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:24:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:24:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:24:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:24:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:24:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:24:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:24:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:24:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:24:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:24:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:24:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:24:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:24:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:24:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:24:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:24:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:24:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:24:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:24:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:24:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:24:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:24:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:24:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:24:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:24:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:24:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:24:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:24:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:24:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:24:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:24:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:24:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:24:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:24:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:24:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:24:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:24:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:24:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:24:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:24:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:24:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:24:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:24:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:24:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:24:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:24:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:24:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:24:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:24:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:24:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:24:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:24:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:24:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:24:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:24:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:24:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:24:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:24:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:24:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:24:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:24:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:24:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:24:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:24:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:24:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:24:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:24:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:24:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:24:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:24:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:24:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:24:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:24:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:24:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:24:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:24:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:24:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:24:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:24:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:24:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:24:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:24:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:24:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:24:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:24:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:24:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:24:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:24:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:24:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:24:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:24:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:24:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:24:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:24:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:24:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:24:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:24:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:24:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:24:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:24:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:24:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:24:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:24:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:24:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:24:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:24:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:24:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:24:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:24:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:24:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:24:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:24:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:24:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:24:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:24:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:24:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:24:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:24:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:24:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:24:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:24:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:24:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:24:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:24:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:24:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:24:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:24:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:24:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:24:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:24:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:24:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:24:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:24:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:24:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:24:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:24:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:24:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:24:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:24:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:24:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:24:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:24:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:24:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:24:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:24:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:24:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:24:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:24:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:24:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:24:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:24:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:24:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:24:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:24:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:24:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:24:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:24:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:24:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:24:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:24:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:24:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:24:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:24:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:24:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:24:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:24:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:24:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:24:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:24:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:24:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:24:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:24:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:24:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:24:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:24:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:24:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:24:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:25:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:25:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:25:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:25:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:25:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:25:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:25:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:25:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:25:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:25:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:25:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:25:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:25:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:25:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:25:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:25:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:25:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:25:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:25:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:25:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:25:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:25:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:25:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:25:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:25:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:25:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:25:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:25:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:25:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:25:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:25:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:25:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:25:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:25:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:25:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:25:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:25:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:25:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:25:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:25:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:25:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:25:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:25:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:25:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:25:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:25:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:25:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:25:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:25:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:25:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:25:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:25:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:25:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:25:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:25:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:25:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:25:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:25:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:25:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:25:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:25:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:25:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:25:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:25:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:25:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:25:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:25:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:25:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:25:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:25:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:25:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:25:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:25:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:25:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:25:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:25:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:25:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:25:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:25:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:25:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:25:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:25:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:25:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:25:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:25:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:25:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:25:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:25:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:25:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:25:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:25:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:25:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:25:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:25:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:25:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:25:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:25:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:25:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:25:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:25:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:25:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:25:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:25:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:25:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:25:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:25:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:25:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:25:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:25:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:25:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:25:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:25:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:25:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:25:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:25:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:25:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:25:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:25:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:25:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:25:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:25:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:25:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:25:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:25:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:25:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:25:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:25:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:25:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:25:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:25:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:25:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:25:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:25:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:25:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:25:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:25:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:25:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:25:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:25:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:25:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:25:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:25:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:25:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:25:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:25:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:25:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:25:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:25:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:25:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:25:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:25:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:25:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:25:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:25:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:25:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:25:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:25:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:25:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:25:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:25:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:25:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:25:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:25:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:25:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:25:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:25:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:25:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:25:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:25:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:25:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:25:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:25:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:25:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:25:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:25:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:25:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:25:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:25:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:25:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:25:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:25:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:25:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:25:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:25:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:25:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:25:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:25:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:25:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:25:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:25:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:25:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:25:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:25:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:25:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:25:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:25:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:25:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:25:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:25:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:25:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:25:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:25:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:25:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:25:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:25:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:25:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:25:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:25:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:25:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:25:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:25:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:25:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:25:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:25:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:25:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:25:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:26:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:26:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:26:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:26:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:26:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:26:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:26:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:26:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:26:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:26:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:26:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:26:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:26:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:26:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:26:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:26:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:26:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:26:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:26:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:26:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:26:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:26:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:26:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:26:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:26:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:26:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:26:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:26:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:26:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:26:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:26:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:26:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:26:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:26:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:26:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:26:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:26:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:26:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:26:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:26:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:26:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:26:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:26:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:26:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:26:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:26:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:26:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:26:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:26:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:26:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:26:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:26:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:26:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:26:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:26:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:26:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:26:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:26:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:26:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:26:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:26:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:26:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:26:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:26:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:26:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:26:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:26:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:26:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:26:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:26:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:26:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:26:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:26:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:26:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:26:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:26:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:26:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:26:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:26:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:26:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:26:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:26:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:26:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:26:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:26:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:26:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:26:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:26:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:26:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:26:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:26:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:26:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:26:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:26:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:26:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:26:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:26:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:26:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:26:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:26:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:26:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:26:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:26:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:26:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:26:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:26:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:26:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:26:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:26:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:26:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:26:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:26:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:26:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:26:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:26:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:26:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:26:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:26:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:26:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:26:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:26:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:26:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:26:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:26:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:26:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:26:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:26:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:26:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:26:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:26:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:26:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:26:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:26:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:26:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:26:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:26:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:26:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:26:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:26:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:26:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:26:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:26:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:26:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:26:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:26:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:26:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:26:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:26:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:26:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:26:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:26:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:26:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:26:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:26:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:26:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:26:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:26:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:26:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:26:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:26:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:26:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:26:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:26:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:26:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:26:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:26:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:26:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:26:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:26:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:26:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:26:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:26:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:26:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:26:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:26:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:26:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:26:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:26:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:26:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:26:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:26:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:26:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:26:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:26:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:26:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:26:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:26:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:26:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:26:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:26:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:26:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:26:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:26:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:26:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:26:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:26:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:26:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:26:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:26:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:26:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:26:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:26:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:26:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:26:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:26:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:26:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:26:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:26:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:26:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:26:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:26:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:26:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:26:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:26:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:26:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:26:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:27:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:27:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:27:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:27:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:27:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:27:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:27:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:27:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:27:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:27:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:27:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:27:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:27:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:27:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:27:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:27:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:27:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:27:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:27:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:27:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:27:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:27:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:27:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:27:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:27:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:27:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:27:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:27:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:27:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:27:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:27:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:27:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:27:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:27:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:27:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:27:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:27:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:27:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:27:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:27:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:27:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:27:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:27:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:27:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:27:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:27:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:27:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:27:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:27:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:27:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:27:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:27:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:27:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:27:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:27:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:27:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:27:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:27:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:27:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:27:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:27:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:27:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:27:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:27:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:27:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:27:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:27:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:27:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:27:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:27:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:27:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:27:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:27:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:27:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:27:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:27:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:27:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:27:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:27:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:27:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:27:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:27:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:27:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:27:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:27:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:27:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:27:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:27:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:27:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:27:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:27:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:27:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:27:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:27:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:27:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:27:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:27:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:27:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:27:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:27:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:27:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:27:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:27:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:27:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:27:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:27:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:27:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:27:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:27:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:27:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:27:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:27:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:27:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:27:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:27:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:27:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:27:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:27:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:27:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:27:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:27:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:27:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:27:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:27:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:27:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:27:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:29:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:29:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:29:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:29:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:29:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:29:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:29:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:29:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:29:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:29:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:29:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:29:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:29:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:29:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:29:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:29:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:29:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:29:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:29:41 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 23:29:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:29:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:29:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:29:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:29:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:29:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:29:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:29:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:29:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:29:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:29:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:29:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:29:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:29:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:29:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:29:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:29:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:29:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:29:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:29:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:29:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:29:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:29:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:29:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:29:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:29:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:29:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:29:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:29:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:29:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:29:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:29:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:29:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:29:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:29:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:29:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:29:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:29:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:29:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:29:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:29:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:29:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:29:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:29:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:29:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:29:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:29:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:29:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:29:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:29:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:29:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:29:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:29:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:29:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:30:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:30:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:30:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:30:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:30:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:30:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:30:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:30:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:30:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:30:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:30:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:30:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:30:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:30:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:30:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:30:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:30:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:30:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:30:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:30:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:30:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:30:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:30:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:30:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:30:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:30:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:30:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:30:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:30:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:30:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:30:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:30:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:30:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:30:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:30:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:30:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:30:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:30:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:30:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:30:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:30:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:30:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:30:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:30:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:30:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:30:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:30:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:30:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:30:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:30:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:30:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:30:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:30:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:30:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:30:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:30:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:30:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:30:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:30:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:30:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:30:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:30:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:30:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:30:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:30:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:30:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:30:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:30:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:30:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:30:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:30:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:30:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:30:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:30:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:30:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:30:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:30:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:30:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:30:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:30:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:30:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:30:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:30:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:30:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:30:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:30:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:30:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:30:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:30:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:30:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:30:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:30:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:30:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:30:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:30:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:30:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:30:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:30:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:30:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:30:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:30:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:30:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:30:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:30:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:30:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:30:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:30:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:30:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:30:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:30:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:30:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:30:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:30:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:30:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:30:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:30:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:30:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:30:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:30:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:30:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:30:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:30:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:30:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:30:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:30:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:30:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:30:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:30:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:30:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:30:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:30:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:30:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:30:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:30:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:30:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:30:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:30:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:30:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:30:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:30:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:30:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:30:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:30:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:30:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:30:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:30:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:30:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:30:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:30:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:30:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:30:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:30:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:30:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:30:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:30:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:30:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:30:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:30:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:30:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:30:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:30:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:30:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:30:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:30:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:30:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:30:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:30:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:30:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:30:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:30:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:30:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:30:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:30:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:30:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:30:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:30:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:30:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:30:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:30:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:30:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:30:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:30:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:30:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:30:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:30:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:30:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:30:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:30:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:30:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:30:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:30:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:30:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:30:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:30:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:30:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:30:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:30:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:30:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:30:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:30:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:30:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:30:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:30:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:30:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:30:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:30:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:30:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:30:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:30:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:30:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:30:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:30:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:30:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:30:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:30:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:30:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:31:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:31:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:31:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:31:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:31:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:31:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:31:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:31:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:31:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:31:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:31:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:31:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:31:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:31:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:31:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:31:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:31:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:31:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:31:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:31:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:31:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:31:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:31:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:31:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:31:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:31:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:31:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:31:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:31:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:31:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:31:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:31:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:31:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:31:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:31:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:31:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:31:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:31:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:31:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:31:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:31:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:31:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:31:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:31:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:31:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:31:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:31:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:31:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:31:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:31:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:31:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:31:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:31:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:31:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:31:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:31:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:31:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:31:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:31:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:31:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:31:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:31:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:31:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:31:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:31:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:31:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:31:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:31:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:31:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:31:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:31:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:31:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:31:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:31:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:31:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:31:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:31:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:31:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:31:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:31:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:31:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:31:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:31:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:31:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:31:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:31:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:31:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:31:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:31:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:31:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:31:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:31:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:31:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:31:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:31:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:31:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:31:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:31:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:31:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:31:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:31:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:31:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:31:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:31:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:31:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:31:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:31:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:31:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:31:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:31:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:31:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:31:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:31:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:31:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:31:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:31:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:31:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:31:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:31:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:31:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:31:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:31:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:31:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:31:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:31:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:31:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:31:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:31:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:31:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:31:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:31:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:31:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:31:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:31:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:31:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:31:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:31:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:31:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:31:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:31:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:31:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:31:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:31:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:31:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:31:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:31:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:31:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:31:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:31:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:31:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:31:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:31:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:31:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:31:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:31:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:31:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:31:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:31:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:31:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:31:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:31:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:31:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:31:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:31:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:31:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:31:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:32:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:32:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:32:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:32:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:32:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:32:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:32:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:32:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:32:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:32:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:32:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:32:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:32:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:32:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:32:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:32:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:32:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:32:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:32:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:32:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:32:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:32:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:32:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:32:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:32:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:32:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:32:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:32:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:32:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:32:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:32:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:32:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:32:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:32:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:32:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:32:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:32:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:32:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:32:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:32:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:32:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:32:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:32:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:32:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:32:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:32:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:32:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:32:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:32:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:32:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:32:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:32:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:32:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:32:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:32:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:32:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:32:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:32:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:32:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:32:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:32:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:32:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:32:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:32:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:32:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:32:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:32:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:32:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:32:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:32:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:32:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:32:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:32:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:32:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:32:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:32:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:32:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:32:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:32:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:32:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:32:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:32:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:32:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:32:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:32:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:32:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:32:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:32:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:32:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:32:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:32:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:32:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:32:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:32:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:32:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:32:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:32:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:32:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:32:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:32:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:32:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:32:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:32:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:32:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:32:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:32:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:32:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:32:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:32:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:32:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:32:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:32:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:32:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:32:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:32:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:32:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:32:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:32:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:32:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:32:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:32:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:32:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:32:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:32:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:32:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:32:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:32:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:32:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:32:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:32:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:32:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:32:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:32:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:32:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:32:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:32:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:32:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:32:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:32:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:32:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:32:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:32:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:32:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:32:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:32:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:32:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:32:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:32:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:32:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:32:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:32:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:32:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:32:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:32:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:32:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:32:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:32:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:32:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:32:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:32:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:32:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:32:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:32:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:32:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:32:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:32:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:32:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:32:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:32:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:32:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:32:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:32:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:32:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:32:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:32:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:32:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:32:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:32:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:32:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:32:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:32:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:32:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:32:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:32:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:32:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:32:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:32:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:32:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:32:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:32:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:32:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:32:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:32:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:32:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:32:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:32:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:32:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:32:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:32:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:32:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:32:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:32:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:32:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:32:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:32:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:32:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:32:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:32:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:32:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:32:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:32:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:32:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:32:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:32:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:32:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:32:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:33:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:33:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:33:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:33:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:33:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:33:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:33:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:33:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:33:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:33:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:33:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:33:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:33:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:33:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:33:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:33:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:33:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:33:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:33:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:33:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:33:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:33:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:33:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:33:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:33:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:33:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:33:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:33:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:33:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:33:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:33:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:33:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:33:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:33:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:33:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:33:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:33:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:33:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:33:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:33:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:33:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:33:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:33:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:33:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:33:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:33:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:33:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:33:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:33:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:33:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:33:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:33:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:33:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:33:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:33:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:33:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:33:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:33:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:33:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:33:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:33:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:33:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:33:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:33:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:33:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:33:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:33:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:33:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:33:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:33:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:33:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:33:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:33:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:33:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:33:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:33:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:33:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:33:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:33:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:33:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:33:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:33:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:33:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:33:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:33:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:33:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:33:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:33:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:33:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:33:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:33:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:33:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:33:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:33:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:33:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:33:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:33:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:33:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:33:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:33:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:33:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:33:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:33:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:33:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:33:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:33:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:33:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:33:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:33:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:33:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:33:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:33:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:33:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:33:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:33:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:33:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:33:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:33:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:33:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:33:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:33:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:33:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:33:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:33:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:33:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:33:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:33:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:33:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:33:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:33:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:33:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:33:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:33:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:33:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:33:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:33:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:33:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:33:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:33:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:33:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:33:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:33:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:33:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:33:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:33:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:33:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:33:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:33:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:33:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:33:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:33:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:33:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:33:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:33:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:33:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:33:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:33:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:33:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:33:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:33:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:33:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:33:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:33:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:33:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:33:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:33:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:33:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:33:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:33:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:33:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:33:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:33:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:33:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:33:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:33:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:33:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:33:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:33:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:33:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:33:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:33:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:33:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:33:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:33:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:33:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:33:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:33:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:33:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:33:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:33:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:33:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:33:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:33:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:33:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:33:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:33:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:33:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:33:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:33:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:33:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:33:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:33:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:33:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:33:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:33:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:33:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:33:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:33:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:33:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:33:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:33:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:33:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:33:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:33:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:33:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:33:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:34:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:34:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:34:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:34:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:34:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:34:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:34:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:34:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:34:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:34:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:34:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:34:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:34:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:34:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:34:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:34:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:34:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:34:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:34:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:34:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:34:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:34:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:34:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:34:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:34:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:34:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:34:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:34:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:34:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:34:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:34:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:34:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:34:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:34:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:34:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:34:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:34:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:34:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:34:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:34:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:34:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:34:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:34:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:34:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:34:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:34:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:34:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:34:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:34:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:34:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:34:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:34:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:34:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:34:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:34:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:34:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:34:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:34:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:34:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:34:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:34:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:34:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:34:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:34:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:34:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:34:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:34:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:34:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:34:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:34:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:34:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:34:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:34:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:34:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:34:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:34:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:34:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:34:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:34:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:34:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:34:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:34:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:34:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:34:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:34:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:34:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:34:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:34:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:34:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:34:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:34:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:34:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:34:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:34:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:34:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:34:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:34:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:34:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:34:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:34:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:34:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:34:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:34:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:34:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:34:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:34:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:34:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:34:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:34:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:34:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:34:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:34:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:34:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:34:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:34:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:34:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:34:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:34:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:34:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:34:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:34:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:34:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:34:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:34:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:34:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:34:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:34:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:34:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:34:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:34:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:34:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:34:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:34:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:34:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:34:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:34:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:34:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:34:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:34:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:34:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:34:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:34:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:34:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:34:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:34:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:34:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:34:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:34:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:34:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:34:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:34:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:34:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:34:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:34:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:34:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:34:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:34:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:34:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:34:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:34:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:34:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:34:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:34:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:34:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:34:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:34:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:34:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:34:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:34:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:34:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:34:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:34:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:34:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:34:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:34:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:34:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:34:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:34:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:34:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:34:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:34:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:34:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:34:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:34:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:34:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:34:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:34:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:34:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:34:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:34:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:34:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:34:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:34:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:34:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:34:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:34:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:34:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:34:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:34:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:34:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:34:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:34:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:34:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:34:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:34:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:34:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:34:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:34:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:34:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:34:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:34:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:34:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:34:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:34:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:34:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:34:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:35:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:35:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:35:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:35:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:35:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:35:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:35:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:35:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:35:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:35:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:35:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:35:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:35:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:35:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:35:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:35:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:35:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:35:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:35:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:35:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:35:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:35:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:35:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:35:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:35:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:35:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:35:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:35:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:35:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:35:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:35:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:35:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:35:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:35:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:35:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:35:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:35:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:35:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:35:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:35:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:35:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:35:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:35:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:35:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:35:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:35:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:35:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:35:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:35:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:35:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:35:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:35:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:35:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:35:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:35:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:35:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:35:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:35:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:35:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:35:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:35:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:35:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:35:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:35:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:35:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:35:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:35:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:35:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:35:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:35:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:35:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:35:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:35:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:35:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:35:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:35:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:35:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:35:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:35:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:35:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:35:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:35:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:35:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:35:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:35:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:35:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:35:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:35:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:35:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:35:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:35:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:35:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:35:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:35:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:35:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:35:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:35:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:35:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:35:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:35:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:35:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:35:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:35:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:35:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:35:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:35:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:35:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:35:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:35:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:35:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:35:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:35:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:35:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:35:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:35:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:35:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:35:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:35:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:35:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:35:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:35:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:35:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:35:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:35:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:35:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:35:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:35:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:35:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:35:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:35:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:35:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:35:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:35:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:35:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:35:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:35:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:35:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:35:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:35:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:35:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:35:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:35:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:35:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:35:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:35:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:35:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:35:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:35:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:35:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:35:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:35:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:35:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:35:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:35:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:35:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:35:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:35:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:35:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:35:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:35:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:35:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:35:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:35:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:35:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:35:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:35:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:35:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:35:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:35:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:35:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:35:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:35:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:35:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:35:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:35:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:35:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:35:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:35:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:35:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:35:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:35:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:35:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:35:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:35:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:35:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:35:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:35:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:35:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:35:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:35:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:35:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:35:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:35:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:35:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:35:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:35:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:35:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:35:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:35:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:35:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:35:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:35:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:35:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:35:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:35:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:35:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:35:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:35:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:35:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:35:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:35:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:35:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:35:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:35:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:35:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:35:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:36:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:36:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:36:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:36:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:36:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:36:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:36:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:36:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:36:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:36:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:36:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:36:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:36:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:36:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:36:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:36:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:36:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:36:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:36:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:36:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:36:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:36:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:36:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:36:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:36:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:36:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:36:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:36:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:36:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:36:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:36:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:36:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:36:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:36:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:36:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:36:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:36:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:36:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:36:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:36:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:36:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:36:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:36:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:36:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:36:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:36:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:36:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:36:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:36:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:36:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:36:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:36:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:36:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:36:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:36:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:36:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:36:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:36:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:36:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:36:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:36:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:36:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:36:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:36:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:36:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:36:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:36:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:36:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:36:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:36:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:36:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:36:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:36:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:36:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:36:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:36:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:36:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:36:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:36:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:36:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:36:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:36:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:36:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:36:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:36:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:36:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:36:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:36:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:36:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:36:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:36:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:36:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:36:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:36:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:36:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:36:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:36:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:36:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:36:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:36:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:36:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:36:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:36:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:36:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:36:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:36:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:36:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:36:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:36:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:36:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:36:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:36:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:36:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:36:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:36:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:36:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:36:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:36:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:36:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:36:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:36:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:36:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:36:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:36:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:36:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:36:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:36:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:36:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:36:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:36:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:36:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:36:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:36:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:36:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:36:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:36:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:36:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:36:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:36:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:36:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:36:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:36:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:36:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:36:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:36:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:36:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:36:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:36:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:36:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:36:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:36:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:36:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:36:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:36:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:36:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:36:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:36:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:36:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:36:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:36:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:36:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:36:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:36:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:36:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:36:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:36:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:36:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:36:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:36:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:36:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:36:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:36:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:36:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:36:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:36:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:36:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:36:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:36:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:36:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:36:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:36:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:36:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:36:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:36:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:36:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:36:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:36:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:36:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:36:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:36:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:36:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:36:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:36:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:36:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:36:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:36:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:36:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:36:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:36:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:36:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:36:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:36:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:36:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:36:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:36:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:36:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:36:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:36:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:36:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:36:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:36:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:36:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:36:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:36:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:36:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:36:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:37:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:37:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:37:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:37:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:37:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:37:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:37:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:37:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:37:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:37:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:37:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:37:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:37:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:37:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:37:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:37:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:37:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:37:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:37:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:37:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:37:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:37:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:37:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:37:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:37:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:37:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:37:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:37:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:37:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:37:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:37:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:37:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:37:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:37:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:37:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:37:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:37:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:37:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:37:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:37:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:37:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:37:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:37:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:37:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:37:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:37:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:37:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:37:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:37:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:37:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:37:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:37:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:37:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:37:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:37:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:37:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:37:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:37:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:37:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:37:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:37:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:37:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:37:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:37:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:37:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:37:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:37:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:37:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:37:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:37:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:37:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:37:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:37:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:37:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:37:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:37:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:37:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:37:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:37:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:37:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:37:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:37:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:37:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:37:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:37:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:37:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:37:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:37:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:37:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:37:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:37:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:37:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:37:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:37:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:37:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:37:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:37:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:37:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:37:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:37:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:37:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:37:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:37:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:37:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:37:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:37:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:37:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:37:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:37:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:37:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:37:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:37:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:37:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:37:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:37:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:37:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:37:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:37:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:37:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:37:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:37:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:37:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:37:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:37:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:37:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:37:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:37:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:37:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:37:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:37:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:37:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:37:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:37:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:37:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:37:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:37:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:37:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:37:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:37:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:37:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:37:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:37:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:37:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:37:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:39:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:39:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:39:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:39:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:39:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:39:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:39:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:39:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:39:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:39:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:39:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:39:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:39:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:39:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:39:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:39:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:39:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:39:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:39:42 rdma06 rsyslogd[1041]: imjournal: 5082 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 23:39:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:39:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:39:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:39:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:39:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:39:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:39:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:39:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:39:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:39:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:39:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:39:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:39:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:39:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:39:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:39:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:39:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:39:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:39:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:39:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:39:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:39:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:39:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:39:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:39:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:39:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:39:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:39:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:39:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:39:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:39:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:39:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:39:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:39:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:39:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:39:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:39:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:39:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:39:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:39:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:39:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:39:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:39:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:39:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:39:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:39:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:39:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:39:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:39:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:39:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:39:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:39:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:39:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:39:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:40:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:40:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:40:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:40:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:40:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:40:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:40:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:40:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:40:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:40:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:40:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:40:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:40:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:40:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:40:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:40:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:40:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:40:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:40:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:40:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:40:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:40:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:40:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:40:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:40:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:40:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:40:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:40:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:40:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:40:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:40:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:40:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:40:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:40:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:40:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:40:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:40:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:40:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:40:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:40:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:40:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:40:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:40:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:40:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:40:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:40:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:40:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:40:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:40:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:40:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:40:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:40:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:40:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:40:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:40:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:40:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:40:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:40:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:40:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:40:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:40:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:40:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:40:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:40:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:40:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:40:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:40:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:40:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:40:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:40:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:40:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:40:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:40:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:40:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:40:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:40:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:40:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:40:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:40:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:40:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:40:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:40:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:40:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:40:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:40:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:40:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:40:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:40:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:40:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:40:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:40:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:40:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:40:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:40:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:40:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:40:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:40:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:40:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:40:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:40:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:40:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:40:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:40:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:40:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:40:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:40:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:40:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:40:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:40:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:40:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:40:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:40:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:40:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:40:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:40:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:40:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:40:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:40:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:40:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:40:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:40:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:40:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:40:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:40:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:40:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:40:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:40:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:40:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:40:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:40:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:40:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:40:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:40:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:40:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:40:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:40:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:40:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:40:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:40:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:40:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:40:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:40:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:40:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:40:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:40:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:40:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:40:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:40:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:40:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:40:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:40:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:40:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:40:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:40:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:40:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:40:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:40:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:40:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:40:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:40:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:40:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:40:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:40:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:40:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:40:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:40:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:40:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:40:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:40:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:40:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:40:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:40:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:40:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:40:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:40:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:40:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:40:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:40:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:40:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:40:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:40:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:40:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:40:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:40:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:40:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:40:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:40:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:40:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:40:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:40:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:40:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:40:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:40:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:40:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:40:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:40:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:40:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:40:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:40:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:40:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:40:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:40:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:40:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:40:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:40:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:40:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:40:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:40:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:40:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:40:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:40:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:40:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:40:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:40:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:40:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:40:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:41:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:41:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:41:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:41:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:41:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:41:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:41:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:41:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:41:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:41:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:41:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:41:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:41:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:41:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:41:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:41:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:41:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:41:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:41:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:41:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:41:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:41:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:41:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:41:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:41:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:41:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:41:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:41:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:41:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:41:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:41:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:41:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:41:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:41:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:41:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:41:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:41:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:41:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:41:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:41:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:41:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:41:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:41:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:41:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:41:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:41:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:41:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:41:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:41:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:41:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:41:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:41:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:41:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:41:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:41:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:41:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:41:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:41:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:41:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:41:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:41:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:41:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:41:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:41:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:41:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:41:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:41:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:41:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:41:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:41:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:41:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:41:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:41:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:41:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:41:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:41:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:41:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:41:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:41:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:41:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:41:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:41:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:41:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:41:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:41:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:41:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:41:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:41:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:41:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:41:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:41:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:41:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:41:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:41:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:41:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:41:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:41:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:41:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:41:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:41:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:41:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:41:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:41:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:41:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:41:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:41:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:41:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:41:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:41:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:41:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:41:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:41:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:41:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:41:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:41:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:41:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:41:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:41:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:41:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:41:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:41:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:41:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:41:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:41:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:41:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:41:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:41:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:41:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:41:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:41:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:41:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:41:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:41:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:41:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:41:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:41:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:41:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:41:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:41:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:41:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:41:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:41:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:41:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:41:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:41:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:41:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:41:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:41:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:41:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:41:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:41:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:41:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:41:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:41:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:41:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:41:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:41:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:41:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:41:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:41:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:41:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:41:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:41:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:41:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:41:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:41:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:41:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:41:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:41:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:41:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:41:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:41:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:41:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:41:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:41:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:41:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:41:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:41:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:41:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:41:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:41:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:41:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:41:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:41:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:41:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:41:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:41:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:41:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:41:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:41:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:41:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:41:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:41:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:41:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:41:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:41:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:41:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:41:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:41:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:41:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:41:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:41:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:41:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:41:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:41:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:41:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:41:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:41:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:41:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:41:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:41:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:41:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:41:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:41:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:41:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:41:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:42:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:42:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:42:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:42:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:42:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:42:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:42:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:42:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:42:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:42:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:42:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:42:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:42:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:42:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:42:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:42:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:42:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:42:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:42:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:42:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:42:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:42:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:42:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:42:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:42:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:42:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:42:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:42:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:42:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:42:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:42:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:42:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:42:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:42:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:42:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:42:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:42:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:42:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:42:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:42:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:42:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:42:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:42:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:42:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:42:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:42:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:42:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:42:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:42:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:42:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:42:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:42:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:42:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:42:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:42:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:42:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:42:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:42:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:42:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:42:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:42:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:42:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:42:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:42:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:42:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:42:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:42:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:42:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:42:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:42:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:42:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:42:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:42:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:42:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:42:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:42:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:42:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:42:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:42:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:42:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:42:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:42:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:42:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:42:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:42:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:42:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:42:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:42:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:42:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:42:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:42:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:42:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:42:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:42:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:42:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:42:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:42:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:42:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:42:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:42:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:42:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:42:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:42:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:42:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:42:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:42:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:42:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:42:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:42:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:42:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:42:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:42:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:42:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:42:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:42:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:42:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:42:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:42:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:42:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:42:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:42:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:42:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:42:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:42:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:42:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:42:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:42:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:42:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:42:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:42:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:42:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:42:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:42:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:42:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:42:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:42:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:42:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:42:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:42:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:42:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:42:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:42:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:42:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:42:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:42:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:42:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:42:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:42:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:42:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:42:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:42:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:42:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:42:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:42:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:42:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:42:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:42:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:42:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:42:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:42:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:42:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:42:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:42:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:42:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:42:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:42:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:42:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:42:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:42:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:42:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:42:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:42:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:42:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:42:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:42:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:42:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:42:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:42:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:42:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:42:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:42:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:42:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:42:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:42:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:42:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:42:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:42:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:42:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:42:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:42:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:42:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:42:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:42:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:42:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:42:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:42:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:42:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:42:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:42:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:42:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:42:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:42:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:42:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:42:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:42:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:42:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:42:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:42:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:42:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:42:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:42:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:42:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:42:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:42:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:42:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:42:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:43:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:43:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:43:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:43:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:43:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:43:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:43:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:43:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:43:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:43:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:43:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:43:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:43:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:43:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:43:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:43:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:43:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:43:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:43:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:43:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:43:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:43:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:43:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:43:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:43:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:43:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:43:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:43:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:43:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:43:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:43:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:43:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:43:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:43:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:43:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:43:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:43:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:43:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:43:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:43:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:43:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:43:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:43:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:43:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:43:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:43:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:43:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:43:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:43:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:43:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:43:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:43:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:43:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:43:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:43:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:43:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:43:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:43:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:43:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:43:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:43:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:43:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:43:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:43:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:43:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:43:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:43:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:43:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:43:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:43:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:43:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:43:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:43:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:43:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:43:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:43:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:43:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:43:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:43:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:43:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:43:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:43:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:43:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:43:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:43:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:43:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:43:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:43:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:43:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:43:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:43:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:43:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:43:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:43:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:43:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:43:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:43:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:43:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:43:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:43:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:43:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:43:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:43:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:43:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:43:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:43:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:43:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:43:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:43:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:43:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:43:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:43:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:43:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:43:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:43:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:43:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:43:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:43:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:43:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:43:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:43:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:43:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:43:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:43:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:43:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:43:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:43:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:43:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:43:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:43:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:43:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:43:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:43:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:43:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:43:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:43:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:43:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:43:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:43:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:43:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:43:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:43:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:43:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:43:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:43:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:43:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:43:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:43:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:43:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:43:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:43:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:43:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:43:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:43:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:43:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:43:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:43:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:43:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:43:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:43:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:43:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:43:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:43:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:43:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:43:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:43:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:43:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:43:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:43:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:43:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:43:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:43:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:43:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:43:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:43:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:43:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:43:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:43:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:43:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:43:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:43:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:43:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:43:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:43:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:43:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:43:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:43:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:43:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:43:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:43:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:43:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:43:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:43:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:43:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:43:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:43:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:43:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:43:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:43:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:43:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:43:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:43:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:43:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:43:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:43:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:43:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:43:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:43:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:43:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:43:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:43:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:43:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:43:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:43:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:43:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:43:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:44:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:44:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:44:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:44:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:44:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:44:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:44:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:44:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:44:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:44:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:44:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:44:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:44:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:44:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:44:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:44:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:44:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:44:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:44:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:44:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:44:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:44:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:44:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:44:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:44:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:44:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:44:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:44:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:44:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:44:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:44:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:44:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:44:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:44:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:44:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:44:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:44:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:44:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:44:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:44:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:44:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:44:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:44:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:44:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:44:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:44:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:44:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:44:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:44:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:44:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:44:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:44:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:44:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:44:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:44:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:44:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:44:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:44:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:44:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:44:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:44:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:44:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:44:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:44:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:44:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:44:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:44:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:44:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:44:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:44:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:44:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:44:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:44:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:44:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:44:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:44:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:44:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:44:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:44:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:44:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:44:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:44:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:44:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:44:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:44:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:44:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:44:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:44:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:44:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:44:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:44:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:44:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:44:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:44:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:44:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:44:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:44:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:44:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:44:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:44:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:44:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:44:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:44:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:44:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:44:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:44:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:44:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:44:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:44:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:44:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:44:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:44:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:44:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:44:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:44:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:44:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:44:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:44:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:44:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:44:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:44:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:44:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:44:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:44:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:44:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:44:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:44:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:44:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:44:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:44:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:44:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:44:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:44:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:44:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:44:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:44:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:44:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:44:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:44:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:44:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:44:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:44:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:44:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:44:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:44:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:44:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:44:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:44:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:44:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:44:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:44:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:44:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:44:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:44:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:44:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:44:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:44:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:44:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:44:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:44:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:44:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:44:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:44:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:44:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:44:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:44:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:44:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:44:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:44:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:44:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:44:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:44:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:44:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:44:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:44:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:44:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:44:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:44:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:44:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:44:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:44:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:44:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:44:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:44:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:44:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:44:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:44:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:44:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:44:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:44:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:44:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:44:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:44:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:44:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:44:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:44:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:44:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:44:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:44:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:44:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:44:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:44:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:44:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:44:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:44:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:44:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:44:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:44:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:44:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:44:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:44:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:44:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:44:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:44:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:44:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:44:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:45:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:45:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:45:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:45:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:45:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:45:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:45:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:45:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:45:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:45:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:45:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:45:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:45:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:45:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:45:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:45:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:45:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:45:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:45:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:45:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:45:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:45:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:45:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:45:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:45:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:45:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:45:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:45:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:45:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:45:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:45:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:45:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:45:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:45:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:45:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:45:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:45:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:45:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:45:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:45:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:45:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:45:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:45:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:45:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:45:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:45:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:45:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:45:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:45:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:45:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:45:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:45:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:45:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:45:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:45:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:45:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:45:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:45:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:45:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:45:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:45:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:45:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:45:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:45:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:45:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:45:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:45:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:45:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:45:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:45:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:45:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:45:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:45:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:45:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:45:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:45:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:45:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:45:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:45:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:45:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:45:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:45:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:45:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:45:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:45:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:45:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:45:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:45:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:45:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:45:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:45:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:45:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:45:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:45:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:45:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:45:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:45:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:45:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:45:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:45:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:45:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:45:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:45:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:45:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:45:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:45:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:45:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:45:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:45:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:45:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:45:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:45:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:45:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:45:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:45:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:45:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:45:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:45:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:45:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:45:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:45:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:45:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:45:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:45:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:45:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:45:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:45:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:45:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:45:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:45:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:45:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:45:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:45:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:45:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:45:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:45:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:45:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:45:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:45:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:45:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:45:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:45:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:45:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:45:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:45:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:45:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:45:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:45:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:45:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:45:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:45:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:45:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:45:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:45:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:45:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:45:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:45:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:45:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:45:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:45:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:45:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:45:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:45:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:45:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:45:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:45:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:45:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:45:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:45:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:45:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:45:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:45:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:45:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:45:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:45:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:45:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:45:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:45:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:45:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:45:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:45:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:45:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:45:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:45:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:45:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:45:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:45:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:45:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:45:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:45:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:45:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:45:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:45:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:45:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:45:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:45:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:45:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:45:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:45:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:45:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:45:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:45:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:45:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:45:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:45:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:45:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:45:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:45:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:45:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:45:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:45:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:45:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:45:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:45:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:45:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:45:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:46:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:46:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:46:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:46:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:46:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:46:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:46:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:46:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:46:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:46:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:46:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:46:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:46:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:46:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:46:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:46:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:46:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:46:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:46:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:46:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:46:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:46:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:46:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:46:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:46:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:46:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:46:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:46:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:46:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:46:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:46:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:46:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:46:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:46:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:46:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:46:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:46:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:46:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:46:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:46:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:46:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:46:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:46:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:46:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:46:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:46:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:46:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:46:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:46:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:46:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:46:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:46:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:46:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:46:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:46:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:46:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:46:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:46:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:46:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:46:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:46:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:46:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:46:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:46:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:46:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:46:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:46:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:46:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:46:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:46:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:46:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:46:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:46:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:46:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:46:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:46:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:46:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:46:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:46:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:46:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:46:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:46:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:46:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:46:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:46:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:46:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:46:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:46:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:46:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:46:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:46:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:46:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:46:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:46:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:46:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:46:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:46:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:46:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:46:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:46:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:46:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:46:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:46:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:46:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:46:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:46:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:46:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:46:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:46:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:46:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:46:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:46:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:46:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:46:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:46:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:46:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:46:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:46:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:46:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:46:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:46:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:46:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:46:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:46:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:46:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:46:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:46:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:46:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:46:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:46:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:46:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:46:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:46:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:46:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:46:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:46:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:46:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:46:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:46:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:46:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:46:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:46:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:46:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:46:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:46:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:46:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:46:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:46:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:46:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:46:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:46:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:46:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:46:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:46:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:46:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:46:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:46:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:46:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:46:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:46:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:46:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:46:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:46:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:46:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:46:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:46:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:46:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:46:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:46:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:46:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:46:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:46:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:46:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:46:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:46:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:46:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:46:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:46:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:46:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:46:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:46:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:46:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:46:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:46:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:46:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:46:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:46:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:46:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:46:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:46:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:46:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:46:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:46:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:46:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:46:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:46:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:46:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:46:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:46:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:46:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:46:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:46:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:46:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:46:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:46:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:46:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:46:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:46:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:46:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:46:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:46:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:46:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:46:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:46:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:46:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:46:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:47:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:47:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:47:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:47:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:47:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:47:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:47:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:47:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:47:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:47:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:47:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:47:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:47:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:47:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:47:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:47:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:47:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:47:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:47:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:47:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:47:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:47:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:47:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:47:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:47:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:47:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:47:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:47:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:47:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:47:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:47:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:47:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:47:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:47:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:47:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:47:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:47:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:47:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:47:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:47:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:47:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:47:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:47:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:47:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:47:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:47:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:47:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:47:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:47:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:47:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:47:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:47:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:47:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:47:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:47:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:47:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:47:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:47:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:47:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:47:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:47:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:47:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:47:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:47:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:47:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:47:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:47:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:47:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:47:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:47:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:47:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:47:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:47:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:47:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:47:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:47:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:47:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:47:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:47:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:47:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:47:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:47:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:47:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:47:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:47:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:47:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:47:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:47:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:47:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:47:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:47:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:47:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:47:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:47:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:47:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:47:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:47:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:47:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:47:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:47:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:47:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:47:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:47:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:47:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:47:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:47:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:47:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:47:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:47:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:47:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:47:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:47:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:47:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:47:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:47:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:47:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:47:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:47:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:47:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:47:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:47:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:47:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:47:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:47:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:47:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:47:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:47:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:47:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:47:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:47:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:47:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:47:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:47:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:47:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:47:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:47:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:47:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:47:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:47:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:47:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:47:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:47:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:47:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:47:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:49:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:49:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:49:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:49:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:49:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:49:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:49:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:49:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:49:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:49:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:49:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:49:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:49:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:49:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:49:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:49:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:49:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:49:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:49:47 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 23:49:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:49:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:49:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:49:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:49:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:49:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:49:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:49:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:49:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:49:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:49:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:49:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:49:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:49:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:49:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:49:52 rdma06 systemd-journald[790]: Data hash table of /run/log/journal/95379d2f723847b68d9e976dee8967a9/system.journal has a fill level at 75.0 (163750 of 218332 items, 75497472 file size, 461 bytes per hash table item), suggesting rotation.
Apr 18 23:49:52 rdma06 systemd-journald[790]: /run/log/journal/95379d2f723847b68d9e976dee8967a9/system.journal: Journal header limits reached or header out-of-date, rotating.
Apr 18 23:49:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:49:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:49:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:49:52 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 18 23:49:52 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 18 23:49:52 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 18 23:49:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:49:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:49:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:49:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:49:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:49:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:49:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:49:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:49:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:49:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:49:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:49:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:49:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:49:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:49:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:49:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:49:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:49:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:50:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:50:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:50:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:50:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:50:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:50:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:50:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:50:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:50:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:50:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:50:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:50:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:50:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:50:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:50:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:50:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:50:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:50:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:50:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:50:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:50:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:50:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:50:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:50:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:50:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:50:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:50:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:50:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:50:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:50:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:50:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:50:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:50:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:50:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:50:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:50:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:50:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:50:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:50:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:50:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:50:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:50:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:50:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:50:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:50:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:50:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:50:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:50:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:50:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:50:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:50:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:50:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:50:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:50:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:50:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:50:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:50:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:50:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:50:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:50:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:50:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:50:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:50:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:50:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:50:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:50:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:50:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:50:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:50:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:50:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:50:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:50:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:50:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:50:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:50:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:50:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:50:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:50:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:50:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:50:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:50:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:50:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:50:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:50:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:50:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:50:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:50:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:50:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:50:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:50:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:50:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:50:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:50:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:50:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:50:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:50:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:50:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:50:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:50:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:50:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:50:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:50:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:50:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:50:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:50:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:50:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:50:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:50:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:50:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:50:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:50:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:50:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:50:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:50:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:50:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:50:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:50:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:50:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:50:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:50:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:50:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:50:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:50:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:50:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:50:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:50:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:50:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:50:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:50:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:50:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:50:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:50:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:50:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:50:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:50:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:50:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:50:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:50:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:50:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:50:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:50:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:50:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:50:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:50:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:50:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:50:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:50:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:50:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:50:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:50:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:50:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:50:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:50:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:50:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:50:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:50:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:50:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:50:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:50:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:50:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:50:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:50:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:50:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:50:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:50:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:50:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:50:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:50:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:50:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:50:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:50:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:50:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:50:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:50:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:50:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:50:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:50:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:50:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:50:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:50:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:50:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:50:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:50:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:50:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:50:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:50:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:50:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:50:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:50:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:50:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:50:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:50:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:50:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:50:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:50:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:50:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:50:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:50:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:50:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:50:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:50:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:50:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:50:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:50:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:50:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:50:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:50:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:50:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:50:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:50:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:50:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:50:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:50:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:50:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:50:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:50:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:51:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:51:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:51:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:51:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:51:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:51:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:51:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:51:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:51:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:51:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:51:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:51:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:51:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:51:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:51:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:51:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:51:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:51:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:51:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:51:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:51:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:51:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:51:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:51:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:51:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:51:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:51:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:51:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:51:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:51:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:51:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:51:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:51:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:51:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:51:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:51:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:51:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:51:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:51:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:51:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:51:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:51:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:51:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:51:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:51:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:51:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:51:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:51:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:51:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:51:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:51:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:51:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:51:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:51:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:51:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:51:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:51:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:51:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:51:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:51:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:51:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:51:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:51:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:51:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:51:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:51:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:51:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:51:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:51:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:51:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:51:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:51:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:51:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:51:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:51:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:51:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:51:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:51:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:51:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:51:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:51:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:51:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:51:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:51:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:51:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:51:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:51:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:51:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:51:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:51:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:51:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:51:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:51:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:51:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:51:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:51:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:51:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:51:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:51:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:51:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:51:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:51:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:51:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:51:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:51:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:51:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:51:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:51:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:51:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:51:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:51:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:51:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:51:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:51:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:51:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:51:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:51:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:51:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:51:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:51:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:51:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:51:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:51:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:51:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:51:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:51:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:51:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:51:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:51:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:51:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:51:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:51:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:51:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:51:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:51:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:51:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:51:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:51:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:51:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:51:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:51:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:51:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:51:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:51:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:51:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:51:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:51:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:51:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:51:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:51:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:51:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:51:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:51:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:51:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:51:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:51:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:51:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:51:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:51:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:51:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:51:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:51:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:51:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:51:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:51:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:51:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:51:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:51:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:51:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:51:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:51:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:51:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:51:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:51:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:51:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:51:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:51:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:51:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:51:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:51:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:51:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:51:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:51:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:51:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:51:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:51:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:51:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:51:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:51:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:51:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:51:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:51:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:51:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:51:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:51:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:51:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:51:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:51:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:51:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:51:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:51:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:51:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:51:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:51:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:51:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:51:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:51:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:51:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:51:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:51:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:51:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:51:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:51:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:51:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:51:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:51:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:52:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:52:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:52:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:52:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:52:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:52:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:52:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:52:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:52:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:52:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:52:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:52:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:52:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:52:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:52:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:52:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:52:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:52:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:52:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:52:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:52:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:52:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:52:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:52:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:52:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:52:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:52:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:52:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:52:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:52:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:52:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:52:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:52:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:52:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:52:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:52:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:52:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:52:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:52:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:52:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:52:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:52:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:52:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:52:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:52:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:52:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:52:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:52:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:52:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:52:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:52:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:52:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:52:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:52:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:52:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:52:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:52:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:52:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:52:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:52:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:52:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:52:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:52:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:52:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:52:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:52:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:52:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:52:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:52:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:52:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:52:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:52:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:52:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:52:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:52:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:52:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:52:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:52:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:52:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:52:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:52:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:52:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:52:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:52:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:52:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:52:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:52:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:52:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:52:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:52:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:52:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:52:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:52:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:52:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:52:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:52:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:52:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:52:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:52:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:52:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:52:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:52:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:52:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:52:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:52:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:52:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:52:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:52:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:52:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:52:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:52:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:52:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:52:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:52:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:52:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:52:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:52:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:52:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:52:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:52:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:52:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:52:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:52:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:52:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:52:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:52:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:52:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:52:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:52:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:52:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:52:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:52:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:52:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:52:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:52:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:52:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:52:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:52:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:52:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:52:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:52:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:52:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:52:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:52:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:52:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:52:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:52:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:52:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:52:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:52:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:52:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:52:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:52:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:52:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:52:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:52:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:52:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:52:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:52:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:52:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:52:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:52:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:52:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:52:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:52:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:52:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:52:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:52:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:52:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:52:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:52:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:52:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:52:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:52:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:52:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:52:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:52:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:52:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:52:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:52:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:52:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:52:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:52:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:52:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:52:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:52:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:52:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:52:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:52:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:52:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:52:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:52:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:52:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:52:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:52:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:52:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:52:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:52:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:52:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:52:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:52:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:52:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:52:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:52:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:52:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:52:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:52:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:52:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:52:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:52:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:52:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:52:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:52:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:52:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:52:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:52:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:53:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:53:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:53:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:53:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:53:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:53:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:53:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:53:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:53:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:53:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:53:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:53:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:53:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:53:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:53:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:53:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:53:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:53:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:53:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:53:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:53:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:53:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:53:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:53:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:53:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:53:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:53:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:53:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:53:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:53:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:53:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:53:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:53:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:53:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:53:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:53:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:53:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:53:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:53:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:53:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:53:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:53:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:53:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:53:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:53:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:53:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:53:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:53:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:53:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:53:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:53:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:53:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:53:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:53:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:53:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:53:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:53:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:53:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:53:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:53:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:53:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:53:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:53:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:53:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:53:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:53:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:53:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:53:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:53:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:53:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:53:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:53:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:53:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:53:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:53:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:53:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:53:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:53:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:53:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:53:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:53:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:53:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:53:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:53:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:53:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:53:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:53:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:53:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:53:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:53:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:53:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:53:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:53:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:53:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:53:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:53:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:53:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:53:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:53:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:53:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:53:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:53:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:53:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:53:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:53:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:53:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:53:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:53:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:53:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:53:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:53:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:53:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:53:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:53:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:53:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:53:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:53:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:53:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:53:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:53:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:53:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:53:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:53:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:53:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:53:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:53:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:53:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:53:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:53:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:53:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:53:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:53:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:53:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:53:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:53:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:53:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:53:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:53:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:53:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:53:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:53:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:53:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:53:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:53:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:53:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:53:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:53:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:53:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:53:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:53:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:53:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:53:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:53:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:53:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:53:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:53:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:53:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:53:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:53:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:53:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:53:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:53:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:53:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:53:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:53:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:53:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:53:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:53:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:53:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:53:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:53:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:53:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:53:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:53:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:53:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:53:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:53:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:53:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:53:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:53:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:53:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:53:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:53:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:53:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:53:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:53:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:53:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:53:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:53:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:53:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:53:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:53:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:53:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:53:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:53:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:53:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:53:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:53:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:53:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:53:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:53:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:53:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:53:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:53:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:53:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:53:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:53:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:53:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:53:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:53:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:53:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:53:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:53:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:53:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:53:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:53:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:54:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:54:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:54:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:54:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:54:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:54:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:54:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:54:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:54:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:54:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:54:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:54:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:54:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:54:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:54:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:54:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:54:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:54:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:54:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:54:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:54:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:54:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:54:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:54:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:54:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:54:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:54:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:54:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:54:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:54:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:54:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:54:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:54:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:54:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:54:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:54:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:54:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:54:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:54:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:54:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:54:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:54:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:54:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:54:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:54:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:54:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:54:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:54:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:54:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:54:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:54:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:54:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:54:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:54:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:54:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:54:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:54:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:54:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:54:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:54:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:54:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:54:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:54:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:54:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:54:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:54:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:54:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:54:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:54:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:54:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:54:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:54:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:54:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:54:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:54:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:54:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:54:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:54:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:54:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:54:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:54:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:54:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:54:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:54:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:54:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:54:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:54:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:54:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:54:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:54:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:54:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:54:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:54:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:54:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:54:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:54:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:54:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:54:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:54:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:54:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:54:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:54:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:54:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:54:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:54:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:54:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:54:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:54:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:54:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:54:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:54:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:54:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:54:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:54:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:54:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:54:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:54:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:54:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:54:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:54:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:54:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:54:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:54:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:54:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:54:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:54:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:54:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:54:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:54:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:54:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:54:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:54:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:54:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:54:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:54:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:54:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:54:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:54:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:54:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:54:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:54:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:54:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:54:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:54:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:54:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:54:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:54:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:54:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:54:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:54:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:54:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:54:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:54:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:54:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:54:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:54:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:54:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:54:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:54:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:54:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:54:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:54:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:54:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:54:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:54:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:54:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:54:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:54:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:54:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:54:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:54:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:54:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:54:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:54:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:54:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:54:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:54:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:54:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:54:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:54:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:54:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:54:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:54:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:54:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:54:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:54:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:54:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:54:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:54:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:54:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:54:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:54:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:54:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:54:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:54:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:54:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:54:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:54:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:54:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:54:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:54:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:54:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:54:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:54:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:54:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:54:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:54:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:54:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:54:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:54:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:54:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:54:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:54:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:54:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:54:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:54:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:55:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:55:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:55:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:55:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:55:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:55:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:55:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:55:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:55:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:55:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:55:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:55:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:55:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:55:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:55:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:55:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:55:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:55:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:55:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:55:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:55:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:55:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:55:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:55:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:55:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:55:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:55:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:55:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:55:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:55:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:55:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:55:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:55:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:55:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:55:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:55:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:55:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:55:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:55:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:55:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:55:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:55:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:55:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:55:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:55:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:55:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:55:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:55:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:55:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:55:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:55:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:55:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:55:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:55:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:55:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:55:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:55:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:55:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:55:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:55:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:55:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:55:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:55:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:55:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:55:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:55:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:55:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:55:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:55:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:55:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:55:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:55:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:55:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:55:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:55:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:55:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:55:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:55:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:55:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:55:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:55:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:55:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:55:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:55:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:55:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:55:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:55:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:55:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:55:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:55:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:55:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:55:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:55:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:55:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:55:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:55:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:55:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:55:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:55:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:55:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:55:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:55:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:55:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:55:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:55:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:55:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:55:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:55:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:55:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:55:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:55:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:55:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:55:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:55:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:55:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:55:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:55:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:55:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:55:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:55:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:55:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:55:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:55:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:55:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:55:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:55:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:55:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:55:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:55:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:55:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:55:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:55:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:55:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:55:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:55:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:55:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:55:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:55:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:55:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:55:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:55:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:55:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:55:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:55:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:55:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:55:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:55:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:55:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:55:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:55:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:55:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:55:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:55:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:55:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:55:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:55:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:55:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:55:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:55:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:55:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:55:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:55:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:55:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:55:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:55:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:55:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:55:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:55:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:55:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:55:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:55:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:55:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:55:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:55:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:55:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:55:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:55:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:55:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:55:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:55:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:55:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:55:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:55:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:55:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:55:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:55:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:55:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:55:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:55:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:55:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:55:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:55:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:55:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:55:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:55:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:55:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:55:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:55:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:55:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:55:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:55:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:55:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:55:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:55:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:55:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:55:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:55:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:55:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:55:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:55:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:55:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:55:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:55:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:55:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:55:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:55:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:56:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:56:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:56:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:56:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:56:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:56:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:56:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:56:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:56:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:56:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:56:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:56:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:56:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:56:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:56:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:56:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:56:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:56:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:56:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:56:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:56:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:56:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:56:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:56:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:56:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:56:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:56:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:56:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:56:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:56:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:56:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:56:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:56:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:56:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:56:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:56:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:56:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:56:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:56:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:56:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:56:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:56:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:56:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:56:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:56:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:56:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:56:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:56:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:56:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:56:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:56:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:56:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:56:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:56:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:56:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:56:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:56:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:56:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:56:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:56:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:56:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:56:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:56:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:56:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:56:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:56:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:56:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:56:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:56:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:56:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:56:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:56:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:56:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:56:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:56:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:56:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:56:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:56:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:56:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:56:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:56:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:56:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:56:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:56:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:56:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:56:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:56:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:56:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:56:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:56:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:56:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:56:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:56:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:56:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:56:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:56:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:56:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:56:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:56:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:56:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:56:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:56:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:56:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:56:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:56:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:56:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:56:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:56:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:56:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:56:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:56:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:56:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:56:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:56:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:56:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:56:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:56:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:56:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:56:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:56:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:56:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:56:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:56:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:56:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:56:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:56:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:56:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:56:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:56:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:56:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:56:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:56:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:56:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:56:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:56:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:56:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:56:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:56:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:56:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:56:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:56:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:56:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:56:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:56:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:56:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:56:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:56:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:56:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:56:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:56:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:56:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:56:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:56:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:56:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:56:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:56:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:56:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:56:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:56:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:56:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:56:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:56:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:56:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:56:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:56:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:56:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:56:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:56:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:56:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:56:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:56:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:56:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:56:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:56:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:56:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:56:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:56:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:56:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:56:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:56:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:56:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:56:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:56:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:56:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:56:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:56:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:56:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:56:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:56:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:56:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:56:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:56:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:56:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:56:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:56:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:56:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:56:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:56:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:56:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:56:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:56:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:56:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:56:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:56:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:56:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:56:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:56:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:56:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:56:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:56:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:56:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:56:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:56:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:56:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:56:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:56:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:57:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:57:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:57:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:57:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:57:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:57:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:57:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:57:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:57:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:57:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:57:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:57:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:57:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:57:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:57:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:57:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:57:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:57:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:57:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:57:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:57:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:57:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:57:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:57:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:57:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:57:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:57:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:57:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:57:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:57:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:57:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:57:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:57:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:57:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:57:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:57:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:57:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:57:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:57:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:57:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:57:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:57:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:57:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:57:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:57:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:57:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:57:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:57:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:57:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:57:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:57:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:57:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:57:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:57:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:57:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:57:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:57:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:57:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:57:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:57:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:57:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:57:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:57:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:57:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:57:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:57:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:57:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:57:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:57:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:57:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:57:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:57:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:57:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:57:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:57:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:57:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:57:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:57:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:57:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:57:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:57:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:57:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:57:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:57:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:57:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:57:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:57:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:57:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:57:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:57:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:57:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:57:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:57:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:57:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:57:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:57:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:57:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:57:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:57:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:57:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:57:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:57:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:57:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:57:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:57:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:57:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:57:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:57:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:57:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:57:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:57:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:57:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:57:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:57:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:57:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:57:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:57:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:57:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:57:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:57:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:57:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:57:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:57:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:57:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:57:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:57:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:57:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:57:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:57:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:57:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:57:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:57:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:57:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:57:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:57:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:57:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:57:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:57:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:57:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:57:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:57:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:57:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:57:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:57:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:57:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:57:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:57:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:57:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:57:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:57:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:57:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:57:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:57:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:57:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:57:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:57:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:57:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:57:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:57:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:57:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:57:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:57:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:59:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:59:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:59:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:59:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:59:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:59:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:59:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:59:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:59:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:59:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:59:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:59:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:59:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:59:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:59:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:59:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:59:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:59:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 18 23:59:53 rdma06 rsyslogd[1041]: imjournal: 5296 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 18 23:59:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 18 23:59:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:59:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:59:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:59:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 18 23:59:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 18 23:59:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 18 23:59:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 18 23:59:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 18 23:59:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 18 23:59:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 18 23:59:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 18 23:59:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 18 23:59:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 18 23:59:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 18 23:59:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 18 23:59:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 18 23:59:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:00:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:00:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:00:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:00:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:00:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:00:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:00:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:00:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:00:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:00:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:00:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:00:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:00:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:00:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:00:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:00:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:00:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:00:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:00:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:00:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:00:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:00:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:00:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:00:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:00:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:00:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:00:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:00:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:00:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:00:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:00:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:00:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:00:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:00:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:00:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:00:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:00:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:00:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:00:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:00:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:00:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:00:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:00:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:00:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:00:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:00:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:00:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:00:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:00:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:00:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:00:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:00:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:00:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:00:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:00:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:00:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:00:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:00:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:00:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:00:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:00:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:00:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:00:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:00:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:00:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:00:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:00:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:00:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:00:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:00:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:00:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:00:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:00:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:00:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:00:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:00:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:00:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:00:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:00:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:00:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:00:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:00:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:00:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:00:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:00:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:00:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:00:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:00:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:00:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:00:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:00:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:00:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:00:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:00:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:00:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:00:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:00:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:00:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:00:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:00:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:00:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:00:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:00:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:00:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:00:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:00:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:00:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:00:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:00:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:00:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:00:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:00:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:00:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:00:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:00:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:00:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:00:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:00:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:00:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:00:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:00:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:00:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:00:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:00:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:00:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:00:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:00:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:00:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:00:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:00:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:00:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:00:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:00:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:00:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:00:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:00:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:00:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:00:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:00:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:00:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:00:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:00:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:00:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:00:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:00:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:00:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:00:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:00:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:00:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:00:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:00:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:00:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:00:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:00:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:00:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:00:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:00:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:00:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:00:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:00:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:00:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:00:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:00:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:00:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:00:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:00:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:00:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:00:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:00:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:00:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:00:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:00:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:00:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:00:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:00:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:00:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:00:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:00:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:00:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:00:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:00:47 rdma06 systemd[1]: Starting Rotate log files...
Apr 19 00:00:47 rdma06 systemd[1]: logrotate.service: Deactivated successfully.
Apr 19 00:00:47 rdma06 systemd[1]: Finished Rotate log files.
Apr 19 00:00:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:00:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:00:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:00:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:00:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:00:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:00:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:00:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:00:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:00:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:00:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:00:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:00:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:00:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:00:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:00:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:00:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:00:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:00:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:00:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:00:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:00:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:00:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:00:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:00:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:00:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:00:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:00:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:00:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:00:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:00:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:00:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:00:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:00:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:00:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:00:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:01:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:01:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:01:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:01:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:01:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:01:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:01:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:01:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:01:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:01:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:01:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:01:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:01:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:01:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:01:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:01:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:01:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:01:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:01:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:01:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:01:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:01:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:01:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:01:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:01:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:01:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:01:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:01:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:01:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:01:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:01:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:01:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:01:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:01:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:01:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:01:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:01:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:01:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:01:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:01:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:01:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:01:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:01:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:01:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:01:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:01:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:01:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:01:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:01:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:01:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:01:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:01:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:01:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:01:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:01:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:01:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:01:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:01:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:01:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:01:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:01:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:01:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:01:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:01:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:01:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:01:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:01:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:01:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:01:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:01:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:01:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:01:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:01:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:01:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:01:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:01:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:01:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:01:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:01:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:01:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:01:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:01:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:01:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:01:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:01:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:01:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:01:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:01:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:01:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:01:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:01:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:01:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:01:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:01:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:01:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:01:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:01:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:01:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:01:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:01:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:01:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:01:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:01:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:01:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:01:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:01:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:01:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:01:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:01:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:01:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:01:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:01:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:01:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:01:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:01:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:01:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:01:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:01:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:01:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:01:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:01:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:01:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:01:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:01:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:01:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:01:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:01:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:01:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:01:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:01:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:01:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:01:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:01:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:01:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:01:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:01:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:01:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:01:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:01:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:01:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:01:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:01:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:01:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:01:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:01:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:01:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:01:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:01:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:01:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:01:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:01:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:01:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:01:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:01:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:01:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:01:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:01:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:01:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:01:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:01:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:01:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:01:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:01:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:01:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:01:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:01:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:01:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:01:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:01:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:01:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:01:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:01:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:01:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:01:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:01:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:01:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:01:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:01:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:01:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:01:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:01:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:01:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:01:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:01:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:01:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:01:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:01:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:01:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:01:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:01:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:01:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:01:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:01:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:01:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:01:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:01:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:01:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:01:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:01:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:01:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:01:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:01:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:01:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:01:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:01:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:01:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:01:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:01:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:01:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:01:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:01:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:01:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:01:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:01:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:01:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:01:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:02:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:02:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:02:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:02:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:02:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:02:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:02:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:02:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:02:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:02:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:02:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:02:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:02:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:02:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:02:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:02:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:02:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:02:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:02:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:02:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:02:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:02:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:02:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:02:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:02:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:02:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:02:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:02:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:02:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:02:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:02:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:02:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:02:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:02:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:02:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:02:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:02:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:02:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:02:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:02:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:02:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:02:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:02:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:02:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:02:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:02:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:02:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:02:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:02:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:02:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:02:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:02:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:02:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:02:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:02:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:02:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:02:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:02:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:02:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:02:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:02:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:02:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:02:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:02:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:02:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:02:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:02:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:02:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:02:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:02:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:02:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:02:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:02:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:02:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:02:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:02:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:02:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:02:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:02:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:02:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:02:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:02:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:02:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:02:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:02:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:02:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:02:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:02:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:02:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:02:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:02:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:02:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:02:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:02:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:02:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:02:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:02:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:02:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:02:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:02:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:02:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:02:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:02:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:02:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:02:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:02:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:02:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:02:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:02:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:02:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:02:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:02:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:02:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:02:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:02:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:02:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:02:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:02:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:02:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:02:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:02:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:02:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:02:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:02:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:02:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:02:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:02:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:02:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:02:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:02:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:02:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:02:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:02:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:02:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:02:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:02:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:02:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:02:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:02:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:02:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:02:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:02:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:02:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:02:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:02:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:02:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:02:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:02:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:02:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:02:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:02:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:02:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:02:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:02:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:02:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:02:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:02:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:02:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:02:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:02:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:02:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:02:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:02:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:02:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:02:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:02:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:02:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:02:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:02:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:02:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:02:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:02:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:02:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:02:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:02:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:02:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:02:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:02:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:02:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:02:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:02:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:02:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:02:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:02:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:02:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:02:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:02:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:02:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:02:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:02:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:02:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:02:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:02:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:02:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:02:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:02:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:02:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:02:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:02:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:02:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:02:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:02:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:02:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:02:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:02:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:02:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:02:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:02:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:02:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:02:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:02:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:02:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:02:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:02:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:02:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:02:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:03:00 rdma06 systemd[1]: Starting dnf makecache...
Apr 19 00:03:01 rdma06 dnf[1207701]: Updating Subscription Management repositories.
Apr 19 00:03:01 rdma06 dnf[1207701]: Unable to read consumer identity
Apr 19 00:03:01 rdma06 dnf[1207701]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 19 00:03:01 rdma06 dnf[1207701]: Metadata cache refreshed recently.
Apr 19 00:03:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 19 00:03:01 rdma06 systemd[1]: Finished dnf makecache.
Apr 19 00:03:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:03:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:03:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:03:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:03:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:03:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:03:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:03:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:03:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:03:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:03:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:03:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:03:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:03:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:03:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:03:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:03:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:03:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:03:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:03:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:03:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:03:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:03:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:03:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:03:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:03:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:03:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:03:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:03:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:03:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:03:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:03:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:03:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:03:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:03:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:03:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:03:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:03:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:03:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:03:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:03:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:03:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:03:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:03:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:03:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:03:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:03:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:03:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:03:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:03:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:03:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:03:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:03:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:03:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:03:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:03:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:03:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:03:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:03:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:03:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:03:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:03:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:03:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:03:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:03:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:03:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:03:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:03:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:03:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:03:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:03:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:03:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:03:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:03:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:03:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:03:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:03:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:03:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:03:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:03:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:03:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:03:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:03:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:03:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:03:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:03:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:03:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:03:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:03:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:03:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:03:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:03:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:03:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:03:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:03:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:03:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:03:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:03:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:03:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:03:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:03:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:03:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:03:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:03:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:03:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:03:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:03:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:03:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:03:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:03:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:03:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:03:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:03:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:03:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:03:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:03:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:03:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:03:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:03:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:03:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:03:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:03:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:03:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:03:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:03:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:03:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:03:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:03:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:03:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:03:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:03:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:03:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:03:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:03:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:03:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:03:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:03:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:03:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:03:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:03:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:03:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:03:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:03:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:03:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:03:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:03:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:03:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:03:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:03:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:03:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:03:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:03:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:03:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:03:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:03:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:03:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:03:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:03:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:03:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:03:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:03:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:03:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:03:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:03:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:03:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:03:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:03:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:03:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:03:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:03:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:03:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:03:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:03:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:03:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:03:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:03:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:03:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:03:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:03:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:03:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:03:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:03:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:03:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:03:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:03:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:03:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:03:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:03:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:03:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:03:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:03:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:03:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:03:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:03:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:03:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:03:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:03:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:03:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:03:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:03:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:03:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:03:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:03:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:03:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:03:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:03:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:03:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:03:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:03:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:03:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:03:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:03:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:03:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:03:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:03:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:03:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:04:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:04:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:04:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:04:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:04:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:04:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:04:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:04:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:04:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:04:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:04:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:04:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:04:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:04:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:04:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:04:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:04:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:04:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:04:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:04:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:04:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:04:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:04:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:04:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:04:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:04:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:04:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:04:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:04:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:04:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:04:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:04:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:04:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:04:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:04:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:04:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:04:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:04:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:04:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:04:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:04:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:04:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:04:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:04:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:04:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:04:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:04:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:04:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:04:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:04:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:04:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:04:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:04:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:04:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:04:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:04:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:04:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:04:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:04:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:04:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:04:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:04:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:04:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:04:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:04:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:04:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:04:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:04:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:04:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:04:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:04:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:04:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:04:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:04:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:04:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:04:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:04:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:04:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:04:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:04:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:04:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:04:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:04:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:04:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:04:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:04:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:04:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:04:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:04:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:04:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:04:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:04:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:04:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:04:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:04:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:04:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:04:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:04:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:04:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:04:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:04:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:04:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:04:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:04:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:04:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:04:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:04:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:04:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:04:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:04:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:04:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:04:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:04:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:04:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:04:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:04:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:04:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:04:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:04:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:04:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:04:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:04:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:04:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:04:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:04:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:04:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:04:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:04:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:04:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:04:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:04:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:04:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:04:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:04:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:04:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:04:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:04:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:04:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:04:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:04:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:04:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:04:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:04:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:04:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:04:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:04:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:04:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:04:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:04:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:04:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:04:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:04:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:04:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:04:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:04:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:04:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:04:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:04:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:04:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:04:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:04:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:04:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:04:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:04:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:04:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:04:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:04:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:04:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:04:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:04:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:04:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:04:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:04:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:04:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:04:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:04:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:04:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:04:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:04:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:04:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:04:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:04:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:04:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:04:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:04:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:04:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:04:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:04:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:04:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:04:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:04:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:04:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:04:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:04:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:04:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:04:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:04:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:04:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:04:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:04:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:04:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:04:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:04:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:04:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:04:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:04:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:04:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:04:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:04:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:04:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:04:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:04:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:04:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:04:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:04:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:04:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:05:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:05:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:05:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:05:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:05:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:05:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:05:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:05:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:05:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:05:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:05:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:05:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:05:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:05:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:05:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:05:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:05:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:05:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:05:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:05:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:05:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:05:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:05:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:05:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:05:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:05:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:05:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:05:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:05:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:05:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:05:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:05:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:05:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:05:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:05:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:05:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:05:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:05:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:05:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:05:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:05:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:05:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:05:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:05:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:05:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:05:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:05:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:05:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:05:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:05:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:05:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:05:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:05:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:05:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:05:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:05:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:05:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:05:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:05:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:05:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:05:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:05:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:05:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:05:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:05:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:05:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:05:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:05:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:05:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:05:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:05:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:05:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:05:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:05:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:05:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:05:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:05:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:05:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:05:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:05:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:05:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:05:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:05:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:05:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:05:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:05:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:05:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:05:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:05:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:05:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:05:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:05:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:05:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:05:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:05:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:05:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:05:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:05:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:05:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:05:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:05:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:05:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:05:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:05:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:05:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:05:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:05:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:05:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:05:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:05:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:05:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:05:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:05:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:05:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:05:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:05:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:05:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:05:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:05:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:05:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:05:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:05:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:05:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:05:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:05:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:05:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:05:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:05:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:05:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:05:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:05:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:05:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:05:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:05:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:05:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:05:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:05:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:05:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:05:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:05:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:05:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:05:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:05:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:05:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:05:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:05:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:05:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:05:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:05:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:05:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:05:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:05:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:05:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:05:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:05:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:05:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:05:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:05:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:05:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:05:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:05:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:05:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:05:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:05:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:05:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:05:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:05:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:05:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:05:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:05:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:05:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:05:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:05:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:05:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:05:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:05:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:05:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:05:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:05:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:05:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:05:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:05:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:05:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:05:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:05:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:05:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:05:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:05:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:05:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:05:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:05:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:05:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:05:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:05:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:05:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:05:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:05:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:05:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:05:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:05:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:05:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:05:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:05:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:05:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:05:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:05:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:05:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:05:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:05:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:05:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:05:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:05:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:05:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:05:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:05:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:05:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:06:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:06:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:06:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:06:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:06:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:06:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:06:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:06:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:06:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:06:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:06:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:06:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:06:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:06:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:06:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:06:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:06:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:06:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:06:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:06:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:06:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:06:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:06:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:06:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:06:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:06:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:06:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:06:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:06:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:06:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:06:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:06:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:06:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:06:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:06:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:06:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:06:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:06:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:06:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:06:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:06:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:06:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:06:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:06:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:06:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:06:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:06:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:06:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:06:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:06:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:06:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:06:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:06:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:06:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:06:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:06:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:06:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:06:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:06:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:06:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:06:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:06:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:06:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:06:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:06:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:06:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:06:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:06:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:06:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:06:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:06:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:06:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:06:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:06:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:06:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:06:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:06:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:06:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:06:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:06:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:06:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:06:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:06:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:06:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:06:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:06:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:06:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:06:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:06:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:06:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:06:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:06:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:06:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:06:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:06:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:06:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:06:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:06:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:06:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:06:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:06:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:06:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:06:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:06:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:06:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:06:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:06:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:06:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:06:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:06:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:06:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:06:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:06:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:06:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:06:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:06:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:06:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:06:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:06:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:06:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:06:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:06:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:06:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:06:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:06:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:06:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:06:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:06:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:06:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:06:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:06:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:06:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:06:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:06:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:06:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:06:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:06:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:06:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:06:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:06:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:06:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:06:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:06:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:06:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:06:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:06:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:06:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:06:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:06:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:06:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:06:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:06:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:06:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:06:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:06:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:06:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:06:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:06:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:06:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:06:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:06:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:06:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:06:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:06:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:06:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:06:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:06:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:06:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:06:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:06:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:06:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:06:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:06:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:06:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:06:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:06:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:06:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:06:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:06:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:06:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:06:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:06:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:06:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:06:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:06:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:06:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:06:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:06:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:06:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:06:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:06:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:06:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:06:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:06:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:06:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:06:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:06:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:06:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:06:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:06:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:06:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:06:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:06:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:06:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:06:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:06:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:06:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:06:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:06:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:06:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:06:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:06:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:06:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:06:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:06:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:06:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:07:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:07:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:07:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:07:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:07:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:07:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:07:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:07:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:07:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:07:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:07:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:07:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:07:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:07:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:07:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:07:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:07:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:07:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:07:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:07:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:07:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:07:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:07:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:07:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:07:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:07:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:07:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:07:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:07:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:07:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:07:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:07:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:07:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:07:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:07:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:07:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:07:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:07:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:07:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:07:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:07:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:07:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:07:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:07:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:07:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:07:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:07:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:07:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:07:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:07:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:07:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:07:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:07:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:07:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:07:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:07:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:07:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:07:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:07:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:07:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:07:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:07:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:07:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:07:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:07:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:07:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:07:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:07:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:07:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:07:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:07:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:07:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:07:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:07:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:07:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:07:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:07:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:07:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:07:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:07:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:07:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:07:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:07:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:07:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:07:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:07:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:07:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:07:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:07:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:07:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:07:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:07:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:07:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:07:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:07:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:07:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:07:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:07:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:07:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:07:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:07:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:07:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:07:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:07:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:07:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:07:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:07:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:07:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:07:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:07:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:07:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:07:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:07:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:07:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:07:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:07:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:07:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:07:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:07:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:07:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:07:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:07:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:07:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:07:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:07:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:07:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:07:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:07:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:07:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:07:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:07:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:07:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:07:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:07:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:07:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:07:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:07:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:07:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:07:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:07:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:07:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:07:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:07:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:07:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:07:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:07:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:07:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:07:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:07:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:07:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:07:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:07:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:07:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:07:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:07:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:07:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:07:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:07:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:07:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:07:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:07:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:07:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:07:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:07:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:07:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:07:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:07:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:07:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:07:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:07:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:07:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:07:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:07:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:07:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:07:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:07:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:07:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:07:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:07:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:07:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:09:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:09:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:09:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:09:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:09:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:09:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:09:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:09:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:09:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:09:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:09:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:09:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:09:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:09:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:09:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:09:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:09:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:09:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:09:58 rdma06 rsyslogd[1041]: imjournal: 5307 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 19 00:10:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:10:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:10:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:10:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:10:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:10:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:10:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:10:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:10:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:10:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:10:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:10:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:10:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:10:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:10:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:10:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:10:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:10:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:10:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:10:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:10:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:10:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:10:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:10:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:10:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:10:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:10:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:10:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:10:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:10:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:10:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:10:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:10:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:10:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:10:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:10:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:10:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:10:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:10:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:10:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:10:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:10:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:10:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:10:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:10:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:10:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:10:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:10:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:10:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:10:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:10:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:10:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:10:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:10:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:10:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:10:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:10:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:10:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:10:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:10:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:10:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:10:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:10:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:10:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:10:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:10:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:10:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:10:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:10:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:10:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:10:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:10:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:10:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:10:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:10:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:10:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:10:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:10:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:10:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:10:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:10:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:10:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:10:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:10:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:10:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:10:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:10:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:10:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:10:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:10:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:10:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:10:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:10:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:10:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:10:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:10:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:10:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:10:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:10:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:10:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:10:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:10:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:10:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:10:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:10:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:10:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:10:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:10:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:10:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:10:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:10:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:10:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:10:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:10:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:10:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:10:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:10:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:10:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:10:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:10:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:10:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:10:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:10:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:10:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:10:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:10:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:10:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:10:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:10:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:10:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:10:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:10:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:10:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:10:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:10:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:10:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:10:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:10:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:10:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:10:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:10:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:10:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:10:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:10:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:10:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:10:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:10:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:10:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:10:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:10:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:10:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:10:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:10:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:10:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:10:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:10:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:10:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:10:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:10:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:10:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:10:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:10:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:10:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:10:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:10:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:10:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:10:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:10:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:10:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:10:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:10:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:10:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:10:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:10:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:10:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:10:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:10:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:10:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:10:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:10:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:10:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:10:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:10:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:10:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:10:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:10:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:10:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:10:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:10:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:10:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:10:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:10:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:10:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:10:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:10:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:10:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:10:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:10:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:10:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:10:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:10:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:10:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:10:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:10:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:10:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:10:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:10:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:10:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:10:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:10:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:10:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:10:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:10:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:10:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:10:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:10:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:11:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:11:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:11:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:11:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:11:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:11:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:11:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:11:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:11:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:11:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:11:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:11:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:11:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:11:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:11:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:11:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:11:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:11:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:11:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:11:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:11:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:11:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:11:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:11:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:11:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:11:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:11:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:11:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:11:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:11:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:11:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:11:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:11:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:11:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:11:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:11:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:11:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:11:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:11:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:11:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:11:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:11:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:11:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:11:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:11:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:11:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:11:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:11:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:11:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:11:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:11:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:11:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:11:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:11:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:11:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:11:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:11:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:11:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:11:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:11:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:11:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:11:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:11:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:11:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:11:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:11:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:11:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:11:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:11:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:11:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:11:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:11:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:11:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:11:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:11:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:11:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:11:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:11:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:11:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:11:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:11:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:11:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:11:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:11:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:11:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:11:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:11:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:11:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:11:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:11:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:11:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:11:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:11:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:11:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:11:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:11:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:11:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:11:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:11:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:11:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:11:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:11:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:11:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:11:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:11:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:11:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:11:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:11:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:11:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:11:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:11:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:11:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:11:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:11:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:11:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:11:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:11:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:11:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:11:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:11:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:11:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:11:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:11:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:11:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:11:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:11:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:11:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:11:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:11:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:11:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:11:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:11:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:11:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:11:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:11:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:11:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:11:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:11:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:11:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:11:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:11:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:11:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:11:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:11:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:11:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:11:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:11:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:11:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:11:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:11:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:11:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:11:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:11:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:11:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:11:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:11:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:11:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:11:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:11:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:11:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:11:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:11:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:11:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:11:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:11:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:11:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:11:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:11:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:11:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:11:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:11:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:11:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:11:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:11:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:11:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:11:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:11:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:11:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:11:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:11:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:11:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:11:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:11:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:11:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:11:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:11:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:11:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:11:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:11:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:11:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:11:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:11:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:11:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:11:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:11:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:11:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:11:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:11:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:11:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:11:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:11:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:11:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:11:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:11:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:11:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:11:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:11:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:11:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:11:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:11:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:11:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:11:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:11:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:11:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:11:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:11:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:12:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:12:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:12:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:12:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:12:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:12:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:12:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:12:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:12:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:12:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:12:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:12:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:12:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:12:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:12:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:12:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:12:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:12:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:12:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:12:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:12:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:12:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:12:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:12:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:12:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:12:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:12:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:12:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:12:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:12:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:12:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:12:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:12:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:12:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:12:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:12:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:12:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:12:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:12:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:12:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:12:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:12:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:12:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:12:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:12:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:12:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:12:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:12:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:12:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:12:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:12:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:12:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:12:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:12:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:12:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:12:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:12:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:12:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:12:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:12:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:12:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:12:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:12:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:12:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:12:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:12:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:12:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:12:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:12:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:12:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:12:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:12:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:12:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:12:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:12:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:12:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:12:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:12:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:12:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:12:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:12:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:12:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:12:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:12:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:12:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:12:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:12:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:12:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:12:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:12:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:12:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:12:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:12:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:12:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:12:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:12:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:12:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:12:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:12:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:12:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:12:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:12:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:12:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:12:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:12:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:12:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:12:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:12:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:12:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:12:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:12:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:12:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:12:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:12:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:12:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:12:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:12:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:12:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:12:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:12:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:12:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:12:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:12:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:12:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:12:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:12:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:12:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:12:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:12:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:12:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:12:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:12:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:12:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:12:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:12:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:12:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:12:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:12:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:12:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:12:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:12:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:12:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:12:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:12:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:12:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:12:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:12:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:12:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:12:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:12:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:12:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:12:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:12:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:12:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:12:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:12:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:12:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:12:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:12:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:12:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:12:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:12:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:12:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:12:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:12:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:12:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:12:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:12:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:12:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:12:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:12:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:12:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:12:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:12:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:12:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:12:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:12:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:12:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:12:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:12:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:12:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:12:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:12:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:12:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:12:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:12:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:12:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:12:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:12:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:12:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:12:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:12:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:12:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:12:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:12:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:12:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:12:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:12:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:12:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:12:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:12:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:12:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:12:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:12:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:12:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:12:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:12:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:12:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:12:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:12:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:12:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:12:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:12:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:12:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:12:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:12:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:13:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:13:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:13:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:13:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:13:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:13:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:13:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:13:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:13:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:13:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:13:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:13:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:13:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:13:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:13:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:13:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:13:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:13:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:13:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:13:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:13:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:13:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:13:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:13:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:13:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:13:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:13:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:13:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:13:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:13:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:13:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:13:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:13:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:13:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:13:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:13:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:13:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:13:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:13:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:13:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:13:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:13:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:13:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:13:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:13:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:13:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:13:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:13:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:13:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:13:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:13:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:13:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:13:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:13:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:13:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:13:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:13:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:13:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:13:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:13:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:13:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:13:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:13:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:13:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:13:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:13:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:13:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:13:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:13:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:13:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:13:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:13:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:13:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:13:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:13:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:13:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:13:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:13:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:13:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:13:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:13:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:13:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:13:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:13:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:13:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:13:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:13:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:13:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:13:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:13:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:13:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:13:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:13:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:13:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:13:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:13:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:13:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:13:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:13:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:13:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:13:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:13:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:13:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:13:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:13:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:13:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:13:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:13:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:13:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:13:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:13:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:13:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:13:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:13:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:13:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:13:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:13:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:13:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:13:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:13:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:13:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:13:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:13:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:13:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:13:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:13:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:13:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:13:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:13:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:13:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:13:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:13:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:13:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:13:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:13:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:13:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:13:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:13:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:13:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:13:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:13:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:13:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:13:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:13:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:13:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:13:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:13:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:13:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:13:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:13:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:13:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:13:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:13:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:13:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:13:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:13:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:13:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:13:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:13:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:13:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:13:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:13:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:13:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:13:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:13:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:13:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:13:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:13:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:13:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:13:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:13:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:13:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:13:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:13:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:13:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:13:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:13:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:13:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:13:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:13:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:13:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:13:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:13:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:13:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:13:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:13:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:13:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:13:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:13:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:13:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:13:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:13:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:13:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:13:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:13:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:13:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:13:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:13:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:13:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:13:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:13:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:13:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:13:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:13:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:13:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:13:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:13:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:13:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:13:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:13:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:13:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:13:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:13:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:13:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:13:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:13:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:14:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:14:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:14:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:14:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:14:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:14:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:14:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:14:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:14:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:14:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:14:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:14:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:14:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:14:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:14:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:14:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:14:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:14:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:14:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:14:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:14:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:14:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:14:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:14:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:14:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:14:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:14:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:14:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:14:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:14:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:14:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:14:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:14:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:14:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:14:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:14:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:14:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:14:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:14:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:14:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:14:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:14:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:14:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:14:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:14:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:14:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:14:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:14:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:14:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:14:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:14:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:14:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:14:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:14:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:14:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:14:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:14:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:14:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:14:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:14:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:14:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:14:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:14:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:14:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:14:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:14:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:14:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:14:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:14:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:14:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:14:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:14:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:14:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:14:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:14:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:14:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:14:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:14:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:14:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:14:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:14:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:14:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:14:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:14:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:14:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:14:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:14:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:14:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:14:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:14:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:14:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:14:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:14:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:14:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:14:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:14:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:14:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:14:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:14:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:14:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:14:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:14:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:14:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:14:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:14:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:14:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:14:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:14:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:14:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:14:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:14:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:14:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:14:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:14:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:14:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:14:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:14:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:14:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:14:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:14:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:14:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:14:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:14:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:14:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:14:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:14:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:14:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:14:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:14:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:14:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:14:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:14:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:14:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:14:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:14:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:14:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:14:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:14:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:14:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:14:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:14:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:14:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:14:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:14:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:14:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:14:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:14:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:14:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:14:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:14:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:14:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:14:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:14:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:14:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:14:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:14:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:14:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:14:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:14:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:14:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:14:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:14:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:14:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:14:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:14:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:14:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:14:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:14:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:14:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:14:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:14:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:14:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:14:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:14:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:14:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:14:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:14:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:14:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:14:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:14:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:14:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:14:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:14:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:14:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:14:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:14:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:14:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:14:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:14:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:14:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:14:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:14:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:14:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:14:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:14:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:14:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:14:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:14:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:14:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:14:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:14:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:14:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:14:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:14:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:14:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:14:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:14:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:14:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:14:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:14:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:14:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:14:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:14:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:14:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:14:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:14:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:15:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:15:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:15:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:15:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:15:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:15:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:15:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:15:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:15:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:15:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:15:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:15:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:15:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:15:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:15:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:15:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:15:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:15:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:15:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:15:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:15:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:15:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:15:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:15:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:15:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:15:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:15:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:15:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:15:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:15:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:15:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:15:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:15:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:15:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:15:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:15:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:15:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:15:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:15:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:15:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:15:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:15:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:15:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:15:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:15:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:15:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:15:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:15:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:15:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:15:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:15:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:15:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:15:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:15:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:15:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:15:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:15:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:15:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:15:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:15:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:15:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:15:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:15:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:15:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:15:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:15:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:15:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:15:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:15:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:15:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:15:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:15:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:15:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:15:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:15:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:15:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:15:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:15:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:15:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:15:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:15:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:15:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:15:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:15:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:15:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:15:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:15:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:15:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:15:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:15:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:15:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:15:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:15:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:15:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:15:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:15:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:15:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:15:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:15:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:15:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:15:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:15:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:15:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:15:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:15:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:15:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:15:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:15:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:15:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:15:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:15:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:15:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:15:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:15:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:15:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:15:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:15:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:15:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:15:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:15:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:15:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:15:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:15:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:15:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:15:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:15:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:15:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:15:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:15:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:15:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:15:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:15:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:15:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:15:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:15:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:15:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:15:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:15:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:15:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:15:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:15:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:15:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:15:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:15:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:15:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:15:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:15:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:15:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:15:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:15:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:15:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:15:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:15:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:15:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:15:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:15:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:15:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:15:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:15:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:15:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:15:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:15:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:15:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:15:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:15:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:15:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:15:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:15:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:15:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:15:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:15:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:15:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:15:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:15:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:15:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:15:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:15:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:15:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:15:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:15:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:15:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:15:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:15:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:15:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:15:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:15:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:15:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:15:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:15:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:15:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:15:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:15:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:15:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:15:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:15:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:15:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:15:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:15:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:15:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:15:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:15:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:15:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:15:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:15:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:15:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:15:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:15:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:15:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:15:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:15:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:15:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:15:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:15:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:15:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:15:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:15:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:16:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:16:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:16:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:16:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:16:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:16:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:16:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:16:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:16:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:16:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:16:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:16:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:16:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:16:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:16:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:16:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:16:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:16:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:16:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:16:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:16:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:16:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:16:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:16:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:16:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:16:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:16:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:16:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:16:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:16:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:16:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:16:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:16:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:16:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:16:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:16:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:16:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:16:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:16:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:16:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:16:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:16:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:16:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:16:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:16:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:16:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:16:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:16:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:16:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:16:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:16:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:16:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:16:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:16:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:16:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:16:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:16:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:16:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:16:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:16:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:16:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:16:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:16:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:16:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:16:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:16:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:16:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:16:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:16:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:16:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:16:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:16:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:16:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:16:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:16:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:16:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:16:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:16:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:16:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:16:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:16:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:16:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:16:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:16:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:16:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:16:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:16:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:16:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:16:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:16:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:16:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:16:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:16:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:16:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:16:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:16:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:16:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:16:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:16:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:16:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:16:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:16:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:16:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:16:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:16:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:16:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:16:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:16:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:16:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:16:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:16:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:16:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:16:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:16:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:16:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:16:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:16:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:16:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:16:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:16:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:16:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:16:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:16:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:16:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:16:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:16:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:16:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:16:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:16:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:16:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:16:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:16:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:16:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:16:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:16:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:16:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:16:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:16:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:16:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:16:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:16:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:16:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:16:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:16:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:16:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:16:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:16:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:16:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:16:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:16:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:16:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:16:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:16:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:16:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:16:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:16:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:16:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:16:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:16:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:16:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:16:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:16:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:16:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:16:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:16:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:16:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:16:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:16:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:16:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:16:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:16:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:16:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:16:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:16:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:16:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:16:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:16:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:16:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:16:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:16:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:16:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:16:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:16:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:16:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:16:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:16:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:16:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:16:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:16:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:16:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:16:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:16:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:16:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:16:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:16:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:16:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:16:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:16:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:16:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:16:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:16:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:16:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:16:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:16:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:16:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:16:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:16:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:16:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:16:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:16:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:16:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:16:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:16:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:16:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:16:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:16:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:17:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:17:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:17:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:17:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:17:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:17:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:17:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:17:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:17:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:17:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:17:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:17:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:17:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:17:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:17:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:17:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:17:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:17:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:17:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:17:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:17:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:17:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:17:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:17:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:17:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:17:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:17:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:17:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:17:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:17:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:17:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:17:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:17:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:17:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:17:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:17:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:17:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:17:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:17:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:17:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:17:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:17:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:17:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:17:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:17:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:17:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:17:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:17:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:17:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:17:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:17:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:17:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:17:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:17:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:17:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:17:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:17:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:17:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:17:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:17:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:17:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:17:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:17:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:17:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:17:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:17:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:17:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:17:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:17:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:17:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:17:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:17:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:17:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:17:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:17:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:17:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:17:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:17:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:17:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:17:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:17:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:17:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:17:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:17:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:17:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:17:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:17:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:17:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:17:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:17:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:17:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:17:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:17:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:17:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:17:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:17:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:17:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:17:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:17:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:17:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:17:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:17:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:17:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:17:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:17:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:17:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:17:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:17:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:17:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:17:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:17:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:17:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:17:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:17:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:17:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:17:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:17:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:17:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:17:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:17:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:17:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:17:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:17:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:17:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:17:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:17:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:17:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:17:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:17:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:17:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:17:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:17:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:17:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:17:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:17:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:17:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:17:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:17:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:17:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:17:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:17:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:17:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:17:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:17:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:17:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:17:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:17:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:17:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:17:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:17:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:17:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:17:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:17:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:17:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:17:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:17:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:17:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:17:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:17:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:17:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:17:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:17:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:17:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:17:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:17:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:17:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:17:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:17:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:17:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:17:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:17:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:17:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:17:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:17:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:17:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:17:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:17:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:17:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:17:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:17:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:17:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:17:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:17:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:17:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:17:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:17:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:17:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:17:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:17:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:17:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:17:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:17:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:17:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:17:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:17:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:17:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:17:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:17:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:20:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:20:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:20:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:20:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:20:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:20:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:20:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:20:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:20:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:20:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:20:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:20:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:20:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:20:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:20:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:20:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:20:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:20:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:20:03 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 19 00:20:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:20:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:20:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:20:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:20:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:20:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:20:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:20:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:20:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:20:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:20:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:20:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:20:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:20:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:20:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:20:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:20:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:20:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:20:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:20:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:20:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:20:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:20:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:20:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:20:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:20:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:20:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:20:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:20:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:20:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:20:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:20:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:20:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:20:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:20:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:20:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:20:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:20:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:20:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:20:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:20:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:20:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:20:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:20:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:20:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:20:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:20:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:20:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:20:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:20:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:20:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:20:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:20:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:20:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:20:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:20:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:20:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:20:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:20:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:20:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:20:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:20:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:20:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:20:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:20:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:20:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:20:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:20:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:20:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:20:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:20:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:20:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:20:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:20:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:20:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:20:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:20:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:20:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:20:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:20:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:20:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:20:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:20:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:20:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:20:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:20:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:20:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:20:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:20:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:20:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:20:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:20:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:20:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:20:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:20:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:20:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:20:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:20:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:20:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:20:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:20:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:20:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:20:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:20:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:20:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:20:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:20:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:20:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:20:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:20:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:20:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:20:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:20:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:20:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:20:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:20:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:20:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:20:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:20:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:20:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:20:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:20:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:20:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:20:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:20:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:20:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:20:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:20:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:20:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:20:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:20:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:20:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:20:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:20:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:20:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:20:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:20:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:20:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:20:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:20:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:20:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:20:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:20:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:20:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:20:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:20:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:20:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:20:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:20:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:20:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:20:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:20:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:20:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:20:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:20:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:20:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:20:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:20:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:20:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:20:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:20:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:20:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:20:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:20:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:20:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:20:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:20:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:20:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:20:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:20:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:20:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:20:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:20:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:20:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:20:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:20:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:20:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:20:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:20:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:20:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:20:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:20:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:20:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:20:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:20:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:20:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:20:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:20:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:20:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:20:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:20:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:20:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:20:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:20:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:20:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:20:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:20:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:20:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:21:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:21:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:21:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:21:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:21:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:21:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:21:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:21:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:21:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:21:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:21:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:21:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:21:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:21:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:21:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:21:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:21:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:21:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:21:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:21:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:21:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:21:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:21:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:21:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:21:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:21:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:21:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:21:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:21:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:21:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:21:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:21:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:21:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:21:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:21:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:21:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:21:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:21:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:21:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:21:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:21:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:21:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:21:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:21:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:21:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:21:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:21:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:21:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:21:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:21:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:21:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:21:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:21:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:21:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:21:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:21:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:21:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:21:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:21:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:21:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:21:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:21:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:21:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:21:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:21:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:21:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:21:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:21:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:21:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:21:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:21:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:21:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:21:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:21:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:21:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:21:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:21:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:21:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:21:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:21:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:21:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:21:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:21:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:21:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:21:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:21:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:21:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:21:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:21:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:21:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:21:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:21:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:21:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:21:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:21:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:21:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:21:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:21:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:21:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:21:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:21:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:21:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:21:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:21:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:21:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:21:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:21:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:21:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:21:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:21:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:21:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:21:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:21:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:21:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:21:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:21:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:21:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:21:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:21:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:21:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:21:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:21:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:21:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:21:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:21:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:21:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:21:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:21:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:21:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:21:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:21:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:21:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:21:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:21:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:21:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:21:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:21:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:21:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:21:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:21:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:21:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:21:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:21:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:21:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:21:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:21:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:21:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:21:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:21:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:21:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:21:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:21:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:21:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:21:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:21:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:21:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:21:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:21:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:21:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:21:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:21:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:21:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:21:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:21:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:21:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:21:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:21:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:21:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:21:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:21:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:21:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:21:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:21:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:21:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:21:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:21:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:21:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:21:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:21:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:21:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:21:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:21:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:21:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:21:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:21:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:21:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:21:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:21:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:21:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:21:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:21:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:21:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:21:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:21:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:21:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:21:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:21:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:21:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:21:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:21:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:21:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:21:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:21:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:21:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:21:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:21:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:21:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:21:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:21:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:21:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:21:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:21:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:21:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:21:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:21:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:21:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:22:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:22:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:22:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:22:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:22:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:22:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:22:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:22:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:22:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:22:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:22:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:22:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:22:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:22:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:22:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:22:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:22:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:22:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:22:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:22:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:22:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:22:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:22:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:22:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:22:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:22:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:22:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:22:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:22:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:22:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:22:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:22:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:22:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:22:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:22:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:22:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:22:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:22:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:22:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:22:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:22:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:22:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:22:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:22:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:22:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:22:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:22:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:22:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:22:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:22:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:22:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:22:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:22:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:22:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:22:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:22:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:22:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:22:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:22:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:22:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:22:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:22:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:22:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:22:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:22:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:22:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:22:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:22:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:22:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:22:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:22:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:22:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:22:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:22:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:22:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:22:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:22:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:22:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:22:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:22:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:22:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:22:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:22:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:22:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:22:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:22:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:22:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:22:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:22:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:22:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:22:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:22:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:22:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:22:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:22:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:22:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:22:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:22:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:22:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:22:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:22:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:22:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:22:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:22:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:22:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:22:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:22:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:22:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:22:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:22:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:22:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:22:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:22:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:22:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:22:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:22:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:22:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:22:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:22:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:22:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:22:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:22:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:22:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:22:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:22:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:22:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:22:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:22:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:22:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:22:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:22:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:22:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:22:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:22:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:22:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:22:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:22:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:22:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:22:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:22:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:22:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:22:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:22:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:22:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:22:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:22:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:22:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:22:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:22:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:22:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:22:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:22:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:22:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:22:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:22:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:22:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:22:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:22:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:22:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:22:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:22:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:22:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:22:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:22:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:22:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:22:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:22:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:22:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:22:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:22:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:22:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:22:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:22:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:22:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:22:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:22:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:22:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:22:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:22:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:22:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:22:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:22:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:22:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:22:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:22:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:22:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:22:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:22:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:22:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:22:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:22:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:22:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:22:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:22:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:22:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:22:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:22:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:22:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:22:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:22:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:22:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:22:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:22:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:22:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:22:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:22:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:22:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:22:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:22:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:22:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:22:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:22:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:22:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:22:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:22:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:22:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:23:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:23:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:23:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:23:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:23:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:23:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:23:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:23:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:23:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:23:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:23:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:23:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:23:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:23:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:23:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:23:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:23:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:23:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:23:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:23:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:23:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:23:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:23:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:23:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:23:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:23:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:23:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:23:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:23:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:23:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:23:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:23:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:23:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:23:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:23:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:23:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:23:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:23:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:23:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:23:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:23:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:23:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:23:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:23:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:23:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:23:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:23:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:23:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:23:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:23:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:23:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:23:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:23:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:23:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:23:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:23:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:23:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:23:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:23:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:23:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:23:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:23:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:23:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:23:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:23:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:23:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:23:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:23:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:23:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:23:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:23:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:23:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:23:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:23:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:23:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:23:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:23:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:23:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:23:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:23:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:23:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:23:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:23:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:23:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:23:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:23:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:23:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:23:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:23:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:23:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:23:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:23:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:23:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:23:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:23:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:23:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:23:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:23:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:23:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:23:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:23:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:23:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:23:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:23:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:23:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:23:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:23:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:23:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:23:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:23:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:23:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:23:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:23:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:23:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:23:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:23:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:23:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:23:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:23:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:23:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:23:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:23:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:23:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:23:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:23:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:23:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:23:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:23:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:23:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:23:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:23:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:23:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:23:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:23:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:23:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:23:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:23:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:23:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:23:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:23:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:23:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:23:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:23:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:23:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:23:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:23:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:23:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:23:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:23:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:23:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:23:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:23:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:23:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:23:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:23:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:23:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:23:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:23:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:23:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:23:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:23:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:23:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:23:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:23:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:23:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:23:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:24:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:24:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:24:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:24:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:24:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:24:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:24:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:24:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:24:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:24:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:24:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:24:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:24:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:24:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:24:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:24:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:24:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:24:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:24:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:24:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:24:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:24:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:24:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:24:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:24:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:24:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:24:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:24:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:24:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:24:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:24:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:24:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:24:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:24:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:24:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:24:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:24:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:24:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:24:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:24:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:24:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:24:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:24:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:24:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:24:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:24:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:24:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:24:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:24:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:24:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:24:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:24:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:24:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:24:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:24:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:24:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:24:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:24:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:24:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:24:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:24:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:24:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:24:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:24:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:24:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:24:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:24:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:24:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:24:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:24:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:24:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:24:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:24:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:24:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:24:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:24:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:24:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:24:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:24:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:24:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:24:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:24:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:24:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:24:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:24:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:24:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:24:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:24:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:24:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:24:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:24:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:24:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:24:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:24:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:24:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:24:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:24:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:24:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:24:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:24:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:24:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:24:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:24:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:24:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:24:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:24:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:24:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:24:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:24:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:24:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:24:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:24:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:24:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:24:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:24:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:24:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:24:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:24:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:24:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:24:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:24:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:24:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:24:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:24:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:24:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:24:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:24:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:24:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:24:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:24:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:24:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:24:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:24:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:24:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:24:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:24:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:24:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:24:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:24:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:24:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:24:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:24:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:24:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:24:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:24:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:24:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:24:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:24:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:24:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:24:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:24:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:24:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:24:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:24:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:24:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:24:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:24:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:24:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:24:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:24:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:24:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:24:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:24:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:24:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:24:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:24:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:24:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:24:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:24:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:24:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:24:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:24:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:24:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:24:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:24:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:24:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:24:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:24:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:24:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:24:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:24:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:24:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:24:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:24:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:24:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:24:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:24:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:24:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:24:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:24:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:24:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:24:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:24:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:24:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:24:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:24:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:25:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:25:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:25:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:25:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:25:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:25:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:25:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:25:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:25:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:25:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:25:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:25:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:25:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:25:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:25:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:25:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:25:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:25:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:25:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:25:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:25:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:25:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:25:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:25:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:25:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:25:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:25:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:25:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:25:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:25:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:25:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:25:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:25:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:25:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:25:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:25:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:25:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:25:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:25:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:25:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:25:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:25:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:25:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:25:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:25:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:25:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:25:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:25:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:25:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:25:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:25:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:25:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:25:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:25:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:25:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:25:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:25:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:25:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:25:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:25:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:25:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:25:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:25:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:25:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:25:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:25:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:25:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:25:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:25:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:25:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:25:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:25:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:25:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:25:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:25:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:25:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:25:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:25:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:25:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:25:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:25:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:25:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:25:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:25:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:25:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:25:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:25:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:25:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:25:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:25:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:25:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:25:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:25:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:25:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:25:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:25:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:25:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:25:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:25:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:25:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:25:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:25:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:25:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:25:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:25:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:25:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:25:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:25:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:25:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:25:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:25:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:25:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:25:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:25:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:25:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:25:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:25:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:25:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:25:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:25:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:25:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:25:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:25:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:25:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:25:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:25:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:25:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:25:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:25:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:25:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:25:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:25:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:25:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:25:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:25:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:25:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:25:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:25:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:25:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:25:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:25:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:25:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:25:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:25:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:25:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:25:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:25:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:25:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:25:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:25:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:25:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:25:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:25:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:25:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:25:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:25:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:25:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:25:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:25:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:25:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:25:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:25:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:25:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:25:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:25:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:25:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:26:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:26:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:26:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:26:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:26:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:26:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:26:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:26:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:26:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:26:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:26:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:26:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:26:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:26:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:26:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:26:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:26:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:26:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:26:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:26:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:26:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:26:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:26:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:26:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:26:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:26:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:26:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:26:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:26:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:26:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:26:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:26:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:26:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:26:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:26:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:26:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:26:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:26:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:26:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:26:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:26:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:26:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:26:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:26:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:26:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:26:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:26:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:26:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:26:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:26:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:26:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:26:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:26:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:26:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:26:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:26:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:26:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:26:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:26:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:26:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:26:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:26:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:26:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:26:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:26:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:26:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:26:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:26:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:26:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:26:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:26:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:26:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:26:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:26:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:26:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:26:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:26:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:26:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:26:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:26:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:26:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:26:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:26:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:26:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:26:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:26:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:26:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:26:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:26:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:26:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:26:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:26:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:26:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:26:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:26:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:26:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:26:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:26:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:26:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:26:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:26:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:26:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:26:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:26:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:26:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:26:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:26:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:26:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:26:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:26:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:26:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:26:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:26:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:26:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:26:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:26:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:26:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:26:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:26:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:26:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:26:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:26:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:26:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:26:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:26:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:26:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:26:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:26:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:26:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:26:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:26:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:26:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:26:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:26:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:26:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:26:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:26:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:26:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:26:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:26:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:26:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:26:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:26:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:26:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:26:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:26:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:26:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:26:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:26:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:26:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:26:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:26:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:26:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:26:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:26:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:26:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:26:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:26:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:26:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:26:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:26:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:26:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:26:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:26:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:26:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:26:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:26:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:26:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:26:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:26:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:26:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:26:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:26:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:26:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:26:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:26:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:26:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:26:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:26:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:26:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:26:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:26:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:26:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:26:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:26:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:26:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:26:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:26:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:26:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:26:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:26:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:26:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:26:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:26:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:26:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:26:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:26:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:26:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:26:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:26:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:26:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:26:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:26:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:26:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:26:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:26:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:26:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:26:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:26:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:26:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:26:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:26:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:26:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:26:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:26:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:26:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:27:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:27:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:27:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:27:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:27:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:27:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:27:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:27:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:27:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:27:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:27:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:27:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:27:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:27:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:27:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:27:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:27:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:27:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:27:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:27:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:27:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:27:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:27:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:27:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:27:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:27:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:27:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:27:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:27:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:27:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:27:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:27:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:27:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:27:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:27:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:27:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:27:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:27:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:27:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:27:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:27:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:27:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:27:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:27:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:27:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:27:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:27:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:27:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:27:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:27:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:27:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:27:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:27:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:27:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:27:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:27:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:27:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:27:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:27:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:27:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:27:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:27:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:27:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:27:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:27:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:27:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:27:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:27:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:27:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:27:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:27:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:27:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:27:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:27:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:27:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:27:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:27:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:27:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:27:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:27:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:27:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:27:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:27:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:27:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:27:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:27:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:27:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:27:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:27:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:27:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:27:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:27:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:27:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:27:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:27:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:27:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:27:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:27:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:27:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:27:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:27:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:27:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:27:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:27:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:27:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:27:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:27:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:27:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:27:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:27:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:27:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:27:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:27:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:27:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:27:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:27:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:27:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:27:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:27:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:27:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:27:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:27:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:27:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:27:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:27:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:27:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:27:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:27:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:27:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:27:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:27:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:27:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:27:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:27:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:27:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:27:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:27:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:27:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:27:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:27:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:27:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:27:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:27:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:27:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:27:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:27:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:27:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:27:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:27:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:27:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:27:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:27:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:27:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:27:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:27:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:27:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:27:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:27:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:27:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:27:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:27:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:27:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:27:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:27:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:27:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:27:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:27:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:27:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:27:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:27:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:27:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:27:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:27:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:27:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:27:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:27:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:27:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:27:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:27:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:27:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:27:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:27:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:27:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:27:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:27:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:27:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:27:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:27:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:27:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:27:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:27:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:27:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:27:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:27:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:27:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:27:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:27:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:27:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:27:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:27:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:27:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:27:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:27:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:27:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:27:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:27:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:27:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:27:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:27:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:27:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:27:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:27:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:27:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:27:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:27:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:27:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:30:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:30:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:30:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:30:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:30:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:30:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:30:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:30:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:30:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:30:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:30:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:30:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:30:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:30:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:30:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:30:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:30:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:30:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:30:04 rdma06 rsyslogd[1041]: imjournal: 5082 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 19 00:30:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:30:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:30:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:30:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:30:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:30:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:30:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:30:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:30:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:30:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:30:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:30:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:30:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:30:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:30:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:30:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:30:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:30:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:30:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:30:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:30:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:30:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:30:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:30:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:30:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:30:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:30:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:30:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:30:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:30:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:30:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:30:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:30:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:30:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:30:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:30:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:30:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:30:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:30:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:30:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:30:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:30:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:30:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:30:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:30:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:30:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:30:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:30:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:30:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:30:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:30:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:30:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:30:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:30:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:30:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:30:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:30:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:30:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:30:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:30:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:30:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:30:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:30:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:30:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:30:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:30:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:30:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:30:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:30:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:30:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:30:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:30:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:30:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:30:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:30:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:30:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:30:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:30:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:30:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:30:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:30:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:30:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:30:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:30:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:30:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:30:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:30:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:30:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:30:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:30:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:30:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:30:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:30:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:30:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:30:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:30:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:30:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:30:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:30:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:30:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:30:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:30:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:30:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:30:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:30:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:30:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:30:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:30:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:31:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:31:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:31:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:31:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:31:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:31:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:31:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:31:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:31:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:31:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:31:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:31:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:31:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:31:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:31:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:31:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:31:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:31:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:31:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:31:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:31:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:31:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:31:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:31:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:31:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:31:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:31:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:31:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:31:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:31:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:31:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:31:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:31:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:31:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:31:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:31:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:31:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:31:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:31:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:31:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:31:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:31:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:31:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:31:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:31:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:31:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:31:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:31:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:31:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:31:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:31:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:31:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:31:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:31:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:31:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:31:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:31:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:31:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:31:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:31:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:31:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:31:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:31:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:31:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:31:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:31:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:31:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:31:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:31:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:31:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:31:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:31:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:31:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:31:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:31:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:31:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:31:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:31:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:31:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:31:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:31:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:31:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:31:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:31:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:31:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:31:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:31:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:31:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:31:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:31:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:31:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:31:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:31:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:31:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:31:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:31:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:31:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:31:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:31:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:31:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:31:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:31:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:31:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:31:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:31:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:31:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:31:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:31:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:31:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:31:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:31:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:31:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:31:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:31:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:31:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:31:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:31:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:31:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:31:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:31:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:31:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:31:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:31:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:31:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:31:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:31:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:31:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:31:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:31:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:31:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:31:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:31:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:31:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:31:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:31:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:31:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:31:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:31:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:31:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:31:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:31:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:31:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:31:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:31:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:31:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:31:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:31:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:31:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:31:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:31:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:31:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:31:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:31:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:31:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:31:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:31:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:31:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:31:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:31:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:31:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:31:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:31:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:31:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:31:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:31:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:31:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:31:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:31:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:31:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:31:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:31:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:31:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:31:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:31:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:31:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:31:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:31:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:31:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:31:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:31:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:31:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:31:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:31:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:31:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:31:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:31:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:31:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:31:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:31:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:31:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:31:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:31:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:31:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:31:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:31:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:31:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:31:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:31:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:31:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:31:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:31:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:31:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:31:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:31:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:31:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:31:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:31:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:31:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:31:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:31:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:31:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:31:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:31:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:31:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:31:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:31:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:32:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:32:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:32:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:32:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:32:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:32:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:32:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:32:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:32:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:32:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:32:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:32:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:32:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:32:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:32:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:32:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:32:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:32:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:32:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:32:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:32:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:32:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:32:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:32:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:32:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:32:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:32:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:32:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:32:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:32:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:32:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:32:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:32:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:32:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:32:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:32:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:32:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:32:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:32:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:32:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:32:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:32:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:32:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:32:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:32:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:32:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:32:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:32:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:32:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:32:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:32:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:32:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:32:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:32:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:32:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:32:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:32:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:32:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:32:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:32:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:32:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:32:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:32:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:32:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:32:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:32:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:32:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:32:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:32:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:32:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:32:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:32:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:32:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:32:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:32:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:32:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:32:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:32:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:32:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:32:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:32:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:32:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:32:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:32:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:32:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:32:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:32:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:32:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:32:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:32:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:32:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:32:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:32:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:32:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:32:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:32:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:32:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:32:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:32:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:32:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:32:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:32:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:32:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:32:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:32:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:32:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:32:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:32:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:32:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:32:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:32:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:32:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:32:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:32:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:32:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:32:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:32:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:32:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:32:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:32:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:32:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:32:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:32:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:32:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:32:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:32:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:32:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:32:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:32:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:32:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:32:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:32:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:32:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:32:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:32:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:32:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:32:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:32:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:32:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:32:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:32:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:32:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:32:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:32:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:32:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:32:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:32:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:32:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:32:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:32:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:32:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:32:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:32:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:32:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:32:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:32:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:32:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:32:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:32:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:32:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:32:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:32:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:32:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:32:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:32:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:32:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:32:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:32:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:32:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:32:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:32:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:32:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:32:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:32:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:32:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:32:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:32:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:32:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:32:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:32:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:32:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:32:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:32:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:32:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:32:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:32:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:32:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:32:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:32:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:32:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:32:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:32:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:32:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:32:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:32:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:32:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:32:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:32:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:32:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:32:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:32:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:32:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:32:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:32:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:32:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:32:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:32:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:32:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:32:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:32:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:32:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:32:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:32:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:32:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:32:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:32:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:33:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:33:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:33:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:33:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:33:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:33:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:33:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:33:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:33:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:33:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:33:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:33:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:33:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:33:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:33:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:33:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:33:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:33:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:33:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:33:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:33:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:33:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:33:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:33:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:33:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:33:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:33:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:33:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:33:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:33:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:33:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:33:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:33:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:33:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:33:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:33:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:33:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:33:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:33:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:33:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:33:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:33:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:33:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:33:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:33:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:33:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:33:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:33:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:33:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:33:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:33:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:33:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:33:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:33:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:33:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:33:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:33:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:33:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:33:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:33:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:33:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:33:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:33:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:33:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:33:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:33:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:33:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:33:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:33:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:33:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:33:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:33:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:33:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:33:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:33:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:33:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:33:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:33:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:33:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:33:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:33:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:33:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:33:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:33:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:33:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:33:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:33:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:33:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:33:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:33:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:33:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:33:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:33:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:33:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:33:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:33:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:33:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:33:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:33:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:33:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:33:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:33:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:33:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:33:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:33:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:33:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:33:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:33:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:33:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:33:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:33:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:33:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:33:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:33:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:33:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:33:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:33:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:33:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:33:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:33:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:33:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:33:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:33:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:33:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:33:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:33:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:33:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:33:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:33:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:33:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:33:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:33:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:33:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:33:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:33:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:33:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:33:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:33:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:33:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:33:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:33:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:33:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:33:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:33:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:33:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:33:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:33:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:33:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:33:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:33:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:33:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:33:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:33:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:33:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:33:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:33:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:33:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:33:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:33:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:33:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:33:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:33:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:33:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:33:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:33:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:33:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:33:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:33:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:33:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:33:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:33:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:33:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:33:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:33:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:33:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:33:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:33:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:33:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:33:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:33:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:33:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:33:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:33:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:33:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:33:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:33:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:33:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:33:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:33:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:33:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:33:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:33:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:33:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:33:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:33:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:33:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:33:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:33:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:33:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:33:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:33:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:33:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:33:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:33:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:33:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:33:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:33:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:33:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:33:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:33:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:33:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:33:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:33:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:33:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:33:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:33:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:34:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:34:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:34:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:34:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:34:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:34:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:34:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:34:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:34:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:34:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:34:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:34:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:34:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:34:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:34:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:34:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:34:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:34:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:34:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:34:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:34:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:34:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:34:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:34:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:34:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:34:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:34:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:34:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:34:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:34:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:34:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:34:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:34:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:34:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:34:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:34:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:34:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:34:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:34:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:34:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:34:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:34:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:34:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:34:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:34:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:34:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:34:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:34:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:34:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:34:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:34:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:34:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:34:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:34:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:34:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:34:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:34:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:34:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:34:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:34:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:34:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:34:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:34:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:34:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:34:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:34:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:34:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:34:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:34:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:34:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:34:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:34:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:34:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:34:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:34:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:34:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:34:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:34:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:34:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:34:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:34:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:34:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:34:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:34:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:34:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:34:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:34:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:34:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:34:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:34:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:34:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:34:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:34:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:34:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:34:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:34:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:34:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:34:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:34:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:34:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:34:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:34:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:34:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:34:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:34:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:34:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:34:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:34:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:34:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:34:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:34:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:34:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:34:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:34:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:34:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:34:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:34:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:34:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:34:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:34:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:34:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:34:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:34:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:34:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:34:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:34:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:34:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:34:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:34:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:34:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:34:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:34:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:34:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:34:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:34:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:34:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:34:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:34:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:34:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:34:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:34:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:34:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:34:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:34:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:34:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:34:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:34:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:34:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:34:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:34:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:34:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:34:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:34:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:34:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:34:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:34:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:34:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:34:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:34:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:34:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:34:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:34:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:34:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:34:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:34:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:34:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:34:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:34:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:34:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:34:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:34:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:34:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:34:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:34:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:34:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:34:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:34:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:34:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:34:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:34:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:34:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:34:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:34:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:34:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:34:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:34:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:34:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:34:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:34:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:34:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:34:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:34:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:34:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:34:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:34:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:34:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:34:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:34:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:34:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:34:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:34:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:34:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:34:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:34:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:34:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:34:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:34:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:34:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:34:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:34:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:34:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:34:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:34:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:34:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:34:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:34:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:35:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:35:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:35:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:35:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:35:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:35:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:35:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:35:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:35:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:35:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:35:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:35:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:35:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:35:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:35:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:35:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:35:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:35:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:35:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:35:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:35:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:35:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:35:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:35:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:35:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:35:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:35:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:35:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:35:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:35:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:35:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:35:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:35:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:35:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:35:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:35:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:35:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:35:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:35:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:35:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:35:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:35:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:35:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:35:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:35:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:35:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:35:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:35:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:35:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:35:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:35:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:35:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:35:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:35:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:35:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:35:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:35:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:35:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:35:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:35:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:35:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:35:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:35:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:35:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:35:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:35:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:35:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:35:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:35:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:35:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:35:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:35:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:35:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:35:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:35:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:35:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:35:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:35:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:35:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:35:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:35:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:35:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:35:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:35:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:35:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:35:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:35:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:35:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:35:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:35:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:35:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:35:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:35:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:35:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:35:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:35:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:35:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:35:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:35:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:35:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:35:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:35:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:35:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:35:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:35:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:35:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:35:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:35:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:35:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:35:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:35:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:35:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:35:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:35:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:35:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:35:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:35:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:35:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:35:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:35:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:35:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:35:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:35:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:35:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:35:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:35:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:35:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:35:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:35:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:35:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:35:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:35:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:35:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:35:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:35:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:35:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:35:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:35:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:35:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:35:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:35:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:35:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:35:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:35:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:35:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:35:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:35:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:35:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:35:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:35:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:35:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:35:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:35:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:35:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:35:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:35:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:35:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:35:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:35:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:35:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:35:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:35:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:35:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:35:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:35:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:35:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:35:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:35:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:35:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:35:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:35:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:35:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:35:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:35:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:35:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:35:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:35:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:35:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:35:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:35:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:35:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:35:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:35:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:35:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:35:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:35:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:35:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:35:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:35:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:35:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:35:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:35:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:35:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:35:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:35:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:35:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:35:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:35:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:35:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:35:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:35:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:35:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:35:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:35:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:35:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:35:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:35:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:35:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:35:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:35:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:35:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:35:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:35:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:35:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:35:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:35:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:36:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:36:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:36:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:36:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:36:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:36:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:36:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:36:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:36:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:36:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:36:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:36:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:36:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:36:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:36:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:36:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:36:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:36:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:36:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:36:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:36:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:36:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:36:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:36:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:36:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:36:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:36:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:36:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:36:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:36:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:36:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:36:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:36:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:36:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:36:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:36:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:36:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:36:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:36:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:36:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:36:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:36:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:36:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:36:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:36:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:36:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:36:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:36:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:36:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:36:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:36:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:36:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:36:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:36:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:36:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:36:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:36:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:36:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:36:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:36:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:36:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:36:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:36:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:36:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:36:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:36:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:36:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:36:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:36:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:36:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:36:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:36:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:36:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:36:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:36:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:36:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:36:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:36:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:36:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:36:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:36:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:36:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:36:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:36:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:36:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:36:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:36:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:36:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:36:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:36:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:36:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:36:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:36:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:36:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:36:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:36:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:36:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:36:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:36:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:36:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:36:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:36:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:36:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:36:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:36:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:36:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:36:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:36:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:36:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:36:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:36:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:36:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:36:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:36:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:36:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:36:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:36:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:36:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:36:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:36:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:36:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:36:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:36:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:36:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:36:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:36:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:36:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:36:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:36:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:36:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:36:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:36:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:36:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:36:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:36:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:36:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:36:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:36:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:36:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:36:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:36:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:36:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:36:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:36:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:36:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:36:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:36:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:36:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:36:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:36:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:36:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:36:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:36:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:36:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:36:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:36:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:36:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:36:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:36:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:36:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:36:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:36:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:36:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:36:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:36:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:36:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:36:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:36:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:36:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:36:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:36:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:36:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:36:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:36:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:36:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:36:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:36:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:36:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:36:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:36:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:36:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:36:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:36:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:36:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:36:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:36:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:36:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:36:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:36:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:36:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:36:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:36:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:36:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:36:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:36:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:36:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:36:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:36:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:36:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:36:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:36:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:36:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:36:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:36:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:36:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:36:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:36:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:36:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:36:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:36:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:36:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:36:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:36:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:36:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:36:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:36:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:37:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:37:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:37:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:37:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:37:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:37:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:37:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:37:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:37:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:37:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:37:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:37:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:37:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:37:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:37:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:37:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:37:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:37:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:37:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:37:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:37:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:37:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:37:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:37:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:37:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:37:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:37:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:37:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:37:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:37:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:37:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:37:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:37:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:37:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:37:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:37:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:37:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:37:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:37:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:37:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:37:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:37:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:37:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:37:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:37:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:37:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:37:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:37:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:37:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:37:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:37:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:37:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:37:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:37:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:37:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:37:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:37:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:37:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:37:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:37:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:37:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:37:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:37:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:37:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:37:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:37:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:37:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:37:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:37:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:37:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:37:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:37:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:37:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:37:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:37:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:37:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:37:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:37:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:37:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:37:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:37:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:37:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:37:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:37:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:37:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:37:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:37:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:37:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:37:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:37:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:37:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:37:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:37:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:37:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:37:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:37:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:37:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:37:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:37:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:37:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:37:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:37:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:37:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:37:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:37:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:37:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:37:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:37:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:37:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:37:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:37:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:37:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:37:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:37:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:37:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:37:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:37:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:37:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:37:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:37:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:37:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:37:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:37:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:37:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:37:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:37:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:37:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:37:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:37:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:37:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:37:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:37:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:37:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:37:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:37:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:37:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:37:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:37:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:37:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:37:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:37:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:37:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:37:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:37:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:37:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:37:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:37:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:37:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:37:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:37:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:37:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:37:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:37:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:37:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:37:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:37:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:37:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:37:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:37:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:37:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:37:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:37:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:37:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:37:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:37:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:37:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:37:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:37:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:37:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:37:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:37:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:37:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:37:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:37:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:37:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:37:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:37:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:37:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:37:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:37:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:37:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:37:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:37:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:37:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:37:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:37:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:37:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:37:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:37:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:37:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:37:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:37:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:37:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:37:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:37:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:37:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:37:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:37:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:37:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:37:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:37:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:37:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:37:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:37:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:37:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:37:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:37:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:37:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:37:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:37:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:37:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:37:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:37:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:37:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:37:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:37:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:40:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:40:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:40:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:40:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:40:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:40:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:40:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:40:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:40:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:40:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:40:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:40:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:40:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:40:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:40:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:40:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:40:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:40:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:40:09 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 19 00:40:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:40:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:40:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:40:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:40:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:40:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:40:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:40:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:40:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:40:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:40:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:40:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:40:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:40:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:40:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:40:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:40:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:40:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:40:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:40:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:40:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:40:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:40:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:40:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:40:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:40:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:40:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:40:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:40:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:40:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:40:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:40:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:40:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:40:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:40:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:40:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:40:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:40:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:40:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:40:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:40:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:40:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:40:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:40:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:40:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:40:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:40:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:40:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:40:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:40:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:40:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:40:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:40:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:40:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:40:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:40:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:40:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:40:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:40:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:40:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:40:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:40:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:40:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:40:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:40:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:40:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:40:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:40:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:40:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:40:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:40:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:40:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:40:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:40:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:40:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:40:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:40:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:40:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:40:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:40:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:40:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:40:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:40:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:40:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:40:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:40:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:40:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:40:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:40:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:40:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:40:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:40:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:40:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:40:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:40:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:40:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:40:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:40:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:40:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:40:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:40:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:40:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:40:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:40:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:40:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:40:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:40:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:40:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:40:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:40:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:40:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:40:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:40:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:40:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:40:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:40:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:40:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:40:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:40:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:40:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:40:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:40:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:40:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:40:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:40:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:40:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:40:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:40:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:40:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:40:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:40:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:40:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:40:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:40:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:40:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:40:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:40:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:40:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:40:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:40:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:40:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:40:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:40:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:40:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:40:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:40:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:40:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:40:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:40:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:40:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:40:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:40:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:40:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:40:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:40:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:40:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:40:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:40:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:40:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:40:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:40:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:40:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:40:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:40:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:40:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:40:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:40:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:40:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:40:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:40:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:40:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:40:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:40:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:40:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:40:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:40:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:40:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:40:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:40:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:40:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:41:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:41:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:41:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:41:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:41:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:41:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:41:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:41:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:41:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:41:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:41:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:41:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:41:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:41:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:41:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:41:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:41:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:41:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:41:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:41:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:41:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:41:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:41:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:41:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:41:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:41:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:41:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:41:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:41:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:41:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:41:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:41:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:41:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:41:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:41:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:41:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:41:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:41:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:41:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:41:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:41:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:41:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:41:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:41:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:41:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:41:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:41:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:41:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:41:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:41:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:41:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:41:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:41:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:41:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:41:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:41:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:41:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:41:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:41:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:41:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:41:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:41:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:41:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:41:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:41:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:41:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:41:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:41:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:41:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:41:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:41:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:41:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:41:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:41:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:41:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:41:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:41:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:41:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:41:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:41:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:41:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:41:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:41:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:41:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:41:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:41:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:41:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:41:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:41:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:41:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:41:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:41:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:41:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:41:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:41:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:41:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:41:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:41:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:41:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:41:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:41:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:41:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:41:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:41:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:41:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:41:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:41:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:41:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:41:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:41:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:41:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:41:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:41:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:41:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:41:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:41:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:41:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:41:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:41:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:41:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:41:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:41:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:41:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:41:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:41:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:41:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:41:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:41:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:41:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:41:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:41:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:41:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:41:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:41:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:41:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:41:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:41:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:41:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:41:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:41:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:41:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:41:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:41:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:41:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:41:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:41:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:41:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:41:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:41:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:41:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:41:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:41:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:41:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:41:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:41:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:41:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:41:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:41:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:41:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:41:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:41:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:41:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:41:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:41:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:41:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:41:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:41:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:41:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:41:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:41:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:41:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:41:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:41:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:41:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:41:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:41:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:41:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:41:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:41:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:41:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:41:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:41:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:41:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:41:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:41:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:41:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:41:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:41:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:41:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:41:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:41:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:41:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:41:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:41:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:41:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:41:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:41:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:41:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:41:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:41:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:41:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:41:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:41:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:41:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:41:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:41:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:41:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:41:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:41:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:41:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:41:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:41:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:41:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:41:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:41:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:41:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:42:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:42:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:42:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:42:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:42:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:42:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:42:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:42:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:42:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:42:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:42:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:42:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:42:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:42:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:42:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:42:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:42:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:42:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:42:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:42:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:42:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:42:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:42:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:42:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:42:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:42:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:42:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:42:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:42:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:42:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:42:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:42:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:42:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:42:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:42:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:42:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:42:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:42:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:42:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:42:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:42:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:42:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:42:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:42:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:42:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:42:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:42:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:42:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:42:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:42:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:42:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:42:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:42:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:42:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:42:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:42:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:42:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:42:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:42:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:42:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:42:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:42:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:42:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:42:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:42:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:42:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:42:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:42:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:42:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:42:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:42:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:42:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:42:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:42:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:42:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:42:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:42:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:42:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:42:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:42:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:42:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:42:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:42:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:42:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:42:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:42:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:42:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:42:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:42:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:42:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:42:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:42:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:42:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:42:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:42:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:42:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:42:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:42:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:42:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:42:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:42:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:42:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:42:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:42:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:42:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:42:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:42:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:42:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:42:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:42:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:42:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:42:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:42:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:42:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:42:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:42:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:42:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:42:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:42:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:42:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:42:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:42:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:42:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:42:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:42:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:42:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:42:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:42:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:42:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:42:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:42:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:42:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:42:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:42:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:42:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:42:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:42:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:42:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:42:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:42:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:42:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:42:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:42:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:42:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:42:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:42:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:42:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:42:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:42:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:42:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:42:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:42:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:42:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:42:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:42:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:42:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:42:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:42:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:42:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:42:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:42:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:42:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:42:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:42:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:42:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:42:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:42:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:42:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:42:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:42:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:42:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:42:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:42:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:42:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:42:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:42:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:42:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:42:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:42:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:42:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:42:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:42:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:42:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:42:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:42:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:42:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:42:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:42:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:42:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:42:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:42:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:42:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:42:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:42:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:42:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:42:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:42:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:42:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:42:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:42:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:42:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:42:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:42:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:42:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:42:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:42:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:42:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:42:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:42:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:42:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:42:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:42:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:42:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:42:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:42:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:42:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:43:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:43:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:43:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:43:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:43:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:43:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:43:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:43:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:43:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:43:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:43:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:43:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:43:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:43:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:43:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:43:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:43:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:43:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:43:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:43:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:43:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:43:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:43:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:43:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:43:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:43:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:43:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:43:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:43:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:43:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:43:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:43:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:43:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:43:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:43:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:43:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:43:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:43:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:43:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:43:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:43:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:43:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:43:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:43:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:43:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:43:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:43:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:43:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:43:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:43:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:43:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:43:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:43:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:43:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:43:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:43:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:43:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:43:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:43:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:43:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:43:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:43:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:43:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:43:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:43:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:43:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:43:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:43:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:43:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:43:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:43:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:43:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:43:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:43:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:43:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:43:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:43:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:43:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:43:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:43:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:43:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:43:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:43:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:43:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:43:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:43:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:43:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:43:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:43:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:43:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:43:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:43:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:43:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:43:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:43:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:43:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:43:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:43:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:43:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:43:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:43:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:43:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:43:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:43:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:43:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:43:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:43:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:43:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:43:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:43:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:43:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:43:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:43:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:43:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:43:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:43:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:43:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:43:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:43:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:43:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:43:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:43:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:43:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:43:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:43:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:43:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:43:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:43:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:43:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:43:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:43:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:43:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:43:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:43:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:43:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:43:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:43:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:43:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:43:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:43:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:43:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:43:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:43:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:43:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:43:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:43:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:43:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:43:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:43:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:43:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:43:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:43:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:43:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:43:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:43:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:43:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:43:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:43:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:43:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:43:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:43:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:43:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:43:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:43:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:43:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:43:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:43:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:43:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:43:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:43:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:43:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:43:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:43:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:43:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:43:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:43:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:43:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:43:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:43:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:43:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:43:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:43:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:43:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:43:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:43:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:43:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:43:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:43:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:43:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:43:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:43:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:43:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:43:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:43:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:43:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:43:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:43:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:43:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:43:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:43:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:43:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:43:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:43:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:43:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:43:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:43:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:43:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:43:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:43:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:43:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:43:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:43:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:43:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:43:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:43:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:43:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:44:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:44:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:44:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:44:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:44:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:44:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:44:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:44:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:44:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:44:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:44:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:44:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:44:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:44:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:44:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:44:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:44:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:44:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:44:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:44:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:44:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:44:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:44:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:44:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:44:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:44:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:44:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:44:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:44:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:44:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:44:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:44:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:44:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:44:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:44:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:44:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:44:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:44:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:44:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:44:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:44:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:44:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:44:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:44:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:44:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:44:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:44:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:44:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:44:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:44:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:44:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:44:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:44:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:44:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:44:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:44:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:44:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:44:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:44:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:44:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:44:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:44:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:44:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:44:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:44:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:44:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:44:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:44:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:44:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:44:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:44:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:44:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:44:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:44:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:44:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:44:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:44:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:44:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:44:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:44:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:44:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:44:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:44:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:44:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:44:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:44:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:44:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:44:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:44:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:44:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:44:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:44:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:44:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:44:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:44:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:44:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:44:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:44:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:44:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:44:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:44:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:44:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:44:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:44:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:44:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:44:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:44:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:44:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:44:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:44:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:44:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:44:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:44:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:44:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:44:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:44:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:44:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:44:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:44:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:44:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:44:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:44:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:44:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:44:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:44:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:44:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:44:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:44:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:44:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:44:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:44:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:44:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:44:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:44:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:44:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:44:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:44:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:44:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:44:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:44:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:44:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:44:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:44:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:44:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:44:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:44:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:44:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:44:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:44:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:44:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:44:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:44:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:44:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:44:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:44:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:44:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:44:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:44:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:44:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:44:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:44:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:44:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:44:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:44:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:44:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:44:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:44:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:44:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:44:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:44:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:44:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:44:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:44:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:44:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:44:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:44:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:44:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:44:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:44:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:44:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:44:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:44:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:44:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:44:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:44:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:44:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:44:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:44:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:44:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:44:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:44:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:44:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:44:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:44:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:44:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:44:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:44:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:44:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:44:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:44:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:44:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:44:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:44:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:44:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:44:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:44:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:44:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:44:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:44:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:44:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:44:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:44:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:44:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:44:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:44:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:44:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:45:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:45:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:45:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:45:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:45:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:45:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:45:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:45:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:45:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:45:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:45:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:45:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:45:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:45:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:45:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:45:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:45:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:45:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:45:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:45:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:45:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:45:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:45:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:45:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:45:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:45:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:45:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:45:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:45:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:45:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:45:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:45:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:45:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:45:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:45:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:45:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:45:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:45:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:45:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:45:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:45:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:45:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:45:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:45:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:45:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:45:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:45:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:45:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:45:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:45:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:45:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:45:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:45:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:45:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:45:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:45:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:45:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:45:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:45:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:45:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:45:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:45:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:45:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:45:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:45:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:45:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:45:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:45:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:45:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:45:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:45:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:45:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:45:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:45:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:45:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:45:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:45:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:45:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:45:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:45:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:45:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:45:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:45:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:45:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:45:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:45:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:45:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:45:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:45:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:45:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:45:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:45:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:45:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:45:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:45:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:45:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:45:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:45:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:45:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:45:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:45:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:45:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:45:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:45:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:45:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:45:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:45:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:45:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:45:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:45:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:45:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:45:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:45:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:45:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:45:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:45:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:45:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:45:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:45:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:45:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:45:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:45:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:45:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:45:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:45:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:45:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:45:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:45:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:45:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:45:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:45:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:45:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:45:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:45:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:45:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:45:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:45:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:45:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:45:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:45:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:45:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:45:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:45:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:45:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:45:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:45:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:45:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:45:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:45:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:45:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:45:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:45:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:45:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:45:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:45:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:45:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:45:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:45:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:45:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:45:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:45:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:45:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:45:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:45:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:45:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:45:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:45:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:45:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:45:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:45:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:45:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:45:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:45:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:45:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:45:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:45:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:45:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:45:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:45:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:45:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:45:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:45:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:45:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:45:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:45:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:45:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:45:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:45:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:45:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:45:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:45:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:45:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:45:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:45:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:45:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:45:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:45:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:45:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:45:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:45:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:45:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:45:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:45:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:45:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:45:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:45:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:45:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:45:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:45:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:45:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:45:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:45:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:45:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:45:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:45:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:45:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:46:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:46:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:46:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:46:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:46:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:46:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:46:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:46:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:46:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:46:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:46:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:46:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:46:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:46:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:46:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:46:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:46:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:46:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:46:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:46:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:46:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:46:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:46:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:46:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:46:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:46:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:46:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:46:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:46:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:46:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:46:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:46:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:46:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:46:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:46:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:46:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:46:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:46:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:46:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:46:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:46:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:46:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:46:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:46:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:46:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:46:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:46:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:46:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:46:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:46:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:46:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:46:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:46:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:46:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:46:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:46:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:46:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:46:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:46:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:46:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:46:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:46:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:46:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:46:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:46:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:46:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:46:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:46:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:46:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:46:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:46:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:46:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:46:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:46:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:46:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:46:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:46:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:46:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:46:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:46:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:46:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:46:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:46:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:46:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:46:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:46:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:46:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:46:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:46:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:46:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:46:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:46:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:46:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:46:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:46:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:46:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:46:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:46:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:46:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:46:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:46:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:46:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:46:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:46:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:46:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:46:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:46:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:46:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:46:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:46:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:46:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:46:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:46:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:46:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:46:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:46:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:46:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:46:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:46:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:46:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:46:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:46:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:46:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:46:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:46:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:46:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:46:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:46:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:46:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:46:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:46:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:46:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:46:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:46:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:46:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:46:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:46:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:46:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:46:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:46:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:46:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:46:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:46:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:46:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:46:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:46:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:46:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:46:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:46:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:46:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:46:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:46:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:46:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:46:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:46:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:46:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:46:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:46:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:46:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:46:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:46:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:46:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:46:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:46:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:46:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:46:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:46:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:46:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:46:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:46:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:46:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:46:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:46:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:46:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:46:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:46:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:46:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:46:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:46:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:46:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:46:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:46:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:46:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:46:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:46:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:46:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:46:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:46:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:46:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:46:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:46:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:46:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:46:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:46:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:46:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:46:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:46:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:46:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:46:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:46:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:46:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:46:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:46:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:46:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:46:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:46:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:46:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:46:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:46:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:46:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:46:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:46:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:46:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:46:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:46:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:46:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:47:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:47:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:47:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:47:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:47:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:47:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:47:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:47:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:47:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:47:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:47:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:47:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:47:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:47:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:47:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:47:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:47:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:47:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:47:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:47:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:47:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:47:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:47:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:47:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:47:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:47:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:47:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:47:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:47:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:47:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:47:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:47:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:47:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:47:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:47:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:47:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:47:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:47:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:47:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:47:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:47:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:47:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:47:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:47:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:47:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:47:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:47:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:47:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:47:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:47:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:47:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:47:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:47:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:47:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:47:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:47:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:47:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:47:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:47:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:47:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:47:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:47:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:47:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:47:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:47:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:47:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:47:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:47:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:47:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:47:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:47:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:47:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:47:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:47:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:47:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:47:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:47:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:47:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:47:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:47:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:47:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:47:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:47:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:47:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:47:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:47:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:47:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:47:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:47:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:47:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:47:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:47:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:47:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:47:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:47:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:47:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:47:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:47:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:47:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:47:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:47:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:47:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:47:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:47:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:47:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:47:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:47:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:47:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:47:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:47:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:47:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:47:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:47:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:47:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:47:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:47:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:47:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:47:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:47:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:47:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:47:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:47:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:47:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:47:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:47:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:47:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:47:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:47:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:47:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:47:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:47:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:47:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:47:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:47:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:47:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:47:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:47:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:47:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:47:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:47:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:47:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:47:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:47:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:47:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:47:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:47:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:47:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:47:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:47:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:47:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:47:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:47:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:47:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:47:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:47:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:47:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:47:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:47:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:47:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:47:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:47:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:47:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:47:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:47:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:47:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:47:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:47:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:47:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:47:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:47:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:47:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:47:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:47:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:47:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:47:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:47:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:48:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:48:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:48:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:48:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:48:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:48:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:48:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:48:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:48:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:48:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:48:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:48:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:48:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:48:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:48:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:48:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:48:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:48:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:50:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:50:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:50:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:50:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:50:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:50:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:50:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:50:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:50:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:50:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:50:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:50:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:50:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:50:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:50:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:50:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:50:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:50:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:50:14 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 19 00:50:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:50:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:50:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:50:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:50:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:50:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:50:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:50:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:50:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:50:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:50:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:50:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:50:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:50:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:50:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:50:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:50:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:50:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:50:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:50:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:50:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:50:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:50:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:50:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:50:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:50:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:50:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:50:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:50:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:50:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:50:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:50:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:50:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:50:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:50:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:50:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:50:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:50:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:50:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:50:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:50:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:50:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:50:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:50:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:50:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:50:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:50:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:50:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:50:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:50:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:50:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:50:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:50:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:50:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:50:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:50:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:50:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:50:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:50:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:50:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:50:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:50:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:50:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:50:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:50:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:50:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:50:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:50:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:50:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:50:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:50:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:50:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:50:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:50:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:50:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:50:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:50:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:50:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:50:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:50:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:50:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:50:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:50:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:50:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:50:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:50:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:50:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:50:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:50:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:50:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:50:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:50:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:50:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:50:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:50:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:50:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:50:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:50:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:50:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:50:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:50:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:50:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:50:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:50:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:50:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:50:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:50:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:50:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:50:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:50:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:50:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:50:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:50:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:50:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:50:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:50:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:50:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:50:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:50:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:50:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:50:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:50:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:50:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:50:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:50:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:50:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:50:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:50:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:50:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:50:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:50:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:50:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:50:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:50:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:50:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:50:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:50:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:50:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:50:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:50:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:50:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:50:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:50:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:50:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:50:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:50:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:50:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:50:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:50:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:50:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:50:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:50:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:50:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:50:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:50:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:50:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:50:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:50:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:50:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:50:59 rdma06 systemd-journald[790]: Data hash table of /run/log/journal/95379d2f723847b68d9e976dee8967a9/system.journal has a fill level at 75.0 (163750 of 218332 items, 75497472 file size, 461 bytes per hash table item), suggesting rotation.
Apr 19 00:50:59 rdma06 systemd-journald[790]: /run/log/journal/95379d2f723847b68d9e976dee8967a9/system.journal: Journal header limits reached or header out-of-date, rotating.
Apr 19 00:50:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:50:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:50:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:50:59 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 19 00:50:59 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 19 00:50:59 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 19 00:50:59 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 19 00:51:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:51:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:51:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:51:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:51:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:51:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:51:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:51:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:51:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:51:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:51:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:51:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:51:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:51:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:51:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:51:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:51:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:51:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:51:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:51:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:51:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:51:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:51:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:51:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:51:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:51:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:51:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:51:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:51:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:51:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:51:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:51:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:51:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:51:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:51:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:51:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:51:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:51:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:51:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:51:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:51:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:51:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:51:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:51:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:51:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:51:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:51:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:51:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:51:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:51:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:51:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:51:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:51:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:51:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:51:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:51:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:51:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:51:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:51:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:51:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:51:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:51:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:51:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:51:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:51:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:51:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:51:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:51:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:51:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:51:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:51:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:51:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:51:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:51:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:51:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:51:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:51:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:51:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:51:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:51:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:51:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:51:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:51:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:51:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:51:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:51:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:51:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:51:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:51:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:51:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:51:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:51:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:51:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:51:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:51:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:51:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:51:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:51:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:51:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:51:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:51:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:51:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:51:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:51:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:51:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:51:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:51:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:51:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:51:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:51:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:51:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:51:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:51:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:51:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:51:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:51:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:51:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:51:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:51:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:51:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:51:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:51:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:51:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:51:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:51:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:51:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:51:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:51:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:51:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:51:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:51:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:51:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:51:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:51:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:51:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:51:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:51:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:51:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:51:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:51:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:51:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:51:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:51:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:51:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:51:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:51:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:51:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:51:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:51:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:51:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:51:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:51:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:51:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:51:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:51:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:51:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:51:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:51:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:51:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:51:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:51:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:51:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:51:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:51:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:51:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:51:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:51:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:51:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:51:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:51:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:51:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:51:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:51:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:51:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:51:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:51:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:51:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:51:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:51:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:51:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:51:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:51:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:51:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:51:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:51:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:51:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:51:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:51:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:51:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:51:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:51:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:51:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:51:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:51:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:51:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:51:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:51:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:51:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:51:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:51:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:51:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:51:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:51:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:51:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:51:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:51:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:51:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:51:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:51:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:51:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:51:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:51:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:51:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:51:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:51:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:51:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:52:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:52:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:52:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:52:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:52:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:52:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:52:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:52:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:52:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:52:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:52:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:52:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:52:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:52:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:52:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:52:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:52:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:52:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:52:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:52:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:52:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:52:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:52:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:52:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:52:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:52:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:52:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:52:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:52:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:52:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:52:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:52:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:52:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:52:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:52:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:52:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:52:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:52:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:52:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:52:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:52:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:52:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:52:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:52:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:52:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:52:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:52:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:52:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:52:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:52:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:52:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:52:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:52:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:52:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:52:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:52:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:52:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:52:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:52:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:52:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:52:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:52:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:52:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:52:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:52:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:52:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:52:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:52:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:52:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:52:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:52:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:52:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:52:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:52:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:52:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:52:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:52:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:52:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:52:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:52:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:52:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:52:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:52:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:52:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:52:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:52:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:52:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:52:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:52:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:52:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:52:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:52:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:52:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:52:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:52:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:52:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:52:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:52:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:52:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:52:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:52:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:52:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:52:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:52:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:52:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:52:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:52:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:52:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:52:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:52:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:52:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:52:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:52:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:52:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:52:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:52:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:52:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:52:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:52:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:52:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:52:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:52:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:52:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:52:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:52:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:52:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:52:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:52:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:52:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:52:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:52:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:52:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:52:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:52:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:52:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:52:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:52:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:52:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:52:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:52:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:52:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:52:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:52:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:52:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:52:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:52:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:52:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:52:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:52:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:52:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:52:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:52:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:52:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:52:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:52:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:52:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:52:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:52:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:52:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:52:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:52:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:52:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:52:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:52:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:52:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:52:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:52:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:52:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:52:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:52:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:52:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:52:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:52:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:52:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:52:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:52:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:52:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:52:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:52:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:52:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:52:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:52:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:52:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:52:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:52:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:52:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:52:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:52:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:52:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:52:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:52:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:52:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:52:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:52:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:52:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:52:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:52:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:52:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:52:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:52:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:52:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:52:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:52:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:52:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:52:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:52:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:52:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:52:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:52:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:52:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:52:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:52:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:52:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:52:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:52:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:52:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:53:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:53:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:53:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:53:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:53:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:53:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:53:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:53:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:53:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:53:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:53:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:53:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:53:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:53:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:53:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:53:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:53:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:53:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:53:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:53:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:53:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:53:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:53:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:53:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:53:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:53:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:53:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:53:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:53:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:53:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:53:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:53:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:53:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:53:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:53:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:53:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:53:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:53:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:53:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:53:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:53:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:53:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:53:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:53:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:53:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:53:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:53:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:53:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:53:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:53:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:53:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:53:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:53:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:53:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:53:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:53:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:53:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:53:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:53:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:53:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:53:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:53:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:53:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:53:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:53:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:53:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:53:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:53:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:53:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:53:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:53:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:53:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:53:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:53:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:53:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:53:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:53:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:53:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:53:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:53:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:53:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:53:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:53:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:53:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:53:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:53:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:53:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:53:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:53:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:53:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:53:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:53:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:53:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:53:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:53:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:53:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:53:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:53:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:53:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:53:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:53:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:53:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:53:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:53:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:53:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:53:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:53:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:53:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:53:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:53:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:53:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:53:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:53:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:53:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:53:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:53:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:53:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:53:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:53:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:53:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:53:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:53:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:53:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:53:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:53:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:53:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:53:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:53:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:53:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:53:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:53:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:53:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:53:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:53:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:53:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:53:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:54:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:54:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:54:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:54:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:54:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:54:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:54:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:54:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:54:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:54:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:54:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:54:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:54:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:54:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:54:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:54:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:54:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:54:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:54:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:54:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:54:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:54:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:54:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:54:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:54:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:54:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:54:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:54:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:54:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:54:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:54:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:54:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:54:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:54:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:54:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:54:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:54:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:54:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:54:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:54:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:54:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:54:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:54:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:54:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:54:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:54:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:54:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:54:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:54:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:54:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:54:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:54:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:54:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:54:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:54:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:54:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:54:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:54:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:54:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:54:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:54:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:54:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:54:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:54:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:54:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:54:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:54:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:54:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:54:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:54:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:54:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:54:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:54:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:54:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:54:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:54:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:54:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:54:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:54:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:54:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:54:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:54:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:54:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:54:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:54:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:54:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:54:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:54:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:54:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:54:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:54:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:54:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:54:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:54:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:54:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:54:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:54:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:54:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:54:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:54:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:54:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:54:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:54:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:54:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:54:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:54:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:54:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:54:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:54:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:54:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:54:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:54:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:54:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:54:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:54:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:54:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:54:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:54:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:54:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:54:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:54:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:54:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:54:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:54:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:54:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:54:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:54:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:54:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:54:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:54:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:54:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:54:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:54:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:54:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:54:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:54:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:54:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:54:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:54:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:54:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:54:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:54:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:54:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:54:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:54:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:54:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:54:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:54:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:54:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:54:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:54:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:54:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:54:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:54:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:54:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:54:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:54:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:54:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:54:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:54:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:54:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:54:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:54:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:54:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:54:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:54:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:54:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:54:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:54:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:54:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:54:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:54:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:54:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:54:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:54:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:54:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:55:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:55:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:55:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:55:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:55:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:55:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:55:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:55:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:55:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:55:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:55:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:55:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:55:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:55:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:55:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:55:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:55:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:55:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:55:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:55:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:55:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:55:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:55:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:55:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:55:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:55:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:55:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:55:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:55:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:55:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:55:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:55:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:55:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:55:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:55:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:55:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:55:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:55:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:55:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:55:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:55:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:55:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:55:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:55:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:55:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:55:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:55:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:55:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:55:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:55:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:55:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:55:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:55:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:55:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:55:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:55:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:55:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:55:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:55:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:55:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:55:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:55:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:55:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:55:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:55:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:55:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:55:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:55:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:55:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:55:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:55:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:55:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:55:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:55:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:55:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:55:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:55:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:55:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:55:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:55:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:55:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:55:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:55:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:55:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:55:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:55:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:55:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:55:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:55:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:55:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:55:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:55:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:55:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:55:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:55:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:55:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:55:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:55:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:55:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:55:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:55:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:55:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:55:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:55:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:55:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:55:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:55:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:55:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:55:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:55:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:55:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:55:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:55:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:55:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:55:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:55:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:55:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:55:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:55:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:55:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:55:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:55:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:55:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:55:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:55:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:55:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:55:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:55:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:55:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:55:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:55:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:55:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:55:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:55:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:55:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:55:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:55:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:55:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:55:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:55:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:55:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:55:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:55:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:55:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:55:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:55:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:55:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:55:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:55:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:55:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:55:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:55:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:55:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:55:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:55:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:55:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:55:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:55:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:55:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:55:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:55:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:55:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:55:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:55:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:55:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:55:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:55:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:55:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:55:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:55:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:55:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:55:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:55:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:55:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:55:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:55:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:55:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:55:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:55:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:55:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:55:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:55:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:55:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:55:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:55:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:55:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:55:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:55:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:55:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:55:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:55:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:55:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:55:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:55:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:55:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:55:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:55:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:55:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:55:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:55:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:55:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:55:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:55:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:55:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:55:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:55:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:55:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:55:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:55:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:55:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:55:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:55:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:55:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:55:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:55:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:55:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:56:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:56:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:56:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:56:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:56:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:56:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:56:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:56:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:56:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:56:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:56:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:56:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:56:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:56:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:56:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:56:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:56:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:56:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:56:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:56:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:56:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:56:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:56:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:56:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:56:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:56:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:56:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:56:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:56:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:56:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:56:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:56:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:56:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:56:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:56:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:56:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:56:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:56:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:56:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:56:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:56:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:56:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:56:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:56:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:56:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:56:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:56:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:56:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:56:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:56:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:56:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:56:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:56:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:56:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:56:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:56:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:56:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:56:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:56:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:56:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:56:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:56:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:56:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:56:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:56:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:56:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:56:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:56:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:56:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:56:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:56:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:56:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:56:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:56:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:56:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:56:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:56:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:56:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:56:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:56:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:56:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:56:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:56:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:56:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:56:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:56:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:56:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:56:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:56:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:56:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:56:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:56:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:56:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:56:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:56:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:56:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:56:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:56:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:56:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:56:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:56:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:56:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:56:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:56:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:56:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:56:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:56:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:56:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:56:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:56:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:56:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:56:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:56:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:56:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:56:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:56:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:56:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:56:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:56:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:56:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:56:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:56:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:56:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:56:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:56:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:56:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:56:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:56:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:56:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:56:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:56:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:56:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:56:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:56:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:56:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:56:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:56:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:56:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:56:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:56:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:56:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:56:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:56:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:56:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:56:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:56:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:56:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:56:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:56:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:56:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:56:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:56:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:56:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:56:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:56:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:56:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:56:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:56:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:56:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:56:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:56:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:56:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:56:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:56:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:56:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:56:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:56:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:56:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:56:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:56:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:56:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:56:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:56:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:56:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:56:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:56:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:56:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:56:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:56:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:56:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:56:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:56:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:56:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:56:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:56:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:56:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:56:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:56:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:56:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:56:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:56:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:56:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:56:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:56:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:56:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:56:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:56:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:56:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:56:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:56:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:56:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:56:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:56:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:56:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:56:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:56:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:56:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:56:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:56:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:56:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:56:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:56:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:56:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:56:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:56:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:56:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:57:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:57:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:57:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:57:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:57:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:57:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:57:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:57:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:57:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:57:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:57:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:57:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:57:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:57:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:57:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:57:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:57:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:57:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:57:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:57:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:57:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:57:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:57:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:57:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:57:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:57:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:57:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:57:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:57:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:57:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:57:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:57:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:57:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:57:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:57:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:57:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:57:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:57:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:57:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:57:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:57:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:57:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:57:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:57:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:57:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:57:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:57:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:57:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:57:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:57:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:57:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:57:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:57:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:57:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:57:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:57:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:57:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:57:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:57:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:57:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:57:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:57:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:57:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:57:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:57:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:57:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:57:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:57:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:57:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:57:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:57:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:57:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:57:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:57:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:57:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:57:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:57:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:57:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:57:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:57:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:57:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:57:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:57:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:57:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:57:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:57:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:57:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:57:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:57:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:57:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:57:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:57:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:57:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:57:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:57:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:57:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:57:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:57:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:57:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:57:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:57:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:57:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:57:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:57:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:57:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:57:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:57:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:57:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:57:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:57:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:57:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:57:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:57:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:57:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:57:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:57:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:57:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:57:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:57:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:57:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:57:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:57:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:57:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:57:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:57:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:57:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:57:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:57:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:57:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:57:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:57:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:57:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:57:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:57:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:57:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:57:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:57:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:57:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:57:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:57:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:57:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:57:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:57:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:57:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:57:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:57:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:57:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:57:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:57:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:57:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:57:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:57:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:57:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:57:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:57:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:57:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:57:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:57:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:57:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:57:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:57:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:57:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:57:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:57:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:57:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:57:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:57:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:57:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:57:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:57:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:57:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:57:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:57:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:57:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:57:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:57:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:57:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:57:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:57:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:57:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:57:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:57:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:57:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:57:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:57:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:57:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:57:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:57:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:57:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:57:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:57:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:57:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:57:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:57:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:57:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:57:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:57:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:57:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:57:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:57:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:57:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:57:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:57:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:57:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:57:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:57:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:57:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:57:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:57:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:57:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:57:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:57:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:57:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:57:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:57:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:57:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:58:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:58:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:58:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:58:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:58:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:58:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:58:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:58:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:58:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:58:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:58:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:58:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:58:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:58:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:58:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:58:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:58:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:58:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 00:58:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 00:58:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:58:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:58:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:58:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 00:58:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 00:58:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 00:58:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 00:58:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 00:58:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 00:58:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 00:58:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 00:58:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 00:58:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 00:58:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 00:58:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 00:58:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 00:58:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:00:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:00:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:00:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:00:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:00:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:00:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:00:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:00:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:00:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:00:15 rdma06 rsyslogd[1041]: imjournal: 5088 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 19 01:00:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:00:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:00:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:00:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:00:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:00:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:00:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:00:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:00:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:00:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:00:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:00:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:00:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:00:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:00:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:00:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:00:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:00:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:00:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:00:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:00:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:00:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:00:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:00:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:00:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:00:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:00:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:00:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:00:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:00:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:00:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:00:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:00:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:00:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:00:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:00:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:00:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:00:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:00:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:00:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:00:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:00:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:00:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:00:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:00:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:00:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:00:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:00:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:00:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:00:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:00:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:00:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:00:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:00:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:00:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:00:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:00:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:00:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:00:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:00:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:00:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:00:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:00:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:00:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:00:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:00:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:00:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:00:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:00:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:00:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:00:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:00:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:00:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:00:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:00:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:00:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:00:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:00:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:00:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:00:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:00:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:00:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:00:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:00:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:00:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:00:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:00:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:00:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:00:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:00:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:00:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:00:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:00:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:00:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:00:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:00:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:00:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:00:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:00:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:00:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:00:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:00:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:00:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:00:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:00:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:00:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:00:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:00:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:00:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:00:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:00:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:00:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:00:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:00:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:00:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:00:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:00:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:00:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:00:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:00:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:00:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:00:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:00:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:00:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:00:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:00:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:00:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:00:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:00:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:00:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:00:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:00:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:00:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:00:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:00:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:00:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:00:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:00:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:00:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:00:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:00:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:00:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:00:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:00:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:01:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:01:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:01:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:01:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:01:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:01:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:01:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:01:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:01:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:01:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:01:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:01:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:01:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:01:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:01:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:01:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:01:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:01:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:01:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:01:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:01:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:01:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:01:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:01:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:01:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:01:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:01:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:01:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:01:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:01:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:01:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:01:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:01:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:01:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:01:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:01:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:01:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:01:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:01:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:01:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:01:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:01:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:01:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:01:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:01:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:01:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:01:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:01:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:01:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:01:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:01:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:01:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:01:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:01:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:01:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:01:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:01:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:01:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:01:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:01:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:01:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:01:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:01:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:01:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:01:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:01:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:01:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:01:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:01:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:01:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:01:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:01:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:01:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:01:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:01:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:01:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:01:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:01:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:01:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:01:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:01:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:01:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:01:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:01:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:01:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:01:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:01:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:01:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:01:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:01:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:01:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:01:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:01:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:01:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:01:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:01:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:01:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:01:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:01:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:01:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:01:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:01:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:01:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:01:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:01:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:01:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:01:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:01:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:01:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:01:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:01:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:01:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:01:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:01:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:01:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:01:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:01:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:01:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:01:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:01:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:01:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:01:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:01:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:01:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:01:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:01:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:01:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:01:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:01:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:01:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:01:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:01:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:01:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:01:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:01:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:01:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:01:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:01:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:01:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:01:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:01:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:01:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:01:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:01:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:01:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:01:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:01:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:01:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:01:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:01:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:01:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:01:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:01:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:01:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:01:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:01:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:01:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:01:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:01:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:01:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:01:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:01:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:01:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:01:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:01:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:01:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:01:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:01:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:01:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:01:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:01:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:01:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:01:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:01:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:01:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:01:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:01:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:01:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:01:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:01:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:01:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:01:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:01:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:01:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:01:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:01:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:01:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:01:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:01:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:01:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:01:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:01:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:01:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:01:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:01:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:01:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:01:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:01:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:01:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:01:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:01:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:01:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:01:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:01:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:01:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:01:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:01:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:01:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:01:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:01:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:01:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:01:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:01:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:01:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:01:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:01:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:02:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:02:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:02:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:02:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:02:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:02:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:02:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:02:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:02:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:02:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:02:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:02:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:02:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:02:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:02:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:02:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:02:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:02:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:02:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:02:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:02:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:02:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:02:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:02:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:02:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:02:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:02:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:02:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:02:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:02:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:02:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:02:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:02:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:02:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:02:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:02:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:02:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:02:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:02:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:02:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:02:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:02:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:02:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:02:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:02:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:02:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:02:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:02:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:02:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:02:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:02:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:02:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:02:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:02:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:02:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:02:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:02:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:02:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:02:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:02:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:02:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:02:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:02:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:02:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:02:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:02:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:02:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:02:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:02:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:02:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:02:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:02:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:02:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:02:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:02:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:02:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:02:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:02:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:02:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:02:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:02:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:02:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:02:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:02:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:02:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:02:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:02:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:02:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:02:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:02:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:02:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:02:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:02:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:02:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:02:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:02:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:02:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:02:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:02:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:02:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:02:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:02:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:02:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:02:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:02:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:02:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:02:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:02:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:02:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:02:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:02:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:02:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:02:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:02:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:02:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:02:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:02:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:02:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:02:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:02:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:02:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:02:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:02:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:02:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:02:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:02:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:02:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:02:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:02:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:02:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:02:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:02:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:02:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:02:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:02:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:02:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:02:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:02:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:02:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:02:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:02:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:02:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:02:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:02:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:02:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:02:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:02:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:02:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:02:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:02:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:02:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:02:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:02:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:02:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:02:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:02:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:02:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:02:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:02:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:02:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:02:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:02:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:02:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:02:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:02:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:02:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:02:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:02:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:02:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:02:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:02:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:02:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:02:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:02:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:02:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:02:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:02:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:02:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:02:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:02:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:02:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:02:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:02:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:02:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:02:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:02:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:02:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:02:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:02:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:02:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:02:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:02:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:02:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:02:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:02:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:02:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:02:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:02:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:02:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:02:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:02:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:02:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:02:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:02:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:02:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:02:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:02:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:02:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:02:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:02:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:02:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:02:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:02:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:02:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:02:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:02:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:03:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:03:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:03:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:03:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:03:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:03:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:03:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:03:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:03:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:03:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:03:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:03:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:03:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:03:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:03:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:03:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:03:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:03:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:03:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:03:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:03:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:03:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:03:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:03:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:03:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:03:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:03:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:03:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:03:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:03:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:03:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:03:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:03:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:03:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:03:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:03:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:03:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:03:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:03:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:03:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:03:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:03:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:03:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:03:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:03:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:03:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:03:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:03:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:03:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:03:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:03:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:03:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:03:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:03:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:03:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:03:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:03:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:03:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:03:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:03:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:03:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:03:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:03:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:03:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:03:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:03:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:03:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:03:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:03:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:03:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:03:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:03:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:03:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:03:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:03:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:03:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:03:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:03:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:03:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:03:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:03:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:03:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:03:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:03:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:03:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:03:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:03:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:03:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:03:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:03:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:03:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:03:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:03:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:03:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:03:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:03:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:03:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:03:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:03:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:03:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:03:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:03:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:03:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:03:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:03:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:03:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:03:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:03:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:03:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:03:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:03:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:03:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:03:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:03:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:03:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:03:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:03:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:03:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:03:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:03:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:03:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:03:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:03:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:03:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:03:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:03:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:03:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:03:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:03:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:03:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:03:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:03:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:03:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:03:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:03:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:03:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:03:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:03:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:03:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:03:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:03:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:03:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:03:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:03:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:03:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:03:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:03:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:03:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:03:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:03:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:03:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:03:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:03:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:03:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:03:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:03:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:03:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:03:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:03:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:03:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:03:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:03:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:03:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:03:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:03:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:03:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:03:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:03:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:03:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:03:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:03:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:03:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:03:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:03:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:03:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:03:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:03:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:03:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:03:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:03:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:03:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:03:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:03:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:03:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:03:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:03:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:03:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:03:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:03:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:03:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:03:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:03:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:03:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:03:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:03:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:03:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:03:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:03:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:03:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:03:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:03:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:03:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:03:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:03:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:03:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:03:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:03:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:03:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:03:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:03:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:03:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:03:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:03:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:03:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:03:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:03:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:04:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:04:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:04:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:04:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:04:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:04:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:04:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:04:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:04:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:04:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:04:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:04:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:04:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:04:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:04:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:04:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:04:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:04:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:04:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:04:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:04:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:04:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:04:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:04:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:04:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:04:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:04:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:04:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:04:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:04:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:04:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:04:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:04:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:04:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:04:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:04:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:04:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:04:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:04:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:04:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:04:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:04:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:04:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:04:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:04:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:04:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:04:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:04:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:04:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:04:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:04:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:04:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:04:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:04:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:04:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:04:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:04:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:04:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:04:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:04:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:04:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:04:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:04:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:04:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:04:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:04:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:04:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:04:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:04:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:04:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:04:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:04:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:04:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:04:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:04:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:04:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:04:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:04:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:04:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:04:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:04:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:04:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:04:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:04:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:04:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:04:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:04:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:04:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:04:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:04:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:04:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:04:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:04:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:04:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:04:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:04:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:04:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:04:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:04:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:04:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:04:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:04:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:04:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:04:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:04:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:04:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:04:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:04:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:04:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:04:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:04:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:04:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:04:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:04:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:04:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:04:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:04:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:04:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:04:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:04:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:04:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:04:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:04:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:04:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:04:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:04:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:04:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:04:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:04:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:04:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:04:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:04:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:04:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:04:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:04:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:04:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:04:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:04:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:04:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:04:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:04:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:04:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:04:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:04:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:04:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:04:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:04:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:04:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:04:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:04:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:04:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:04:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:04:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:04:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:04:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:04:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:04:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:04:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:04:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:04:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:04:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:04:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:04:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:04:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:04:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:04:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:04:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:04:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:04:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:04:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:04:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:04:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:04:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:04:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:04:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:04:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:04:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:04:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:04:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:04:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:04:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:04:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:04:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:04:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:04:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:04:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:04:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:04:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:04:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:04:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:04:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:04:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:04:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:04:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:04:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:04:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:04:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:04:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:04:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:04:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:04:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:04:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:04:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:04:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:04:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:04:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:04:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:04:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:04:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:04:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:04:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:04:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:04:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:04:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:04:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:04:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:05:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:05:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:05:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:05:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:05:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:05:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:05:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:05:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:05:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:05:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:05:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:05:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:05:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:05:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:05:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:05:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:05:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:05:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:05:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:05:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:05:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:05:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:05:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:05:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:05:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:05:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:05:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:05:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:05:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:05:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:05:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:05:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:05:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:05:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:05:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:05:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:05:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:05:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:05:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:05:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:05:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:05:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:05:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:05:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:05:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:05:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:05:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:05:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:05:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:05:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:05:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:05:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:05:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:05:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:05:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:05:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:05:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:05:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:05:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:05:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:05:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:05:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:05:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:05:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:05:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:05:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:05:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:05:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:05:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:05:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:05:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:05:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:05:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:05:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:05:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:05:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:05:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:05:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:05:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:05:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:05:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:05:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:05:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:05:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:05:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:05:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:05:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:05:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:05:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:05:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:05:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:05:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:05:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:05:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:05:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:05:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:05:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:05:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:05:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:05:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:05:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:05:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:05:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:05:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:05:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:05:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:05:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:05:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:05:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:05:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:05:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:05:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:05:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:05:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:05:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:05:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:05:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:05:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:05:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:05:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:05:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:05:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:05:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:05:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:05:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:05:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:05:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:05:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:05:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:05:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:05:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:05:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:05:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:05:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:05:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:05:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:05:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:05:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:05:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:05:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:05:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:05:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:05:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:05:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:05:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:05:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:05:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:05:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:05:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:05:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:05:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:05:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:05:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:05:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:05:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:05:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:05:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:05:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:05:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:05:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:05:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:05:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:05:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:05:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:05:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:05:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:05:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:05:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:05:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:05:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:05:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:05:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:05:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:05:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:05:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:05:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:05:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:05:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:05:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:05:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:05:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:05:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:05:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:05:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:05:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:05:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:05:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:05:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:05:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:05:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:05:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:05:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:05:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:05:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:05:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:05:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:05:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:05:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:05:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:05:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:05:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:05:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:05:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:05:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:05:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:05:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:05:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:05:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:05:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:05:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:05:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:05:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:05:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:05:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:05:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:05:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:06:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:06:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:06:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:06:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:06:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:06:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:06:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:06:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:06:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:06:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:06:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:06:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:06:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:06:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:06:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:06:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:06:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:06:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:06:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:06:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:06:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:06:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:06:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:06:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:06:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:06:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:06:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:06:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:06:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:06:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:06:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:06:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:06:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:06:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:06:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:06:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:06:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:06:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:06:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:06:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:06:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:06:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:06:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:06:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:06:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:06:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:06:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:06:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:06:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:06:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:06:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:06:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:06:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:06:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:06:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:06:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:06:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:06:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:06:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:06:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:06:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:06:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:06:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:06:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:06:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:06:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:06:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:06:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:06:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:06:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:06:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:06:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:06:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:06:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:06:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:06:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:06:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:06:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:06:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:06:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:06:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:06:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:06:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:06:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:06:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:06:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:06:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:06:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:06:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:06:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:06:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:06:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:06:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:06:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:06:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:06:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:06:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:06:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:06:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:06:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:06:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:06:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:06:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:06:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:06:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:06:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:06:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:06:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:06:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:06:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:06:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:06:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:06:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:06:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:06:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:06:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:06:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:06:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:06:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:06:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:06:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:06:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:06:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:06:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:06:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:06:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:06:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:06:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:06:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:06:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:06:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:06:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:06:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:06:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:06:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:06:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:06:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:06:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:06:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:06:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:06:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:06:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:06:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:06:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:06:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:06:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:06:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:06:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:06:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:06:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:06:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:06:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:06:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:06:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:06:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:06:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:06:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:06:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:06:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:06:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:06:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:06:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:06:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:06:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:06:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:06:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:06:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:06:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:06:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:06:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:06:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:06:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:06:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:06:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:06:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:06:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:06:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:06:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:06:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:06:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:06:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:06:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:06:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:06:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:06:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:06:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:06:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:06:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:06:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:06:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:06:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:06:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:06:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:06:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:06:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:06:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:06:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:06:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:06:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:06:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:06:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:06:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:06:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:06:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:06:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:06:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:06:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:06:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:06:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:06:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:06:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:06:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:06:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:06:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:06:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:06:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:07:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:07:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:07:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:07:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:07:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:07:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:07:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:07:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:07:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:07:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:07:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:07:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:07:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:07:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:07:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:07:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:07:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:07:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:07:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:07:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:07:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:07:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:07:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:07:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:07:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:07:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:07:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:07:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:07:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:07:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:07:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:07:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:07:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:07:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:07:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:07:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:07:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:07:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:07:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:07:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:07:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:07:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:07:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:07:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:07:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:07:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:07:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:07:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:07:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:07:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:07:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:07:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:07:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:07:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:07:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:07:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:07:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:07:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:07:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:07:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:07:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:07:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:07:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:07:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:07:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:07:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:07:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:07:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:07:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:07:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:07:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:07:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:07:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:07:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:07:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:07:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:07:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:07:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:07:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:07:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:07:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:07:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:07:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:07:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:07:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:07:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:07:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:07:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:07:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:07:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:07:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:07:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:07:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:07:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:07:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:07:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:07:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:07:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:07:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:07:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:07:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:07:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:07:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:07:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:07:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:07:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:07:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:07:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:07:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:07:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:07:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:07:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:07:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:07:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:07:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:07:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:07:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:07:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:07:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:07:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:07:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:07:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:07:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:07:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:07:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:07:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:07:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:07:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:07:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:07:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:07:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:07:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:07:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:07:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:07:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:07:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:07:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:07:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:07:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:07:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:07:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:07:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:07:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:07:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:07:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:07:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:07:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:07:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:07:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:07:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:07:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:07:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:07:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:07:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:07:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:07:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:07:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:07:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:07:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:07:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:07:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:07:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:07:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:07:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:07:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:07:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:07:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:07:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:07:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:07:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:07:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:07:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:07:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:07:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:07:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:07:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:07:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:07:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:07:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:07:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:07:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:07:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:07:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:07:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:07:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:07:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:07:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:07:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:07:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:07:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:07:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:07:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:07:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:07:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:07:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:07:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:07:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:07:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:07:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:07:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:07:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:07:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:07:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:07:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:07:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:07:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:07:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:07:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:07:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:07:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:07:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:07:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:07:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:07:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:07:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:07:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:08:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:08:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:08:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:08:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:08:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:08:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:08:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:08:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:08:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:08:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:08:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:08:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:08:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:08:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:08:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:08:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:08:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:08:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:08:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:08:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:08:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:08:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:08:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:08:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:08:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:08:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:08:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:08:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:08:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:08:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:08:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:08:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:08:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:08:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:08:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:08:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:10:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:10:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:10:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:10:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:10:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:10:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:10:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:10:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:10:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:10:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:10:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:10:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:10:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:10:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:10:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:10:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:10:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:10:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:10:20 rdma06 rsyslogd[1041]: imjournal: 5297 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 19 01:10:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:10:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:10:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:10:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:10:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:10:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:10:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:10:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:10:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:10:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:10:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:10:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:10:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:10:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:10:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:10:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:10:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:10:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:10:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:10:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:10:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:10:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:10:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:10:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:10:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:10:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:10:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:10:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:10:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:10:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:10:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:10:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:10:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:10:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:10:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:10:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:10:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:10:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:10:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:10:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:10:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:10:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:10:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:10:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:10:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:10:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:10:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:10:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:10:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:10:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:10:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:10:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:10:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:10:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:10:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:10:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:10:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:10:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:10:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:10:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:10:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:10:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:10:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:10:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:10:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:10:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:10:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:10:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:10:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:10:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:10:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:10:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:10:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:10:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:10:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:10:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:10:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:10:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:10:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:10:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:10:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:10:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:10:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:10:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:10:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:10:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:10:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:10:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:10:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:10:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:10:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:10:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:10:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:10:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:10:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:10:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:10:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:10:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:10:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:10:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:10:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:10:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:10:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:10:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:10:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:10:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:10:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:10:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:10:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:10:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:10:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:10:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:10:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:10:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:10:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:10:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:10:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:10:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:10:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:10:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:10:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:10:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:10:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:10:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:10:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:10:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:10:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:10:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:10:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:10:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:10:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:10:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:10:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:10:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:10:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:10:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:10:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:10:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:10:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:10:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:10:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:10:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:10:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:10:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:11:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:11:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:11:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:11:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:11:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:11:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:11:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:11:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:11:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:11:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:11:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:11:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:11:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:11:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:11:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:11:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:11:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:11:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:11:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:11:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:11:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:11:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:11:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:11:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:11:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:11:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:11:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:11:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:11:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:11:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:11:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:11:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:11:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:11:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:11:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:11:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:11:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:11:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:11:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:11:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:11:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:11:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:11:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:11:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:11:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:11:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:11:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:11:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:11:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:11:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:11:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:11:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:11:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:11:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:11:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:11:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:11:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:11:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:11:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:11:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:11:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:11:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:11:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:11:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:11:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:11:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:11:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:11:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:11:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:11:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:11:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:11:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:11:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:11:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:11:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:11:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:11:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:11:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:11:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:11:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:11:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:11:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:11:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:11:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:11:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:11:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:11:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:11:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:11:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:11:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:11:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:11:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:11:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:11:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:11:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:11:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:11:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:11:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:11:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:11:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:11:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:11:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:11:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:11:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:11:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:11:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:11:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:11:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:11:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:11:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:11:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:11:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:11:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:11:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:11:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:11:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:11:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:11:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:11:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:11:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:11:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:11:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:11:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:11:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:11:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:11:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:11:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:11:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:11:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:11:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:11:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:11:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:11:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:11:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:11:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:11:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:11:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:11:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:11:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:11:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:11:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:11:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:11:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:11:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:11:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:11:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:11:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:11:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:11:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:11:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:11:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:11:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:11:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:11:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:11:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:11:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:11:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:11:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:11:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:11:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:11:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:11:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:11:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:11:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:11:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:11:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:11:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:11:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:11:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:11:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:11:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:12:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:12:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:12:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:12:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:12:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:12:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:12:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:12:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:12:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:12:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:12:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:12:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:12:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:12:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:12:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:12:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:12:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:12:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:12:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:12:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:12:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:12:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:12:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:12:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:12:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:12:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:12:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:12:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:12:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:12:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:12:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:12:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:12:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:12:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:12:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:12:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:12:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:12:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:12:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:12:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:12:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:12:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:12:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:12:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:12:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:12:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:12:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:12:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:12:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:12:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:12:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:12:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:12:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:12:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:12:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:12:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:12:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:12:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:12:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:12:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:12:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:12:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:12:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:12:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:12:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:12:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:12:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:12:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:12:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:12:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:12:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:12:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:12:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:12:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:12:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:12:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:12:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:12:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:12:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:12:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:12:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:12:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:12:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:12:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:12:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:12:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:12:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:12:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:12:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:12:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:12:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:12:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:12:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:12:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:12:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:12:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:12:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:12:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:12:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:12:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:12:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:12:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:12:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:12:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:12:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:12:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:12:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:12:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:12:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:12:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:12:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:12:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:12:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:12:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:12:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:12:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:12:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:12:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:12:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:12:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:12:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:12:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:12:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:12:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:12:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:12:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:12:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:12:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:12:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:12:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:12:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:12:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:12:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:12:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:12:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:12:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:12:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:12:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:12:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:12:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:12:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:12:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:12:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:12:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:12:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:12:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:12:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:12:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:12:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:12:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:12:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:12:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:12:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:12:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:12:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:12:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:12:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:12:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:12:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:12:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:12:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:12:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:12:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:12:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:12:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:12:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:12:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:12:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:12:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:12:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:12:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:12:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:12:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:12:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:12:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:12:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:12:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:12:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:12:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:12:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:12:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:12:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:12:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:12:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:12:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:12:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:12:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:12:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:12:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:12:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:12:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:12:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:12:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:12:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:12:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:12:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:12:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:12:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:12:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:12:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:12:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:12:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:12:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:12:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:12:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:12:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:12:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:12:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:12:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:12:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:12:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:12:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:12:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:12:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:12:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:12:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:13:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:13:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:13:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:13:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:13:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:13:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:13:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:13:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:13:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:13:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:13:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:13:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:13:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:13:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:13:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:13:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:13:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:13:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:13:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:13:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:13:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:13:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:13:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:13:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:13:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:13:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:13:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:13:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:13:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:13:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:13:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:13:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:13:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:13:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:13:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:13:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:13:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:13:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:13:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:13:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:13:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:13:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:13:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:13:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:13:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:13:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:13:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:13:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:13:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:13:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:13:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:13:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:13:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:13:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:13:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:13:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:13:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:13:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:13:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:13:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:13:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:13:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:13:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:13:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:13:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:13:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:13:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:13:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:13:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:13:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:13:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:13:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:13:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:13:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:13:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:13:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:13:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:13:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:13:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:13:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:13:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:13:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:13:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:13:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:13:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:13:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:13:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:13:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:13:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:13:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:13:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:13:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:13:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:13:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:13:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:13:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:13:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:13:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:13:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:13:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:13:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:13:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:13:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:13:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:13:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:13:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:13:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:13:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:13:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:13:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:13:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:13:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:13:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:13:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:13:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:13:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:13:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:13:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:13:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:13:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:13:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:13:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:13:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:13:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:13:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:13:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:13:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:13:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:13:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:13:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:13:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:13:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:13:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:13:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:13:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:13:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:13:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:13:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:13:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:13:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:13:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:13:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:13:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:13:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:13:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:13:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:13:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:13:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:13:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:13:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:13:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:13:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:13:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:13:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:13:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:13:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:13:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:13:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:13:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:13:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:13:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:13:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:13:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:13:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:13:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:13:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:13:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:13:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:13:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:13:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:13:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:13:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:13:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:13:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:13:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:13:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:13:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:13:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:13:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:13:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:13:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:13:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:13:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:13:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:13:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:13:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:13:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:13:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:13:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:13:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:13:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:13:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:13:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:13:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:13:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:13:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:13:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:13:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:13:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:13:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:13:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:13:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:13:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:13:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:13:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:13:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:13:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:13:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:13:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:13:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:13:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:13:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:13:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:13:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:13:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:13:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:14:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:14:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:14:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:14:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:14:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:14:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:14:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:14:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:14:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:14:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:14:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:14:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:14:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:14:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:14:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:14:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:14:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:14:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:14:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:14:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:14:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:14:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:14:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:14:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:14:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:14:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:14:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:14:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:14:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:14:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:14:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:14:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:14:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:14:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:14:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:14:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:14:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:14:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:14:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:14:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:14:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:14:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:14:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:14:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:14:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:14:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:14:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:14:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:14:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:14:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:14:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:14:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:14:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:14:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:14:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:14:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:14:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:14:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:14:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:14:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:14:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:14:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:14:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:14:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:14:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:14:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:14:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:14:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:14:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:14:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:14:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:14:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:14:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:14:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:14:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:14:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:14:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:14:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:14:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:14:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:14:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:14:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:14:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:14:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:14:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:14:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:14:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:14:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:14:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:14:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:14:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:14:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:14:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:14:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:14:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:14:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:14:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:14:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:14:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:14:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:14:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:14:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:14:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:14:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:14:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:14:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:14:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:14:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:14:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:14:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:14:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:14:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:14:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:14:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:14:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:14:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:14:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:14:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:14:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:14:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:14:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:14:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:14:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:14:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:14:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:14:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:14:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:14:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:14:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:14:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:14:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:14:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:14:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:14:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:14:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:14:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:14:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:14:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:14:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:14:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:14:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:14:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:14:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:14:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:14:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:14:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:14:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:14:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:14:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:14:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:14:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:14:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:14:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:14:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:14:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:14:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:14:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:14:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:14:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:14:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:14:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:14:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:14:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:14:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:14:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:14:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:14:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:14:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:14:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:14:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:14:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:14:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:14:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:14:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:14:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:14:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:14:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:14:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:14:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:14:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:14:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:14:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:14:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:14:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:14:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:14:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:14:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:14:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:14:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:14:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:14:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:14:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:14:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:14:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:14:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:14:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:14:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:14:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:14:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:14:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:14:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:14:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:14:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:14:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:14:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:14:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:14:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:14:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:14:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:14:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:14:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:14:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:14:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:14:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:14:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:14:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:15:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:15:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:15:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:15:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:15:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:15:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:15:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:15:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:15:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:15:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:15:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:15:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:15:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:15:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:15:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:15:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:15:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:15:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:15:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:15:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:15:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:15:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:15:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:15:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:15:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:15:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:15:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:15:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:15:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:15:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:15:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:15:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:15:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:15:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:15:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:15:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:15:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:15:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:15:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:15:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:15:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:15:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:15:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:15:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:15:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:15:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:15:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:15:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:15:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:15:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:15:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:15:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:15:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:15:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:15:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:15:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:15:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:15:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:15:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:15:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:15:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:15:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:15:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:15:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:15:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:15:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:15:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:15:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:15:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:15:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:15:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:15:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:15:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:15:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:15:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:15:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:15:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:15:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:15:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:15:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:15:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:15:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:15:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:15:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:15:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:15:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:15:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:15:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:15:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:15:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:15:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:15:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:15:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:15:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:15:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:15:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:15:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:15:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:15:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:15:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:15:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:15:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:15:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:15:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:15:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:15:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:15:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:15:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:15:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:15:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:15:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:15:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:15:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:15:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:15:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:15:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:15:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:15:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:15:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:15:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:15:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:15:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:15:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:15:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:15:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:15:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:15:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:15:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:15:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:15:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:15:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:15:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:15:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:15:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:15:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:15:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:15:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:15:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:15:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:15:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:15:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:15:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:15:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:15:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:15:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:15:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:15:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:15:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:15:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:15:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:15:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:15:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:15:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:15:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:15:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:15:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:15:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:15:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:15:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:15:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:15:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:15:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:15:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:15:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:15:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:15:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:15:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:15:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:15:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:15:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:15:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:15:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:15:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:15:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:15:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:15:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:15:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:15:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:15:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:15:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:15:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:15:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:15:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:15:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:15:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:15:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:15:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:15:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:15:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:15:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:15:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:15:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:15:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:15:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:15:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:15:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:15:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:15:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:15:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:15:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:15:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:15:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:15:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:15:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:15:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:15:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:15:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:15:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:15:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:15:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:15:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:15:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:15:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:15:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:15:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:15:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:16:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:16:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:16:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:16:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:16:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:16:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:16:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:16:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:16:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:16:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:16:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:16:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:16:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:16:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:16:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:16:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:16:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:16:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:16:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:16:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:16:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:16:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:16:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:16:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:16:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:16:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:16:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:16:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:16:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:16:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:16:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:16:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:16:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:16:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:16:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:16:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:16:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:16:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:16:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:16:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:16:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:16:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:16:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:16:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:16:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:16:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:16:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:16:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:16:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:16:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:16:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:16:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:16:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:16:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:16:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:16:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:16:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:16:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:16:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:16:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:16:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:16:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:16:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:16:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:16:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:16:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:16:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:16:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:16:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:16:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:16:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:16:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:16:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:16:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:16:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:16:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:16:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:16:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:16:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:16:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:16:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:16:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:16:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:16:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:16:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:16:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:16:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:16:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:16:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:16:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:16:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:16:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:16:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:16:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:16:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:16:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:16:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:16:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:16:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:16:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:16:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:16:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:16:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:16:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:16:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:16:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:16:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:16:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:16:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:16:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:16:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:16:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:16:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:16:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:16:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:16:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:16:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:16:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:16:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:16:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:16:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:16:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:16:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:16:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:16:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:16:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:16:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:16:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:16:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:16:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:16:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:16:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:16:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:16:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:16:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:16:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:16:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:16:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:16:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:16:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:16:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:16:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:16:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:16:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:16:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:16:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:16:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:16:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:16:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:16:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:16:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:16:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:16:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:16:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:16:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:16:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:16:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:16:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:16:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:16:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:16:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:16:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:16:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:16:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:16:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:16:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:16:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:16:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:16:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:16:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:16:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:16:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:16:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:16:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:16:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:16:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:16:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:16:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:16:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:16:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:16:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:16:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:16:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:16:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:16:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:16:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:16:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:16:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:16:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:16:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:16:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:16:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:16:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:16:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:16:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:16:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:16:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:16:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:16:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:16:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:16:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:16:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:16:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:16:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:16:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:16:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:16:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:16:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:16:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:16:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:16:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:16:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:16:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:16:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:16:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:16:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:17:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:17:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:17:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:17:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:17:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:17:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:17:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:17:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:17:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:17:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:17:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:17:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:17:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:17:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:17:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:17:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:17:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:17:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:17:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:17:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:17:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:17:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:17:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:17:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:17:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:17:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:17:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:17:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:17:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:17:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:17:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:17:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:17:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:17:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:17:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:17:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:17:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:17:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:17:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:17:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:17:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:17:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:17:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:17:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:17:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:17:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:17:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:17:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:17:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:17:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:17:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:17:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:17:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:17:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:17:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:17:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:17:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:17:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:17:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:17:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:17:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:17:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:17:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:17:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:17:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:17:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:17:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:17:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:17:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:17:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:17:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:17:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:17:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:17:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:17:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:17:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:17:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:17:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:17:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:17:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:17:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:17:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:17:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:17:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:17:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:17:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:17:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:17:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:17:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:17:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:17:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:17:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:17:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:17:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:17:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:17:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:17:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:17:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:17:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:17:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:17:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:17:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:17:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:17:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:17:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:17:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:17:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:17:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:17:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:17:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:17:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:17:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:17:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:17:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:17:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:17:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:17:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:17:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:17:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:17:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:17:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:17:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:17:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:17:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:17:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:17:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:17:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:17:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:17:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:17:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:17:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:17:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:17:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:17:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:17:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:17:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:17:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:17:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:17:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:17:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:17:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:17:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:17:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:17:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:17:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:17:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:17:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:17:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:17:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:17:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:17:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:17:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:17:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:17:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:17:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:17:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:17:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:17:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:17:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:17:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:17:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:17:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:17:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:17:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:17:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:17:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:17:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:17:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:17:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:17:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:17:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:17:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:17:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:17:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:17:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:17:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:17:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:17:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:17:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:17:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:17:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:17:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:17:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:17:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:17:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:17:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:17:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:17:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:17:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:17:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:17:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:17:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:17:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:17:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:17:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:17:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:17:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:17:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:17:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:17:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:17:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:17:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:17:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:17:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:17:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:17:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:17:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:17:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:17:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:17:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:17:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:17:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:17:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:17:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:17:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:17:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:18:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:18:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:18:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:18:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:18:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:18:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:18:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:18:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:18:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:18:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:18:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:18:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:18:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:18:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:18:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:18:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:18:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:18:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:18:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:18:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:18:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:18:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:18:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:18:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:18:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:18:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:18:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:18:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:18:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:18:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:18:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:18:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:18:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:18:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:18:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:18:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:18:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:18:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:18:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:18:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:18:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:18:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:18:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:18:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:18:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:18:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:18:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:18:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:18:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:18:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:18:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:18:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:18:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:18:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:18:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:18:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:18:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:18:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:18:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:18:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:18:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:18:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:18:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:18:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:18:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:18:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:18:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:18:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:18:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:18:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:18:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:18:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:20:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:20:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:20:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:20:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:20:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:20:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:20:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:20:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:20:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:20:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:20:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:20:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:20:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:20:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:20:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:20:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:20:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:20:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:20:25 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 19 01:20:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:20:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:20:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:20:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:20:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:20:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:20:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:20:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:20:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:20:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:20:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:20:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:20:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:20:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:20:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:20:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:20:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:20:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:20:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:20:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:20:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:20:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:20:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:20:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:20:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:20:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:20:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:20:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:20:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:20:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:20:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:20:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:20:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:20:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:20:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:20:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:20:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:20:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:20:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:20:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:20:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:20:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:20:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:20:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:20:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:20:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:20:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:20:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:20:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:20:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:20:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:20:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:20:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:20:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:20:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:20:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:20:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:20:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:20:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:20:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:20:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:20:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:20:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:20:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:20:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:20:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:20:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:20:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:20:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:20:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:20:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:20:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:20:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:20:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:20:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:20:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:20:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:20:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:20:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:20:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:20:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:20:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:20:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:20:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:20:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:20:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:20:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:20:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:20:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:20:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:20:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:20:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:20:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:20:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:20:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:20:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:20:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:20:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:20:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:20:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:20:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:20:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:20:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:20:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:20:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:20:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:20:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:20:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:21:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:21:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:21:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:21:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:21:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:21:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:21:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:21:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:21:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:21:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:21:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:21:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:21:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:21:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:21:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:21:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:21:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:21:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:21:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:21:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:21:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:21:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:21:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:21:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:21:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:21:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:21:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:21:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:21:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:21:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:21:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:21:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:21:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:21:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:21:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:21:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:21:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:21:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:21:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:21:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:21:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:21:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:21:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:21:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:21:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:21:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:21:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:21:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:21:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:21:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:21:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:21:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:21:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:21:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:21:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:21:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:21:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:21:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:21:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:21:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:21:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:21:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:21:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:21:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:21:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:21:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:21:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:21:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:21:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:21:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:21:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:21:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:21:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:21:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:21:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:21:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:21:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:21:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:21:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:21:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:21:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:21:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:21:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:21:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:21:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:21:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:21:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:21:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:21:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:21:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:21:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:21:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:21:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:21:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:21:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:21:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:21:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:21:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:21:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:21:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:21:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:21:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:21:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:21:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:21:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:21:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:21:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:21:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:21:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:21:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:21:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:21:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:21:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:21:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:21:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:21:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:21:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:21:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:21:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:21:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:21:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:21:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:21:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:21:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:21:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:21:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:21:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:21:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:21:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:21:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:21:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:21:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:21:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:21:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:21:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:21:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:21:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:21:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:21:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:21:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:21:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:21:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:21:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:21:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:21:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:21:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:21:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:21:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:21:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:21:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:21:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:21:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:21:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:21:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:21:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:21:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:21:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:21:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:21:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:21:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:21:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:21:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:21:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:21:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:21:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:21:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:21:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:21:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:21:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:21:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:21:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:21:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:21:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:21:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:21:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:21:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:21:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:21:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:21:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:21:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:21:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:21:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:21:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:21:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:21:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:21:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:21:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:21:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:21:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:21:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:21:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:21:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:21:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:21:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:21:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:21:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:21:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:21:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:21:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:21:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:21:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:21:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:21:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:21:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:21:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:21:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:21:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:21:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:21:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:21:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:21:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:21:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:21:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:21:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:21:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:21:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:22:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:22:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:22:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:22:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:22:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:22:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:22:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:22:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:22:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:22:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:22:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:22:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:22:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:22:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:22:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:22:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:22:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:22:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:22:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:22:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:22:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:22:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:22:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:22:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:22:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:22:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:22:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:22:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:22:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:22:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:22:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:22:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:22:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:22:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:22:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:22:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:22:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:22:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:22:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:22:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:22:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:22:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:22:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:22:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:22:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:22:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:22:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:22:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:22:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:22:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:22:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:22:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:22:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:22:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:22:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:22:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:22:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:22:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:22:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:22:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:22:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:22:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:22:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:22:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:22:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:22:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:22:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:22:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:22:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:22:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:22:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:22:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:22:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:22:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:22:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:22:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:22:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:22:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:22:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:22:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:22:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:22:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:22:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:22:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:22:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:22:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:22:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:22:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:22:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:22:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:22:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:22:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:22:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:22:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:22:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:22:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:22:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:22:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:22:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:22:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:22:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:22:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:22:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:22:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:22:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:22:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:22:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:22:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:22:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:22:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:22:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:22:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:22:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:22:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:22:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:22:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:22:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:22:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:22:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:22:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:22:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:22:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:22:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:22:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:22:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:22:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:22:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:22:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:22:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:22:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:22:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:22:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:22:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:22:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:22:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:22:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:22:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:22:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:22:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:22:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:22:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:22:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:22:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:22:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:22:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:22:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:22:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:22:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:22:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:22:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:22:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:22:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:22:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:22:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:22:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:22:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:22:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:22:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:22:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:22:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:22:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:22:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:22:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:22:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:22:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:22:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:22:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:22:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:22:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:22:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:22:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:22:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:22:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:22:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:22:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:22:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:22:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:22:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:22:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:22:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:22:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:22:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:22:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:22:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:22:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:22:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:22:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:22:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:22:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:22:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:22:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:22:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:22:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:22:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:22:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:22:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:22:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:22:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:22:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:22:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:22:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:22:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:22:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:22:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:22:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:22:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:22:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:22:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:22:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:22:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:22:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:22:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:22:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:22:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:22:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:22:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:23:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:23:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:23:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:23:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:23:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:23:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:23:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:23:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:23:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:23:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:23:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:23:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:23:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:23:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:23:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:23:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:23:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:23:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:23:00 rdma06 systemd[1]: Starting dnf makecache...
Apr 19 01:23:01 rdma06 dnf[1209662]: Updating Subscription Management repositories.
Apr 19 01:23:01 rdma06 dnf[1209662]: Unable to read consumer identity
Apr 19 01:23:01 rdma06 dnf[1209662]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 19 01:23:01 rdma06 dnf[1209662]: Metadata cache refreshed recently.
Apr 19 01:23:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 19 01:23:01 rdma06 systemd[1]: Finished dnf makecache.
Apr 19 01:23:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:23:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:23:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:23:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:23:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:23:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:23:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:23:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:23:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:23:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:23:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:23:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:23:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:23:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:23:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:23:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:23:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:23:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:23:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:23:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:23:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:23:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:23:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:23:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:23:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:23:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:23:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:23:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:23:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:23:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:23:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:23:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:23:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:23:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:23:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:23:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:23:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:23:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:23:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:23:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:23:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:23:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:23:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:23:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:23:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:23:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:23:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:23:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:23:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:23:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:23:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:23:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:23:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:23:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:23:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:23:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:23:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:23:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:23:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:23:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:23:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:23:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:23:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:23:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:23:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:23:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:23:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:23:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:23:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:23:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:23:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:23:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:23:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:23:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:23:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:23:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:23:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:23:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:23:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:23:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:23:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:23:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:23:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:23:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:23:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:23:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:23:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:23:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:23:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:23:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:23:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:23:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:23:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:23:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:23:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:23:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:23:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:23:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:23:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:23:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:23:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:23:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:23:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:23:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:23:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:23:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:23:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:23:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:23:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:23:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:23:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:23:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:23:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:23:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:23:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:23:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:23:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:23:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:23:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:23:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:23:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:23:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:23:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:23:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:23:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:23:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:23:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:23:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:23:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:23:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:23:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:23:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:23:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:23:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:23:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:23:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:23:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:23:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:23:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:23:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:23:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:23:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:23:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:23:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:23:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:23:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:23:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:23:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:23:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:23:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:23:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:23:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:23:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:23:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:23:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:23:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:23:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:23:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:23:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:23:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:23:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:23:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:23:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:23:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:23:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:23:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:23:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:23:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:23:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:23:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:23:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:23:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:23:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:23:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:23:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:23:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:23:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:23:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:23:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:23:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:23:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:23:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:23:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:23:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:23:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:23:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:23:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:23:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:23:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:23:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:23:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:23:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:23:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:23:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:23:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:23:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:23:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:23:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:24:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:24:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:24:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:24:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:24:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:24:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:24:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:24:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:24:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:24:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:24:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:24:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:24:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:24:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:24:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:24:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:24:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:24:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:24:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:24:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:24:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:24:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:24:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:24:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:24:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:24:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:24:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:24:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:24:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:24:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:24:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:24:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:24:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:24:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:24:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:24:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:24:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:24:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:24:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:24:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:24:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:24:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:24:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:24:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:24:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:24:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:24:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:24:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:24:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:24:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:24:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:24:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:24:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:24:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:24:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:24:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:24:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:24:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:24:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:24:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:24:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:24:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:24:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:24:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:24:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:24:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:24:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:24:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:24:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:24:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:24:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:24:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:24:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:24:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:24:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:24:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:24:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:24:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:24:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:24:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:24:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:24:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:24:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:24:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:24:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:24:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:24:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:24:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:24:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:24:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:24:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:24:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:24:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:24:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:24:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:24:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:24:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:24:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:24:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:24:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:24:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:24:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:24:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:24:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:24:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:24:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:24:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:24:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:24:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:24:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:24:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:24:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:24:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:24:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:24:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:24:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:24:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:24:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:24:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:24:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:24:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:24:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:24:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:24:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:24:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:24:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:24:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:24:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:24:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:24:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:24:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:24:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:24:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:24:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:24:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:24:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:24:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:24:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:24:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:24:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:24:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:24:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:24:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:24:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:24:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:24:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:24:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:24:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:24:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:24:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:24:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:24:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:24:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:24:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:24:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:24:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:24:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:24:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:24:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:24:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:24:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:24:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:24:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:24:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:24:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:24:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:24:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:24:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:24:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:24:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:24:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:24:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:24:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:24:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:24:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:24:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:24:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:24:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:24:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:24:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:24:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:24:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:24:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:24:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:24:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:24:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:24:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:24:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:24:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:24:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:24:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:24:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:24:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:24:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:24:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:24:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:24:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:24:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:24:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:24:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:24:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:24:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:24:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:24:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:24:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:24:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:24:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:24:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:24:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:24:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:24:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:24:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:24:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:24:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:24:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:24:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:25:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:25:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:25:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:25:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:25:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:25:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:25:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:25:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:25:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:25:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:25:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:25:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:25:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:25:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:25:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:25:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:25:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:25:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:25:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:25:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:25:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:25:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:25:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:25:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:25:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:25:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:25:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:25:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:25:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:25:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:25:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:25:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:25:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:25:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:25:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:25:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:25:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:25:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:25:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:25:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:25:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:25:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:25:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:25:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:25:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:25:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:25:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:25:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:25:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:25:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:25:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:25:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:25:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:25:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:25:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:25:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:25:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:25:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:25:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:25:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:25:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:25:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:25:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:25:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:25:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:25:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:25:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:25:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:25:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:25:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:25:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:25:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:25:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:25:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:25:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:25:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:25:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:25:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:25:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:25:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:25:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:25:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:25:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:25:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:25:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:25:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:25:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:25:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:25:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:25:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:25:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:25:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:25:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:25:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:25:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:25:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:25:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:25:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:25:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:25:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:25:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:25:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:25:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:25:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:25:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:25:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:25:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:25:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:25:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:25:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:25:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:25:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:25:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:25:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:25:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:25:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:25:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:25:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:25:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:25:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:25:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:25:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:25:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:25:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:25:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:25:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:25:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:25:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:25:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:25:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:25:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:25:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:25:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:25:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:25:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:25:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:25:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:25:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:25:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:25:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:25:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:25:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:25:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:25:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:25:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:25:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:25:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:25:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:25:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:25:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:25:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:25:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:25:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:25:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:25:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:25:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:25:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:25:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:25:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:25:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:25:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:25:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:25:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:25:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:25:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:25:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:25:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:25:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:25:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:25:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:25:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:25:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:25:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:25:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:25:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:25:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:25:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:25:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:25:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:25:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:25:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:25:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:25:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:25:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:25:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:25:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:25:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:25:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:25:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:25:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:25:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:25:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:25:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:25:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:25:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:25:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:25:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:25:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:25:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:25:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:25:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:25:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:25:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:25:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:25:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:25:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:25:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:25:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:25:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:25:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:25:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:25:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:25:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:25:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:25:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:25:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:26:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:26:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:26:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:26:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:26:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:26:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:26:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:26:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:26:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:26:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:26:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:26:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:26:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:26:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:26:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:26:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:26:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:26:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:26:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:26:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:26:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:26:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:26:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:26:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:26:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:26:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:26:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:26:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:26:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:26:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:26:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:26:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:26:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:26:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:26:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:26:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:26:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:26:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:26:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:26:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:26:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:26:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:26:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:26:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:26:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:26:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:26:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:26:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:26:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:26:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:26:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:26:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:26:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:26:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:26:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:26:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:26:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:26:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:26:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:26:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:26:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:26:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:26:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:26:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:26:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:26:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:26:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:26:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:26:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:26:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:26:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:26:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:26:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:26:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:26:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:26:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:26:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:26:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:26:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:26:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:26:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:26:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:26:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:26:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:26:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:26:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:26:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:26:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:26:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:26:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:26:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:26:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:26:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:26:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:26:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:26:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:26:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:26:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:26:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:26:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:26:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:26:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:26:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:26:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:26:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:26:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:26:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:26:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:26:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:26:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:26:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:26:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:26:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:26:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:26:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:26:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:26:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:26:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:26:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:26:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:26:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:26:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:26:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:26:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:26:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:26:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:26:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:26:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:26:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:26:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:26:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:26:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:26:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:26:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:26:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:26:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:26:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:26:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:26:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:26:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:26:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:26:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:26:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:26:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:26:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:26:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:26:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:26:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:26:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:26:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:26:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:26:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:26:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:26:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:26:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:26:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:26:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:26:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:26:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:26:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:26:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:26:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:26:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:26:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:26:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:26:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:26:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:26:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:26:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:26:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:26:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:26:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:26:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:26:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:26:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:26:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:26:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:26:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:26:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:26:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:26:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:26:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:26:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:26:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:26:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:26:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:26:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:26:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:26:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:26:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:26:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:26:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:26:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:26:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:26:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:26:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:26:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:26:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:26:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:26:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:26:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:26:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:26:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:26:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:26:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:26:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:26:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:26:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:26:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:26:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:26:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:26:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:26:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:26:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:26:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:26:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:27:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:27:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:27:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:27:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:27:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:27:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:27:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:27:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:27:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:27:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:27:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:27:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:27:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:27:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:27:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:27:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:27:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:27:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:27:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:27:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:27:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:27:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:27:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:27:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:27:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:27:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:27:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:27:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:27:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:27:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:27:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:27:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:27:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:27:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:27:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:27:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:27:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:27:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:27:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:27:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:27:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:27:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:27:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:27:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:27:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:27:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:27:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:27:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:27:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:27:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:27:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:27:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:27:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:27:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:27:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:27:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:27:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:27:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:27:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:27:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:27:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:27:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:27:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:27:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:27:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:27:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:27:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:27:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:27:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:27:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:27:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:27:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:27:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:27:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:27:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:27:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:27:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:27:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:27:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:27:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:27:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:27:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:27:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:27:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:27:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:27:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:27:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:27:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:27:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:27:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:27:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:27:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:27:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:27:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:27:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:27:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:27:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:27:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:27:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:27:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:27:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:27:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:27:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:27:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:27:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:27:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:27:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:27:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:27:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:27:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:27:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:27:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:27:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:27:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:27:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:27:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:27:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:27:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:27:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:27:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:27:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:27:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:27:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:27:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:27:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:27:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:27:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:27:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:27:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:27:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:27:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:27:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:27:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:27:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:27:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:27:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:27:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:27:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:27:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:27:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:27:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:27:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:27:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:27:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:27:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:27:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:27:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:27:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:27:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:27:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:27:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:27:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:27:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:27:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:27:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:27:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:27:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:27:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:27:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:27:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:27:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:27:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:27:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:27:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:27:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:27:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:27:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:27:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:27:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:27:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:27:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:27:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:27:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:27:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:27:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:27:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:27:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:27:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:27:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:27:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:27:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:27:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:27:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:27:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:27:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:27:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:27:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:27:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:27:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:27:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:27:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:27:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:27:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:27:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:27:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:27:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:27:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:27:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:27:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:27:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:27:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:27:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:27:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:27:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:27:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:27:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:27:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:27:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:27:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:27:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:27:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:27:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:27:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:27:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:27:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:27:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:28:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:28:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:28:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:28:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:28:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:28:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:28:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:28:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:28:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:28:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:28:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:28:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:28:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:28:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:28:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:28:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:28:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:28:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:28:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:28:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:28:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:28:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:28:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:28:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:28:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:28:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:28:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:28:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:28:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:28:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:28:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:28:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:28:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:28:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:28:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:28:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:28:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:28:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:28:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:28:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:28:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:28:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:28:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:28:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:28:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:28:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:28:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:28:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:28:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:28:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:28:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:28:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:28:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:28:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:28:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:28:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:28:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:28:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:28:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:28:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:28:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:28:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:28:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:28:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:28:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:28:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:28:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:28:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:28:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:28:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:28:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:28:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:28:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:28:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:28:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:28:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:28:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:28:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:28:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:28:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:28:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:28:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:28:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:28:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:28:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:28:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:28:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:28:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:28:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:28:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:30:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:30:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:30:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:30:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:30:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:30:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:30:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:30:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:30:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:30:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:30:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:30:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:30:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:30:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:30:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:30:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:30:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:30:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:30:26 rdma06 rsyslogd[1041]: imjournal: 5089 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 19 01:30:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:30:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:30:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:30:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:30:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:30:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:30:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:30:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:30:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:30:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:30:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:30:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:30:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:30:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:30:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:30:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:30:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:30:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:30:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:30:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:30:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:30:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:30:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:30:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:30:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:30:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:30:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:30:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:30:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:30:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:30:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:30:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:30:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:30:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:30:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:30:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:30:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:30:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:30:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:30:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:30:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:30:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:30:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:30:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:30:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:30:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:30:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:30:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:30:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:30:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:30:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:30:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:30:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:30:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:30:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:30:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:30:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:30:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:30:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:30:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:30:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:30:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:30:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:30:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:30:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:30:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:30:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:30:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:30:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:30:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:30:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:30:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:30:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:30:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:30:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:30:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:30:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:30:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:30:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:30:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:30:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:30:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:30:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:30:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:30:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:30:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:30:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:30:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:30:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:30:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:30:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:30:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:30:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:30:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:30:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:30:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:30:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:30:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:30:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:30:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:30:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:30:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:30:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:30:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:30:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:30:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:30:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:30:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:31:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:31:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:31:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:31:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:31:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:31:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:31:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:31:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:31:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:31:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:31:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:31:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:31:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:31:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:31:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:31:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:31:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:31:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:31:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:31:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:31:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:31:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:31:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:31:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:31:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:31:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:31:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:31:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:31:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:31:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:31:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:31:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:31:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:31:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:31:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:31:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:31:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:31:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:31:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:31:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:31:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:31:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:31:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:31:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:31:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:31:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:31:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:31:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:31:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:31:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:31:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:31:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:31:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:31:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:31:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:31:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:31:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:31:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:31:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:31:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:31:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:31:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:31:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:31:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:31:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:31:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:31:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:31:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:31:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:31:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:31:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:31:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:31:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:31:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:31:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:31:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:31:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:31:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:31:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:31:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:31:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:31:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:31:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:31:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:31:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:31:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:31:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:31:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:31:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:31:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:31:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:31:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:31:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:31:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:31:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:31:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:31:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:31:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:31:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:31:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:31:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:31:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:31:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:31:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:31:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:31:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:31:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:31:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:31:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:31:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:31:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:31:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:31:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:31:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:31:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:31:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:31:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:31:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:31:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:31:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:31:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:31:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:31:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:31:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:31:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:31:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:31:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:31:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:31:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:31:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:31:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:31:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:31:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:31:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:31:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:31:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:31:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:31:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:31:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:31:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:31:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:31:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:31:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:31:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:31:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:31:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:31:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:31:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:31:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:31:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:31:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:31:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:31:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:31:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:31:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:31:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:31:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:31:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:31:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:31:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:31:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:31:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:31:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:31:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:31:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:31:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:31:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:31:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:31:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:31:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:31:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:31:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:31:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:31:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:31:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:31:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:31:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:31:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:31:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:31:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:31:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:31:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:31:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:31:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:31:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:31:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:31:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:31:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:31:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:31:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:31:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:31:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:31:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:31:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:31:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:31:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:31:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:31:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:31:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:31:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:31:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:31:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:31:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:31:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:31:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:31:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:31:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:31:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:31:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:31:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:31:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:31:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:31:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:31:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:31:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:31:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:32:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:32:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:32:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:32:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:32:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:32:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:32:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:32:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:32:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:32:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:32:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:32:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:32:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:32:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:32:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:32:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:32:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:32:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:32:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:32:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:32:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:32:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:32:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:32:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:32:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:32:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:32:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:32:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:32:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:32:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:32:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:32:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:32:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:32:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:32:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:32:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:32:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:32:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:32:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:32:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:32:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:32:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:32:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:32:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:32:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:32:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:32:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:32:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:32:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:32:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:32:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:32:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:32:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:32:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:32:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:32:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:32:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:32:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:32:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:32:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:32:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:32:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:32:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:32:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:32:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:32:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:32:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:32:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:32:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:32:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:32:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:32:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:32:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:32:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:32:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:32:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:32:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:32:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:32:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:32:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:32:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:32:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:32:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:32:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:32:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:32:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:32:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:32:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:32:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:32:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:32:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:32:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:32:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:32:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:32:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:32:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:32:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:32:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:32:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:32:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:32:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:32:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:32:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:32:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:32:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:32:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:32:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:32:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:32:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:32:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:32:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:32:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:32:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:32:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:32:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:32:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:32:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:32:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:32:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:32:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:32:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:32:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:32:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:32:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:32:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:32:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:32:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:32:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:32:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:32:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:32:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:32:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:32:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:32:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:32:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:32:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:32:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:32:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:32:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:32:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:32:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:32:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:32:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:32:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:32:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:32:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:32:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:32:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:32:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:32:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:32:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:32:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:32:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:32:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:32:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:32:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:32:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:32:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:32:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:32:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:32:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:32:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:32:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:32:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:32:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:32:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:32:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:32:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:32:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:32:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:32:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:32:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:32:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:32:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:32:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:32:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:32:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:32:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:32:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:32:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:32:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:32:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:32:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:32:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:32:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:32:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:32:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:32:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:32:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:32:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:32:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:32:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:32:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:32:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:32:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:32:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:32:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:32:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:32:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:32:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:32:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:32:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:32:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:32:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:32:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:32:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:32:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:32:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:32:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:32:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:32:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:32:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:32:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:32:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:32:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:32:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:33:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:33:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:33:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:33:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:33:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:33:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:33:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:33:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:33:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:33:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:33:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:33:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:33:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:33:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:33:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:33:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:33:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:33:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:33:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:33:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:33:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:33:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:33:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:33:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:33:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:33:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:33:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:33:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:33:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:33:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:33:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:33:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:33:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:33:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:33:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:33:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:33:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:33:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:33:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:33:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:33:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:33:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:33:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:33:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:33:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:33:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:33:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:33:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:33:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:33:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:33:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:33:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:33:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:33:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:33:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:33:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:33:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:33:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:33:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:33:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:33:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:33:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:33:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:33:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:33:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:33:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:33:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:33:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:33:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:33:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:33:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:33:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:33:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:33:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:33:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:33:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:33:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:33:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:33:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:33:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:33:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:33:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:33:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:33:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:33:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:33:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:33:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:33:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:33:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:33:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:33:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:33:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:33:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:33:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:33:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:33:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:33:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:33:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:33:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:33:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:33:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:33:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:33:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:33:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:33:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:33:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:33:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:33:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:33:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:33:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:33:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:33:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:33:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:33:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:33:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:33:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:33:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:33:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:33:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:33:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:33:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:33:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:33:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:33:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:33:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:33:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:33:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:33:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:33:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:33:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:33:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:33:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:33:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:33:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:33:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:33:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:33:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:33:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:33:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:33:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:33:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:33:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:33:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:33:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:33:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:33:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:33:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:33:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:33:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:33:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:33:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:33:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:33:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:33:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:33:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:33:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:33:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:33:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:33:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:33:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:33:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:33:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:33:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:33:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:33:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:33:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:33:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:33:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:33:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:33:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:33:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:33:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:33:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:33:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:33:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:33:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:33:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:33:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:33:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:33:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:33:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:33:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:33:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:33:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:33:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:33:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:33:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:33:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:33:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:33:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:33:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:33:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:33:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:33:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:33:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:33:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:33:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:33:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:33:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:33:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:33:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:33:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:33:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:33:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:33:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:33:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:33:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:33:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:33:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:33:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:33:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:33:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:33:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:33:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:33:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:33:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:34:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:34:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:34:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:34:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:34:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:34:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:34:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:34:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:34:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:34:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:34:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:34:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:34:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:34:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:34:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:34:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:34:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:34:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:34:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:34:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:34:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:34:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:34:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:34:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:34:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:34:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:34:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:34:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:34:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:34:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:34:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:34:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:34:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:34:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:34:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:34:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:34:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:34:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:34:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:34:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:34:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:34:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:34:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:34:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:34:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:34:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:34:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:34:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:34:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:34:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:34:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:34:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:34:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:34:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:34:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:34:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:34:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:34:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:34:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:34:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:34:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:34:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:34:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:34:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:34:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:34:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:34:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:34:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:34:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:34:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:34:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:34:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:34:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:34:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:34:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:34:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:34:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:34:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:34:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:34:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:34:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:34:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:34:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:34:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:34:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:34:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:34:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:34:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:34:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:34:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:34:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:34:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:34:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:34:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:34:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:34:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:34:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:34:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:34:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:34:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:34:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:34:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:34:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:34:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:34:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:34:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:34:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:34:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:34:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:34:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:34:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:34:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:34:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:34:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:34:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:34:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:34:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:34:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:34:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:34:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:34:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:34:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:34:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:34:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:34:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:34:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:34:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:34:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:34:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:34:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:34:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:34:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:34:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:34:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:34:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:34:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:34:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:34:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:34:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:34:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:34:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:34:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:34:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:34:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:34:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:34:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:34:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:34:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:34:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:34:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:34:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:34:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:34:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:34:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:34:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:34:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:34:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:34:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:34:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:34:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:34:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:34:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:34:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:34:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:34:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:34:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:34:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:34:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:34:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:34:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:34:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:34:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:34:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:34:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:34:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:34:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:34:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:34:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:34:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:34:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:34:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:34:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:34:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:34:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:34:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:34:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:34:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:34:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:34:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:34:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:34:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:34:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:34:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:34:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:34:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:34:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:34:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:34:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:34:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:34:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:34:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:34:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:34:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:34:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:34:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:34:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:34:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:34:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:34:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:34:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:34:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:34:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:34:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:34:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:34:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:34:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:35:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:35:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:35:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:35:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:35:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:35:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:35:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:35:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:35:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:35:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:35:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:35:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:35:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:35:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:35:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:35:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:35:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:35:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:35:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:35:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:35:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:35:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:35:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:35:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:35:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:35:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:35:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:35:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:35:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:35:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:35:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:35:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:35:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:35:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:35:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:35:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:35:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:35:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:35:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:35:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:35:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:35:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:35:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:35:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:35:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:35:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:35:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:35:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:35:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:35:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:35:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:35:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:35:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:35:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:35:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:35:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:35:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:35:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:35:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:35:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:35:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:35:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:35:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:35:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:35:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:35:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:35:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:35:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:35:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:35:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:35:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:35:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:35:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:35:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:35:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:35:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:35:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:35:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:35:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:35:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:35:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:35:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:35:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:35:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:35:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:35:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:35:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:35:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:35:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:35:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:35:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:35:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:35:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:35:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:35:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:35:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:35:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:35:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:35:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:35:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:35:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:35:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:35:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:35:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:35:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:35:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:35:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:35:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:35:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:35:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:35:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:35:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:35:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:35:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:35:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:35:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:35:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:35:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:35:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:35:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:35:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:35:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:35:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:35:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:35:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:35:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:35:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:35:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:35:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:35:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:35:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:35:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:35:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:35:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:35:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:35:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:35:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:35:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:35:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:35:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:35:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:35:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:35:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:35:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:35:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:35:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:35:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:35:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:35:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:35:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:35:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:35:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:35:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:35:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:35:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:35:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:35:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:35:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:35:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:35:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:35:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:35:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:35:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:35:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:35:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:35:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:35:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:35:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:35:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:35:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:35:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:35:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:35:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:35:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:35:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:35:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:35:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:35:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:35:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:35:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:35:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:35:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:35:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:35:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:35:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:35:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:35:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:35:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:35:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:35:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:35:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:35:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:35:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:35:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:35:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:35:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:35:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:35:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:35:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:35:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:35:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:35:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:35:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:35:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:35:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:35:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:35:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:35:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:35:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:35:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:35:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:35:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:35:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:35:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:35:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:35:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:36:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:36:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:36:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:36:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:36:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:36:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:36:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:36:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:36:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:36:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:36:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:36:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:36:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:36:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:36:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:36:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:36:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:36:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:36:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:36:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:36:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:36:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:36:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:36:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:36:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:36:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:36:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:36:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:36:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:36:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:36:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:36:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:36:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:36:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:36:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:36:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:36:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:36:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:36:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:36:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:36:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:36:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:36:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:36:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:36:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:36:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:36:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:36:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:36:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:36:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:36:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:36:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:36:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:36:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:36:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:36:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:36:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:36:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:36:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:36:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:36:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:36:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:36:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:36:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:36:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:36:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:36:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:36:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:36:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:36:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:36:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:36:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:36:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:36:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:36:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:36:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:36:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:36:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:36:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:36:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:36:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:36:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:36:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:36:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:36:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:36:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:36:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:36:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:36:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:36:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:36:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:36:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:36:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:36:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:36:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:36:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:36:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:36:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:36:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:36:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:36:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:36:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:36:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:36:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:36:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:36:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:36:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:36:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:36:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:36:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:36:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:36:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:36:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:36:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:36:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:36:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:36:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:36:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:36:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:36:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:36:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:36:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:36:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:36:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:36:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:36:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:36:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:36:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:36:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:36:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:36:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:36:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:36:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:36:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:36:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:36:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:36:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:36:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:36:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:36:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:36:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:36:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:36:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:36:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:36:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:36:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:36:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:36:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:36:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:36:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:36:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:36:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:36:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:36:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:36:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:36:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:36:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:36:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:36:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:36:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:36:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:36:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:36:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:36:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:36:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:36:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:36:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:36:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:36:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:36:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:36:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:36:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:36:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:36:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:36:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:36:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:36:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:36:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:36:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:36:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:36:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:36:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:36:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:36:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:36:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:36:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:36:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:36:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:36:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:36:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:36:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:36:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:36:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:36:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:36:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:36:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:36:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:36:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:36:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:36:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:36:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:36:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:36:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:36:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:36:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:36:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:36:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:36:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:36:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:36:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:36:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:36:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:36:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:36:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:36:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:36:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:37:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:37:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:37:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:37:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:37:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:37:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:37:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:37:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:37:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:37:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:37:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:37:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:37:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:37:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:37:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:37:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:37:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:37:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:37:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:37:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:37:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:37:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:37:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:37:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:37:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:37:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:37:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:37:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:37:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:37:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:37:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:37:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:37:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:37:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:37:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:37:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:37:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:37:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:37:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:37:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:37:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:37:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:37:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:37:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:37:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:37:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:37:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:37:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:37:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:37:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:37:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:37:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:37:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:37:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:37:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:37:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:37:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:37:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:37:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:37:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:37:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:37:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:37:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:37:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:37:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:37:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:37:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:37:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:37:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:37:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:37:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:37:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:37:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:37:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:37:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:37:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:37:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:37:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:37:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:37:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:37:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:37:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:37:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:37:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:37:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:37:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:37:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:37:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:37:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:37:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:37:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:37:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:37:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:37:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:37:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:37:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:37:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:37:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:37:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:37:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:37:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:37:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:37:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:37:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:37:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:37:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:37:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:37:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:37:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:37:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:37:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:37:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:37:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:37:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:37:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:37:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:37:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:37:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:37:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:37:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:37:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:37:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:37:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:37:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:37:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:37:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:37:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:37:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:37:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:37:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:37:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:37:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:37:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:37:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:37:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:37:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:37:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:37:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:37:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:37:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:37:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:37:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:37:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:37:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:37:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:37:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:37:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:37:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:37:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:37:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:37:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:37:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:37:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:37:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:37:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:37:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:37:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:37:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:37:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:37:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:37:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:37:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:37:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:37:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:37:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:37:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:37:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:37:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:37:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:37:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:37:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:37:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:37:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:37:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:37:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:37:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:37:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:37:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:37:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:37:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:37:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:37:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:37:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:37:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:37:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:37:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:37:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:37:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:37:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:37:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:37:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:37:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:37:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:37:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:37:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:37:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:37:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:37:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:37:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:37:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:37:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:37:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:37:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:37:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:37:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:37:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:37:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:37:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:37:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:37:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:37:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:37:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:37:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:37:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:37:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:37:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:38:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:38:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:38:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:38:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:38:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:38:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:38:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:38:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:38:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:38:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:38:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:38:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:38:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:38:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:38:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:38:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:38:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:38:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:38:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:38:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:38:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:38:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:38:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:38:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:38:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:38:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:38:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:38:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:38:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:38:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:38:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:38:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:38:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:38:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:38:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:38:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:38:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:38:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:38:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:38:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:38:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:38:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:38:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:38:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:38:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:38:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:38:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:38:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:38:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:38:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:38:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:38:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:38:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:38:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:38:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:38:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:38:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:38:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:38:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:38:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:38:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:38:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:38:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:38:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:38:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:38:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:38:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:38:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:38:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:38:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:38:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:38:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:38:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:38:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:38:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:38:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:38:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:38:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:38:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:38:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:38:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:38:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:38:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:38:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:38:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:38:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:38:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:38:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:38:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:38:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:40:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:40:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:40:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:40:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:40:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:40:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:40:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:40:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:40:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:40:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:40:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:40:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:40:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:40:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:40:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:40:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:40:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:40:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:40:31 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 19 01:40:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:40:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:40:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:40:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:40:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:40:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:40:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:40:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:40:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:40:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:40:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:40:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:40:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:40:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:40:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:40:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:40:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:40:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:40:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:40:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:40:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:40:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:40:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:40:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:40:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:40:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:40:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:40:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:40:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:40:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:40:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:40:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:40:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:40:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:40:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:40:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:40:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:40:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:40:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:40:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:40:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:40:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:40:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:40:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:40:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:40:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:40:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:40:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:40:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:40:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:40:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:40:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:40:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:40:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:40:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:40:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:40:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:40:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:40:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:40:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:40:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:40:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:40:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:40:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:40:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:40:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:40:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:40:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:40:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:40:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:40:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:40:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:40:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:40:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:40:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:40:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:40:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:40:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:40:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:40:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:40:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:40:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:40:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:40:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:40:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:40:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:40:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:40:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:40:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:40:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:41:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:41:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:41:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:41:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:41:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:41:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:41:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:41:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:41:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:41:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:41:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:41:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:41:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:41:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:41:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:41:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:41:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:41:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:41:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:41:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:41:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:41:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:41:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:41:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:41:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:41:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:41:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:41:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:41:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:41:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:41:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:41:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:41:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:41:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:41:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:41:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:41:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:41:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:41:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:41:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:41:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:41:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:41:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:41:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:41:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:41:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:41:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:41:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:41:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:41:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:41:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:41:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:41:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:41:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:41:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:41:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:41:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:41:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:41:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:41:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:41:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:41:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:41:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:41:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:41:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:41:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:41:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:41:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:41:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:41:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:41:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:41:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:41:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:41:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:41:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:41:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:41:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:41:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:41:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:41:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:41:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:41:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:41:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:41:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:41:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:41:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:41:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:41:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:41:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:41:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:41:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:41:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:41:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:41:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:41:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:41:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:41:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:41:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:41:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:41:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:41:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:41:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:41:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:41:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:41:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:41:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:41:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:41:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:41:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:41:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:41:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:41:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:41:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:41:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:41:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:41:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:41:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:41:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:41:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:41:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:41:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:41:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:41:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:41:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:41:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:41:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:41:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:41:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:41:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:41:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:41:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:41:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:41:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:41:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:41:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:41:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:41:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:41:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:41:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:41:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:41:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:41:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:41:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:41:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:41:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:41:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:41:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:41:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:41:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:41:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:41:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:41:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:41:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:41:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:41:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:41:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:41:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:41:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:41:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:41:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:41:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:41:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:41:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:41:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:41:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:41:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:41:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:41:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:41:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:41:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:41:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:41:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:41:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:41:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:41:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:41:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:41:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:41:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:41:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:41:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:41:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:41:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:41:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:41:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:41:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:41:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:41:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:41:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:41:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:41:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:41:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:41:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:41:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:41:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:41:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:41:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:41:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:41:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:41:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:41:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:41:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:41:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:41:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:41:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:41:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:41:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:41:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:41:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:41:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:41:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:41:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:41:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:41:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:41:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:41:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:41:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:42:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:42:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:42:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:42:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:42:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:42:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:42:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:42:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:42:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:42:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:42:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:42:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:42:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:42:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:42:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:42:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:42:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:42:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:42:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:42:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:42:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:42:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:42:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:42:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:42:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:42:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:42:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:42:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:42:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:42:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:42:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:42:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:42:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:42:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:42:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:42:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:42:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:42:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:42:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:42:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:42:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:42:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:42:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:42:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:42:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:42:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:42:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:42:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:42:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:42:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:42:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:42:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:42:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:42:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:42:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:42:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:42:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:42:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:42:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:42:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:42:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:42:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:42:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:42:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:42:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:42:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:42:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:42:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:42:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:42:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:42:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:42:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:42:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:42:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:42:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:42:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:42:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:42:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:42:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:42:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:42:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:42:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:42:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:42:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:42:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:42:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:42:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:42:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:42:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:42:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:42:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:42:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:42:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:42:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:42:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:42:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:42:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:42:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:42:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:42:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:42:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:42:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:42:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:42:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:42:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:42:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:42:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:42:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:42:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:42:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:42:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:42:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:42:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:42:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:42:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:42:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:42:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:42:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:42:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:42:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:42:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:42:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:42:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:42:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:42:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:42:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:42:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:42:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:42:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:42:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:42:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:42:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:42:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:42:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:42:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:42:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:42:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:42:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:42:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:42:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:42:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:42:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:42:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:42:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:42:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:42:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:42:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:42:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:42:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:42:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:42:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:42:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:42:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:42:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:42:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:42:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:42:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:42:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:42:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:42:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:42:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:42:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:42:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:42:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:42:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:42:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:42:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:42:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:42:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:42:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:42:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:42:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:42:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:42:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:42:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:42:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:42:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:42:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:42:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:42:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:42:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:42:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:42:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:42:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:42:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:42:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:42:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:42:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:42:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:42:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:42:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:42:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:42:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:42:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:42:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:42:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:42:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:42:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:42:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:42:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:42:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:42:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:42:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:42:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:42:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:42:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:42:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:42:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:42:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:42:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:42:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:42:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:42:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:42:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:42:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:42:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:43:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:43:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:43:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:43:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:43:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:43:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:43:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:43:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:43:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:43:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:43:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:43:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:43:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:43:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:43:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:43:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:43:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:43:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:43:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:43:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:43:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:43:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:43:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:43:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:43:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:43:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:43:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:43:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:43:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:43:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:43:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:43:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:43:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:43:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:43:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:43:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:43:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:43:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:43:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:43:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:43:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:43:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:43:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:43:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:43:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:43:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:43:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:43:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:43:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:43:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:43:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:43:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:43:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:43:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:43:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:43:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:43:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:43:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:43:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:43:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:43:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:43:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:43:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:43:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:43:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:43:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:43:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:43:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:43:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:43:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:43:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:43:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:43:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:43:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:43:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:43:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:43:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:43:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:43:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:43:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:43:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:43:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:43:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:43:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:43:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:43:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:43:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:43:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:43:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:43:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:43:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:43:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:43:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:43:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:43:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:43:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:43:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:43:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:43:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:43:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:43:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:43:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:43:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:43:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:43:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:43:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:43:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:43:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:43:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:43:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:43:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:43:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:43:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:43:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:43:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:43:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:43:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:43:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:43:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:43:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:43:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:43:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:43:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:43:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:43:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:43:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:43:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:43:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:43:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:43:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:43:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:43:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:43:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:43:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:43:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:43:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:43:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:43:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:43:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:43:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:43:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:43:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:43:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:43:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:43:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:43:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:43:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:43:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:43:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:43:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:43:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:43:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:43:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:43:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:43:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:43:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:43:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:43:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:43:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:43:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:43:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:43:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:43:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:43:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:43:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:43:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:43:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:43:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:43:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:43:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:43:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:43:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:43:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:43:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:43:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:43:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:43:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:43:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:43:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:43:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:43:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:43:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:43:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:43:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:43:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:43:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:43:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:43:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:43:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:43:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:43:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:43:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:43:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:43:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:43:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:43:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:43:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:43:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:43:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:43:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:43:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:43:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:43:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:43:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:43:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:43:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:43:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:43:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:43:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:43:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:43:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:43:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:43:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:43:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:43:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:43:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:44:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:44:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:44:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:44:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:44:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:44:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:44:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:44:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:44:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:44:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:44:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:44:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:44:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:44:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:44:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:44:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:44:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:44:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:44:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:44:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:44:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:44:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:44:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:44:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:44:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:44:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:44:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:44:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:44:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:44:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:44:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:44:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:44:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:44:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:44:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:44:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:44:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:44:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:44:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:44:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:44:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:44:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:44:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:44:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:44:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:44:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:44:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:44:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:44:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:44:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:44:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:44:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:44:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:44:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:44:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:44:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:44:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:44:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:44:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:44:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:44:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:44:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:44:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:44:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:44:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:44:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:44:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:44:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:44:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:44:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:44:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:44:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:44:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:44:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:44:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:44:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:44:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:44:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:44:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:44:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:44:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:44:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:44:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:44:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:44:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:44:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:44:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:44:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:44:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:44:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:44:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:44:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:44:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:44:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:44:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:44:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:44:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:44:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:44:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:44:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:44:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:44:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:44:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:44:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:44:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:44:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:44:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:44:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:44:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:44:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:44:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:44:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:44:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:44:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:44:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:44:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:44:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:44:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:44:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:44:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:44:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:44:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:44:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:44:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:44:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:44:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:44:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:44:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:44:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:44:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:44:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:44:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:44:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:44:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:44:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:44:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:44:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:44:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:44:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:44:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:44:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:44:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:44:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:44:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:44:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:44:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:44:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:44:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:44:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:44:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:44:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:44:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:44:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:44:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:44:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:44:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:44:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:44:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:44:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:44:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:44:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:44:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:44:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:44:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:44:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:44:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:44:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:44:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:44:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:44:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:44:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:44:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:44:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:44:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:44:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:44:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:44:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:44:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:44:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:44:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:44:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:44:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:44:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:44:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:44:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:44:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:44:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:44:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:44:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:44:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:44:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:44:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:44:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:44:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:44:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:44:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:44:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:44:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:44:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:44:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:44:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:44:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:44:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:44:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:44:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:44:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:44:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:44:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:44:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:44:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:44:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:44:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:44:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:44:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:44:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:44:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:45:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:45:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:45:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:45:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:45:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:45:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:45:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:45:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:45:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:45:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:45:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:45:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:45:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:45:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:45:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:45:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:45:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:45:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:45:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:45:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:45:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:45:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:45:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:45:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:45:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:45:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:45:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:45:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:45:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:45:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:45:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:45:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:45:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:45:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:45:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:45:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:45:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:45:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:45:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:45:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:45:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:45:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:45:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:45:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:45:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:45:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:45:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:45:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:45:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:45:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:45:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:45:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:45:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:45:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:45:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:45:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:45:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:45:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:45:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:45:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:45:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:45:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:45:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:45:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:45:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:45:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:45:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:45:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:45:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:45:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:45:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:45:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:45:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:45:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:45:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:45:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:45:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:45:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:45:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:45:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:45:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:45:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:45:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:45:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:45:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:45:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:45:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:45:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:45:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:45:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:45:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:45:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:45:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:45:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:45:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:45:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:45:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:45:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:45:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:45:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:45:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:45:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:45:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:45:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:45:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:45:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:45:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:45:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:45:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:45:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:45:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:45:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:45:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:45:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:45:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:45:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:45:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:45:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:45:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:45:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:45:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:45:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:45:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:45:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:45:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:45:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:45:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:45:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:45:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:45:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:45:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:45:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:45:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:45:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:45:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:45:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:45:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:45:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:45:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:45:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:45:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:45:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:45:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:45:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:45:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:45:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:45:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:45:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:45:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:45:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:45:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:45:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:45:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:45:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:45:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:45:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:45:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:45:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:45:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:45:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:45:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:45:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:45:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:45:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:45:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:45:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:45:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:45:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:45:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:45:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:45:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:45:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:45:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:45:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:45:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:45:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:45:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:45:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:45:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:45:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:45:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:45:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:45:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:45:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:45:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:45:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:45:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:45:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:45:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:45:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:45:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:45:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:45:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:45:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:45:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:45:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:45:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:45:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:45:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:45:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:45:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:45:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:45:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:45:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:45:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:45:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:45:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:45:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:45:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:45:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:45:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:45:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:45:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:45:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:45:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:45:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:46:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:46:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:46:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:46:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:46:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:46:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:46:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:46:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:46:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:46:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:46:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:46:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:46:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:46:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:46:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:46:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:46:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:46:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:46:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:46:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:46:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:46:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:46:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:46:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:46:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:46:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:46:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:46:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:46:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:46:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:46:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:46:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:46:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:46:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:46:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:46:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:46:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:46:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:46:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:46:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:46:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:46:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:46:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:46:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:46:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:46:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:46:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:46:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:46:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:46:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:46:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:46:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:46:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:46:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:46:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:46:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:46:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:46:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:46:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:46:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:46:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:46:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:46:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:46:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:46:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:46:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:46:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:46:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:46:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:46:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:46:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:46:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:46:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:46:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:46:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:46:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:46:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:46:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:46:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:46:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:46:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:46:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:46:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:46:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:46:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:46:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:46:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:46:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:46:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:46:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:46:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:46:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:46:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:46:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:46:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:46:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:46:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:46:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:46:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:46:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:46:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:46:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:46:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:46:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:46:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:46:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:46:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:46:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:46:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:46:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:46:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:46:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:46:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:46:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:46:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:46:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:46:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:46:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:46:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:46:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:46:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:46:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:46:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:46:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:46:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:46:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:46:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:46:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:46:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:46:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:46:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:46:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:46:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:46:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:46:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:46:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:46:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:46:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:46:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:46:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:46:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:46:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:46:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:46:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:46:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:46:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:46:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:46:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:46:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:46:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:46:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:46:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:46:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:46:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:46:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:46:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:46:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:46:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:46:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:46:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:46:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:46:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:46:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:46:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:46:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:46:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:46:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:46:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:46:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:46:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:46:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:46:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:46:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:46:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:46:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:46:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:46:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:46:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:46:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:46:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:46:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:46:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:46:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:46:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:46:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:46:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:46:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:46:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:46:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:46:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:46:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:46:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:46:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:46:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:46:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:46:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:46:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:46:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:46:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:46:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:46:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:46:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:46:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:46:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:46:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:46:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:46:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:46:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:46:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:46:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:46:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:46:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:46:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:46:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:46:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:46:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:47:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:47:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:47:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:47:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:47:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:47:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:47:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:47:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:47:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:47:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:47:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:47:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:47:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:47:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:47:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:47:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:47:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:47:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:47:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:47:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:47:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:47:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:47:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:47:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:47:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:47:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:47:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:47:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:47:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:47:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:47:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:47:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:47:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:47:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:47:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:47:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:47:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:47:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:47:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:47:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:47:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:47:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:47:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:47:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:47:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:47:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:47:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:47:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:47:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:47:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:47:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:47:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:47:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:47:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:47:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:47:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:47:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:47:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:47:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:47:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:47:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:47:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:47:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:47:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:47:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:47:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:47:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:47:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:47:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:47:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:47:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:47:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:47:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:47:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:47:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:47:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:47:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:47:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:47:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:47:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:47:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:47:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:47:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:47:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:47:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:47:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:47:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:47:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:47:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:47:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:47:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:47:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:47:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:47:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:47:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:47:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:47:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:47:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:47:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:47:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:47:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:47:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:47:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:47:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:47:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:47:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:47:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:47:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:47:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:47:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:47:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:47:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:47:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:47:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:47:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:47:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:47:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:47:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:47:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:47:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:47:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:47:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:47:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:47:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:47:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:47:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:47:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:47:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:47:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:47:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:47:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:47:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:47:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:47:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:47:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:47:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:47:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:47:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:47:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:47:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:47:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:47:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:47:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:47:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:47:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:47:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:47:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:47:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:47:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:47:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:47:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:47:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:47:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:47:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:47:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:47:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:47:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:47:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:47:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:47:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:47:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:47:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:47:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:47:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:47:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:47:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:47:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:47:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:47:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:47:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:47:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:47:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:47:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:47:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:47:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:47:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:47:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:47:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:47:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:47:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:47:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:47:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:47:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:47:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:47:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:47:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:47:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:47:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:47:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:47:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:47:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:47:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:47:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:47:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:47:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:47:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:47:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:47:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:47:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:47:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:47:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:47:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:47:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:47:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:47:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:47:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:47:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:47:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:47:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:47:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:47:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:47:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:47:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:47:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:47:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:47:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:48:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:48:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:48:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:48:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:48:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:48:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:48:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:48:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:48:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:48:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:48:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:48:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:48:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:48:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:48:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:48:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:48:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:48:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:48:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:48:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:48:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:48:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:48:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:48:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:48:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:48:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:48:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:48:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:48:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:48:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:48:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:48:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:48:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:48:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:48:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:48:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:48:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:48:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:48:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:48:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:48:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:48:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:48:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:48:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:48:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:48:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:48:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:48:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:48:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:48:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:48:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:48:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:48:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:48:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:48:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:48:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:48:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:48:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:48:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:48:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:48:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:48:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:48:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:48:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:48:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:48:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:48:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:48:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:48:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:48:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:48:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:48:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:48:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:48:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:48:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:48:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:48:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:48:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:48:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:48:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:48:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:48:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:48:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:48:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:48:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:48:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:48:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:48:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:48:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:48:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:48:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:48:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:48:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:48:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:48:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:48:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:48:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:48:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:48:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:48:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:48:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:48:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:48:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:48:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:48:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:48:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:48:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:48:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:50:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:50:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:50:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:50:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:50:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:50:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:50:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:50:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:50:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:50:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:50:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:50:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:50:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:50:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:50:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:50:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:50:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:50:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:50:36 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 19 01:50:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:50:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:50:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:50:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:50:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:50:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:50:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:50:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:50:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:50:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:50:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:50:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:50:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:50:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:50:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:50:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:50:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:50:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:50:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:50:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:50:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:50:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:50:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:50:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:50:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:50:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:50:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:50:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:50:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:50:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:50:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:50:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:50:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:50:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:50:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:50:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:50:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:50:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:50:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:50:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:50:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:50:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:50:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:50:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:50:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:50:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:50:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:50:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:50:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:50:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:50:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:50:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:50:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:50:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:50:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:50:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:50:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:50:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:50:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:50:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:50:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:50:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:50:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:50:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:50:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:50:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:50:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:50:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:50:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:50:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:50:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:50:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:51:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:51:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:51:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:51:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:51:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:51:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:51:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:51:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:51:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:51:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:51:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:51:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:51:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:51:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:51:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:51:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:51:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:51:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:51:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:51:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:51:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:51:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:51:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:51:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:51:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:51:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:51:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:51:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:51:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:51:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:51:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:51:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:51:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:51:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:51:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:51:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:51:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:51:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:51:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:51:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:51:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:51:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:51:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:51:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:51:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:51:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:51:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:51:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:51:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:51:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:51:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:51:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:51:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:51:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:51:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:51:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:51:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:51:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:51:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:51:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:51:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:51:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:51:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:51:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:51:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:51:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:51:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:51:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:51:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:51:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:51:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:51:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:51:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:51:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:51:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:51:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:51:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:51:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:51:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:51:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:51:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:51:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:51:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:51:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:51:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:51:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:51:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:51:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:51:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:51:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:51:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:51:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:51:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:51:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:51:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:51:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:51:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:51:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:51:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:51:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:51:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:51:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:51:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:51:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:51:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:51:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:51:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:51:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:51:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:51:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:51:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:51:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:51:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:51:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:51:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:51:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:51:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:51:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:51:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:51:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:51:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:51:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:51:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:51:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:51:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:51:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:51:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:51:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:51:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:51:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:51:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:51:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:51:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:51:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:51:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:51:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:51:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:51:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:51:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:51:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:51:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:51:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:51:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:51:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:51:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:51:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:51:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:51:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:51:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:51:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:51:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:51:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:51:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:51:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:51:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:51:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:51:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:51:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:51:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:51:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:51:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:51:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:51:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:51:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:51:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:51:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:51:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:51:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:51:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:51:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:51:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:51:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:51:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:51:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:51:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:51:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:51:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:51:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:51:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:51:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:51:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:51:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:51:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:51:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:51:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:51:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:51:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:51:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:51:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:51:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:51:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:51:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:51:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:51:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:51:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:51:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:51:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:51:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:51:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:51:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:51:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:51:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:51:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:51:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:51:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:51:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:51:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:51:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:51:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:51:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:51:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:51:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:51:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:51:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:51:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:51:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:52:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:52:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:52:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:52:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:52:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:52:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:52:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:52:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:52:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:52:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:52:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:52:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:52:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:52:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:52:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:52:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:52:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:52:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:52:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:52:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:52:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:52:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:52:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:52:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:52:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:52:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:52:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:52:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:52:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:52:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:52:06 rdma06 systemd-journald[790]: Data hash table of /run/log/journal/95379d2f723847b68d9e976dee8967a9/system.journal has a fill level at 75.0 (163751 of 218332 items, 75497472 file size, 461 bytes per hash table item), suggesting rotation.
Apr 19 01:52:06 rdma06 systemd-journald[790]: /run/log/journal/95379d2f723847b68d9e976dee8967a9/system.journal: Journal header limits reached or header out-of-date, rotating.
Apr 19 01:52:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:52:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:52:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:52:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:52:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:52:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:52:06 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 19 01:52:06 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 19 01:52:06 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 19 01:52:06 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 19 01:52:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:52:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:52:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:52:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:52:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:52:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:52:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:52:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:52:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:52:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:52:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:52:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:52:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:52:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:52:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:52:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:52:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:52:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:52:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:52:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:52:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:52:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:52:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:52:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:52:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:52:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:52:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:52:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:52:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:52:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:52:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:52:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:52:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:52:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:52:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:52:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:52:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:52:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:52:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:52:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:52:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:52:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:52:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:52:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:52:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:52:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:52:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:52:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:52:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:52:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:52:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:52:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:52:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:52:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:52:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:52:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:52:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:52:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:52:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:52:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:52:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:52:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:52:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:52:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:52:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:52:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:52:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:52:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:52:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:52:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:52:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:52:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:52:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:52:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:52:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:52:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:52:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:52:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:52:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:52:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:52:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:52:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:52:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:52:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:52:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:52:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:52:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:52:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:52:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:52:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:52:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:52:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:52:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:52:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:52:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:52:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:52:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:52:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:52:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:52:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:52:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:52:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:52:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:52:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:52:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:52:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:52:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:52:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:52:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:52:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:52:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:52:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:52:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:52:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:52:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:52:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:52:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:52:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:52:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:52:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:52:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:52:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:52:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:52:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:52:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:52:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:52:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:52:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:52:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:52:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:52:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:52:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:52:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:52:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:52:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:52:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:52:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:52:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:52:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:52:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:52:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:52:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:52:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:52:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:52:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:52:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:52:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:52:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:52:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:52:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:52:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:52:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:52:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:52:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:52:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:52:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:52:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:52:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:52:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:52:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:52:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:52:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:52:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:52:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:52:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:52:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:52:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:52:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:52:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:52:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:52:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:52:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:52:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:52:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:52:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:52:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:52:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:52:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:52:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:52:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:53:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:53:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:53:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:53:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:53:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:53:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:53:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:53:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:53:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:53:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:53:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:53:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:53:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:53:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:53:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:53:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:53:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:53:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:53:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:53:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:53:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:53:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:53:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:53:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:53:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:53:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:53:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:53:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:53:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:53:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:53:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:53:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:53:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:53:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:53:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:53:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:53:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:53:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:53:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:53:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:53:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:53:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:53:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:53:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:53:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:53:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:53:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:53:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:53:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:53:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:53:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:53:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:53:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:53:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:53:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:53:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:53:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:53:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:53:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:53:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:53:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:53:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:53:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:53:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:53:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:53:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:53:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:53:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:53:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:53:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:53:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:53:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:53:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:53:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:53:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:53:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:53:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:53:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:53:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:53:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:53:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:53:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:53:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:53:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:53:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:53:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:53:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:53:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:53:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:53:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:53:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:53:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:53:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:53:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:53:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:53:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:53:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:53:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:53:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:53:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:53:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:53:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:53:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:53:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:53:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:53:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:53:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:53:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:53:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:53:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:53:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:53:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:53:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:53:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:53:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:53:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:53:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:53:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:53:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:53:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:53:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:53:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:53:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:53:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:53:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:53:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:53:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:53:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:53:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:53:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:53:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:53:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:53:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:53:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:53:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:53:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:53:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:53:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:53:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:53:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:53:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:53:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:53:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:53:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:53:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:53:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:53:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:53:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:53:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:53:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:53:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:53:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:53:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:53:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:53:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:53:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:53:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:53:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:53:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:53:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:53:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:53:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:53:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:53:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:53:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:53:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:53:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:53:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:53:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:53:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:53:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:53:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:53:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:53:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:53:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:53:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:53:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:53:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:53:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:53:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:53:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:53:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:53:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:53:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:53:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:53:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:53:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:53:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:53:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:53:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:53:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:53:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:53:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:53:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:53:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:53:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:53:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:53:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:53:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:53:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:53:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:53:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:53:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:53:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:53:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:53:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:53:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:53:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:53:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:53:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:53:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:53:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:53:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:53:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:53:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:53:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:54:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:54:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:54:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:54:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:54:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:54:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:54:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:54:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:54:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:54:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:54:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:54:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:54:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:54:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:54:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:54:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:54:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:54:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:54:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:54:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:54:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:54:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:54:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:54:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:54:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:54:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:54:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:54:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:54:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:54:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:54:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:54:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:54:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:54:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:54:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:54:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:54:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:54:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:54:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:54:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:54:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:54:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:54:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:54:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:54:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:54:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:54:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:54:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:54:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:54:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:54:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:54:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:54:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:54:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:54:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:54:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:54:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:54:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:54:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:54:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:54:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:54:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:54:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:54:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:54:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:54:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:54:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:54:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:54:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:54:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:54:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:54:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:54:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:54:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:54:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:54:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:54:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:54:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:54:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:54:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:54:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:54:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:54:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:54:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:54:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:54:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:54:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:54:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:54:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:54:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:54:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:54:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:54:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:54:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:54:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:54:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:54:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:54:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:54:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:54:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:54:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:54:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:54:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:54:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:54:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:54:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:54:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:54:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:54:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:54:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:54:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:54:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:54:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:54:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:54:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:54:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:54:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:54:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:54:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:54:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:54:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:54:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:54:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:54:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:54:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:54:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:54:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:54:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:54:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:54:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:54:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:54:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:54:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:54:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:54:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:54:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:54:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:54:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:54:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:54:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:54:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:54:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:54:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:54:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:54:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:54:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:54:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:54:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:54:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:54:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:54:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:54:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:54:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:54:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:54:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:54:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:54:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:54:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:54:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:54:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:54:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:54:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:54:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:54:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:54:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:54:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:54:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:54:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:54:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:54:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:54:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:54:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:54:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:54:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:54:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:54:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:54:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:54:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:54:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:54:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:54:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:54:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:54:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:54:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:54:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:54:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:54:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:54:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:54:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:54:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:54:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:54:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:54:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:54:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:54:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:54:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:54:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:54:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:54:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:54:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:54:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:54:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:54:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:54:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:54:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:54:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:54:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:54:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:54:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:54:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:54:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:54:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:54:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:54:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:54:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:54:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:55:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:55:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:55:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:55:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:55:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:55:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:55:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:55:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:55:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:55:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:55:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:55:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:55:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:55:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:55:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:55:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:55:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:55:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:55:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:55:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:55:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:55:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:55:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:55:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:55:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:55:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:55:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:55:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:55:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:55:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:55:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:55:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:55:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:55:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:55:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:55:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:55:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:55:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:55:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:55:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:55:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:55:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:55:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:55:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:55:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:55:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:55:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:55:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:55:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:55:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:55:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:55:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:55:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:55:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:55:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:55:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:55:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:55:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:55:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:55:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:55:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:55:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:55:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:55:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:55:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:55:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:55:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:55:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:55:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:55:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:55:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:55:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:55:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:55:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:55:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:55:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:55:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:55:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:55:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:55:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:55:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:55:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:55:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:55:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:55:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:55:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:55:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:55:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:55:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:55:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:55:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:55:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:55:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:55:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:55:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:55:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:55:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:55:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:55:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:55:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:55:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:55:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:55:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:55:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:55:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:55:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:55:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:55:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:55:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:55:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:55:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:55:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:55:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:55:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:55:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:55:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:55:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:55:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:55:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:55:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:55:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:55:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:55:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:55:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:55:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:55:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:55:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:55:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:55:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:55:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:55:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:55:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:55:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:55:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:55:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:55:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:55:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:55:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:55:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:55:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:55:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:55:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:55:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:55:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:55:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:55:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:55:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:55:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:55:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:55:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:55:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:55:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:55:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:55:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:55:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:55:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:55:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:55:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:55:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:55:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:55:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:55:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:55:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:55:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:55:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:55:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:55:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:55:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:55:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:55:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:55:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:55:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:55:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:55:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:55:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:55:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:55:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:55:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:55:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:55:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:55:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:55:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:55:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:55:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:55:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:55:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:55:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:55:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:55:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:55:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:55:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:55:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:55:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:55:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:55:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:55:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:55:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:55:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:55:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:55:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:55:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:55:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:55:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:55:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:55:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:55:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:55:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:55:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:55:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:55:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:55:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:55:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:55:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:55:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:55:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:55:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:56:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:56:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:56:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:56:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:56:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:56:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:56:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:56:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:56:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:56:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:56:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:56:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:56:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:56:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:56:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:56:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:56:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:56:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:56:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:56:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:56:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:56:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:56:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:56:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:56:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:56:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:56:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:56:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:56:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:56:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:56:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:56:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:56:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:56:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:56:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:56:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:56:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:56:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:56:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:56:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:56:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:56:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:56:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:56:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:56:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:56:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:56:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:56:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:56:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:56:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:56:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:56:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:56:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:56:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:56:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:56:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:56:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:56:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:56:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:56:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:56:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:56:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:56:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:56:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:56:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:56:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:56:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:56:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:56:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:56:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:56:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:56:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:56:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:56:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:56:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:56:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:56:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:56:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:56:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:56:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:56:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:56:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:56:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:56:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:56:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:56:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:56:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:56:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:56:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:56:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:56:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:56:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:56:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:56:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:56:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:56:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:56:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:56:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:56:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:56:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:56:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:56:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:56:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:56:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:56:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:56:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:56:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:56:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:56:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:56:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:56:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:56:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:56:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:56:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:56:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:56:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:56:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:56:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:56:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:56:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:56:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:56:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:56:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:56:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:56:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:56:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:56:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:56:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:56:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:56:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:56:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:56:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:56:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:56:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:56:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:56:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:56:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:56:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:56:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:56:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:56:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:56:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:56:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:56:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:56:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:56:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:56:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:56:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:56:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:56:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:56:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:56:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:56:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:56:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:56:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:56:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:56:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:56:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:56:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:56:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:56:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:56:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:56:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:56:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:56:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:56:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:56:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:56:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:56:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:56:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:56:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:56:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:56:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:56:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:56:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:56:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:56:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:56:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:56:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:56:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:56:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:56:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:56:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:56:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:56:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:56:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:56:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:56:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:56:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:56:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:56:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:56:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:56:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:56:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:56:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:56:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:56:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:56:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:56:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:56:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:56:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:56:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:56:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:56:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:56:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:56:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:56:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:56:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:56:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:56:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:56:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:56:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:56:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:56:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:56:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:56:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:57:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:57:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:57:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:57:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:57:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:57:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:57:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:57:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:57:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:57:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:57:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:57:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:57:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:57:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:57:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:57:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:57:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:57:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:57:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:57:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:57:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:57:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:57:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:57:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:57:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:57:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:57:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:57:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:57:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:57:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:57:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:57:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:57:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:57:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:57:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:57:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:57:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:57:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:57:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:57:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:57:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:57:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:57:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:57:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:57:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:57:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:57:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:57:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:57:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:57:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:57:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:57:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:57:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:57:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:57:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:57:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:57:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:57:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:57:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:57:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:57:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:57:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:57:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:57:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:57:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:57:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:57:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:57:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:57:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:57:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:57:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:57:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:57:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:57:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:57:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:57:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:57:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:57:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:57:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:57:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:57:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:57:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:57:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:57:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:57:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:57:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:57:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:57:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:57:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:57:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:57:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:57:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:57:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:57:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:57:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:57:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:57:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:57:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:57:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:57:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:57:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:57:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:57:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:57:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:57:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:57:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:57:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:57:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:57:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:57:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:57:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:57:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:57:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:57:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:57:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:57:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:57:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:57:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:57:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:57:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:57:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:57:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:57:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:57:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:57:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:57:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:57:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:57:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:57:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:57:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:57:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:57:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:57:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:57:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:57:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:57:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:57:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:57:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:57:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:57:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:57:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:57:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:57:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:57:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:57:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:57:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:57:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:57:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:57:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:57:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:57:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:57:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:57:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:57:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:57:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:57:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:57:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:57:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:57:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:57:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:57:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:57:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:57:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:57:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:57:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:57:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:57:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:57:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:57:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:57:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:57:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:57:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:57:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:57:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:57:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:57:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:57:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:57:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:57:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:57:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:57:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:57:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:57:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:57:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:57:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:57:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:57:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:57:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:57:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:57:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:57:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:57:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:57:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:57:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:57:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:57:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:57:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:57:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:57:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:57:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:57:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:57:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:57:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:57:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:57:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:57:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:57:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:57:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:57:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:57:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:57:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:57:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:57:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:57:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:57:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:57:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:58:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:58:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:58:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:58:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:58:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:58:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:58:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:58:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:58:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:58:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:58:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:58:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:58:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:58:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:58:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:58:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:58:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:58:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:58:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:58:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:58:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:58:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:58:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:58:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:58:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:58:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:58:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:58:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:58:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:58:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:58:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:58:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:58:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:58:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:58:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:58:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:58:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:58:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:58:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:58:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:58:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:58:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:58:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:58:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:58:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:58:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:58:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:58:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:58:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:58:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:58:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:58:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:58:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:58:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:58:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:58:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:58:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:58:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:58:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:58:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:58:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:58:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:58:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:58:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:58:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:58:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:58:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:58:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:58:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:58:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:58:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:58:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:58:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:58:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:58:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:58:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:58:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:58:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:58:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:58:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:58:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:58:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:58:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:58:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:58:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:58:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:58:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:58:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:58:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:58:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:58:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:58:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:58:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:58:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:58:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:58:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:58:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:58:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:58:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:58:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:58:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:58:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:58:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:58:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:58:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:58:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:58:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:58:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 01:58:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 01:58:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:58:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:58:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:58:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 01:58:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 01:58:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 01:58:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 01:58:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 01:58:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 01:58:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 01:58:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 01:58:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 01:58:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 01:58:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 01:58:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 01:58:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 01:58:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:00:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:00:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:00:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:00:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:00:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:00:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:00:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:00:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:00:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:00:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:00:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:00:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:00:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:00:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:00:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:00:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:00:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:00:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:00:37 rdma06 rsyslogd[1041]: imjournal: 5088 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 19 02:00:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:00:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:00:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:00:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:00:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:00:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:00:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:00:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:00:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:00:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:00:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:00:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:00:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:00:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:00:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:00:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:00:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:00:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:00:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:00:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:00:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:00:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:00:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:00:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:00:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:00:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:00:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:00:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:00:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:00:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:00:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:00:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:00:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:00:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:00:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:00:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:00:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:00:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:00:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:00:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:00:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:00:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:00:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:00:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:00:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:00:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:00:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:00:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:00:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:00:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:00:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:00:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:00:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:00:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:00:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:00:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:00:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:00:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:00:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:00:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:00:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:00:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:00:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:00:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:00:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:00:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:00:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:00:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:00:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:00:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:00:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:00:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:01:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:01:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:01:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:01:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:01:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:01:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:01:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:01:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:01:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:01:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:01:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:01:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:01:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:01:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:01:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:01:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:01:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:01:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:01:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:01:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:01:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:01:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:01:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:01:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:01:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:01:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:01:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:01:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:01:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:01:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:01:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:01:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:01:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:01:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:01:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:01:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:01:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:01:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:01:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:01:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:01:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:01:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:01:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:01:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:01:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:01:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:01:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:01:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:01:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:01:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:01:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:01:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:01:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:01:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:01:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:01:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:01:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:01:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:01:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:01:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:01:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:01:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:01:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:01:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:01:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:01:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:01:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:01:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:01:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:01:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:01:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:01:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:01:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:01:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:01:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:01:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:01:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:01:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:01:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:01:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:01:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:01:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:01:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:01:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:01:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:01:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:01:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:01:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:01:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:01:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:01:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:01:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:01:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:01:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:01:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:01:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:01:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:01:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:01:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:01:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:01:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:01:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:01:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:01:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:01:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:01:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:01:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:01:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:01:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:01:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:01:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:01:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:01:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:01:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:01:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:01:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:01:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:01:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:01:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:01:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:01:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:01:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:01:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:01:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:01:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:01:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:01:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:01:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:01:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:01:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:01:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:01:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:01:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:01:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:01:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:01:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:01:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:01:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:01:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:01:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:01:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:01:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:01:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:01:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:01:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:01:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:01:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:01:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:01:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:01:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:01:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:01:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:01:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:01:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:01:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:01:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:01:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:01:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:01:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:01:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:01:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:01:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:01:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:01:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:01:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:01:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:01:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:01:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:01:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:01:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:01:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:01:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:01:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:01:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:01:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:01:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:01:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:01:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:01:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:01:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:01:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:01:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:01:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:01:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:01:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:01:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:01:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:01:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:01:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:01:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:01:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:01:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:01:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:01:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:01:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:01:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:01:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:01:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:01:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:01:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:01:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:01:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:01:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:01:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:01:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:01:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:01:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:01:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:01:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:01:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:01:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:01:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:01:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:01:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:01:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:01:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:02:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:02:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:02:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:02:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:02:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:02:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:02:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:02:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:02:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:02:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:02:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:02:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:02:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:02:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:02:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:02:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:02:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:02:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:02:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:02:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:02:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:02:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:02:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:02:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:02:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:02:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:02:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:02:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:02:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:02:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:02:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:02:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:02:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:02:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:02:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:02:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:02:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:02:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:02:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:02:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:02:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:02:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:02:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:02:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:02:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:02:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:02:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:02:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:02:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:02:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:02:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:02:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:02:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:02:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:02:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:02:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:02:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:02:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:02:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:02:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:02:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:02:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:02:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:02:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:02:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:02:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:02:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:02:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:02:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:02:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:02:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:02:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:02:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:02:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:02:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:02:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:02:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:02:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:02:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:02:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:02:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:02:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:02:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:02:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:02:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:02:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:02:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:02:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:02:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:02:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:02:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:02:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:02:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:02:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:02:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:02:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:02:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:02:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:02:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:02:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:02:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:02:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:02:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:02:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:02:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:02:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:02:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:02:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:02:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:02:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:02:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:02:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:02:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:02:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:02:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:02:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:02:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:02:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:02:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:02:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:02:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:02:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:02:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:02:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:02:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:02:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:02:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:02:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:02:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:02:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:02:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:02:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:02:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:02:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:02:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:02:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:02:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:02:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:02:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:02:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:02:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:02:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:02:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:02:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:02:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:02:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:02:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:02:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:02:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:02:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:02:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:02:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:02:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:02:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:02:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:02:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:02:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:02:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:02:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:02:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:02:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:02:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:02:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:02:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:02:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:02:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:02:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:02:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:02:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:02:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:02:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:02:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:02:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:02:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:02:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:02:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:02:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:02:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:02:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:02:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:02:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:02:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:02:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:02:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:02:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:02:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:02:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:02:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:02:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:02:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:02:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:02:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:02:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:02:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:02:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:02:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:02:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:02:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:02:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:02:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:02:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:02:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:02:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:02:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:02:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:02:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:02:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:02:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:02:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:02:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:02:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:02:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:02:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:02:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:02:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:02:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:03:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:03:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:03:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:03:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:03:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:03:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:03:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:03:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:03:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:03:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:03:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:03:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:03:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:03:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:03:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:03:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:03:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:03:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:03:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:03:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:03:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:03:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:03:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:03:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:03:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:03:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:03:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:03:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:03:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:03:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:03:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:03:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:03:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:03:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:03:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:03:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:03:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:03:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:03:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:03:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:03:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:03:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:03:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:03:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:03:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:03:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:03:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:03:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:03:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:03:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:03:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:03:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:03:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:03:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:03:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:03:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:03:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:03:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:03:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:03:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:03:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:03:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:03:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:03:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:03:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:03:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:03:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:03:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:03:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:03:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:03:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:03:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:03:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:03:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:03:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:03:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:03:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:03:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:03:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:03:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:03:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:03:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:03:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:03:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:03:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:03:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:03:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:03:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:03:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:03:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:03:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:03:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:03:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:03:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:03:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:03:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:03:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:03:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:03:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:03:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:03:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:03:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:03:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:03:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:03:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:03:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:03:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:03:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:03:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:03:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:03:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:03:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:03:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:03:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:03:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:03:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:03:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:03:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:03:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:03:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:03:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:03:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:03:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:03:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:03:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:03:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:03:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:03:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:03:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:03:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:03:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:03:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:03:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:03:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:03:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:03:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:03:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:03:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:03:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:03:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:03:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:03:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:03:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:03:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:03:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:03:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:03:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:03:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:03:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:03:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:03:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:03:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:03:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:03:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:03:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:03:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:03:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:03:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:03:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:03:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:03:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:03:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:03:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:03:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:03:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:03:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:03:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:03:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:03:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:03:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:03:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:03:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:03:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:03:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:03:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:03:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:03:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:03:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:03:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:03:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:03:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:03:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:03:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:03:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:03:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:03:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:03:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:03:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:03:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:03:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:03:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:03:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:03:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:03:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:03:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:03:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:03:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:03:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:03:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:03:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:03:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:03:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:03:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:03:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:03:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:03:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:03:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:03:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:03:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:03:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:03:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:03:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:03:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:03:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:03:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:03:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:04:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:04:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:04:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:04:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:04:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:04:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:04:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:04:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:04:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:04:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:04:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:04:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:04:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:04:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:04:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:04:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:04:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:04:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:04:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:04:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:04:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:04:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:04:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:04:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:04:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:04:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:04:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:04:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:04:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:04:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:04:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:04:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:04:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:04:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:04:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:04:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:04:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:04:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:04:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:04:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:04:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:04:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:04:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:04:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:04:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:04:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:04:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:04:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:04:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:04:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:04:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:04:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:04:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:04:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:04:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:04:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:04:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:04:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:04:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:04:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:04:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:04:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:04:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:04:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:04:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:04:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:04:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:04:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:04:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:04:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:04:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:04:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:04:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:04:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:04:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:04:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:04:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:04:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:04:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:04:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:04:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:04:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:04:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:04:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:04:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:04:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:04:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:04:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:04:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:04:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:04:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:04:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:04:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:04:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:04:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:04:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:04:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:04:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:04:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:04:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:04:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:04:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:04:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:04:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:04:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:04:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:04:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:04:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:04:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:04:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:04:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:04:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:04:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:04:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:04:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:04:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:04:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:04:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:04:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:04:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:04:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:04:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:04:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:04:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:04:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:04:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:04:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:04:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:04:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:04:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:04:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:04:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:04:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:04:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:04:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:04:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:04:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:04:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:04:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:04:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:04:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:04:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:04:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:04:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:04:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:04:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:04:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:04:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:04:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:04:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:04:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:04:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:04:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:04:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:04:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:04:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:04:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:04:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:04:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:04:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:04:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:04:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:04:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:04:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:04:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:04:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:04:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:04:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:04:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:04:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:04:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:04:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:04:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:04:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:04:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:04:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:04:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:04:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:04:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:04:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:04:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:04:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:04:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:04:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:04:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:04:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:04:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:04:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:04:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:04:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:04:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:04:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:04:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:04:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:04:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:04:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:04:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:04:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:04:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:04:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:04:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:04:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:04:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:04:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:04:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:04:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:04:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:04:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:04:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:04:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:04:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:04:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:04:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:04:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:04:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:04:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:05:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:05:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:05:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:05:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:05:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:05:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:05:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:05:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:05:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:05:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:05:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:05:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:05:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:05:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:05:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:05:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:05:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:05:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:05:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:05:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:05:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:05:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:05:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:05:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:05:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:05:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:05:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:05:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:05:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:05:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:05:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:05:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:05:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:05:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:05:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:05:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:05:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:05:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:05:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:05:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:05:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:05:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:05:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:05:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:05:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:05:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:05:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:05:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:05:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:05:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:05:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:05:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:05:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:05:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:05:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:05:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:05:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:05:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:05:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:05:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:05:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:05:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:05:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:05:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:05:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:05:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:05:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:05:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:05:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:05:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:05:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:05:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:05:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:05:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:05:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:05:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:05:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:05:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:05:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:05:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:05:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:05:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:05:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:05:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:05:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:05:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:05:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:05:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:05:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:05:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:05:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:05:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:05:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:05:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:05:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:05:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:05:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:05:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:05:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:05:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:05:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:05:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:05:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:05:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:05:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:05:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:05:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:05:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:05:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:05:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:05:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:05:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:05:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:05:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:05:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:05:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:05:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:05:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:05:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:05:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:05:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:05:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:05:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:05:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:05:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:05:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:05:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:05:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:05:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:05:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:05:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:05:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:05:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:05:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:05:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:05:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:05:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:05:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:05:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:05:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:05:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:05:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:05:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:05:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:05:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:05:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:05:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:05:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:05:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:05:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:05:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:05:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:05:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:05:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:05:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:05:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:05:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:05:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:05:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:05:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:05:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:05:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:05:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:05:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:05:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:05:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:05:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:05:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:05:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:05:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:05:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:05:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:05:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:05:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:05:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:05:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:05:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:05:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:05:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:05:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:05:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:05:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:05:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:05:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:05:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:05:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:05:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:05:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:05:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:05:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:05:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:05:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:05:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:05:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:05:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:05:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:05:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:05:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:05:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:05:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:05:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:05:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:05:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:05:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:05:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:05:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:05:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:05:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:05:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:05:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:05:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:05:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:05:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:05:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:05:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:05:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:06:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:06:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:06:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:06:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:06:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:06:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:06:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:06:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:06:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:06:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:06:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:06:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:06:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:06:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:06:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:06:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:06:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:06:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:06:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:06:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:06:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:06:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:06:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:06:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:06:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:06:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:06:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:06:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:06:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:06:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:06:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:06:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:06:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:06:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:06:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:06:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:06:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:06:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:06:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:06:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:06:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:06:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:06:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:06:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:06:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:06:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:06:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:06:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:06:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:06:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:06:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:06:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:06:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:06:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:06:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:06:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:06:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:06:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:06:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:06:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:06:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:06:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:06:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:06:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:06:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:06:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:06:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:06:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:06:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:06:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:06:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:06:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:06:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:06:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:06:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:06:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:06:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:06:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:06:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:06:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:06:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:06:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:06:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:06:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:06:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:06:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:06:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:06:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:06:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:06:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:06:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:06:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:06:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:06:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:06:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:06:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:06:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:06:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:06:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:06:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:06:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:06:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:06:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:06:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:06:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:06:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:06:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:06:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:06:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:06:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:06:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:06:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:06:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:06:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:06:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:06:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:06:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:06:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:06:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:06:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:06:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:06:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:06:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:06:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:06:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:06:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:06:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:06:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:06:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:06:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:06:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:06:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:06:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:06:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:06:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:06:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:06:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:06:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:06:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:06:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:06:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:06:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:06:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:06:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:06:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:06:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:06:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:06:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:06:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:06:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:06:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:06:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:06:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:06:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:06:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:06:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:06:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:06:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:06:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:06:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:06:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:06:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:06:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:06:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:06:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:06:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:06:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:06:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:06:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:06:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:06:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:06:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:06:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:06:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:06:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:06:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:06:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:06:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:06:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:06:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:06:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:06:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:06:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:06:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:06:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:06:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:06:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:06:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:06:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:06:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:06:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:06:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:06:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:06:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:06:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:06:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:06:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:06:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:06:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:06:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:06:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:06:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:06:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:06:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:06:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:06:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:06:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:06:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:06:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:06:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:06:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:06:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:06:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:06:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:06:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:06:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:07:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:07:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:07:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:07:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:07:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:07:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:07:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:07:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:07:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:07:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:07:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:07:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:07:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:07:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:07:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:07:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:07:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:07:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:07:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:07:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:07:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:07:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:07:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:07:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:07:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:07:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:07:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:07:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:07:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:07:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:07:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:07:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:07:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:07:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:07:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:07:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:07:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:07:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:07:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:07:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:07:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:07:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:07:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:07:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:07:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:07:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:07:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:07:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:07:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:07:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:07:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:07:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:07:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:07:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:07:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:07:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:07:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:07:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:07:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:07:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:07:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:07:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:07:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:07:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:07:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:07:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:07:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:07:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:07:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:07:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:07:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:07:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:07:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:07:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:07:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:07:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:07:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:07:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:07:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:07:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:07:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:07:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:07:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:07:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:07:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:07:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:07:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:07:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:07:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:07:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:07:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:07:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:07:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:07:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:07:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:07:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:07:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:07:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:07:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:07:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:07:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:07:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:07:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:07:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:07:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:07:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:07:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:07:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:07:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:07:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:07:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:07:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:07:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:07:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:07:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:07:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:07:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:07:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:07:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:07:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:07:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:07:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:07:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:07:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:07:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:07:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:07:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:07:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:07:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:07:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:07:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:07:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:07:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:07:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:07:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:07:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:07:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:07:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:07:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:07:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:07:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:07:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:07:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:07:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:07:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:07:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:07:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:07:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:07:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:07:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:07:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:07:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:07:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:07:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:07:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:07:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:07:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:07:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:07:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:07:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:07:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:07:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:07:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:07:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:07:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:07:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:07:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:07:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:07:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:07:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:07:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:07:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:07:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:07:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:07:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:07:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:07:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:07:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:07:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:07:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:07:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:07:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:07:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:07:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:07:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:07:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:07:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:07:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:07:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:07:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:07:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:07:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:07:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:07:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:07:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:07:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:07:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:07:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:07:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:07:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:07:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:07:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:07:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:07:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:07:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:07:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:07:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:07:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:07:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:07:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:07:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:07:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:07:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:07:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:07:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:07:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:08:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:08:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:08:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:08:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:08:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:08:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:08:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:08:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:08:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:08:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:08:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:08:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:08:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:08:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:08:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:08:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:08:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:08:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:08:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:08:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:08:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:08:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:08:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:08:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:08:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:08:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:08:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:08:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:08:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:08:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:08:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:08:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:08:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:08:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:08:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:08:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:08:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:08:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:08:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:08:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:08:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:08:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:08:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:08:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:08:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:08:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:08:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:08:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:08:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:08:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:08:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:08:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:08:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:08:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:08:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:08:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:08:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:08:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:08:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:08:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:08:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:08:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:08:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:08:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:08:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:08:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:08:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:08:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:08:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:08:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:08:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:08:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:08:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:08:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:08:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:08:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:08:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:08:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:08:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:08:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:08:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:08:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:08:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:08:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:08:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:08:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:08:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:08:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:08:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:08:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:08:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:08:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:08:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:08:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:08:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:08:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:08:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:08:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:08:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:08:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:08:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:08:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:08:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:08:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:08:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:08:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:08:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:08:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:08:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:08:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:08:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:08:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:08:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:08:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:08:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:08:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:08:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:08:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:08:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:08:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:08:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:08:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:08:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:08:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:08:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:08:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:10:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:10:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:10:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:10:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:10:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:10:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:10:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:10:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:10:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:10:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:10:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:10:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:10:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:10:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:10:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:10:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:10:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:10:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:10:42 rdma06 rsyslogd[1041]: imjournal: 5297 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 19 02:10:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:10:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:10:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:10:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:10:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:10:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:10:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:10:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:10:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:10:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:10:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:10:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:10:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:10:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:10:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:10:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:10:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:10:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:10:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:10:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:10:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:10:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:10:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:10:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:10:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:10:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:10:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:10:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:10:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:10:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:10:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:10:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:10:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:10:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:10:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:10:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:10:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:10:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:10:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:10:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:10:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:10:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:10:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:10:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:10:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:10:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:10:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:10:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:10:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:10:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:10:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:10:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:10:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:10:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:11:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:11:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:11:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:11:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:11:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:11:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:11:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:11:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:11:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:11:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:11:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:11:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:11:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:11:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:11:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:11:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:11:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:11:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:11:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:11:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:11:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:11:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:11:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:11:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:11:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:11:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:11:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:11:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:11:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:11:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:11:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:11:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:11:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:11:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:11:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:11:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:11:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:11:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:11:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:11:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:11:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:11:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:11:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:11:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:11:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:11:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:11:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:11:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:11:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:11:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:11:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:11:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:11:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:11:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:11:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:11:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:11:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:11:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:11:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:11:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:11:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:11:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:11:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:11:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:11:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:11:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:11:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:11:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:11:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:11:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:11:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:11:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:11:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:11:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:11:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:11:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:11:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:11:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:11:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:11:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:11:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:11:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:11:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:11:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:11:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:11:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:11:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:11:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:11:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:11:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:11:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:11:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:11:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:11:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:11:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:11:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:11:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:11:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:11:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:11:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:11:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:11:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:11:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:11:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:11:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:11:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:11:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:11:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:11:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:11:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:11:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:11:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:11:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:11:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:11:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:11:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:11:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:11:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:11:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:11:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:11:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:11:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:11:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:11:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:11:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:11:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:11:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:11:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:11:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:11:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:11:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:11:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:11:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:11:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:11:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:11:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:11:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:11:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:11:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:11:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:11:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:11:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:11:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:11:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:11:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:11:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:11:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:11:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:11:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:11:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:11:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:11:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:11:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:11:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:11:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:11:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:11:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:11:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:11:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:11:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:11:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:11:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:11:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:11:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:11:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:11:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:11:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:11:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:11:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:11:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:11:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:11:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:11:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:11:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:11:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:11:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:11:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:11:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:11:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:11:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:11:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:11:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:11:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:11:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:11:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:11:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:11:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:11:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:11:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:11:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:11:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:11:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:11:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:11:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:11:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:11:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:11:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:11:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:11:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:11:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:11:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:11:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:11:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:11:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:11:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:11:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:11:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:11:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:11:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:11:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:11:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:11:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:11:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:11:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:11:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:11:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:12:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:12:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:12:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:12:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:12:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:12:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:12:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:12:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:12:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:12:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:12:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:12:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:12:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:12:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:12:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:12:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:12:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:12:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:12:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:12:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:12:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:12:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:12:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:12:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:12:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:12:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:12:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:12:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:12:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:12:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:12:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:12:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:12:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:12:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:12:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:12:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:12:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:12:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:12:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:12:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:12:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:12:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:12:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:12:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:12:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:12:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:12:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:12:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:12:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:12:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:12:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:12:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:12:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:12:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:12:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:12:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:12:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:12:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:12:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:12:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:12:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:12:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:12:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:12:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:12:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:12:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:12:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:12:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:12:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:12:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:12:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:12:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:12:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:12:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:12:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:12:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:12:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:12:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:12:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:12:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:12:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:12:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:12:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:12:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:12:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:12:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:12:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:12:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:12:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:12:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:12:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:12:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:12:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:12:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:12:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:12:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:12:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:12:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:12:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:12:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:12:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:12:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:12:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:12:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:12:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:12:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:12:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:12:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:12:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:12:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:12:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:12:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:12:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:12:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:12:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:12:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:12:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:12:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:12:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:12:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:12:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:12:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:12:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:12:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:12:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:12:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:12:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:12:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:12:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:12:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:12:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:12:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:12:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:12:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:12:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:12:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:12:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:12:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:12:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:12:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:12:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:12:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:12:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:12:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:12:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:12:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:12:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:12:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:12:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:12:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:12:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:12:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:12:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:12:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:12:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:12:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:12:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:12:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:12:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:12:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:12:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:12:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:12:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:12:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:12:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:12:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:12:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:12:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:12:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:12:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:12:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:12:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:12:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:12:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:12:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:12:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:12:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:12:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:12:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:12:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:12:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:12:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:12:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:12:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:12:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:12:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:12:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:12:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:12:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:12:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:12:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:12:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:12:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:12:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:12:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:12:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:12:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:12:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:12:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:12:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:12:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:12:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:12:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:12:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:12:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:12:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:12:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:12:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:12:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:12:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:12:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:12:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:12:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:12:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:12:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:12:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:13:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:13:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:13:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:13:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:13:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:13:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:13:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:13:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:13:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:13:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:13:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:13:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:13:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:13:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:13:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:13:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:13:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:13:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:13:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:13:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:13:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:13:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:13:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:13:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:13:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:13:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:13:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:13:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:13:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:13:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:13:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:13:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:13:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:13:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:13:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:13:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:13:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:13:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:13:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:13:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:13:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:13:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:13:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:13:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:13:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:13:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:13:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:13:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:13:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:13:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:13:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:13:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:13:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:13:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:13:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:13:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:13:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:13:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:13:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:13:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:13:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:13:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:13:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:13:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:13:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:13:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:13:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:13:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:13:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:13:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:13:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:13:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:13:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:13:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:13:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:13:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:13:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:13:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:13:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:13:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:13:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:13:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:13:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:13:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:13:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:13:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:13:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:13:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:13:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:13:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:13:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:13:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:13:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:13:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:13:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:13:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:13:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:13:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:13:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:13:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:13:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:13:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:13:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:13:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:13:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:13:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:13:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:13:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:13:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:13:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:13:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:13:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:13:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:13:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:13:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:13:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:13:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:13:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:13:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:13:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:13:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:13:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:13:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:13:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:13:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:13:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:13:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:13:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:13:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:13:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:13:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:13:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:13:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:13:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:13:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:13:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:13:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:13:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:13:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:13:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:13:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:13:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:13:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:13:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:13:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:13:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:13:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:13:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:13:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:13:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:13:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:13:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:13:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:13:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:13:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:13:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:13:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:13:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:13:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:13:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:13:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:13:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:13:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:13:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:13:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:13:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:13:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:13:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:13:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:13:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:13:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:13:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:13:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:13:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:13:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:13:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:13:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:13:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:13:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:13:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:13:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:13:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:13:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:13:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:13:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:13:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:13:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:13:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:13:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:13:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:13:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:13:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:13:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:13:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:13:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:13:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:13:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:13:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:13:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:13:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:13:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:13:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:13:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:13:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:13:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:13:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:13:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:13:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:13:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:13:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:13:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:13:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:13:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:13:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:13:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:13:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:14:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:14:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:14:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:14:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:14:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:14:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:14:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:14:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:14:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:14:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:14:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:14:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:14:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:14:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:14:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:14:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:14:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:14:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:14:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:14:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:14:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:14:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:14:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:14:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:14:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:14:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:14:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:14:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:14:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:14:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:14:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:14:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:14:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:14:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:14:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:14:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:14:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:14:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:14:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:14:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:14:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:14:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:14:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:14:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:14:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:14:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:14:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:14:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:14:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:14:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:14:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:14:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:14:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:14:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:14:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:14:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:14:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:14:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:14:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:14:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:14:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:14:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:14:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:14:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:14:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:14:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:14:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:14:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:14:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:14:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:14:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:14:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:14:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:14:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:14:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:14:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:14:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:14:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:14:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:14:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:14:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:14:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:14:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:14:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:14:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:14:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:14:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:14:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:14:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:14:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:14:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:14:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:14:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:14:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:14:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:14:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:14:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:14:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:14:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:14:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:14:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:14:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:14:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:14:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:14:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:14:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:14:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:14:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:14:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:14:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:14:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:14:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:14:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:14:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:14:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:14:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:14:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:14:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:14:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:14:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:14:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:14:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:14:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:14:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:14:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:14:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:14:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:14:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:14:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:14:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:14:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:14:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:14:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:14:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:14:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:14:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:14:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:14:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:14:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:14:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:14:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:14:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:14:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:14:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:14:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:14:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:14:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:14:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:14:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:14:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:14:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:14:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:14:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:14:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:14:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:14:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:14:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:14:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:14:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:14:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:14:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:14:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:14:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:14:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:14:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:14:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:14:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:14:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:14:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:14:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:14:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:14:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:14:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:14:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:14:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:14:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:14:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:14:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:14:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:14:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:14:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:14:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:14:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:14:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:14:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:14:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:14:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:14:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:14:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:14:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:14:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:14:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:14:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:14:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:14:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:14:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:14:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:14:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:14:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:14:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:14:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:14:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:14:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:14:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:14:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:14:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:14:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:14:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:14:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:14:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:14:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:14:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:14:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:14:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:14:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:14:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:15:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:15:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:15:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:15:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:15:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:15:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:15:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:15:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:15:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:15:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:15:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:15:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:15:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:15:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:15:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:15:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:15:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:15:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:15:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:15:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:15:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:15:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:15:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:15:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:15:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:15:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:15:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:15:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:15:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:15:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:15:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:15:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:15:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:15:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:15:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:15:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:15:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:15:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:15:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:15:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:15:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:15:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:15:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:15:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:15:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:15:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:15:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:15:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:15:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:15:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:15:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:15:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:15:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:15:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:15:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:15:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:15:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:15:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:15:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:15:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:15:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:15:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:15:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:15:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:15:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:15:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:15:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:15:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:15:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:15:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:15:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:15:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:15:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:15:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:15:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:15:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:15:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:15:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:15:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:15:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:15:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:15:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:15:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:15:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:15:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:15:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:15:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:15:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:15:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:15:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:15:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:15:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:15:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:15:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:15:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:15:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:15:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:15:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:15:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:15:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:15:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:15:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:15:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:15:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:15:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:15:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:15:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:15:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:15:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:15:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:15:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:15:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:15:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:15:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:15:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:15:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:15:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:15:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:15:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:15:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:15:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:15:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:15:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:15:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:15:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:15:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:15:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:15:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:15:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:15:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:15:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:15:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:15:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:15:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:15:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:15:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:15:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:15:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:15:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:15:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:15:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:15:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:15:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:15:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:15:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:15:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:15:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:15:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:15:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:15:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:15:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:15:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:15:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:15:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:15:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:15:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:15:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:15:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:15:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:15:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:15:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:15:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:15:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:15:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:15:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:15:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:15:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:15:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:15:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:15:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:15:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:15:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:15:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:15:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:15:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:15:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:15:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:15:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:15:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:15:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:15:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:15:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:15:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:15:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:15:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:15:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:15:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:15:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:15:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:15:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:15:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:15:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:15:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:15:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:15:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:15:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:15:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:15:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:15:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:15:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:15:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:15:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:15:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:15:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:15:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:15:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:15:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:15:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:15:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:15:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:15:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:15:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:15:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:15:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:15:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:15:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:16:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:16:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:16:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:16:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:16:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:16:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:16:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:16:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:16:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:16:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:16:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:16:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:16:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:16:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:16:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:16:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:16:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:16:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:16:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:16:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:16:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:16:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:16:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:16:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:16:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:16:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:16:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:16:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:16:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:16:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:16:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:16:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:16:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:16:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:16:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:16:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:16:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:16:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:16:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:16:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:16:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:16:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:16:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:16:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:16:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:16:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:16:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:16:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:16:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:16:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:16:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:16:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:16:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:16:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:16:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:16:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:16:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:16:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:16:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:16:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:16:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:16:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:16:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:16:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:16:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:16:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:16:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:16:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:16:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:16:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:16:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:16:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:16:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:16:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:16:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:16:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:16:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:16:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:16:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:16:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:16:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:16:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:16:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:16:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:16:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:16:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:16:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:16:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:16:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:16:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:16:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:16:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:16:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:16:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:16:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:16:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:16:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:16:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:16:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:16:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:16:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:16:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:16:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:16:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:16:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:16:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:16:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:16:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:16:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:16:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:16:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:16:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:16:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:16:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:16:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:16:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:16:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:16:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:16:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:16:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:16:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:16:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:16:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:16:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:16:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:16:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:16:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:16:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:16:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:16:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:16:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:16:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:16:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:16:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:16:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:16:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:16:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:16:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:16:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:16:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:16:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:16:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:16:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:16:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:16:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:16:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:16:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:16:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:16:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:16:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:16:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:16:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:16:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:16:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:16:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:16:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:16:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:16:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:16:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:16:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:16:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:16:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:16:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:16:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:16:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:16:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:16:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:16:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:16:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:16:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:16:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:16:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:16:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:16:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:16:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:16:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:16:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:16:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:16:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:16:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:16:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:16:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:16:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:16:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:16:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:16:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:16:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:16:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:16:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:16:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:16:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:16:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:16:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:16:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:16:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:16:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:16:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:16:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:16:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:16:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:16:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:16:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:16:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:16:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:16:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:16:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:16:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:16:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:16:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:16:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:16:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:16:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:16:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:16:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:16:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:16:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:17:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:17:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:17:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:17:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:17:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:17:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:17:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:17:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:17:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:17:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:17:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:17:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:17:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:17:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:17:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:17:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:17:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:17:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:17:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:17:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:17:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:17:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:17:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:17:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:17:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:17:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:17:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:17:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:17:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:17:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:17:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:17:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:17:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:17:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:17:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:17:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:17:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:17:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:17:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:17:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:17:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:17:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:17:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:17:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:17:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:17:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:17:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:17:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:17:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:17:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:17:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:17:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:17:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:17:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:17:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:17:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:17:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:17:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:17:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:17:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:17:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:17:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:17:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:17:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:17:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:17:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:17:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:17:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:17:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:17:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:17:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:17:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:17:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:17:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:17:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:17:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:17:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:17:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:17:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:17:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:17:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:17:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:17:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:17:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:17:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:17:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:17:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:17:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:17:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:17:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:17:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:17:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:17:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:17:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:17:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:17:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:17:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:17:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:17:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:17:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:17:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:17:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:17:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:17:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:17:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:17:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:17:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:17:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:17:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:17:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:17:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:17:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:17:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:17:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:17:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:17:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:17:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:17:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:17:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:17:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:17:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:17:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:17:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:17:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:17:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:17:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:17:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:17:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:17:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:17:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:17:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:17:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:17:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:17:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:17:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:17:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:17:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:17:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:17:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:17:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:17:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:17:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:17:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:17:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:17:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:17:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:17:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:17:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:17:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:17:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:17:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:17:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:17:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:17:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:17:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:17:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:17:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:17:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:17:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:17:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:17:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:17:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:17:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:17:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:17:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:17:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:17:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:17:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:17:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:17:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:17:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:17:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:17:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:17:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:17:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:17:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:17:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:17:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:17:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:17:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:17:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:17:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:17:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:17:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:17:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:17:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:17:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:17:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:17:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:17:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:17:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:17:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:17:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:17:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:17:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:17:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:17:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:17:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:17:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:17:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:17:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:17:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:17:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:17:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:17:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:17:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:17:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:17:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:17:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:17:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:17:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:17:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:17:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:17:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:17:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:17:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:18:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:18:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:18:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:18:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:18:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:18:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:18:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:18:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:18:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:18:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:18:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:18:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:18:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:18:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:18:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:18:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:18:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:18:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:18:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:18:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:18:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:18:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:18:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:18:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:18:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:18:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:18:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:18:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:18:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:18:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:18:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:18:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:18:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:18:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:18:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:18:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:18:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:18:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:18:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:18:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:18:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:18:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:18:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:18:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:18:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:18:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:18:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:18:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:18:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:18:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:18:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:18:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:18:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:18:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:18:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:18:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:18:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:18:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:18:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:18:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:18:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:18:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:18:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:18:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:18:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:18:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:18:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:18:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:18:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:18:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:18:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:18:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:18:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:18:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:18:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:18:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:18:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:18:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:18:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:18:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:18:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:18:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:18:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:18:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:18:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:18:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:18:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:18:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:18:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:18:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:18:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:18:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:18:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:18:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:18:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:18:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:18:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:18:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:18:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:18:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:18:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:18:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:18:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:18:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:18:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:18:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:18:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:18:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:18:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:18:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:18:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:18:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:18:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:18:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:18:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:18:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:18:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:18:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:18:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:18:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:18:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:18:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:18:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:18:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:18:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:18:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:18:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:18:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:18:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:18:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:18:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:18:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:18:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:18:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:18:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:18:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:18:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:18:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:18:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:18:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:18:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:18:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:18:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:18:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:20:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:20:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:20:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:20:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:20:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:20:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:20:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:20:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:20:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:20:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:20:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:20:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:20:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:20:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:20:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:20:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:20:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:20:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:20:47 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 19 02:20:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:20:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:20:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:20:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:20:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:20:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:20:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:20:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:20:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:20:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:20:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:20:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:20:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:20:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:20:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:20:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:20:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:20:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:20:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:20:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:20:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:20:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:20:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:20:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:20:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:20:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:20:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:20:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:20:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:20:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:20:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:20:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:20:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:20:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:20:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:20:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:21:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:21:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:21:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:21:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:21:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:21:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:21:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:21:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:21:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:21:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:21:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:21:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:21:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:21:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:21:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:21:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:21:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:21:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:21:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:21:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:21:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:21:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:21:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:21:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:21:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:21:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:21:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:21:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:21:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:21:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:21:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:21:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:21:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:21:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:21:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:21:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:21:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:21:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:21:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:21:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:21:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:21:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:21:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:21:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:21:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:21:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:21:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:21:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:21:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:21:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:21:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:21:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:21:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:21:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:21:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:21:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:21:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:21:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:21:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:21:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:21:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:21:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:21:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:21:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:21:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:21:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:21:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:21:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:21:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:21:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:21:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:21:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:21:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:21:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:21:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:21:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:21:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:21:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:21:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:21:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:21:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:21:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:21:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:21:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:21:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:21:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:21:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:21:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:21:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:21:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:21:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:21:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:21:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:21:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:21:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:21:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:21:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:21:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:21:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:21:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:21:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:21:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:21:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:21:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:21:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:21:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:21:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:21:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:21:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:21:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:21:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:21:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:21:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:21:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:21:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:21:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:21:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:21:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:21:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:21:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:21:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:21:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:21:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:21:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:21:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:21:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:21:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:21:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:21:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:21:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:21:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:21:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:21:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:21:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:21:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:21:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:21:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:21:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:21:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:21:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:21:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:21:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:21:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:21:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:21:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:21:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:21:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:21:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:21:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:21:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:21:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:21:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:21:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:21:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:21:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:21:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:21:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:21:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:21:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:21:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:21:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:21:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:21:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:21:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:21:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:21:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:21:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:21:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:21:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:21:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:21:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:21:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:21:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:21:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:21:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:21:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:21:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:21:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:21:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:21:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:21:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:21:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:21:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:21:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:21:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:21:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:21:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:21:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:21:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:21:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:21:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:21:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:21:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:21:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:21:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:21:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:21:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:21:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:21:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:21:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:21:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:21:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:21:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:21:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:21:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:21:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:21:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:22:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:22:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:22:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:22:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:22:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:22:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:22:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:22:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:22:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:22:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:22:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:22:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:22:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:22:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:22:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:22:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:22:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:22:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:22:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:22:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:22:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:22:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:22:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:22:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:22:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:22:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:22:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:22:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:22:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:22:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:22:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:22:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:22:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:22:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:22:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:22:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:22:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:22:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:22:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:22:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:22:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:22:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:22:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:22:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:22:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:22:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:22:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:22:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:22:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:22:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:22:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:22:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:22:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:22:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:22:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:22:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:22:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:22:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:22:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:22:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:22:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:22:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:22:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:22:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:22:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:22:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:22:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:22:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:22:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:22:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:22:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:22:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:22:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:22:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:22:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:22:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:22:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:22:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:22:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:22:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:22:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:22:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:22:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:22:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:22:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:22:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:22:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:22:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:22:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:22:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:22:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:22:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:22:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:22:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:22:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:22:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:22:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:22:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:22:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:22:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:22:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:22:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:22:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:22:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:22:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:22:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:22:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:22:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:22:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:22:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:22:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:22:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:22:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:22:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:22:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:22:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:22:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:22:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:22:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:22:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:22:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:22:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:22:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:22:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:22:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:22:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:22:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:22:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:22:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:22:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:22:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:22:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:22:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:22:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:22:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:22:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:22:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:22:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:22:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:22:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:22:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:22:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:22:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:22:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:22:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:22:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:22:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:22:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:22:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:22:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:22:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:22:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:22:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:22:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:22:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:22:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:22:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:22:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:22:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:22:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:22:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:22:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:22:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:22:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:22:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:22:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:22:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:22:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:22:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:22:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:22:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:22:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:22:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:22:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:22:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:22:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:22:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:22:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:22:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:22:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:22:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:22:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:22:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:22:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:22:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:22:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:22:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:22:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:22:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:22:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:22:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:22:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:22:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:22:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:22:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:22:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:22:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:22:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:23:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:23:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:23:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:23:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:23:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:23:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:23:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:23:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:23:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:23:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:23:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:23:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:23:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:23:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:23:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:23:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:23:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:23:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:23:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:23:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:23:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:23:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:23:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:23:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:23:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:23:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:23:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:23:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:23:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:23:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:23:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:23:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:23:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:23:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:23:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:23:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:23:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:23:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:23:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:23:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:23:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:23:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:23:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:23:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:23:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:23:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:23:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:23:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:23:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:23:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:23:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:23:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:23:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:23:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:23:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:23:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:23:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:23:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:23:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:23:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:23:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:23:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:23:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:23:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:23:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:23:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:23:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:23:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:23:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:23:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:23:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:23:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:23:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:23:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:23:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:23:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:23:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:23:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:23:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:23:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:23:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:23:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:23:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:23:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:23:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:23:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:23:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:23:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:23:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:23:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:23:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:23:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:23:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:23:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:23:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:23:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:23:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:23:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:23:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:23:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:23:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:23:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:23:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:23:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:23:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:23:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:23:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:23:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:23:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:23:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:23:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:23:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:23:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:23:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:23:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:23:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:23:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:23:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:23:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:23:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:23:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:23:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:23:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:23:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:23:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:23:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:23:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:23:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:23:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:23:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:23:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:23:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:23:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:23:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:23:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:23:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:23:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:23:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:23:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:23:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:23:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:23:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:23:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:23:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:23:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:23:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:23:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:23:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:23:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:23:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:23:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:23:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:23:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:23:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:23:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:23:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:23:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:23:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:23:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:23:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:23:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:23:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:23:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:23:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:23:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:23:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:23:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:23:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:23:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:23:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:23:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:23:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:23:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:23:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:23:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:23:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:23:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:23:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:23:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:23:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:23:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:23:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:23:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:23:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:23:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:23:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:23:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:23:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:23:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:23:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:23:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:23:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:23:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:23:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:23:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:23:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:23:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:23:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:23:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:23:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:23:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:23:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:23:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:23:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:23:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:23:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:23:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:23:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:23:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:23:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:23:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:23:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:23:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:23:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:23:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:23:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:24:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:24:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:24:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:24:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:24:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:24:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:24:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:24:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:24:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:24:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:24:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:24:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:24:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:24:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:24:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:24:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:24:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:24:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:24:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:24:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:24:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:24:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:24:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:24:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:24:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:24:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:24:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:24:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:24:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:24:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:24:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:24:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:24:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:24:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:24:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:24:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:24:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:24:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:24:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:24:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:24:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:24:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:24:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:24:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:24:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:24:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:24:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:24:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:24:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:24:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:24:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:24:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:24:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:24:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:24:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:24:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:24:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:24:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:24:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:24:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:24:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:24:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:24:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:24:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:24:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:24:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:24:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:24:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:24:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:24:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:24:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:24:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:24:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:24:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:24:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:24:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:24:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:24:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:24:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:24:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:24:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:24:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:24:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:24:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:24:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:24:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:24:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:24:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:24:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:24:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:24:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:24:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:24:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:24:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:24:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:24:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:24:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:24:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:24:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:24:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:24:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:24:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:24:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:24:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:24:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:24:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:24:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:24:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:24:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:24:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:24:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:24:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:24:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:24:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:24:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:24:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:24:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:24:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:24:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:24:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:24:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:24:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:24:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:24:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:24:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:24:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:24:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:24:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:24:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:24:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:24:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:24:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:24:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:24:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:24:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:24:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:24:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:24:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:24:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:24:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:24:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:24:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:24:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:24:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:24:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:24:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:24:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:24:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:24:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:24:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:24:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:24:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:24:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:24:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:24:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:24:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:24:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:24:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:24:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:24:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:24:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:24:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:24:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:24:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:24:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:24:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:24:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:24:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:24:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:24:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:24:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:24:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:24:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:24:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:24:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:24:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:24:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:24:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:24:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:24:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:24:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:24:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:24:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:24:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:24:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:24:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:24:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:24:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:24:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:24:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:24:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:24:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:24:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:24:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:24:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:24:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:24:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:24:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:24:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:24:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:24:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:24:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:24:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:24:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:24:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:24:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:24:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:24:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:24:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:24:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:24:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:24:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:24:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:24:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:24:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:24:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:25:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:25:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:25:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:25:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:25:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:25:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:25:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:25:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:25:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:25:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:25:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:25:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:25:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:25:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:25:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:25:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:25:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:25:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:25:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:25:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:25:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:25:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:25:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:25:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:25:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:25:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:25:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:25:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:25:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:25:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:25:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:25:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:25:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:25:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:25:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:25:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:25:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:25:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:25:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:25:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:25:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:25:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:25:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:25:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:25:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:25:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:25:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:25:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:25:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:25:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:25:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:25:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:25:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:25:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:25:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:25:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:25:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:25:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:25:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:25:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:25:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:25:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:25:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:25:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:25:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:25:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:25:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:25:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:25:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:25:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:25:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:25:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:25:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:25:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:25:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:25:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:25:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:25:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:25:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:25:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:25:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:25:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:25:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:25:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:25:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:25:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:25:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:25:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:25:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:25:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:25:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:25:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:25:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:25:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:25:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:25:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:25:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:25:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:25:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:25:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:25:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:25:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:25:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:25:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:25:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:25:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:25:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:25:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:25:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:25:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:25:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:25:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:25:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:25:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:25:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:25:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:25:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:25:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:25:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:25:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:25:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:25:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:25:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:25:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:25:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:25:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:25:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:25:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:25:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:25:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:25:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:25:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:25:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:25:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:25:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:25:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:25:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:25:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:25:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:25:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:25:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:25:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:25:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:25:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:25:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:25:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:25:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:25:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:25:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:25:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:25:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:25:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:25:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:25:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:25:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:25:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:25:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:25:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:25:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:25:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:25:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:25:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:25:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:25:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:25:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:25:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:25:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:25:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:25:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:25:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:25:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:25:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:25:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:25:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:25:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:25:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:25:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:25:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:25:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:25:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:25:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:25:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:25:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:25:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:25:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:25:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:25:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:25:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:25:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:25:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:25:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:25:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:25:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:25:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:25:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:25:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:25:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:25:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:25:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:25:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:25:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:25:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:25:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:25:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:25:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:25:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:25:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:25:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:25:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:25:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:25:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:25:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:25:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:25:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:25:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:25:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:26:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:26:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:26:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:26:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:26:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:26:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:26:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:26:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:26:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:26:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:26:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:26:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:26:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:26:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:26:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:26:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:26:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:26:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:26:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:26:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:26:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:26:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:26:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:26:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:26:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:26:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:26:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:26:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:26:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:26:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:26:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:26:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:26:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:26:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:26:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:26:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:26:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:26:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:26:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:26:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:26:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:26:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:26:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:26:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:26:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:26:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:26:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:26:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:26:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:26:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:26:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:26:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:26:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:26:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:26:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:26:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:26:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:26:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:26:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:26:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:26:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:26:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:26:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:26:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:26:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:26:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:26:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:26:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:26:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:26:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:26:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:26:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:26:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:26:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:26:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:26:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:26:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:26:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:26:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:26:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:26:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:26:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:26:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:26:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:26:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:26:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:26:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:26:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:26:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:26:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:26:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:26:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:26:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:26:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:26:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:26:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:26:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:26:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:26:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:26:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:26:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:26:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:26:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:26:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:26:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:26:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:26:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:26:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:26:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:26:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:26:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:26:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:26:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:26:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:26:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:26:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:26:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:26:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:26:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:26:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:26:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:26:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:26:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:26:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:26:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:26:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:26:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:26:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:26:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:26:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:26:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:26:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:26:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:26:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:26:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:26:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:26:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:26:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:26:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:26:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:26:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:26:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:26:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:26:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:26:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:26:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:26:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:26:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:26:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:26:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:26:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:26:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:26:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:26:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:26:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:26:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:26:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:26:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:26:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:26:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:26:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:26:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:26:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:26:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:26:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:26:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:26:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:26:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:26:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:26:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:26:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:26:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:26:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:26:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:26:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:26:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:26:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:26:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:26:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:26:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:26:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:26:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:26:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:26:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:26:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:26:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:26:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:26:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:26:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:26:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:26:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:26:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:26:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:26:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:26:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:26:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:26:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:26:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:26:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:26:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:26:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:26:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:26:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:26:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:26:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:26:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:26:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:26:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:26:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:26:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:26:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:26:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:26:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:26:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:26:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:26:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:27:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:27:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:27:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:27:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:27:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:27:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:27:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:27:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:27:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:27:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:27:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:27:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:27:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:27:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:27:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:27:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:27:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:27:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:27:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:27:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:27:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:27:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:27:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:27:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:27:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:27:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:27:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:27:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:27:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:27:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:27:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:27:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:27:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:27:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:27:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:27:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:27:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:27:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:27:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:27:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:27:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:27:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:27:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:27:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:27:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:27:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:27:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:27:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:27:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:27:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:27:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:27:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:27:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:27:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:27:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:27:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:27:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:27:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:27:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:27:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:27:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:27:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:27:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:27:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:27:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:27:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:27:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:27:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:27:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:27:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:27:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:27:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:27:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:27:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:27:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:27:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:27:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:27:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:27:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:27:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:27:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:27:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:27:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:27:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:27:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:27:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:27:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:27:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:27:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:27:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:27:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:27:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:27:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:27:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:27:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:27:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:27:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:27:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:27:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:27:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:27:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:27:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:27:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:27:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:27:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:27:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:27:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:27:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:27:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:27:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:27:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:27:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:27:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:27:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:27:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:27:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:27:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:27:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:27:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:27:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:27:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:27:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:27:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:27:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:27:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:27:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:27:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:27:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:27:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:27:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:27:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:27:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:27:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:27:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:27:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:27:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:27:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:27:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:27:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:27:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:27:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:27:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:27:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:27:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:27:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:27:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:27:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:27:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:27:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:27:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:27:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:27:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:27:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:27:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:27:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:27:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:27:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:27:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:27:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:27:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:27:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:27:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:27:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:27:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:27:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:27:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:27:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:27:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:27:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:27:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:27:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:27:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:27:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:27:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:27:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:27:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:27:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:27:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:27:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:27:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:27:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:27:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:27:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:27:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:27:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:27:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:27:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:27:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:27:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:27:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:27:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:27:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:27:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:27:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:27:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:27:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:27:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:27:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:27:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:27:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:27:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:27:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:27:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:27:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:27:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:27:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:27:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:27:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:27:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:27:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:27:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:27:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:27:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:27:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:27:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:27:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:28:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:28:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:28:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:28:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:28:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:28:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:28:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:28:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:28:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:28:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:28:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:28:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:28:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:28:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:28:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:28:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:28:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:28:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:28:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:28:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:28:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:28:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:28:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:28:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:28:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:28:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:28:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:28:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:28:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:28:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:28:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:28:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:28:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:28:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:28:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:28:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:28:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:28:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:28:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:28:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:28:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:28:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:28:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:28:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:28:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:28:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:28:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:28:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:28:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:28:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:28:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:28:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:28:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:28:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:28:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:28:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:28:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:28:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:28:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:28:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:28:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:28:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:28:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:28:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:28:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:28:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:28:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:28:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:28:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:28:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:28:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:28:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:28:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:28:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:28:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:28:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:28:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:28:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:28:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:28:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:28:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:28:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:28:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:28:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:28:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:28:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:28:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:28:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:28:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:28:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:28:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:28:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:28:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:28:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:28:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:28:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:28:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:28:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:28:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:28:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:28:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:28:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:28:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:28:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:28:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:28:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:28:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:28:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:28:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:28:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:28:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:28:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:28:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:28:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:28:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:28:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:28:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:28:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:28:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:28:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:28:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:28:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:28:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:28:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:28:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:28:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:28:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:28:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:28:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:28:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:28:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:28:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:28:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:28:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:28:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:28:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:28:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:28:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:28:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:28:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:28:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:28:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:28:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:28:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:28:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:28:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:28:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:28:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:28:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:28:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:28:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:28:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:28:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:28:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:28:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:28:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:28:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:28:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:28:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:28:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:28:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:28:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:30:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:30:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:30:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:30:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:30:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:30:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:30:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:30:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:30:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:30:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:30:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:30:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:30:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:30:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:30:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:30:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:30:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:30:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:30:48 rdma06 rsyslogd[1041]: imjournal: 5082 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 19 02:30:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:30:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:30:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:30:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:30:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:30:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:30:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:30:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:30:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:30:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:30:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:30:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:30:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:30:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:30:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:30:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:30:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:30:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:30:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:30:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:30:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:30:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:30:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:30:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:30:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:30:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:30:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:30:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:30:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:30:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:30:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:30:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:30:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:30:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:30:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:30:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:31:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:31:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:31:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:31:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:31:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:31:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:31:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:31:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:31:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:31:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:31:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:31:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:31:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:31:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:31:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:31:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:31:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:31:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:31:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:31:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:31:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:31:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:31:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:31:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:31:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:31:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:31:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:31:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:31:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:31:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:31:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:31:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:31:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:31:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:31:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:31:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:31:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:31:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:31:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:31:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:31:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:31:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:31:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:31:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:31:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:31:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:31:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:31:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:31:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:31:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:31:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:31:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:31:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:31:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:31:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:31:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:31:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:31:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:31:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:31:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:31:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:31:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:31:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:31:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:31:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:31:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:31:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:31:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:31:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:31:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:31:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:31:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:31:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:31:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:31:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:31:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:31:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:31:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:31:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:31:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:31:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:31:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:31:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:31:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:31:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:31:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:31:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:31:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:31:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:31:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:31:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:31:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:31:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:31:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:31:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:31:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:31:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:31:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:31:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:31:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:31:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:31:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:31:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:31:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:31:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:31:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:31:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:31:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:31:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:31:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:31:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:31:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:31:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:31:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:31:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:31:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:31:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:31:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:31:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:31:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:31:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:31:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:31:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:31:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:31:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:31:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:31:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:31:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:31:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:31:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:31:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:31:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:31:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:31:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:31:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:31:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:31:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:31:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:31:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:31:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:31:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:31:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:31:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:31:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:31:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:31:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:31:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:31:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:31:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:31:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:31:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:31:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:31:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:31:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:31:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:31:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:31:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:31:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:31:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:31:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:31:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:31:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:31:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:31:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:31:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:31:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:31:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:31:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:31:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:31:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:31:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:31:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:31:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:31:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:31:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:31:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:31:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:31:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:31:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:31:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:31:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:31:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:31:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:31:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:31:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:31:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:31:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:31:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:31:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:31:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:31:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:31:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:31:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:31:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:31:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:31:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:31:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:31:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:31:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:31:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:31:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:31:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:31:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:31:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:31:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:31:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:31:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:31:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:31:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:31:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:31:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:31:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:31:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:31:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:31:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:31:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:32:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:32:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:32:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:32:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:32:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:32:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:32:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:32:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:32:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:32:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:32:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:32:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:32:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:32:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:32:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:32:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:32:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:32:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:32:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:32:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:32:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:32:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:32:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:32:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:32:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:32:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:32:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:32:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:32:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:32:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:32:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:32:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:32:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:32:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:32:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:32:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:32:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:32:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:32:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:32:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:32:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:32:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:32:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:32:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:32:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:32:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:32:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:32:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:32:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:32:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:32:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:32:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:32:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:32:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:32:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:32:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:32:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:32:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:32:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:32:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:32:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:32:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:32:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:32:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:32:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:32:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:32:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:32:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:32:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:32:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:32:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:32:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:32:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:32:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:32:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:32:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:32:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:32:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:32:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:32:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:32:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:32:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:32:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:32:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:32:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:32:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:32:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:32:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:32:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:32:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:32:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:32:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:32:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:32:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:32:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:32:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:32:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:32:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:32:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:32:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:32:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:32:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:32:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:32:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:32:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:32:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:32:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:32:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:32:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:32:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:32:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:32:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:32:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:32:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:32:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:32:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:32:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:32:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:32:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:32:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:32:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:32:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:32:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:32:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:32:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:32:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:32:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:32:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:32:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:32:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:32:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:32:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:32:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:32:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:32:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:32:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:32:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:32:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:32:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:32:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:32:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:32:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:32:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:32:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:32:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:32:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:32:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:32:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:32:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:32:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:32:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:32:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:32:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:32:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:32:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:32:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:32:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:32:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:32:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:32:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:32:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:32:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:32:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:32:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:32:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:32:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:32:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:32:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:32:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:32:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:32:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:32:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:32:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:32:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:32:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:32:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:32:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:32:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:32:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:32:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:32:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:32:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:32:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:32:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:32:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:32:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:32:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:32:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:32:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:32:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:32:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:32:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:32:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:32:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:32:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:32:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:32:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:32:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:32:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:32:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:32:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:32:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:32:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:32:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:32:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:32:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:32:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:32:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:32:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:32:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:32:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:32:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:32:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:32:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:32:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:32:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:33:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:33:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:33:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:33:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:33:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:33:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:33:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:33:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:33:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:33:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:33:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:33:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:33:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:33:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:33:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:33:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:33:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:33:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:33:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:33:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:33:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:33:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:33:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:33:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:33:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:33:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:33:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:33:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:33:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:33:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:33:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:33:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:33:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:33:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:33:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:33:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:33:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:33:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:33:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:33:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:33:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:33:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:33:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:33:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:33:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:33:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:33:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:33:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:33:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:33:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:33:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:33:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:33:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:33:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:33:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:33:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:33:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:33:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:33:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:33:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:33:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:33:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:33:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:33:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:33:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:33:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:33:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:33:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:33:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:33:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:33:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:33:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:33:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:33:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:33:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:33:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:33:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:33:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:33:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:33:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:33:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:33:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:33:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:33:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:33:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:33:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:33:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:33:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:33:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:33:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:33:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:33:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:33:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:33:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:33:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:33:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:33:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:33:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:33:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:33:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:33:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:33:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:33:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:33:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:33:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:33:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:33:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:33:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:33:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:33:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:33:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:33:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:33:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:33:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:33:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:33:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:33:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:33:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:33:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:33:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:33:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:33:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:33:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:33:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:33:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:33:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:33:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:33:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:33:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:33:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:33:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:33:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:33:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:33:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:33:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:33:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:33:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:33:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:33:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:33:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:33:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:33:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:33:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:33:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:33:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:33:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:33:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:33:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:33:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:33:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:33:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:33:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:33:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:33:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:33:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:33:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:33:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:33:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:33:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:33:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:33:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:33:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:33:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:33:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:33:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:33:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:33:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:33:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:33:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:33:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:33:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:33:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:33:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:33:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:33:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:33:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:33:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:33:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:33:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:33:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:33:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:33:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:33:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:33:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:33:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:33:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:33:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:33:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:33:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:33:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:33:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:33:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:33:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:33:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:33:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:33:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:33:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:33:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:33:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:33:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:33:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:33:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:33:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:33:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:33:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:33:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:33:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:33:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:33:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:33:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:33:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:33:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:33:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:33:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:33:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:33:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:34:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:34:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:34:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:34:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:34:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:34:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:34:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:34:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:34:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:34:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:34:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:34:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:34:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:34:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:34:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:34:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:34:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:34:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:34:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:34:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:34:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:34:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:34:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:34:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:34:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:34:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:34:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:34:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:34:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:34:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:34:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:34:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:34:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:34:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:34:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:34:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:34:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:34:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:34:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:34:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:34:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:34:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:34:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:34:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:34:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:34:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:34:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:34:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:34:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:34:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:34:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:34:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:34:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:34:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:34:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:34:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:34:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:34:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:34:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:34:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:34:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:34:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:34:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:34:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:34:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:34:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:34:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:34:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:34:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:34:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:34:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:34:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:34:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:34:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:34:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:34:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:34:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:34:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:34:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:34:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:34:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:34:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:34:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:34:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:34:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:34:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:34:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:34:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:34:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:34:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:34:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:34:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:34:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:34:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:34:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:34:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:34:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:34:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:34:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:34:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:34:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:34:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:34:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:34:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:34:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:34:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:34:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:34:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:34:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:34:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:34:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:34:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:34:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:34:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:34:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:34:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:34:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:34:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:34:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:34:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:34:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:34:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:34:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:34:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:34:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:34:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:34:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:34:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:34:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:34:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:34:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:34:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:34:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:34:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:34:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:34:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:34:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:34:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:34:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:34:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:34:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:34:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:34:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:34:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:34:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:34:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:34:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:34:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:34:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:34:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:34:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:34:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:34:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:34:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:34:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:34:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:34:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:34:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:34:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:34:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:34:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:34:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:34:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:34:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:34:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:34:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:34:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:34:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:34:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:34:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:34:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:34:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:34:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:34:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:34:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:34:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:34:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:34:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:34:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:34:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:34:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:34:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:34:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:34:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:34:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:34:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:34:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:34:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:34:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:34:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:34:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:34:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:34:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:34:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:34:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:34:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:34:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:34:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:34:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:34:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:34:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:34:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:34:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:34:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:34:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:34:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:34:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:34:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:34:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:34:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:34:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:34:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:34:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:34:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:34:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:34:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:35:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:35:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:35:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:35:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:35:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:35:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:35:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:35:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:35:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:35:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:35:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:35:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:35:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:35:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:35:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:35:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:35:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:35:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:35:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:35:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:35:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:35:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:35:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:35:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:35:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:35:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:35:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:35:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:35:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:35:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:35:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:35:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:35:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:35:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:35:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:35:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:35:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:35:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:35:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:35:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:35:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:35:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:35:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:35:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:35:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:35:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:35:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:35:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:35:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:35:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:35:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:35:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:35:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:35:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:35:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:35:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:35:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:35:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:35:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:35:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:35:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:35:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:35:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:35:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:35:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:35:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:35:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:35:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:35:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:35:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:35:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:35:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:35:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:35:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:35:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:35:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:35:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:35:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:35:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:35:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:35:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:35:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:35:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:35:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:35:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:35:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:35:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:35:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:35:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:35:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:35:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:35:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:35:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:35:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:35:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:35:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:35:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:35:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:35:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:35:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:35:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:35:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:35:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:35:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:35:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:35:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:35:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:35:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:35:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:35:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:35:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:35:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:35:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:35:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:35:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:35:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:35:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:35:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:35:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:35:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:35:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:35:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:35:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:35:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:35:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:35:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:35:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:35:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:35:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:35:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:35:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:35:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:35:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:35:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:35:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:35:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:35:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:35:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:35:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:35:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:35:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:35:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:35:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:35:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:35:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:35:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:35:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:35:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:35:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:35:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:35:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:35:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:35:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:35:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:35:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:35:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:35:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:35:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:35:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:35:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:35:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:35:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:35:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:35:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:35:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:35:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:35:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:35:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:35:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:35:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:35:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:35:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:35:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:35:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:35:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:35:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:35:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:35:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:35:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:35:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:35:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:35:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:35:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:35:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:35:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:35:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:35:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:35:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:35:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:35:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:35:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:35:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:35:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:35:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:35:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:35:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:35:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:35:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:35:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:35:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:35:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:35:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:35:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:35:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:35:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:35:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:35:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:35:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:35:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:35:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:35:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:35:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:35:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:35:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:35:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:35:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:36:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:36:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:36:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:36:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:36:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:36:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:36:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:36:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:36:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:36:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:36:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:36:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:36:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:36:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:36:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:36:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:36:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:36:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:36:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:36:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:36:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:36:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:36:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:36:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:36:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:36:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:36:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:36:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:36:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:36:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:36:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:36:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:36:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:36:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:36:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:36:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:36:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:36:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:36:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:36:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:36:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:36:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:36:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:36:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:36:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:36:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:36:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:36:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:36:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:36:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:36:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:36:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:36:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:36:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:36:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:36:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:36:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:36:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:36:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:36:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:36:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:36:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:36:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:36:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:36:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:36:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:36:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:36:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:36:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:36:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:36:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:36:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:36:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:36:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:36:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:36:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:36:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:36:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:36:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:36:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:36:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:36:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:36:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:36:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:36:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:36:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:36:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:36:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:36:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:36:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:36:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:36:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:36:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:36:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:36:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:36:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:36:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:36:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:36:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:36:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:36:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:36:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:36:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:36:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:36:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:36:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:36:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:36:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:36:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:36:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:36:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:36:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:36:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:36:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:36:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:36:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:36:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:36:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:36:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:36:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:36:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:36:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:36:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:36:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:36:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:36:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:36:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:36:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:36:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:36:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:36:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:36:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:36:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:36:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:36:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:36:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:36:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:36:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:36:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:36:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:36:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:36:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:36:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:36:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:36:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:36:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:36:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:36:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:36:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:36:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:36:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:36:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:36:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:36:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:36:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:36:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:36:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:36:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:36:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:36:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:36:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:36:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:36:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:36:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:36:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:36:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:36:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:36:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:36:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:36:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:36:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:36:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:36:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:36:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:36:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:36:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:36:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:36:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:36:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:36:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:36:47 rdma06 systemd[1]: Starting dnf makecache...
Apr 19 02:36:48 rdma06 dnf[1211471]: Updating Subscription Management repositories.
Apr 19 02:36:48 rdma06 dnf[1211471]: Unable to read consumer identity
Apr 19 02:36:48 rdma06 dnf[1211471]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 19 02:36:49 rdma06 dnf[1211471]: Copr repo for fmf owned by psss                 3.0 kB/s | 3.3 kB     00:01
Apr 19 02:36:49 rdma06 dnf[1211471]: beaker-AppStream                                 47 kB/s | 168  B     00:00
Apr 19 02:36:49 rdma06 dnf[1211471]: Errors during downloading metadata for repository 'beaker-AppStream':
Apr 19 02:36:49 rdma06 dnf[1211471]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 19 02:36:49 rdma06 dnf[1211471]: Error: Failed to download metadata for repo 'beaker-AppStream': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 19 02:36:49 rdma06 dnf[1211471]: beaker-AppStream-debuginfo                       54 kB/s | 168  B     00:00
Apr 19 02:36:49 rdma06 dnf[1211471]: Errors during downloading metadata for repository 'beaker-AppStream-debuginfo':
Apr 19 02:36:49 rdma06 dnf[1211471]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 19 02:36:49 rdma06 dnf[1211471]: Error: Failed to download metadata for repo 'beaker-AppStream-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/AppStream/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 19 02:36:49 rdma06 dnf[1211471]: beaker-BaseOS                                    62 kB/s | 168  B     00:00
Apr 19 02:36:49 rdma06 dnf[1211471]: Errors during downloading metadata for repository 'beaker-BaseOS':
Apr 19 02:36:49 rdma06 dnf[1211471]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 19 02:36:49 rdma06 dnf[1211471]: Error: Failed to download metadata for repo 'beaker-BaseOS': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 19 02:36:49 rdma06 dnf[1211471]: beaker-BaseOS-debuginfo                          50 kB/s | 168  B     00:00
Apr 19 02:36:49 rdma06 dnf[1211471]: Errors during downloading metadata for repository 'beaker-BaseOS-debuginfo':
Apr 19 02:36:49 rdma06 dnf[1211471]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 19 02:36:49 rdma06 dnf[1211471]: Error: Failed to download metadata for repo 'beaker-BaseOS-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/BaseOS/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 19 02:36:49 rdma06 dnf[1211471]: beaker-CRB                                       57 kB/s | 168  B     00:00
Apr 19 02:36:49 rdma06 dnf[1211471]: Errors during downloading metadata for repository 'beaker-CRB':
Apr 19 02:36:49 rdma06 dnf[1211471]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 19 02:36:49 rdma06 dnf[1211471]: Error: Failed to download metadata for repo 'beaker-CRB': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 19 02:36:49 rdma06 dnf[1211471]: beaker-CRB-debuginfo                             56 kB/s | 168  B     00:00
Apr 19 02:36:49 rdma06 dnf[1211471]: Errors during downloading metadata for repository 'beaker-CRB-debuginfo':
Apr 19 02:36:49 rdma06 dnf[1211471]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 19 02:36:49 rdma06 dnf[1211471]: Error: Failed to download metadata for repo 'beaker-CRB-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/CRB/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 19 02:36:49 rdma06 dnf[1211471]: beaker-HighAvailability                          65 kB/s | 168  B     00:00
Apr 19 02:36:49 rdma06 dnf[1211471]: Errors during downloading metadata for repository 'beaker-HighAvailability':
Apr 19 02:36:49 rdma06 dnf[1211471]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 19 02:36:49 rdma06 dnf[1211471]: Error: Failed to download metadata for repo 'beaker-HighAvailability': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 19 02:36:49 rdma06 dnf[1211471]: beaker-HighAvailability-debuginfo                58 kB/s | 168  B     00:00
Apr 19 02:36:49 rdma06 dnf[1211471]: Errors during downloading metadata for repository 'beaker-HighAvailability-debuginfo':
Apr 19 02:36:49 rdma06 dnf[1211471]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 19 02:36:49 rdma06 dnf[1211471]: Error: Failed to download metadata for repo 'beaker-HighAvailability-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/HighAvailability/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 19 02:36:49 rdma06 dnf[1211471]: beaker-NFV                                       67 kB/s | 168  B     00:00
Apr 19 02:36:49 rdma06 dnf[1211471]: Errors during downloading metadata for repository 'beaker-NFV':
Apr 19 02:36:49 rdma06 dnf[1211471]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 19 02:36:49 rdma06 dnf[1211471]: Error: Failed to download metadata for repo 'beaker-NFV': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 19 02:36:49 rdma06 dnf[1211471]: beaker-NFV-debuginfo                             61 kB/s | 168  B     00:00
Apr 19 02:36:49 rdma06 dnf[1211471]: Errors during downloading metadata for repository 'beaker-NFV-debuginfo':
Apr 19 02:36:49 rdma06 dnf[1211471]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 19 02:36:49 rdma06 dnf[1211471]: Error: Failed to download metadata for repo 'beaker-NFV-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/NFV/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 19 02:36:49 rdma06 dnf[1211471]: beaker-RT                                        65 kB/s | 168  B     00:00
Apr 19 02:36:49 rdma06 dnf[1211471]: Errors during downloading metadata for repository 'beaker-RT':
Apr 19 02:36:49 rdma06 dnf[1211471]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 19 02:36:49 rdma06 dnf[1211471]: Error: Failed to download metadata for repo 'beaker-RT': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 19 02:36:49 rdma06 dnf[1211471]: beaker-RT-debuginfo                              55 kB/s | 168  B     00:00
Apr 19 02:36:49 rdma06 dnf[1211471]: Errors during downloading metadata for repository 'beaker-RT-debuginfo':
Apr 19 02:36:49 rdma06 dnf[1211471]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 19 02:36:49 rdma06 dnf[1211471]: Error: Failed to download metadata for repo 'beaker-RT-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/RT/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 19 02:36:49 rdma06 dnf[1211471]: beaker-ResilientStorage                          66 kB/s | 168  B     00:00
Apr 19 02:36:49 rdma06 dnf[1211471]: Errors during downloading metadata for repository 'beaker-ResilientStorage':
Apr 19 02:36:49 rdma06 dnf[1211471]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 19 02:36:49 rdma06 dnf[1211471]: Error: Failed to download metadata for repo 'beaker-ResilientStorage': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 19 02:36:49 rdma06 dnf[1211471]: beaker-ResilientStorage-debuginfo                55 kB/s | 168  B     00:00
Apr 19 02:36:49 rdma06 dnf[1211471]: Errors during downloading metadata for repository 'beaker-ResilientStorage-debuginfo':
Apr 19 02:36:49 rdma06 dnf[1211471]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 19 02:36:49 rdma06 dnf[1211471]: Error: Failed to download metadata for repo 'beaker-ResilientStorage-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/ResilientStorage/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 19 02:36:49 rdma06 dnf[1211471]: beaker-SAP                                       59 kB/s | 168  B     00:00
Apr 19 02:36:49 rdma06 dnf[1211471]: Errors during downloading metadata for repository 'beaker-SAP':
Apr 19 02:36:49 rdma06 dnf[1211471]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 19 02:36:49 rdma06 dnf[1211471]: Error: Failed to download metadata for repo 'beaker-SAP': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 19 02:36:49 rdma06 dnf[1211471]: beaker-SAP-debuginfo                             58 kB/s | 168  B     00:00
Apr 19 02:36:49 rdma06 dnf[1211471]: Errors during downloading metadata for repository 'beaker-SAP-debuginfo':
Apr 19 02:36:49 rdma06 dnf[1211471]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 19 02:36:49 rdma06 dnf[1211471]: Error: Failed to download metadata for repo 'beaker-SAP-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAP/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 19 02:36:49 rdma06 dnf[1211471]: beaker-SAPHANA                                   61 kB/s | 168  B     00:00
Apr 19 02:36:49 rdma06 dnf[1211471]: Errors during downloading metadata for repository 'beaker-SAPHANA':
Apr 19 02:36:49 rdma06 dnf[1211471]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 19 02:36:49 rdma06 dnf[1211471]: Error: Failed to download metadata for repo 'beaker-SAPHANA': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/os/repodata/repomd.xml (IP: 10.73.2.112)
Apr 19 02:36:49 rdma06 dnf[1211471]: beaker-SAPHANA-debuginfo                         53 kB/s | 168  B     00:00
Apr 19 02:36:49 rdma06 dnf[1211471]: Errors during downloading metadata for repository 'beaker-SAPHANA-debuginfo':
Apr 19 02:36:49 rdma06 dnf[1211471]:  - Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 19 02:36:49 rdma06 dnf[1211471]: Error: Failed to download metadata for repo 'beaker-SAPHANA-debuginfo': Cannot download repomd.xml: Status code: 404 for http://download.eng.pek2.redhat.com/rhel-9/composes/RHEL-9/RHEL-9.2.0-updates-20230328.53/compose/SAPHANA/x86_64/debug/tree/repodata/repomd.xml (IP: 10.73.2.112)
Apr 19 02:36:50 rdma06 dnf[1211471]: beaker-harness                                  2.0 kB/s | 1.3 kB     00:00
Apr 19 02:36:51 rdma06 dnf[1211471]: beaker-tasks                                    2.0 kB/s | 1.3 kB     00:00
Apr 19 02:36:51 rdma06 dnf[1211471]: Ignoring repositories: beaker-AppStream, beaker-AppStream-debuginfo, beaker-BaseOS, beaker-BaseOS-debuginfo, beaker-CRB, beaker-CRB-debuginfo, beaker-HighAvailability, beaker-HighAvailability-debuginfo, beaker-NFV, beaker-NFV-debuginfo, beaker-RT, beaker-RT-debuginfo, beaker-ResilientStorage, beaker-ResilientStorage-debuginfo, beaker-SAP, beaker-SAP-debuginfo, beaker-SAPHANA, beaker-SAPHANA-debuginfo
Apr 19 02:36:51 rdma06 dnf[1211471]: Metadata cache created.
Apr 19 02:36:51 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 19 02:36:51 rdma06 systemd[1]: Finished dnf makecache.
Apr 19 02:36:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:36:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:36:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:36:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:36:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:36:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:36:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:36:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:36:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:36:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:36:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:36:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:36:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:36:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:36:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:36:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:36:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:36:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:36:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:36:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:36:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:36:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:36:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:36:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:36:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:36:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:36:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:36:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:36:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:36:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:36:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:36:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:36:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:36:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:36:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:36:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:37:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:37:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:37:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:37:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:37:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:37:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:37:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:37:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:37:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:37:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:37:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:37:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:37:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:37:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:37:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:37:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:37:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:37:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:37:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:37:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:37:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:37:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:37:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:37:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:37:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:37:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:37:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:37:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:37:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:37:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:37:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:37:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:37:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:37:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:37:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:37:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:37:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:37:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:37:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:37:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:37:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:37:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:37:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:37:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:37:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:37:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:37:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:37:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:37:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:37:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:37:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:37:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:37:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:37:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:37:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:37:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:37:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:37:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:37:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:37:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:37:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:37:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:37:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:37:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:37:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:37:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:37:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:37:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:37:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:37:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:37:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:37:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:37:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:37:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:37:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:37:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:37:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:37:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:37:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:37:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:37:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:37:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:37:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:37:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:37:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:37:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:37:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:37:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:37:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:37:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:37:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:37:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:37:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:37:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:37:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:37:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:37:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:37:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:37:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:37:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:37:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:37:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:37:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:37:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:37:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:37:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:37:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:37:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:37:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:37:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:37:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:37:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:37:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:37:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:37:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:37:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:37:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:37:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:37:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:37:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:37:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:37:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:37:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:37:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:37:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:37:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:37:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:37:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:37:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:37:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:37:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:37:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:37:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:37:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:37:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:37:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:37:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:37:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:37:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:37:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:37:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:37:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:37:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:37:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:37:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:37:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:37:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:37:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:37:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:37:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:37:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:37:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:37:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:37:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:37:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:37:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:37:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:37:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:37:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:37:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:37:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:37:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:37:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:37:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:37:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:37:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:37:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:37:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:37:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:37:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:37:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:37:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:37:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:37:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:37:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:37:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:37:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:37:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:37:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:37:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:37:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:37:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:37:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:37:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:37:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:37:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:37:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:37:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:37:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:37:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:37:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:37:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:37:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:37:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:37:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:37:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:37:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:37:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:37:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:37:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:37:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:37:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:37:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:37:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:37:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:37:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:37:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:37:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:37:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:37:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:37:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:37:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:37:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:37:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:37:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:37:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:38:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:38:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:38:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:38:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:38:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:38:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:38:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:38:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:38:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:38:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:38:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:38:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:38:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:38:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:38:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:38:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:38:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:38:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:38:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:38:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:38:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:38:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:38:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:38:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:38:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:38:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:38:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:38:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:38:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:38:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:38:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:38:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:38:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:38:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:38:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:38:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:38:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:38:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:38:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:38:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:38:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:38:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:38:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:38:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:38:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:38:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:38:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:38:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:38:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:38:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:38:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:38:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:38:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:38:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:38:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:38:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:38:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:38:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:38:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:38:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:38:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:38:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:38:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:38:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:38:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:38:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:38:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:38:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:38:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:38:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:38:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:38:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:38:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:38:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:38:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:38:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:38:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:38:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:38:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:38:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:38:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:38:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:38:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:38:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:38:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:38:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:38:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:38:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:38:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:38:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:38:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:38:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:38:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:38:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:38:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:38:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:38:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:38:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:38:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:38:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:38:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:38:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:38:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:38:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:38:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:38:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:38:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:38:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:38:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:38:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:38:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:38:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:38:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:38:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:38:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:38:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:38:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:38:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:38:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:38:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:38:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:38:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:38:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:38:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:38:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:38:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:38:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:38:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:38:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:38:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:38:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:38:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:38:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:38:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:38:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:38:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:38:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:38:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:38:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:38:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:38:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:38:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:38:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:38:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:38:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:38:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:38:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:38:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:38:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:38:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:38:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:38:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:38:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:38:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:38:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:38:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:38:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:38:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:38:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:40:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:40:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:40:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:40:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:40:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:40:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:40:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:40:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:40:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:40:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:40:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:40:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:40:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:40:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:40:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:40:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:40:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:40:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:40:53 rdma06 rsyslogd[1041]: imjournal: 5374 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 19 02:40:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:40:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:40:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:40:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:40:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:40:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:40:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:40:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:40:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:40:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:40:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:40:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:40:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:40:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:40:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:40:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:40:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:40:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:41:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:41:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:41:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:41:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:41:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:41:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:41:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:41:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:41:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:41:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:41:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:41:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:41:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:41:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:41:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:41:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:41:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:41:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:41:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:41:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:41:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:41:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:41:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:41:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:41:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:41:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:41:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:41:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:41:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:41:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:41:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:41:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:41:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:41:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:41:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:41:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:41:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:41:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:41:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:41:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:41:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:41:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:41:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:41:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:41:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:41:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:41:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:41:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:41:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:41:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:41:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:41:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:41:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:41:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:41:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:41:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:41:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:41:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:41:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:41:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:41:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:41:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:41:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:41:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:41:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:41:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:41:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:41:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:41:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:41:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:41:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:41:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:41:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:41:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:41:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:41:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:41:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:41:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:41:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:41:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:41:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:41:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:41:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:41:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:41:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:41:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:41:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:41:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:41:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:41:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:41:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:41:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:41:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:41:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:41:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:41:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:41:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:41:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:41:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:41:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:41:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:41:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:41:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:41:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:41:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:41:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:41:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:41:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:41:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:41:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:41:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:41:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:41:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:41:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:41:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:41:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:41:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:41:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:41:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:41:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:41:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:41:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:41:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:41:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:41:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:41:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:41:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:41:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:41:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:41:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:41:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:41:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:41:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:41:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:41:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:41:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:41:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:41:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:41:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:41:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:41:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:41:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:41:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:41:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:41:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:41:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:41:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:41:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:41:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:41:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:41:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:41:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:41:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:41:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:41:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:41:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:41:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:41:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:41:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:41:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:41:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:41:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:41:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:41:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:41:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:41:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:41:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:41:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:41:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:41:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:41:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:41:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:41:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:41:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:41:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:41:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:41:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:41:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:41:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:41:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:41:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:41:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:41:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:41:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:41:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:41:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:41:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:41:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:41:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:41:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:41:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:41:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:41:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:41:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:41:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:41:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:41:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:41:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:41:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:41:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:41:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:41:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:41:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:41:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:41:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:41:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:41:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:41:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:41:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:41:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:41:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:41:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:41:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:41:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:41:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:41:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:42:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:42:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:42:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:42:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:42:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:42:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:42:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:42:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:42:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:42:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:42:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:42:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:42:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:42:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:42:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:42:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:42:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:42:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:42:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:42:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:42:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:42:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:42:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:42:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:42:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:42:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:42:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:42:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:42:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:42:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:42:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:42:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:42:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:42:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:42:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:42:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:42:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:42:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:42:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:42:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:42:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:42:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:42:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:42:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:42:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:42:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:42:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:42:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:42:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:42:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:42:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:42:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:42:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:42:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:42:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:42:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:42:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:42:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:42:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:42:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:42:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:42:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:42:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:42:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:42:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:42:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:42:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:42:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:42:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:42:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:42:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:42:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:42:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:42:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:42:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:42:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:42:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:42:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:42:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:42:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:42:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:42:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:42:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:42:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:42:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:42:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:42:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:42:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:42:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:42:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:42:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:42:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:42:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:42:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:42:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:42:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:42:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:42:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:42:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:42:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:42:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:42:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:42:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:42:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:42:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:42:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:42:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:42:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:42:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:42:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:42:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:42:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:42:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:42:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:42:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:42:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:42:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:42:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:42:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:42:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:42:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:42:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:42:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:42:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:42:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:42:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:42:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:42:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:42:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:42:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:42:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:42:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:42:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:42:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:42:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:42:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:42:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:42:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:42:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:42:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:42:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:42:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:42:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:42:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:42:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:42:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:42:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:42:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:42:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:42:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:42:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:42:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:42:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:42:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:42:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:42:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:42:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:42:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:42:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:42:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:42:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:42:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:42:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:42:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:42:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:42:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:42:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:42:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:42:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:42:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:42:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:42:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:42:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:42:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:42:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:42:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:42:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:42:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:42:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:42:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:42:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:42:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:42:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:42:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:42:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:42:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:42:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:42:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:42:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:42:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:42:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:42:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:42:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:42:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:42:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:42:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:42:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:42:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:42:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:42:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:42:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:42:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:42:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:42:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:42:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:42:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:42:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:42:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:42:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:42:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:42:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:42:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:42:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:42:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:42:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:42:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:43:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:43:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:43:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:43:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:43:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:43:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:43:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:43:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:43:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:43:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:43:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:43:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:43:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:43:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:43:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:43:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:43:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:43:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:43:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:43:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:43:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:43:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:43:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:43:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:43:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:43:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:43:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:43:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:43:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:43:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:43:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:43:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:43:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:43:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:43:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:43:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:43:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:43:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:43:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:43:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:43:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:43:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:43:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:43:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:43:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:43:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:43:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:43:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:43:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:43:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:43:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:43:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:43:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:43:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:43:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:43:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:43:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:43:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:43:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:43:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:43:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:43:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:43:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:43:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:43:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:43:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:43:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:43:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:43:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:43:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:43:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:43:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:43:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:43:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:43:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:43:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:43:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:43:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:43:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:43:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:43:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:43:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:43:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:43:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:43:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:43:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:43:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:43:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:43:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:43:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:43:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:43:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:43:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:43:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:43:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:43:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:43:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:43:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:43:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:43:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:43:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:43:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:43:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:43:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:43:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:43:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:43:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:43:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:43:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:43:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:43:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:43:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:43:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:43:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:43:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:43:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:43:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:43:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:43:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:43:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:43:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:43:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:43:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:43:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:43:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:43:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:43:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:43:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:43:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:43:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:43:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:43:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:43:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:43:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:43:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:43:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:43:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:43:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:43:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:43:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:43:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:43:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:43:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:43:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:43:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:43:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:43:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:43:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:43:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:43:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:43:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:43:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:43:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:43:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:43:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:43:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:43:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:43:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:43:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:43:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:43:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:43:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:43:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:43:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:43:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:43:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:43:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:43:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:43:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:43:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:43:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:43:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:43:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:43:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:43:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:43:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:43:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:43:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:43:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:43:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:43:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:43:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:43:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:43:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:43:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:43:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:43:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:43:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:43:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:43:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:43:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:43:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:43:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:43:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:43:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:43:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:43:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:43:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:43:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:43:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:43:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:43:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:43:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:43:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:43:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:43:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:43:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:43:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:43:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:43:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:43:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:43:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:43:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:43:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:43:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:43:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:44:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:44:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:44:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:44:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:44:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:44:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:44:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:44:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:44:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:44:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:44:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:44:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:44:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:44:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:44:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:44:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:44:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:44:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:44:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:44:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:44:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:44:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:44:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:44:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:44:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:44:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:44:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:44:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:44:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:44:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:44:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:44:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:44:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:44:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:44:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:44:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:44:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:44:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:44:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:44:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:44:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:44:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:44:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:44:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:44:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:44:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:44:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:44:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:44:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:44:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:44:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:44:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:44:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:44:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:44:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:44:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:44:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:44:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:44:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:44:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:44:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:44:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:44:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:44:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:44:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:44:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:44:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:44:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:44:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:44:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:44:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:44:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:44:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:44:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:44:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:44:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:44:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:44:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:44:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:44:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:44:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:44:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:44:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:44:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:44:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:44:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:44:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:44:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:44:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:44:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:44:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:44:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:44:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:44:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:44:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:44:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:44:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:44:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:44:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:44:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:44:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:44:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:44:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:44:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:44:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:44:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:44:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:44:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:44:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:44:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:44:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:44:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:44:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:44:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:44:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:44:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:44:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:44:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:44:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:44:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:44:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:44:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:44:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:44:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:44:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:44:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:44:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:44:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:44:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:44:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:44:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:44:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:44:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:44:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:44:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:44:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:44:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:44:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:44:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:44:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:44:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:44:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:44:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:44:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:44:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:44:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:44:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:44:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:44:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:44:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:44:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:44:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:44:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:44:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:44:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:44:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:44:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:44:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:44:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:44:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:44:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:44:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:44:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:44:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:44:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:44:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:44:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:44:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:44:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:44:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:44:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:44:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:44:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:44:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:44:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:44:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:44:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:44:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:44:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:44:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:44:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:44:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:44:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:44:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:44:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:44:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:44:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:44:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:44:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:44:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:44:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:44:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:44:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:44:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:44:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:44:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:44:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:44:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:44:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:44:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:44:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:44:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:44:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:44:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:44:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:44:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:44:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:44:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:44:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:44:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:44:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:44:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:44:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:44:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:44:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:44:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:45:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:45:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:45:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:45:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:45:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:45:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:45:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:45:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:45:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:45:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:45:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:45:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:45:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:45:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:45:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:45:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:45:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:45:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:45:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:45:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:45:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:45:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:45:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:45:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:45:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:45:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:45:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:45:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:45:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:45:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:45:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:45:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:45:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:45:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:45:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:45:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:45:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:45:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:45:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:45:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:45:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:45:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:45:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:45:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:45:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:45:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:45:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:45:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:45:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:45:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:45:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:45:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:45:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:45:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:45:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:45:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:45:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:45:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:45:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:45:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:45:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:45:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:45:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:45:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:45:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:45:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:45:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:45:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:45:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:45:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:45:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:45:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:45:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:45:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:45:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:45:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:45:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:45:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:45:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:45:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:45:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:45:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:45:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:45:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:45:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:45:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:45:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:45:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:45:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:45:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:45:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:45:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:45:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:45:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:45:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:45:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:45:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:45:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:45:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:45:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:45:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:45:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:45:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:45:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:45:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:45:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:45:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:45:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:45:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:45:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:45:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:45:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:45:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:45:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:45:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:45:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:45:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:45:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:45:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:45:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:45:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:45:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:45:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:45:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:45:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:45:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:45:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:45:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:45:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:45:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:45:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:45:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:45:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:45:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:45:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:45:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:45:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:45:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:45:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:45:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:45:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:45:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:45:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:45:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:45:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:45:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:45:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:45:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:45:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:45:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:45:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:45:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:45:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:45:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:45:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:45:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:45:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:45:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:45:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:45:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:45:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:45:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:45:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:45:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:45:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:45:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:45:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:45:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:45:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:45:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:45:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:45:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:45:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:45:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:45:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:45:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:45:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:45:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:45:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:45:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:45:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:45:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:45:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:45:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:45:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:45:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:45:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:45:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:45:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:45:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:45:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:45:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:45:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:45:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:45:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:45:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:45:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:45:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:45:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:45:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:45:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:45:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:45:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:45:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:45:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:45:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:45:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:45:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:45:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:45:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:45:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:45:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:45:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:45:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:45:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:45:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:46:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:46:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:46:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:46:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:46:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:46:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:46:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:46:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:46:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:46:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:46:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:46:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:46:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:46:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:46:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:46:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:46:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:46:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:46:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:46:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:46:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:46:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:46:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:46:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:46:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:46:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:46:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:46:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:46:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:46:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:46:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:46:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:46:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:46:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:46:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:46:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:46:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:46:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:46:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:46:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:46:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:46:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:46:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:46:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:46:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:46:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:46:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:46:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:46:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:46:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:46:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:46:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:46:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:46:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:46:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:46:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:46:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:46:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:46:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:46:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:46:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:46:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:46:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:46:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:46:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:46:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:46:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:46:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:46:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:46:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:46:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:46:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:46:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:46:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:46:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:46:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:46:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:46:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:46:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:46:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:46:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:46:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:46:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:46:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:46:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:46:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:46:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:46:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:46:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:46:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:46:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:46:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:46:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:46:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:46:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:46:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:46:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:46:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:46:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:46:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:46:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:46:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:46:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:46:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:46:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:46:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:46:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:46:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:46:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:46:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:46:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:46:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:46:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:46:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:46:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:46:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:46:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:46:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:46:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:46:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:46:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:46:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:46:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:46:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:46:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:46:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:46:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:46:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:46:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:46:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:46:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:46:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:46:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:46:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:46:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:46:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:46:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:46:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:46:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:46:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:46:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:46:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:46:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:46:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:46:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:46:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:46:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:46:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:46:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:46:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:46:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:46:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:46:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:46:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:46:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:46:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:46:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:46:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:46:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:46:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:46:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:46:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:46:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:46:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:46:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:46:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:46:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:46:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:46:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:46:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:46:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:46:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:46:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:46:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:46:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:46:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:46:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:46:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:46:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:46:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:46:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:46:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:46:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:46:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:46:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:46:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:46:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:46:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:46:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:46:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:46:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:46:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:46:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:46:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:46:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:46:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:46:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:46:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:46:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:46:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:46:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:46:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:46:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:46:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:46:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:46:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:46:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:46:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:46:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:46:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:46:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:46:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:46:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:46:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:46:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:46:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:47:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:47:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:47:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:47:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:47:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:47:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:47:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:47:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:47:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:47:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:47:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:47:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:47:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:47:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:47:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:47:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:47:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:47:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:47:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:47:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:47:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:47:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:47:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:47:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:47:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:47:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:47:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:47:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:47:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:47:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:47:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:47:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:47:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:47:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:47:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:47:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:47:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:47:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:47:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:47:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:47:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:47:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:47:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:47:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:47:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:47:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:47:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:47:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:47:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:47:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:47:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:47:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:47:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:47:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:47:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:47:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:47:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:47:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:47:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:47:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:47:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:47:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:47:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:47:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:47:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:47:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:47:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:47:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:47:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:47:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:47:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:47:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:47:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:47:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:47:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:47:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:47:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:47:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:47:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:47:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:47:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:47:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:47:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:47:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:47:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:47:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:47:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:47:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:47:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:47:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:47:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:47:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:47:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:47:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:47:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:47:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:47:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:47:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:47:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:47:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:47:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:47:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:47:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:47:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:47:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:47:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:47:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:47:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:47:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:47:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:47:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:47:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:47:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:47:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:47:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:47:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:47:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:47:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:47:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:47:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:47:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:47:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:47:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:47:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:47:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:47:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:47:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:47:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:47:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:47:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:47:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:47:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:47:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:47:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:47:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:47:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:47:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:47:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:47:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:47:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:47:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:47:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:47:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:47:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:47:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:47:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:47:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:47:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:47:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:47:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:47:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:47:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:47:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:47:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:47:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:47:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:47:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:47:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:47:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:47:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:47:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:47:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:47:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:47:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:47:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:47:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:47:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:47:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:47:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:47:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:47:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:47:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:47:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:47:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:47:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:47:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:47:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:47:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:47:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:47:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:47:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:47:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:47:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:47:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:47:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:47:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:47:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:47:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:47:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:47:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:47:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:47:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:47:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:47:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:47:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:47:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:47:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:47:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:47:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:47:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:47:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:47:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:47:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:47:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:47:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:47:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:47:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:47:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:47:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:47:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:47:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:47:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:47:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:47:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:47:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:47:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:48:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:48:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:48:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:48:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:48:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:48:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:48:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:48:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:48:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:48:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:48:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:48:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:48:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:48:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:48:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:48:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:48:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:48:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:48:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:48:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:48:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:48:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:48:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:48:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:48:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:48:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:48:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:48:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:48:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:48:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:48:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:48:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:48:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:48:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:48:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:48:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:48:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:48:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:48:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:48:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:48:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:48:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:48:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:48:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:48:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:48:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:48:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:48:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:48:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:48:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:48:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:48:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:48:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:48:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:48:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:48:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:48:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:48:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:48:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:48:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:48:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:48:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:48:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:48:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:48:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:48:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:48:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:48:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:48:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:48:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:48:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:48:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:48:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:48:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:48:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:48:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:48:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:48:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:48:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:48:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:48:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:48:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:48:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:48:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:48:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:48:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:48:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:48:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:48:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:48:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:48:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:48:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:48:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:48:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:48:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:48:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:48:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:48:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:48:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:48:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:48:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:48:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:48:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:48:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:48:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:48:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:48:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:48:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:48:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:48:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:48:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:48:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:48:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:48:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:48:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:48:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:48:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:48:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:48:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:48:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:48:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:48:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:48:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:48:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:48:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:48:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:48:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:48:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:48:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:48:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:48:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:48:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:48:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:48:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:48:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:48:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:48:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:48:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:48:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:48:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:48:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:48:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:48:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:48:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:48:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:48:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:48:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:48:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:48:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:48:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:48:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:48:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:48:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:48:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:48:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:48:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:48:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:48:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:48:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:48:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:48:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:48:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:48:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:48:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:48:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:48:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:48:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:48:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:48:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:48:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:48:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:48:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:48:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:48:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:48:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:48:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:48:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:48:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:48:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:48:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:50:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:50:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:50:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:50:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:50:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:50:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:50:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:50:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:50:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:50:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:50:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:50:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:50:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:50:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:50:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:50:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:50:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:50:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:50:58 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 19 02:51:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:51:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:51:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:51:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:51:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:51:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:51:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:51:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:51:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:51:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:51:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:51:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:51:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:51:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:51:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:51:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:51:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:51:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:51:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:51:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:51:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:51:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:51:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:51:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:51:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:51:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:51:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:51:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:51:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:51:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:51:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:51:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:51:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:51:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:51:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:51:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:51:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:51:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:51:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:51:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:51:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:51:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:51:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:51:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:51:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:51:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:51:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:51:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:51:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:51:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:51:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:51:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:51:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:51:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:51:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:51:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:51:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:51:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:51:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:51:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:51:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:51:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:51:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:51:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:51:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:51:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:51:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:51:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:51:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:51:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:51:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:51:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:51:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:51:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:51:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:51:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:51:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:51:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:51:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:51:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:51:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:51:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:51:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:51:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:51:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:51:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:51:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:51:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:51:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:51:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:51:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:51:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:51:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:51:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:51:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:51:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:51:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:51:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:51:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:51:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:51:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:51:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:51:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:51:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:51:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:51:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:51:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:51:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:51:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:51:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:51:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:51:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:51:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:51:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:51:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:51:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:51:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:51:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:51:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:51:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:51:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:51:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:51:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:51:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:51:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:51:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:51:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:51:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:51:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:51:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:51:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:51:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:51:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:51:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:51:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:51:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:51:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:51:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:51:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:51:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:51:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:51:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:51:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:51:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:51:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:51:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:51:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:51:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:51:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:51:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:51:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:51:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:51:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:52:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:52:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:52:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:52:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:52:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:52:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:52:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:52:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:52:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:52:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:52:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:52:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:52:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:52:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:52:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:52:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:52:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:52:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:52:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:52:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:52:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:52:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:52:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:52:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:52:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:52:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:52:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:52:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:52:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:52:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:52:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:52:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:52:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:52:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:52:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:52:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:52:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:52:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:52:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:52:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:52:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:52:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:52:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:52:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:52:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:52:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:52:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:52:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:52:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:52:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:52:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:52:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:52:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:52:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:52:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:52:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:52:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:52:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:52:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:52:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:52:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:52:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:52:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:52:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:52:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:52:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:52:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:52:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:52:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:52:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:52:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:52:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:52:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:52:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:52:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:52:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:52:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:52:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:52:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:52:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:52:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:52:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:52:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:52:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:52:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:52:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:52:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:52:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:52:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:52:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:52:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:52:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:52:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:52:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:52:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:52:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:52:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:52:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:52:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:52:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:52:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:52:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:52:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:52:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:52:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:52:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:52:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:52:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:52:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:52:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:52:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:52:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:52:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:52:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:52:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:52:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:52:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:52:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:52:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:52:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:52:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:52:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:52:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:52:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:52:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:52:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:52:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:52:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:52:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:52:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:52:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:52:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:52:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:52:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:52:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:52:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:52:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:52:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:52:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:52:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:52:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:52:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:52:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:52:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:52:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:52:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:52:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:52:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:52:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:52:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:52:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:52:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:52:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:52:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:52:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:52:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:52:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:52:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:52:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:52:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:52:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:52:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:52:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:52:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:52:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:52:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:52:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:52:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:52:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:52:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:52:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:52:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:52:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:52:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:52:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:52:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:52:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:52:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:52:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:52:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:53:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:53:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:53:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:53:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:53:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:53:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:53:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:53:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:53:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:53:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:53:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:53:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:53:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:53:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:53:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:53:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:53:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:53:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:53:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:53:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:53:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:53:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:53:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:53:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:53:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:53:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:53:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:53:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:53:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:53:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:53:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:53:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:53:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:53:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:53:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:53:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:53:08 rdma06 systemd-journald[790]: Data hash table of /run/log/journal/95379d2f723847b68d9e976dee8967a9/system.journal has a fill level at 75.0 (163750 of 218332 items, 75497472 file size, 461 bytes per hash table item), suggesting rotation.
Apr 19 02:53:08 rdma06 systemd-journald[790]: /run/log/journal/95379d2f723847b68d9e976dee8967a9/system.journal: Journal header limits reached or header out-of-date, rotating.
Apr 19 02:53:08 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 19 02:53:08 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 19 02:53:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:53:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:53:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:53:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:53:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:53:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:53:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:53:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:53:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:53:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:53:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:53:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:53:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:53:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:53:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:53:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:53:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:53:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:53:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:53:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:53:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:53:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:53:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:53:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:53:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:53:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:53:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:53:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:53:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:53:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:53:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:53:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:53:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:53:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:53:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:53:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:53:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:53:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:53:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:53:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:53:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:53:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:53:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:53:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:53:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:53:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:53:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:53:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:53:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:53:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:53:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:53:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:53:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:53:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:53:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:53:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:53:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:53:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:53:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:53:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:53:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:53:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:53:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:53:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:53:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:53:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:53:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:53:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:53:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:53:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:53:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:53:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:53:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:53:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:53:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:53:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:53:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:53:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:53:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:53:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:53:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:53:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:53:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:53:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:53:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:53:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:53:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:53:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:53:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:53:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:53:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:53:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:53:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:53:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:53:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:53:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:53:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:53:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:53:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:53:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:53:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:53:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:53:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:53:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:53:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:53:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:53:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:53:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:53:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:53:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:53:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:53:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:53:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:53:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:53:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:53:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:53:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:53:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:53:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:53:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:53:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:53:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:53:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:53:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:53:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:53:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:53:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:53:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:53:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:53:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:53:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:53:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:53:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:53:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:53:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:53:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:53:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:53:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:53:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:53:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:53:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:53:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:53:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:53:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:53:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:53:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:53:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:53:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:53:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:53:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:53:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:53:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:53:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:53:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:53:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:53:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:53:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:53:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:53:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:53:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:53:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:53:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:53:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:53:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:53:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:53:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:53:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:53:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:53:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:53:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:53:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:53:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:53:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:53:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:53:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:53:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:53:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:53:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:53:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:53:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:54:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:54:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:54:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:54:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:54:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:54:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:54:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:54:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:54:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:54:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:54:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:54:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:54:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:54:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:54:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:54:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:54:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:54:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:54:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:54:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:54:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:54:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:54:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:54:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:54:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:54:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:54:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:54:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:54:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:54:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:54:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:54:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:54:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:54:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:54:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:54:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:54:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:54:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:54:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:54:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:54:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:54:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:54:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:54:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:54:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:54:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:54:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:54:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:54:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:54:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:54:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:54:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:54:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:54:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:54:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:54:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:54:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:54:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:54:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:54:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:54:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:54:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:54:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:54:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:54:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:54:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:54:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:54:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:54:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:54:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:54:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:54:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:54:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:54:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:54:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:54:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:54:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:54:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:54:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:54:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:54:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:54:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:54:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:54:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:54:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:54:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:54:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:54:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:54:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:54:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:54:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:54:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:54:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:54:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:54:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:54:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:54:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:54:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:54:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:54:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:54:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:54:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:54:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:54:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:54:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:54:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:54:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:54:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:54:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:54:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:54:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:54:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:54:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:54:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:54:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:54:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:54:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:54:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:54:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:54:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:54:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:54:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:54:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:54:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:54:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:54:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:54:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:54:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:54:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:54:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:54:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:54:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:54:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:54:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:54:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:54:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:54:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:54:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:54:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:54:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:54:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:54:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:54:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:54:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:54:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:54:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:54:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:54:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:54:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:54:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:54:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:54:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:54:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:54:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:54:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:54:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:54:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:54:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:54:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:54:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:54:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:54:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:54:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:54:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:54:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:54:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:54:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:54:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:54:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:54:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:54:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:54:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:54:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:54:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:54:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:54:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:54:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:54:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:54:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:54:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:54:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:54:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:54:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:54:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:54:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:54:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:54:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:54:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:54:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:54:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:54:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:54:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:54:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:54:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:54:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:54:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:54:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:54:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:54:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:54:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:54:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:54:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:54:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:54:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:54:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:54:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:54:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:54:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:54:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:54:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:54:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:54:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:54:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:54:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:54:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:54:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:55:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:55:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:55:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:55:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:55:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:55:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:55:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:55:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:55:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:55:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:55:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:55:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:55:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:55:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:55:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:55:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:55:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:55:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:55:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:55:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:55:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:55:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:55:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:55:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:55:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:55:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:55:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:55:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:55:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:55:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:55:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:55:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:55:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:55:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:55:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:55:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:55:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:55:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:55:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:55:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:55:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:55:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:55:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:55:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:55:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:55:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:55:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:55:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:55:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:55:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:55:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:55:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:55:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:55:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:55:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:55:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:55:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:55:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:55:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:55:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:55:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:55:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:55:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:55:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:55:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:55:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:55:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:55:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:55:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:55:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:55:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:55:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:55:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:55:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:55:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:55:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:55:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:55:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:55:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:55:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:55:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:55:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:55:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:55:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:55:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:55:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:55:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:55:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:55:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:55:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:55:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:55:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:55:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:55:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:55:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:55:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:55:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:55:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:55:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:55:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:55:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:55:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:55:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:55:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:55:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:55:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:55:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:55:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:55:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:55:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:55:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:55:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:55:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:55:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:55:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:55:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:55:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:55:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:55:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:55:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:55:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:55:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:55:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:55:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:55:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:55:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:55:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:55:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:55:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:55:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:55:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:55:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:55:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:55:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:55:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:55:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:55:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:55:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:55:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:55:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:55:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:55:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:55:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:55:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:55:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:55:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:55:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:55:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:55:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:55:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:55:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:55:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:55:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:55:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:55:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:55:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:55:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:55:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:55:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:55:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:55:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:55:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:55:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:55:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:55:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:55:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:55:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:55:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:55:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:55:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:55:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:55:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:55:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:55:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:55:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:55:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:55:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:55:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:55:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:55:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:55:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:55:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:55:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:55:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:55:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:55:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:55:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:55:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:55:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:55:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:55:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:55:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:55:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:55:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:55:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:55:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:55:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:55:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:55:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:55:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:55:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:55:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:55:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:55:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:55:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:55:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:55:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:55:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:55:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:55:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:55:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:55:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:55:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:55:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:55:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:55:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:56:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:56:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:56:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:56:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:56:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:56:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:56:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:56:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:56:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:56:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:56:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:56:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:56:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:56:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:56:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:56:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:56:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:56:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:56:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:56:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:56:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:56:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:56:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:56:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:56:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:56:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:56:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:56:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:56:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:56:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:56:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:56:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:56:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:56:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:56:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:56:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:56:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:56:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:56:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:56:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:56:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:56:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:56:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:56:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:56:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:56:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:56:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:56:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:56:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:56:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:56:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:56:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:56:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:56:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:56:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:56:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:56:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:56:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:56:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:56:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:56:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:56:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:56:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:56:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:56:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:56:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:56:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:56:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:56:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:56:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:56:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:56:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:56:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:56:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:56:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:56:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:56:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:56:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:56:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:56:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:56:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:56:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:56:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:56:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:56:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:56:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:56:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:56:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:56:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:56:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:56:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:56:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:56:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:56:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:56:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:56:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:56:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:56:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:56:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:56:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:56:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:56:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:56:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:56:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:56:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:56:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:56:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:56:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:56:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:56:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:56:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:56:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:56:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:56:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:56:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:56:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:56:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:56:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:56:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:56:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:56:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:56:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:56:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:56:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:56:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:56:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:56:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:56:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:56:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:56:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:56:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:56:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:56:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:56:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:56:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:56:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:56:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:56:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:56:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:56:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:56:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:56:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:56:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:56:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:56:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:56:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:56:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:56:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:56:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:56:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:56:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:56:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:56:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:56:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:56:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:56:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:56:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:56:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:56:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:56:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:56:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:56:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:56:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:56:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:56:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:56:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:56:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:56:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:56:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:56:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:56:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:56:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:56:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:56:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:56:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:56:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:56:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:56:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:56:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:56:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:56:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:56:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:56:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:56:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:56:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:56:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:56:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:56:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:56:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:56:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:56:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:56:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:56:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:56:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:56:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:56:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:56:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:56:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:56:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:56:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:56:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:56:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:56:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:56:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:56:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:56:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:56:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:56:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:56:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:56:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:56:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:56:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:56:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:56:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:56:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:56:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:57:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:57:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:57:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:57:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:57:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:57:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:57:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:57:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:57:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:57:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:57:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:57:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:57:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:57:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:57:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:57:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:57:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:57:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:57:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:57:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:57:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:57:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:57:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:57:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:57:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:57:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:57:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:57:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:57:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:57:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:57:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:57:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:57:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:57:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:57:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:57:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:57:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:57:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:57:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:57:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:57:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:57:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:57:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:57:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:57:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:57:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:57:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:57:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:57:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:57:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:57:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:57:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:57:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:57:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:57:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:57:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:57:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:57:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:57:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:57:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:57:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:57:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:57:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:57:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:57:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:57:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:57:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:57:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:57:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:57:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:57:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:57:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:57:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:57:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:57:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:57:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:57:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:57:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:57:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:57:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:57:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:57:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:57:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:57:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:57:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:57:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:57:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:57:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:57:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:57:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:57:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:57:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:57:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:57:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:57:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:57:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:57:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:57:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:57:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:57:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:57:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:57:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:57:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:57:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:57:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:57:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:57:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:57:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:57:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:57:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:57:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:57:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:57:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:57:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:57:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:57:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:57:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:57:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:57:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:57:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:57:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:57:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:57:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:57:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:57:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:57:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:57:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:57:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:57:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:57:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:57:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:57:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:57:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:57:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:57:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:57:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:57:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:57:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:57:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:57:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:57:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:57:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:57:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:57:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:57:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:57:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:57:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:57:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:57:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:57:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:57:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:57:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:57:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:57:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:57:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:57:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:57:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:57:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:57:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:57:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:57:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:57:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:57:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:57:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:57:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:57:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:57:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:57:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:57:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:57:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:57:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:57:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:57:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:57:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:57:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:57:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:57:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:57:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:57:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:57:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:57:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:57:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:57:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:57:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:57:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:57:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:57:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:57:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:57:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:57:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:57:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:57:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:57:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:57:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:57:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:57:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:57:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:57:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:57:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:57:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:57:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:57:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:57:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:57:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:57:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:57:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:57:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:57:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:57:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:57:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:57:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:57:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:57:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:57:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:57:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:57:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:58:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:58:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:58:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:58:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:58:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:58:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:58:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:58:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:58:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:58:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:58:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:58:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:58:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:58:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:58:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:58:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:58:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:58:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:58:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:58:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:58:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:58:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:58:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:58:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:58:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:58:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:58:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:58:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:58:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:58:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:58:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:58:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:58:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:58:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:58:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:58:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:58:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:58:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:58:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:58:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:58:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:58:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:58:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:58:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:58:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:58:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:58:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:58:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:58:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:58:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:58:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:58:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:58:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:58:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:58:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:58:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:58:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:58:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:58:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:58:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:58:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:58:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:58:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:58:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:58:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:58:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:58:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:58:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:58:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:58:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:58:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:58:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:58:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:58:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:58:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:58:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:58:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:58:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:58:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:58:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:58:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:58:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:58:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:58:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:58:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:58:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:58:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:58:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:58:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:58:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:58:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:58:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:58:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:58:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:58:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:58:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:58:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:58:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:58:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:58:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:58:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:58:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:58:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:58:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:58:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:58:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:58:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:58:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:58:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:58:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:58:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:58:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:58:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:58:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:58:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:58:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:58:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:58:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:58:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:58:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:58:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:58:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:58:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:58:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:58:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:58:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:58:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:58:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:58:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:58:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:58:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:58:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:58:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:58:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:58:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:58:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:58:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:58:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:58:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:58:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:58:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:58:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:58:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:58:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:58:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:58:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:58:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:58:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:58:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:58:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:58:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:58:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:58:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:58:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:58:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:58:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:58:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:58:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:58:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:58:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:58:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:58:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:58:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:58:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:58:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:58:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:58:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:58:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:58:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:58:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:58:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:58:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:58:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:58:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:58:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:58:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:58:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:58:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:58:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:58:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 02:58:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 02:58:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:58:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:58:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:58:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 02:58:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 02:58:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 02:58:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 02:58:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 02:58:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 02:58:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 02:58:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 02:58:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 02:58:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 02:58:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 02:58:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 02:58:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 02:58:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:00:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:00:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:00:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:00:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:00:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:00:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:00:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:00:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:00:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:00:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:00:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:00:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:00:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:00:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:00:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:00:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:00:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:00:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:00:59 rdma06 rsyslogd[1041]: imjournal: 5086 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 19 03:01:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:01:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:01:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:01:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:01:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:01:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:01:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:01:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:01:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:01:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:01:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:01:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:01:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:01:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:01:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:01:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:01:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:01:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:01:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:01:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:01:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:01:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:01:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:01:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:01:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:01:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:01:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:01:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:01:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:01:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:01:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:01:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:01:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:01:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:01:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:01:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:01:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:01:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:01:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:01:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:01:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:01:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:01:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:01:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:01:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:01:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:01:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:01:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:01:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:01:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:01:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:01:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:01:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:01:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:01:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:01:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:01:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:01:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:01:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:01:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:01:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:01:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:01:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:01:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:01:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:01:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:01:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:01:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:01:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:01:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:01:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:01:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:01:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:01:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:01:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:01:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:01:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:01:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:01:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:01:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:01:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:01:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:01:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:01:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:01:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:01:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:01:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:01:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:01:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:01:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:01:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:01:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:01:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:01:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:01:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:01:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:01:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:01:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:01:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:01:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:01:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:01:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:01:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:01:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:01:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:01:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:01:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:01:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:01:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:01:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:01:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:01:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:01:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:01:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:01:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:01:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:01:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:01:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:01:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:01:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:01:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:01:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:01:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:01:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:01:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:01:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:01:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:01:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:01:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:01:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:01:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:01:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:01:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:01:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:01:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:01:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:01:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:01:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:01:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:01:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:01:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:01:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:01:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:01:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:01:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:01:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:01:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:01:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:01:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:01:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:01:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:01:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:01:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:01:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:01:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:01:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:01:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:01:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:01:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:01:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:01:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:01:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:01:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:01:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:01:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:01:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:01:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:01:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:01:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:01:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:01:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:01:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:01:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:01:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:01:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:01:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:01:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:01:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:01:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:01:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:01:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:01:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:01:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:01:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:01:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:01:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:01:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:01:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:01:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:01:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:01:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:01:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:01:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:01:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:01:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:01:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:01:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:01:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:01:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:01:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:01:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:01:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:01:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:01:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:01:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:01:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:01:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:01:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:01:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:01:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:01:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:01:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:01:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:01:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:01:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:01:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:02:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:02:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:02:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:02:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:02:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:02:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:02:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:02:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:02:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:02:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:02:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:02:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:02:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:02:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:02:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:02:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:02:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:02:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:02:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:02:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:02:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:02:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:02:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:02:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:02:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:02:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:02:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:02:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:02:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:02:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:02:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:02:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:02:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:02:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:02:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:02:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:02:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:02:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:02:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:02:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:02:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:02:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:02:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:02:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:02:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:02:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:02:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:02:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:02:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:02:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:02:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:02:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:02:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:02:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:02:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:02:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:02:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:02:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:02:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:02:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:02:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:02:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:02:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:02:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:02:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:02:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:02:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:02:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:02:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:02:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:02:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:02:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:02:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:02:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:02:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:02:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:02:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:02:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:02:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:02:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:02:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:02:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:02:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:02:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:02:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:02:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:02:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:02:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:02:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:02:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:02:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:02:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:02:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:02:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:02:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:02:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:02:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:02:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:02:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:02:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:02:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:02:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:02:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:02:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:02:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:02:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:02:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:02:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:02:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:02:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:02:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:02:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:02:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:02:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:02:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:02:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:02:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:02:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:02:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:02:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:02:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:02:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:02:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:02:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:02:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:02:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:02:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:02:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:02:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:02:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:02:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:02:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:02:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:02:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:02:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:02:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:02:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:02:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:02:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:02:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:02:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:02:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:02:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:02:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:02:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:02:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:02:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:02:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:02:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:02:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:02:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:02:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:02:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:02:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:02:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:02:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:02:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:02:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:02:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:02:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:02:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:02:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:02:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:02:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:02:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:02:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:02:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:02:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:02:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:02:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:02:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:02:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:02:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:02:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:02:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:02:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:02:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:02:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:02:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:02:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:02:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:02:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:02:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:02:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:02:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:02:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:02:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:02:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:02:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:02:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:02:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:02:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:02:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:02:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:02:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:02:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:02:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:02:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:02:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:02:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:02:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:02:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:02:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:02:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:02:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:02:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:02:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:02:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:02:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:02:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:02:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:02:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:02:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:02:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:02:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:02:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:03:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:03:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:03:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:03:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:03:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:03:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:03:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:03:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:03:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:03:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:03:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:03:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:03:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:03:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:03:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:03:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:03:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:03:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:03:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:03:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:03:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:03:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:03:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:03:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:03:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:03:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:03:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:03:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:03:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:03:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:03:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:03:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:03:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:03:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:03:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:03:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:03:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:03:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:03:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:03:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:03:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:03:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:03:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:03:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:03:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:03:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:03:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:03:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:03:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:03:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:03:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:03:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:03:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:03:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:03:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:03:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:03:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:03:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:03:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:03:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:03:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:03:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:03:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:03:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:03:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:03:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:03:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:03:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:03:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:03:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:03:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:03:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:03:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:03:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:03:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:03:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:03:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:03:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:03:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:03:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:03:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:03:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:03:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:03:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:03:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:03:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:03:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:03:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:03:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:03:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:03:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:03:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:03:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:03:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:03:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:03:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:03:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:03:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:03:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:03:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:03:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:03:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:03:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:03:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:03:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:03:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:03:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:03:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:03:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:03:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:03:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:03:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:03:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:03:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:03:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:03:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:03:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:03:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:03:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:03:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:03:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:03:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:03:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:03:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:03:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:03:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:03:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:03:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:03:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:03:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:03:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:03:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:03:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:03:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:03:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:03:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:03:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:03:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:03:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:03:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:03:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:03:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:03:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:03:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:03:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:03:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:03:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:03:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:03:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:03:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:03:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:03:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:03:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:03:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:03:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:03:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:03:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:03:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:03:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:03:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:03:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:03:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:03:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:03:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:03:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:03:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:03:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:03:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:03:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:03:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:03:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:03:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:03:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:03:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:03:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:03:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:03:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:03:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:03:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:03:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:03:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:03:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:03:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:03:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:03:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:03:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:03:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:03:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:03:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:03:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:03:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:03:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:03:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:03:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:03:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:03:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:03:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:03:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:03:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:03:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:03:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:03:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:03:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:03:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:03:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:03:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:03:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:03:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:03:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:03:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:03:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:03:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:03:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:03:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:03:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:03:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:04:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:04:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:04:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:04:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:04:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:04:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:04:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:04:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:04:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:04:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:04:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:04:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:04:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:04:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:04:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:04:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:04:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:04:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:04:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:04:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:04:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:04:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:04:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:04:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:04:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:04:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:04:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:04:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:04:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:04:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:04:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:04:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:04:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:04:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:04:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:04:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:04:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:04:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:04:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:04:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:04:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:04:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:04:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:04:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:04:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:04:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:04:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:04:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:04:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:04:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:04:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:04:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:04:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:04:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:04:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:04:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:04:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:04:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:04:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:04:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:04:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:04:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:04:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:04:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:04:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:04:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:04:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:04:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:04:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:04:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:04:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:04:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:04:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:04:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:04:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:04:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:04:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:04:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:04:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:04:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:04:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:04:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:04:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:04:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:04:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:04:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:04:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:04:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:04:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:04:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:04:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:04:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:04:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:04:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:04:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:04:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:04:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:04:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:04:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:04:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:04:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:04:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:04:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:04:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:04:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:04:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:04:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:04:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:04:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:04:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:04:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:04:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:04:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:04:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:04:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:04:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:04:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:04:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:04:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:04:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:04:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:04:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:04:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:04:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:04:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:04:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:04:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:04:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:04:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:04:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:04:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:04:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:04:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:04:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:04:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:04:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:04:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:04:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:04:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:04:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:04:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:04:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:04:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:04:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:04:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:04:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:04:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:04:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:04:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:04:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:04:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:04:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:04:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:04:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:04:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:04:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:04:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:04:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:04:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:04:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:04:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:04:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:04:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:04:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:04:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:04:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:04:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:04:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:04:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:04:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:04:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:04:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:04:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:04:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:04:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:04:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:04:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:04:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:04:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:04:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:04:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:04:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:04:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:04:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:04:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:04:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:04:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:04:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:04:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:04:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:04:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:04:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:04:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:04:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:04:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:04:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:04:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:04:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:04:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:04:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:04:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:04:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:04:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:04:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:04:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:04:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:04:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:04:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:04:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:04:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:04:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:04:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:04:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:04:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:04:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:04:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:05:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:05:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:05:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:05:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:05:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:05:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:05:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:05:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:05:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:05:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:05:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:05:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:05:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:05:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:05:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:05:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:05:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:05:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:05:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:05:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:05:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:05:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:05:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:05:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:05:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:05:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:05:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:05:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:05:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:05:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:05:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:05:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:05:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:05:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:05:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:05:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:05:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:05:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:05:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:05:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:05:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:05:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:05:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:05:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:05:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:05:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:05:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:05:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:05:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:05:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:05:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:05:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:05:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:05:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:05:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:05:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:05:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:05:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:05:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:05:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:05:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:05:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:05:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:05:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:05:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:05:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:05:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:05:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:05:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:05:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:05:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:05:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:05:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:05:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:05:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:05:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:05:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:05:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:05:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:05:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:05:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:05:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:05:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:05:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:05:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:05:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:05:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:05:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:05:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:05:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:05:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:05:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:05:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:05:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:05:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:05:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:05:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:05:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:05:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:05:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:05:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:05:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:05:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:05:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:05:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:05:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:05:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:05:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:05:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:05:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:05:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:05:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:05:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:05:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:05:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:05:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:05:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:05:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:05:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:05:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:05:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:05:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:05:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:05:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:05:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:05:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:05:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:05:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:05:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:05:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:05:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:05:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:05:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:05:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:05:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:05:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:05:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:05:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:05:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:05:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:05:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:05:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:05:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:05:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:05:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:05:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:05:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:05:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:05:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:05:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:05:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:05:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:05:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:05:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:05:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:05:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:05:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:05:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:05:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:05:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:05:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:05:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:05:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:05:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:05:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:05:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:05:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:05:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:05:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:05:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:05:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:05:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:05:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:05:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:05:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:05:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:05:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:05:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:05:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:05:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:05:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:05:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:05:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:05:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:05:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:05:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:05:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:05:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:05:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:05:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:05:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:05:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:05:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:05:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:05:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:05:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:05:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:05:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:05:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:05:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:05:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:05:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:05:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:05:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:05:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:05:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:05:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:05:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:05:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:05:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:05:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:05:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:05:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:05:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:05:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:05:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:06:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:06:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:06:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:06:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:06:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:06:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:06:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:06:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:06:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:06:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:06:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:06:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:06:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:06:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:06:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:06:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:06:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:06:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:06:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:06:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:06:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:06:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:06:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:06:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:06:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:06:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:06:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:06:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:06:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:06:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:06:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:06:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:06:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:06:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:06:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:06:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:06:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:06:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:06:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:06:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:06:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:06:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:06:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:06:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:06:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:06:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:06:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:06:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:06:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:06:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:06:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:06:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:06:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:06:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:06:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:06:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:06:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:06:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:06:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:06:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:06:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:06:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:06:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:06:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:06:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:06:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:06:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:06:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:06:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:06:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:06:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:06:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:06:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:06:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:06:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:06:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:06:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:06:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:06:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:06:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:06:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:06:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:06:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:06:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:06:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:06:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:06:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:06:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:06:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:06:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:06:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:06:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:06:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:06:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:06:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:06:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:06:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:06:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:06:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:06:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:06:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:06:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:06:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:06:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:06:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:06:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:06:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:06:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:06:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:06:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:06:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:06:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:06:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:06:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:06:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:06:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:06:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:06:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:06:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:06:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:06:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:06:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:06:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:06:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:06:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:06:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:06:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:06:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:06:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:06:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:06:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:06:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:06:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:06:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:06:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:06:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:06:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:06:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:06:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:06:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:06:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:06:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:06:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:06:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:06:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:06:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:06:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:06:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:06:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:06:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:06:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:06:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:06:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:06:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:06:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:06:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:06:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:06:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:06:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:06:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:06:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:06:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:06:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:06:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:06:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:06:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:06:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:06:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:06:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:06:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:06:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:06:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:06:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:06:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:06:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:06:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:06:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:06:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:06:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:06:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:06:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:06:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:06:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:06:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:06:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:06:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:06:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:06:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:06:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:06:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:06:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:06:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:06:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:06:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:06:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:06:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:06:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:06:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:06:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:06:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:06:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:06:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:06:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:06:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:06:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:06:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:06:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:06:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:06:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:06:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:06:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:06:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:06:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:06:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:06:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:06:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:07:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:07:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:07:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:07:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:07:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:07:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:07:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:07:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:07:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:07:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:07:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:07:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:07:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:07:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:07:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:07:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:07:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:07:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:07:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:07:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:07:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:07:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:07:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:07:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:07:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:07:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:07:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:07:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:07:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:07:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:07:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:07:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:07:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:07:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:07:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:07:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:07:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:07:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:07:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:07:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:07:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:07:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:07:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:07:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:07:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:07:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:07:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:07:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:07:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:07:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:07:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:07:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:07:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:07:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:07:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:07:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:07:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:07:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:07:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:07:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:07:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:07:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:07:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:07:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:07:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:07:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:07:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:07:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:07:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:07:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:07:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:07:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:07:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:07:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:07:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:07:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:07:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:07:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:07:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:07:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:07:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:07:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:07:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:07:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:07:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:07:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:07:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:07:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:07:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:07:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:07:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:07:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:07:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:07:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:07:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:07:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:07:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:07:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:07:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:07:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:07:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:07:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:07:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:07:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:07:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:07:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:07:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:07:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:07:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:07:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:07:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:07:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:07:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:07:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:07:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:07:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:07:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:07:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:07:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:07:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:07:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:07:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:07:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:07:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:07:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:07:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:07:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:07:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:07:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:07:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:07:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:07:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:07:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:07:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:07:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:07:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:07:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:07:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:07:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:07:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:07:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:07:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:07:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:07:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:07:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:07:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:07:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:07:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:07:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:07:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:07:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:07:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:07:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:07:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:07:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:07:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:07:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:07:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:07:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:07:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:07:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:07:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:07:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:07:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:07:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:07:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:07:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:07:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:07:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:07:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:07:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:07:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:07:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:07:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:07:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:07:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:07:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:07:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:07:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:07:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:07:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:07:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:07:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:07:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:07:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:07:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:07:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:07:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:07:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:07:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:07:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:07:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:07:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:07:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:07:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:07:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:07:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:07:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:07:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:07:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:07:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:07:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:07:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:07:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:07:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:07:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:07:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:07:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:07:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:07:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:07:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:07:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:07:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:07:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:07:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:07:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:08:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:08:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:08:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:08:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:08:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:08:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:08:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:08:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:08:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:08:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:08:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:08:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:08:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:08:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:08:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:08:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:08:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:08:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:08:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:08:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:08:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:08:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:08:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:08:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:08:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:08:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:08:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:08:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:08:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:08:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:08:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:08:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:08:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:08:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:08:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:08:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:08:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:08:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:08:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:08:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:08:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:08:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:08:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:08:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:08:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:08:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:08:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:08:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:08:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:08:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:08:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:08:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:08:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:08:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:08:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:08:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:08:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:08:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:08:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:08:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:08:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:08:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:08:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:08:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:08:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:08:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:08:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:08:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:08:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:08:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:08:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:08:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:08:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:08:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:08:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:08:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:08:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:08:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:08:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:08:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:08:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:08:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:08:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:08:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:08:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:08:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:08:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:08:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:08:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:08:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:08:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:08:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:08:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:08:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:08:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:08:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:08:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:08:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:08:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:08:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:08:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:08:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:08:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:08:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:08:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:08:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:08:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:08:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:08:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:08:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:08:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:08:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:08:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:08:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:08:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:08:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:08:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:08:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:08:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:08:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:08:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:08:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:08:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:08:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:08:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:08:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:08:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:08:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:08:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:08:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:08:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:08:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:08:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:08:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:08:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:08:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:08:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:08:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:08:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:08:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:08:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:08:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:08:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:08:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:08:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:08:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:08:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:08:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:08:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:08:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:08:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:08:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:08:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:08:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:08:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:08:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:08:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:08:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:08:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:08:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:08:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:08:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:08:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:08:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:08:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:08:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:08:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:08:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:08:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:08:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:08:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:08:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:08:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:08:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:08:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:08:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:08:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:08:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:08:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:08:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:08:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:08:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:08:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:08:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:08:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:08:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:08:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:08:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:08:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:08:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:08:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:08:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:08:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:08:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:08:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:08:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:08:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:08:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:11:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:11:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:11:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:11:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:11:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:11:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:11:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:11:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:11:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:11:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:11:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:11:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:11:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:11:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:11:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:11:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:11:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:11:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:11:04 rdma06 rsyslogd[1041]: imjournal: 5298 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 19 03:11:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:11:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:11:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:11:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:11:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:11:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:11:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:11:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:11:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:11:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:11:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:11:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:11:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:11:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:11:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:11:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:11:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:11:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:11:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:11:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:11:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:11:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:11:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:11:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:11:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:11:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:11:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:11:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:11:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:11:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:11:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:11:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:11:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:11:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:11:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:11:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:11:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:11:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:11:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:11:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:11:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:11:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:11:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:11:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:11:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:11:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:11:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:11:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:11:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:11:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:11:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:11:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:11:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:11:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:11:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:11:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:11:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:11:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:11:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:11:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:11:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:11:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:11:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:11:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:11:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:11:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:11:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:11:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:11:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:11:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:11:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:11:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:11:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:11:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:11:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:11:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:11:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:11:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:11:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:11:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:11:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:11:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:11:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:11:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:11:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:11:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:11:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:11:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:11:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:11:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:11:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:11:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:11:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:11:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:11:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:11:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:11:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:11:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:11:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:11:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:11:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:11:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:11:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:11:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:11:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:11:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:11:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:11:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:11:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:11:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:11:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:11:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:11:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:11:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:11:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:11:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:11:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:11:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:11:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:11:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:11:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:11:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:11:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:11:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:11:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:11:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:11:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:11:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:11:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:11:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:11:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:11:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:11:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:11:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:11:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:11:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:11:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:11:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:11:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:11:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:11:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:11:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:11:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:11:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:11:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:11:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:11:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:11:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:11:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:11:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:11:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:11:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:11:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:11:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:11:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:11:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:11:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:11:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:11:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:11:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:11:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:11:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:11:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:11:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:11:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:11:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:11:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:11:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:11:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:11:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:11:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:11:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:11:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:11:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:11:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:11:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:11:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:11:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:11:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:11:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:11:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:11:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:11:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:11:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:11:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:11:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:11:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:11:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:11:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:11:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:11:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:11:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:11:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:11:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:11:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:11:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:11:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:11:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:12:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:12:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:12:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:12:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:12:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:12:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:12:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:12:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:12:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:12:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:12:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:12:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:12:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:12:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:12:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:12:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:12:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:12:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:12:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:12:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:12:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:12:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:12:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:12:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:12:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:12:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:12:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:12:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:12:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:12:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:12:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:12:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:12:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:12:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:12:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:12:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:12:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:12:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:12:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:12:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:12:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:12:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:12:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:12:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:12:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:12:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:12:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:12:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:12:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:12:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:12:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:12:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:12:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:12:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:12:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:12:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:12:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:12:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:12:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:12:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:12:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:12:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:12:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:12:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:12:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:12:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:12:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:12:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:12:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:12:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:12:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:12:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:12:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:12:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:12:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:12:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:12:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:12:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:12:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:12:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:12:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:12:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:12:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:12:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:12:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:12:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:12:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:12:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:12:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:12:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:12:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:12:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:12:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:12:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:12:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:12:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:12:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:12:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:12:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:12:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:12:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:12:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:12:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:12:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:12:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:12:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:12:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:12:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:12:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:12:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:12:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:12:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:12:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:12:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:12:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:12:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:12:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:12:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:12:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:12:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:12:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:12:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:12:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:12:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:12:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:12:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:12:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:12:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:12:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:12:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:12:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:12:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:12:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:12:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:12:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:12:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:12:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:12:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:12:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:12:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:12:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:12:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:12:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:12:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:12:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:12:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:12:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:12:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:12:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:12:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:12:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:12:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:12:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:12:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:12:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:12:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:12:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:12:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:12:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:12:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:12:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:12:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:12:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:12:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:12:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:12:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:12:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:12:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:12:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:12:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:12:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:12:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:12:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:12:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:12:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:12:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:12:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:12:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:12:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:12:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:12:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:12:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:12:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:12:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:12:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:12:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:12:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:12:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:12:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:12:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:12:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:12:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:12:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:12:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:12:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:12:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:12:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:12:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:12:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:12:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:12:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:12:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:12:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:12:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:12:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:12:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:12:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:12:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:12:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:12:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:12:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:12:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:12:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:12:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:12:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:12:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:13:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:13:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:13:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:13:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:13:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:13:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:13:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:13:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:13:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:13:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:13:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:13:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:13:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:13:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:13:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:13:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:13:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:13:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:13:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:13:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:13:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:13:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:13:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:13:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:13:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:13:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:13:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:13:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:13:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:13:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:13:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:13:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:13:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:13:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:13:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:13:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:13:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:13:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:13:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:13:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:13:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:13:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:13:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:13:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:13:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:13:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:13:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:13:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:13:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:13:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:13:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:13:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:13:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:13:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:13:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:13:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:13:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:13:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:13:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:13:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:13:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:13:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:13:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:13:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:13:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:13:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:13:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:13:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:13:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:13:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:13:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:13:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:13:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:13:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:13:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:13:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:13:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:13:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:13:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:13:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:13:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:13:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:13:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:13:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:13:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:13:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:13:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:13:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:13:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:13:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:13:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:13:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:13:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:13:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:13:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:13:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:13:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:13:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:13:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:13:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:13:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:13:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:13:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:13:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:13:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:13:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:13:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:13:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:13:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:13:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:13:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:13:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:13:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:13:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:13:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:13:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:13:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:13:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:13:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:13:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:13:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:13:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:13:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:13:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:13:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:13:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:13:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:13:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:13:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:13:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:13:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:13:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:13:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:13:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:13:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:13:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:13:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:13:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:13:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:13:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:13:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:13:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:13:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:13:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:13:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:13:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:13:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:13:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:13:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:13:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:13:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:13:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:13:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:13:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:13:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:13:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:13:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:13:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:13:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:13:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:13:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:13:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:13:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:13:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:13:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:13:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:13:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:13:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:13:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:13:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:13:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:13:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:13:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:13:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:13:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:13:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:13:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:13:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:13:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:13:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:13:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:13:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:13:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:13:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:13:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:13:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:13:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:13:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:13:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:13:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:13:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:13:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:13:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:13:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:13:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:13:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:13:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:13:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:13:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:13:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:13:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:13:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:13:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:13:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:13:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:13:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:13:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:13:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:13:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:13:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:13:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:13:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:13:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:13:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:13:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:13:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:14:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:14:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:14:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:14:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:14:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:14:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:14:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:14:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:14:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:14:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:14:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:14:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:14:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:14:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:14:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:14:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:14:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:14:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:14:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:14:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:14:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:14:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:14:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:14:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:14:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:14:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:14:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:14:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:14:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:14:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:14:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:14:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:14:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:14:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:14:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:14:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:14:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:14:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:14:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:14:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:14:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:14:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:14:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:14:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:14:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:14:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:14:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:14:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:14:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:14:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:14:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:14:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:14:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:14:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:14:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:14:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:14:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:14:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:14:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:14:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:14:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:14:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:14:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:14:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:14:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:14:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:14:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:14:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:14:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:14:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:14:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:14:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:14:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:14:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:14:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:14:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:14:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:14:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:14:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:14:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:14:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:14:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:14:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:14:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:14:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:14:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:14:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:14:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:14:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:14:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:14:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:14:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:14:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:14:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:14:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:14:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:14:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:14:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:14:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:14:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:14:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:14:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:14:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:14:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:14:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:14:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:14:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:14:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:14:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:14:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:14:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:14:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:14:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:14:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:14:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:14:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:14:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:14:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:14:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:14:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:14:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:14:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:14:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:14:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:14:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:14:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:14:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:14:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:14:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:14:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:14:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:14:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:14:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:14:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:14:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:14:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:14:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:14:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:14:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:14:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:14:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:14:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:14:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:14:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:14:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:14:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:14:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:14:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:14:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:14:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:14:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:14:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:14:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:14:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:14:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:14:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:14:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:14:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:14:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:14:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:14:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:14:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:14:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:14:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:14:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:14:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:14:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:14:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:14:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:14:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:14:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:14:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:14:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:14:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:14:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:14:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:14:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:14:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:14:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:14:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:14:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:14:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:14:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:14:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:14:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:14:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:14:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:14:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:14:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:14:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:14:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:14:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:14:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:14:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:14:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:14:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:14:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:14:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:14:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:14:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:14:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:14:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:14:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:14:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:14:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:14:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:14:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:14:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:14:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:14:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:14:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:14:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:14:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:14:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:14:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:14:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:15:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:15:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:15:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:15:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:15:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:15:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:15:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:15:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:15:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:15:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:15:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:15:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:15:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:15:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:15:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:15:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:15:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:15:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:15:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:15:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:15:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:15:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:15:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:15:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:15:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:15:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:15:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:15:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:15:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:15:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:15:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:15:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:15:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:15:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:15:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:15:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:15:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:15:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:15:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:15:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:15:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:15:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:15:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:15:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:15:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:15:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:15:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:15:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:15:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:15:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:15:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:15:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:15:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:15:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:15:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:15:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:15:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:15:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:15:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:15:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:15:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:15:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:15:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:15:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:15:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:15:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:15:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:15:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:15:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:15:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:15:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:15:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:15:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:15:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:15:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:15:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:15:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:15:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:15:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:15:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:15:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:15:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:15:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:15:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:15:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:15:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:15:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:15:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:15:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:15:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:15:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:15:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:15:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:15:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:15:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:15:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:15:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:15:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:15:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:15:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:15:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:15:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:15:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:15:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:15:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:15:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:15:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:15:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:15:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:15:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:15:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:15:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:15:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:15:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:15:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:15:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:15:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:15:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:15:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:15:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:15:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:15:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:15:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:15:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:15:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:15:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:15:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:15:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:15:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:15:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:15:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:15:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:15:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:15:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:15:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:15:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:15:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:15:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:15:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:15:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:15:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:15:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:15:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:15:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:15:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:15:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:15:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:15:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:15:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:15:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:15:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:15:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:15:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:15:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:15:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:15:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:15:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:15:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:15:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:15:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:15:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:15:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:15:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:15:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:15:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:15:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:15:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:15:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:15:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:15:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:15:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:15:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:15:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:15:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:15:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:15:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:15:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:15:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:15:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:15:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:15:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:15:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:15:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:15:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:15:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:15:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:15:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:15:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:15:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:15:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:15:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:15:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:15:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:15:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:15:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:15:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:15:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:15:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:15:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:15:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:15:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:15:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:15:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:15:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:15:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:15:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:15:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:15:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:15:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:15:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:15:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:15:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:15:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:15:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:15:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:15:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:16:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:16:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:16:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:16:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:16:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:16:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:16:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:16:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:16:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:16:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:16:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:16:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:16:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:16:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:16:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:16:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:16:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:16:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:16:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:16:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:16:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:16:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:16:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:16:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:16:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:16:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:16:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:16:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:16:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:16:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:16:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:16:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:16:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:16:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:16:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:16:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:16:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:16:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:16:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:16:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:16:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:16:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:16:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:16:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:16:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:16:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:16:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:16:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:16:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:16:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:16:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:16:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:16:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:16:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:16:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:16:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:16:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:16:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:16:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:16:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:16:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:16:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:16:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:16:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:16:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:16:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:16:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:16:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:16:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:16:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:16:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:16:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:16:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:16:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:16:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:16:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:16:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:16:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:16:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:16:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:16:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:16:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:16:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:16:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:16:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:16:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:16:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:16:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:16:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:16:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:16:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:16:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:16:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:16:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:16:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:16:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:16:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:16:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:16:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:16:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:16:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:16:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:16:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:16:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:16:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:16:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:16:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:16:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:16:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:16:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:16:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:16:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:16:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:16:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:16:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:16:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:16:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:16:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:16:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:16:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:16:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:16:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:16:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:16:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:16:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:16:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:16:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:16:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:16:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:16:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:16:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:16:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:16:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:16:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:16:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:16:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:16:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:16:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:16:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:16:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:16:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:16:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:16:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:16:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:16:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:16:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:16:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:16:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:16:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:16:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:16:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:16:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:16:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:16:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:16:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:16:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:16:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:16:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:16:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:16:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:16:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:16:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:16:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:16:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:16:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:16:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:16:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:16:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:16:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:16:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:16:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:16:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:16:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:16:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:16:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:16:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:16:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:16:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:16:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:16:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:16:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:16:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:16:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:16:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:16:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:16:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:16:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:16:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:16:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:16:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:16:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:16:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:16:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:16:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:16:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:16:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:16:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:16:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:16:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:16:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:16:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:16:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:16:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:16:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:16:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:16:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:16:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:16:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:16:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:16:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:16:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:16:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:16:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:16:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:16:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:16:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:17:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:17:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:17:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:17:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:17:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:17:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:17:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:17:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:17:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:17:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:17:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:17:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:17:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:17:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:17:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:17:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:17:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:17:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:17:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:17:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:17:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:17:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:17:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:17:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:17:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:17:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:17:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:17:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:17:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:17:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:17:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:17:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:17:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:17:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:17:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:17:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:17:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:17:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:17:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:17:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:17:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:17:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:17:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:17:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:17:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:17:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:17:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:17:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:17:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:17:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:17:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:17:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:17:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:17:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:17:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:17:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:17:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:17:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:17:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:17:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:17:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:17:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:17:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:17:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:17:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:17:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:17:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:17:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:17:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:17:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:17:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:17:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:17:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:17:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:17:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:17:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:17:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:17:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:17:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:17:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:17:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:17:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:17:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:17:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:17:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:17:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:17:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:17:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:17:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:17:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:17:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:17:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:17:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:17:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:17:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:17:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:17:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:17:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:17:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:17:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:17:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:17:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:17:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:17:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:17:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:17:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:17:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:17:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:17:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:17:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:17:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:17:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:17:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:17:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:17:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:17:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:17:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:17:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:17:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:17:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:17:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:17:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:17:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:17:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:17:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:17:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:17:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:17:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:17:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:17:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:17:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:17:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:17:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:17:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:17:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:17:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:17:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:17:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:17:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:17:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:17:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:17:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:17:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:17:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:17:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:17:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:17:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:17:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:17:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:17:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:17:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:17:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:17:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:17:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:17:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:17:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:17:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:17:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:17:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:17:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:17:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:17:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:17:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:17:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:17:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:17:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:17:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:17:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:17:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:17:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:17:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:17:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:17:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:17:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:17:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:17:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:17:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:17:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:17:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:17:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:17:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:17:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:17:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:17:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:17:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:17:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:17:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:17:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:17:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:17:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:17:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:17:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:17:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:17:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:17:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:17:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:17:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:17:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:17:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:17:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:17:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:17:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:17:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:17:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:17:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:17:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:17:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:17:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:17:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:17:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:17:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:17:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:17:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:17:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:17:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:17:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:18:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:18:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:18:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:18:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:18:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:18:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:18:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:18:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:18:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:18:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:18:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:18:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:18:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:18:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:18:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:18:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:18:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:18:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:18:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:18:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:18:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:18:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:18:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:18:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:18:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:18:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:18:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:18:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:18:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:18:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:18:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:18:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:18:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:18:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:18:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:18:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:18:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:18:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:18:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:18:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:18:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:18:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:18:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:18:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:18:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:18:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:18:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:18:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:18:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:18:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:18:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:18:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:18:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:18:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:18:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:18:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:18:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:18:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:18:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:18:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:18:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:18:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:18:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:18:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:18:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:18:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:18:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:18:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:18:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:18:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:18:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:18:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:18:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:18:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:18:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:18:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:18:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:18:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:18:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:18:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:18:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:18:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:18:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:18:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:18:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:18:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:18:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:18:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:18:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:18:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:18:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:18:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:18:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:18:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:18:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:18:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:18:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:18:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:18:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:18:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:18:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:18:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:18:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:18:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:18:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:18:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:18:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:18:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:18:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:18:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:18:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:18:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:18:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:18:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:18:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:18:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:18:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:18:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:18:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:18:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:18:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:18:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:18:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:18:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:18:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:18:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:18:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:18:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:18:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:18:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:18:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:18:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:18:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:18:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:18:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:18:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:18:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:18:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:18:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:18:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:18:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:18:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:18:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:18:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:18:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:18:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:18:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:18:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:18:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:18:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:18:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:18:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:18:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:18:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:18:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:18:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:18:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:18:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:18:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:18:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:18:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:18:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:18:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:18:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:18:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:18:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:18:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:18:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:18:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:18:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:18:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:18:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:18:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:18:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:18:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:18:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:18:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:18:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:18:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:18:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:18:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:18:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:18:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:18:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:18:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:18:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:18:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:18:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:18:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:18:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:18:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:18:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:18:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:18:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:18:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:18:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:18:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:18:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:18:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:18:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:18:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:18:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:18:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:18:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:18:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:18:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:18:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:18:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:18:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:18:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:18:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:18:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:18:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:18:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:18:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:18:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:21:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:21:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:21:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:21:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:21:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:21:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:21:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:21:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:21:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:21:09 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 19 03:21:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:21:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:21:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:21:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:21:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:21:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:21:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:21:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:21:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:21:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:21:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:21:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:21:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:21:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:21:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:21:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:21:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:21:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:21:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:21:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:21:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:21:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:21:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:21:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:21:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:21:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:21:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:21:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:21:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:21:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:21:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:21:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:21:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:21:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:21:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:21:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:21:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:21:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:21:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:21:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:21:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:21:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:21:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:21:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:21:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:21:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:21:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:21:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:21:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:21:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:21:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:21:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:21:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:21:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:21:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:21:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:21:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:21:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:21:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:21:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:21:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:21:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:21:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:21:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:21:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:21:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:21:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:21:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:21:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:21:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:21:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:21:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:21:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:21:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:21:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:21:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:21:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:21:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:21:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:21:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:21:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:21:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:21:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:21:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:21:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:21:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:21:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:21:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:21:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:21:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:21:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:21:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:21:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:21:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:21:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:21:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:21:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:21:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:21:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:22:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:22:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:22:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:22:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:22:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:22:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:22:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:22:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:22:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:22:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:22:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:22:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:22:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:22:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:22:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:22:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:22:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:22:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:22:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:22:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:22:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:22:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:22:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:22:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:22:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:22:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:22:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:22:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:22:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:22:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:22:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:22:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:22:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:22:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:22:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:22:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:22:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:22:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:22:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:22:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:22:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:22:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:22:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:22:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:22:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:22:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:22:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:22:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:22:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:22:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:22:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:22:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:22:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:22:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:22:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:22:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:22:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:22:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:22:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:22:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:22:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:22:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:22:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:22:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:22:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:22:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:22:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:22:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:22:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:22:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:22:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:22:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:22:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:22:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:22:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:22:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:22:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:22:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:22:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:22:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:22:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:22:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:22:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:22:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:22:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:22:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:22:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:22:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:22:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:22:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:22:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:22:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:22:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:22:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:22:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:22:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:22:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:22:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:22:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:22:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:22:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:22:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:22:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:22:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:22:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:22:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:22:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:22:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:22:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:22:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:22:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:22:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:22:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:22:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:22:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:22:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:22:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:22:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:22:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:22:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:22:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:22:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:22:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:22:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:22:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:22:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:22:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:22:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:22:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:22:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:22:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:22:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:22:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:22:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:22:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:22:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:22:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:22:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:22:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:22:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:22:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:22:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:22:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:22:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:22:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:22:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:22:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:22:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:22:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:22:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:22:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:22:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:22:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:22:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:22:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:22:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:22:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:22:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:22:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:22:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:22:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:22:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:22:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:22:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:22:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:22:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:22:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:22:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:22:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:22:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:22:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:22:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:22:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:22:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:22:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:22:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:22:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:22:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:22:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:22:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:22:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:22:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:22:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:22:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:22:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:22:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:22:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:22:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:22:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:22:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:22:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:22:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:22:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:22:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:22:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:22:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:22:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:22:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:22:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:22:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:22:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:22:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:22:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:22:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:22:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:22:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:22:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:22:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:22:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:22:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:22:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:22:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:22:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:22:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:22:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:22:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:23:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:23:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:23:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:23:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:23:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:23:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:23:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:23:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:23:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:23:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:23:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:23:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:23:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:23:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:23:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:23:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:23:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:23:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:23:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:23:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:23:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:23:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:23:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:23:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:23:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:23:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:23:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:23:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:23:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:23:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:23:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:23:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:23:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:23:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:23:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:23:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:23:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:23:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:23:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:23:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:23:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:23:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:23:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:23:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:23:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:23:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:23:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:23:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:23:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:23:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:23:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:23:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:23:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:23:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:23:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:23:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:23:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:23:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:23:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:23:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:23:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:23:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:23:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:23:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:23:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:23:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:23:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:23:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:23:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:23:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:23:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:23:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:23:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:23:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:23:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:23:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:23:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:23:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:23:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:23:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:23:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:23:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:23:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:23:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:23:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:23:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:23:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:23:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:23:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:23:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:23:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:23:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:23:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:23:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:23:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:23:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:23:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:23:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:23:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:23:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:23:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:23:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:23:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:23:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:23:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:23:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:23:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:23:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:23:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:23:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:23:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:23:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:23:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:23:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:23:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:23:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:23:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:23:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:23:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:23:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:23:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:23:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:23:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:23:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:23:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:23:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:23:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:23:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:23:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:23:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:23:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:23:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:23:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:23:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:23:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:23:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:23:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:23:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:23:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:23:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:23:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:23:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:23:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:23:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:23:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:23:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:23:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:23:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:23:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:23:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:23:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:23:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:23:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:23:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:23:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:23:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:23:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:23:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:23:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:23:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:23:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:23:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:23:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:23:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:23:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:23:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:23:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:23:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:23:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:23:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:23:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:23:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:23:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:23:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:23:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:23:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:23:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:23:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:23:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:23:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:23:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:23:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:23:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:23:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:23:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:23:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:23:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:23:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:23:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:23:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:23:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:23:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:23:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:23:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:23:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:23:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:23:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:23:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:23:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:23:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:23:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:23:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:23:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:23:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:23:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:23:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:23:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:23:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:23:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:23:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:23:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:23:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:23:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:23:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:23:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:23:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:24:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:24:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:24:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:24:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:24:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:24:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:24:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:24:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:24:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:24:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:24:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:24:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:24:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:24:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:24:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:24:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:24:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:24:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:24:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:24:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:24:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:24:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:24:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:24:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:24:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:24:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:24:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:24:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:24:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:24:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:24:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:24:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:24:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:24:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:24:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:24:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:24:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:24:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:24:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:24:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:24:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:24:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:24:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:24:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:24:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:24:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:24:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:24:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:24:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:24:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:24:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:24:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:24:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:24:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:24:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:24:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:24:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:24:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:24:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:24:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:24:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:24:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:24:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:24:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:24:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:24:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:24:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:24:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:24:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:24:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:24:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:24:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:24:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:24:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:24:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:24:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:24:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:24:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:24:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:24:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:24:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:24:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:24:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:24:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:24:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:24:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:24:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:24:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:24:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:24:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:24:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:24:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:24:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:24:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:24:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:24:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:24:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:24:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:24:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:24:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:24:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:24:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:24:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:24:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:24:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:24:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:24:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:24:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:24:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:24:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:24:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:24:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:24:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:24:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:24:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:24:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:24:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:24:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:24:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:24:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:24:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:24:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:24:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:24:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:24:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:24:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:24:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:24:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:24:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:24:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:24:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:24:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:24:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:24:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:24:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:24:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:24:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:24:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:24:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:24:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:24:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:24:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:24:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:24:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:24:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:24:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:24:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:24:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:24:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:24:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:24:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:24:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:24:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:24:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:24:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:24:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:24:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:24:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:24:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:24:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:24:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:24:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:24:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:24:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:24:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:24:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:24:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:24:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:24:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:24:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:24:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:24:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:24:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:24:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:24:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:24:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:24:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:24:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:24:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:24:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:24:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:24:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:24:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:24:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:24:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:24:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:24:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:24:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:24:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:24:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:24:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:24:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:24:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:24:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:24:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:24:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:24:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:24:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:24:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:24:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:24:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:24:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:24:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:24:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:24:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:24:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:24:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:24:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:24:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:24:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:24:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:24:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:24:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:24:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:24:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:24:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:25:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:25:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:25:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:25:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:25:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:25:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:25:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:25:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:25:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:25:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:25:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:25:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:25:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:25:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:25:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:25:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:25:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:25:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:25:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:25:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:25:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:25:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:25:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:25:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:25:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:25:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:25:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:25:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:25:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:25:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:25:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:25:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:25:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:25:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:25:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:25:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:25:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:25:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:25:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:25:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:25:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:25:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:25:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:25:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:25:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:25:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:25:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:25:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:25:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:25:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:25:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:25:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:25:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:25:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:25:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:25:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:25:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:25:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:25:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:25:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:25:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:25:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:25:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:25:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:25:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:25:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:25:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:25:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:25:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:25:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:25:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:25:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:25:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:25:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:25:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:25:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:25:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:25:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:25:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:25:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:25:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:25:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:25:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:25:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:25:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:25:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:25:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:25:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:25:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:25:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:25:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:25:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:25:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:25:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:25:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:25:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:25:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:25:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:25:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:25:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:25:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:25:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:25:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:25:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:25:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:25:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:25:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:25:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:25:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:25:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:25:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:25:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:25:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:25:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:25:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:25:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:25:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:25:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:25:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:25:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:25:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:25:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:25:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:25:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:25:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:25:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:25:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:25:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:25:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:25:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:25:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:25:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:25:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:25:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:25:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:25:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:25:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:25:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:25:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:25:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:25:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:25:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:25:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:25:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:25:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:25:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:25:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:25:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:25:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:25:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:25:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:25:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:25:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:25:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:25:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:25:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:25:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:25:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:25:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:25:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:25:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:25:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:25:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:25:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:25:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:25:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:25:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:25:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:25:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:25:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:25:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:25:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:25:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:25:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:25:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:25:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:25:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:25:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:25:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:25:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:25:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:25:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:25:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:25:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:25:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:25:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:25:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:25:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:25:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:25:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:25:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:25:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:25:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:25:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:25:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:25:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:25:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:25:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:25:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:25:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:25:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:25:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:25:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:25:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:25:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:25:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:25:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:25:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:25:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:25:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:25:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:25:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:25:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:25:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:25:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:25:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:26:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:26:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:26:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:26:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:26:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:26:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:26:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:26:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:26:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:26:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:26:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:26:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:26:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:26:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:26:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:26:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:26:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:26:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:26:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:26:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:26:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:26:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:26:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:26:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:26:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:26:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:26:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:26:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:26:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:26:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:26:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:26:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:26:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:26:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:26:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:26:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:26:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:26:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:26:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:26:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:26:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:26:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:26:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:26:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:26:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:26:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:26:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:26:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:26:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:26:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:26:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:26:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:26:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:26:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:26:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:26:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:26:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:26:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:26:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:26:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:26:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:26:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:26:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:26:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:26:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:26:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:26:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:26:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:26:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:26:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:26:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:26:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:26:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:26:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:26:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:26:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:26:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:26:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:26:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:26:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:26:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:26:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:26:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:26:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:26:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:26:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:26:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:26:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:26:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:26:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:26:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:26:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:26:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:26:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:26:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:26:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:26:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:26:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:26:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:26:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:26:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:26:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:26:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:26:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:26:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:26:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:26:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:26:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:26:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:26:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:26:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:26:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:26:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:26:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:26:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:26:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:26:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:26:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:26:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:26:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:26:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:26:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:26:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:26:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:26:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:26:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:26:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:26:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:26:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:26:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:26:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:26:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:26:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:26:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:26:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:26:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:26:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:26:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:26:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:26:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:26:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:26:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:26:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:26:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:26:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:26:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:26:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:26:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:26:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:26:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:26:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:26:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:26:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:26:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:26:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:26:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:26:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:26:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:26:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:26:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:26:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:26:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:26:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:26:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:26:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:26:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:26:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:26:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:26:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:26:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:26:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:26:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:26:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:26:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:26:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:26:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:26:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:26:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:26:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:26:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:26:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:26:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:26:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:26:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:26:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:26:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:26:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:26:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:26:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:26:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:26:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:26:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:26:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:26:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:26:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:26:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:26:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:26:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:26:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:26:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:26:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:26:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:26:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:26:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:26:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:26:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:26:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:26:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:26:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:26:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:26:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:26:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:26:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:26:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:26:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:26:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:27:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:27:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:27:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:27:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:27:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:27:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:27:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:27:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:27:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:27:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:27:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:27:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:27:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:27:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:27:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:27:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:27:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:27:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:27:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:27:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:27:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:27:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:27:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:27:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:27:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:27:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:27:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:27:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:27:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:27:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:27:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:27:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:27:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:27:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:27:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:27:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:27:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:27:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:27:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:27:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:27:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:27:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:27:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:27:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:27:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:27:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:27:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:27:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:27:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:27:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:27:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:27:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:27:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:27:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:27:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:27:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:27:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:27:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:27:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:27:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:27:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:27:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:27:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:27:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:27:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:27:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:27:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:27:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:27:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:27:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:27:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:27:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:27:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:27:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:27:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:27:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:27:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:27:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:27:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:27:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:27:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:27:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:27:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:27:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:27:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:27:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:27:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:27:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:27:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:27:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:27:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:27:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:27:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:27:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:27:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:27:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:27:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:27:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:27:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:27:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:27:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:27:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:27:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:27:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:27:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:27:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:27:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:27:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:27:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:27:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:27:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:27:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:27:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:27:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:27:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:27:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:27:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:27:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:27:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:27:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:27:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:27:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:27:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:27:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:27:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:27:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:27:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:27:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:27:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:27:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:27:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:27:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:27:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:27:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:27:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:27:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:27:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:27:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:27:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:27:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:27:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:27:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:27:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:27:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:27:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:27:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:27:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:27:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:27:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:27:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:27:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:27:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:27:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:27:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:27:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:27:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:27:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:27:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:27:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:27:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:27:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:27:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:27:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:27:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:27:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:27:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:27:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:27:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:27:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:27:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:27:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:27:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:27:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:27:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:27:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:27:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:27:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:27:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:27:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:27:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:27:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:27:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:27:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:27:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:27:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:27:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:27:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:27:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:27:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:27:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:27:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:27:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:27:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:27:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:27:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:27:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:27:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:27:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:27:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:27:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:27:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:27:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:27:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:27:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:27:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:27:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:27:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:27:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:27:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:27:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:27:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:27:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:27:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:27:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:27:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:27:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:28:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:28:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:28:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:28:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:28:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:28:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:28:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:28:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:28:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:28:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:28:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:28:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:28:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:28:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:28:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:28:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:28:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:28:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:28:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:28:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:28:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:28:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:28:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:28:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:28:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:28:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:28:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:28:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:28:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:28:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:28:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:28:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:28:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:28:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:28:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:28:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:28:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:28:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:28:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:28:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:28:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:28:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:28:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:28:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:28:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:28:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:28:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:28:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:28:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:28:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:28:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:28:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:28:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:28:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:28:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:28:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:28:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:28:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:28:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:28:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:28:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:28:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:28:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:28:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:28:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:28:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:28:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:28:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:28:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:28:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:28:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:28:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:28:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:28:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:28:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:28:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:28:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:28:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:28:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:28:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:28:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:28:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:28:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:28:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:28:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:28:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:28:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:28:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:28:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:28:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:28:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:28:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:28:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:28:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:28:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:28:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:28:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:28:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:28:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:28:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:28:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:28:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:28:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:28:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:28:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:28:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:28:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:28:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:28:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:28:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:28:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:28:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:28:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:28:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:28:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:28:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:28:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:28:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:28:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:28:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:28:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:28:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:28:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:28:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:28:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:28:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:28:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:28:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:28:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:28:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:28:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:28:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:28:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:28:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:28:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:28:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:28:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:28:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:28:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:28:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:28:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:28:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:28:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:28:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:28:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:28:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:28:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:28:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:28:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:28:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:28:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:28:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:28:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:28:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:28:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:28:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:28:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:28:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:28:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:28:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:28:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:28:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:28:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:28:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:28:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:28:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:28:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:28:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:28:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:28:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:28:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:28:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:28:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:28:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:28:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:28:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:28:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:28:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:28:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:28:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:28:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:28:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:28:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:28:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:28:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:28:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:28:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:28:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:28:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:28:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:28:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:28:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:28:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:28:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:28:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:28:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:28:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:28:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:28:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:28:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:28:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:28:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:28:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:28:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:28:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:28:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:28:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:28:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:28:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:28:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:28:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:28:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:28:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:28:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:28:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:28:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:29:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:29:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:29:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:29:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:29:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:29:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:29:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:29:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:29:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:29:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:29:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:29:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:29:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:29:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:29:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:29:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:29:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:29:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:31:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:31:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:31:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:31:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:31:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:31:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:31:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:31:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:31:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:31:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:31:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:31:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:31:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:31:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:31:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:31:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:31:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:31:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:31:10 rdma06 rsyslogd[1041]: imjournal: 5082 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 19 03:31:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:31:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:31:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:31:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:31:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:31:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:31:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:31:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:31:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:31:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:31:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:31:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:31:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:31:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:31:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:31:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:31:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:31:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:31:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:31:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:31:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:31:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:31:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:31:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:31:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:31:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:31:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:31:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:31:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:31:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:31:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:31:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:31:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:31:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:31:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:31:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:31:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:31:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:31:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:31:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:31:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:31:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:31:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:31:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:31:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:31:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:31:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:31:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:31:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:31:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:31:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:31:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:31:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:31:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:31:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:31:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:31:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:31:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:31:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:31:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:31:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:31:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:31:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:31:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:31:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:31:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:31:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:31:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:31:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:31:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:31:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:31:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:31:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:31:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:31:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:31:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:31:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:31:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:31:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:31:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:31:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:31:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:31:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:31:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:31:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:31:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:31:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:31:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:31:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:31:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:31:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:31:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:31:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:31:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:31:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:31:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:31:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:31:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:31:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:31:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:31:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:31:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:31:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:31:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:31:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:31:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:31:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:31:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:31:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:31:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:31:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:31:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:31:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:31:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:31:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:31:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:31:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:31:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:31:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:31:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:31:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:31:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:31:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:31:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:31:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:31:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:31:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:31:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:31:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:31:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:31:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:31:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:31:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:31:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:31:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:31:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:31:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:31:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:31:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:31:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:31:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:31:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:31:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:31:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:31:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:31:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:31:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:31:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:31:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:31:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:31:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:31:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:31:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:31:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:31:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:31:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:31:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:31:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:31:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:31:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:31:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:31:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:31:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:31:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:31:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:31:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:31:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:31:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:31:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:31:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:31:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:32:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:32:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:32:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:32:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:32:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:32:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:32:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:32:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:32:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:32:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:32:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:32:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:32:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:32:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:32:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:32:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:32:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:32:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:32:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:32:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:32:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:32:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:32:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:32:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:32:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:32:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:32:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:32:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:32:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:32:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:32:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:32:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:32:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:32:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:32:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:32:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:32:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:32:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:32:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:32:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:32:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:32:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:32:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:32:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:32:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:32:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:32:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:32:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:32:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:32:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:32:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:32:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:32:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:32:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:32:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:32:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:32:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:32:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:32:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:32:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:32:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:32:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:32:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:32:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:32:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:32:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:32:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:32:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:32:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:32:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:32:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:32:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:32:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:32:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:32:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:32:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:32:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:32:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:32:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:32:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:32:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:32:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:32:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:32:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:32:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:32:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:32:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:32:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:32:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:32:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:32:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:32:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:32:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:32:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:32:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:32:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:32:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:32:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:32:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:32:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:32:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:32:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:32:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:32:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:32:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:32:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:32:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:32:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:32:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:32:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:32:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:32:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:32:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:32:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:32:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:32:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:32:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:32:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:32:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:32:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:32:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:32:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:32:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:32:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:32:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:32:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:32:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:32:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:32:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:32:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:32:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:32:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:32:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:32:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:32:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:32:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:32:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:32:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:32:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:32:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:32:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:32:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:32:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:32:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:32:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:32:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:32:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:32:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:32:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:32:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:32:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:32:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:32:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:32:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:32:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:32:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:32:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:32:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:32:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:32:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:32:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:32:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:32:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:32:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:32:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:32:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:32:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:32:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:32:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:32:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:32:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:32:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:32:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:32:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:32:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:32:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:32:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:32:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:32:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:32:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:32:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:32:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:32:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:32:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:32:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:32:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:32:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:32:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:32:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:32:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:32:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:32:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:32:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:32:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:32:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:32:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:32:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:32:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:32:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:32:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:32:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:32:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:32:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:32:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:32:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:32:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:32:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:32:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:32:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:32:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:32:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:32:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:32:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:32:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:32:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:32:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:33:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:33:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:33:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:33:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:33:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:33:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:33:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:33:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:33:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:33:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:33:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:33:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:33:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:33:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:33:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:33:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:33:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:33:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:33:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:33:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:33:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:33:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:33:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:33:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:33:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:33:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:33:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:33:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:33:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:33:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:33:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:33:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:33:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:33:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:33:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:33:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:33:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:33:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:33:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:33:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:33:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:33:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:33:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:33:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:33:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:33:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:33:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:33:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:33:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:33:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:33:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:33:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:33:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:33:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:33:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:33:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:33:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:33:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:33:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:33:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:33:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:33:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:33:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:33:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:33:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:33:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:33:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:33:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:33:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:33:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:33:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:33:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:33:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:33:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:33:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:33:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:33:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:33:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:33:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:33:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:33:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:33:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:33:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:33:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:33:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:33:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:33:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:33:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:33:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:33:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:33:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:33:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:33:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:33:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:33:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:33:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:33:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:33:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:33:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:33:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:33:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:33:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:33:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:33:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:33:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:33:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:33:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:33:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:33:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:33:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:33:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:33:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:33:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:33:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:33:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:33:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:33:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:33:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:33:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:33:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:33:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:33:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:33:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:33:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:33:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:33:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:33:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:33:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:33:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:33:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:33:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:33:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:33:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:33:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:33:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:33:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:33:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:33:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:33:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:33:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:33:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:33:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:33:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:33:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:33:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:33:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:33:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:33:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:33:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:33:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:33:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:33:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:33:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:33:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:33:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:33:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:33:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:33:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:33:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:33:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:33:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:33:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:33:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:33:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:33:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:33:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:33:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:33:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:33:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:33:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:33:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:33:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:33:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:33:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:33:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:33:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:33:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:33:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:33:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:33:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:33:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:33:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:33:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:33:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:33:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:33:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:33:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:33:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:33:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:33:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:33:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:33:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:33:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:33:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:33:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:33:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:33:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:33:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:33:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:33:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:33:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:33:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:33:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:33:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:33:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:33:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:33:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:33:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:33:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:33:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:33:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:33:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:33:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:33:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:33:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:33:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:34:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:34:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:34:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:34:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:34:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:34:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:34:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:34:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:34:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:34:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:34:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:34:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:34:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:34:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:34:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:34:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:34:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:34:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:34:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:34:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:34:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:34:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:34:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:34:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:34:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:34:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:34:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:34:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:34:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:34:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:34:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:34:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:34:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:34:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:34:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:34:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:34:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:34:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:34:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:34:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:34:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:34:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:34:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:34:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:34:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:34:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:34:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:34:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:34:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:34:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:34:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:34:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:34:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:34:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:34:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:34:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:34:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:34:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:34:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:34:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:34:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:34:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:34:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:34:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:34:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:34:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:34:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:34:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:34:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:34:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:34:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:34:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:34:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:34:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:34:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:34:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:34:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:34:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:34:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:34:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:34:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:34:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:34:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:34:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:34:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:34:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:34:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:34:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:34:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:34:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:34:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:34:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:34:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:34:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:34:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:34:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:34:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:34:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:34:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:34:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:34:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:34:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:34:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:34:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:34:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:34:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:34:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:34:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:34:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:34:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:34:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:34:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:34:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:34:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:34:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:34:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:34:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:34:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:34:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:34:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:34:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:34:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:34:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:34:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:34:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:34:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:34:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:34:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:34:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:34:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:34:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:34:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:34:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:34:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:34:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:34:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:34:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:34:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:34:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:34:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:34:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:34:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:34:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:34:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:34:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:34:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:34:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:34:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:34:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:34:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:34:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:34:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:34:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:34:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:34:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:34:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:34:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:34:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:34:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:34:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:34:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:34:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:34:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:34:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:34:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:34:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:34:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:34:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:34:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:34:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:34:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:34:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:34:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:34:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:34:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:34:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:34:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:34:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:34:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:34:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:34:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:34:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:34:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:34:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:34:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:34:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:34:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:34:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:34:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:34:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:34:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:34:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:34:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:34:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:34:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:34:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:34:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:34:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:34:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:34:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:34:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:34:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:34:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:34:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:34:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:34:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:34:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:34:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:34:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:34:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:34:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:34:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:34:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:34:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:34:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:34:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:35:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:35:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:35:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:35:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:35:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:35:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:35:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:35:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:35:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:35:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:35:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:35:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:35:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:35:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:35:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:35:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:35:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:35:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:35:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:35:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:35:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:35:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:35:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:35:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:35:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:35:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:35:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:35:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:35:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:35:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:35:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:35:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:35:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:35:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:35:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:35:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:35:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:35:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:35:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:35:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:35:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:35:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:35:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:35:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:35:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:35:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:35:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:35:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:35:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:35:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:35:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:35:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:35:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:35:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:35:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:35:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:35:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:35:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:35:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:35:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:35:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:35:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:35:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:35:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:35:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:35:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:35:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:35:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:35:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:35:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:35:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:35:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:35:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:35:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:35:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:35:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:35:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:35:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:35:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:35:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:35:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:35:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:35:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:35:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:35:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:35:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:35:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:35:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:35:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:35:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:35:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:35:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:35:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:35:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:35:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:35:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:35:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:35:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:35:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:35:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:35:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:35:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:35:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:35:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:35:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:35:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:35:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:35:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:35:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:35:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:35:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:35:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:35:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:35:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:35:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:35:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:35:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:35:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:35:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:35:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:35:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:35:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:35:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:35:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:35:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:35:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:35:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:35:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:35:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:35:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:35:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:35:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:35:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:35:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:35:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:35:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:35:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:35:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:35:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:35:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:35:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:35:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:35:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:35:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:35:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:35:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:35:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:35:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:35:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:35:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:35:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:35:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:35:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:35:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:35:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:35:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:35:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:35:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:35:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:35:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:35:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:35:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:35:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:35:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:35:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:35:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:35:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:35:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:35:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:35:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:35:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:35:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:35:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:35:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:35:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:35:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:35:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:35:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:35:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:35:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:35:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:35:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:35:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:35:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:35:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:35:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:35:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:35:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:35:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:35:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:35:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:35:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:35:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:35:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:35:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:35:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:35:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:35:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:35:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:35:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:35:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:35:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:35:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:35:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:35:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:35:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:35:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:35:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:35:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:35:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:35:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:35:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:35:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:35:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:35:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:35:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:36:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:36:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:36:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:36:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:36:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:36:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:36:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:36:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:36:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:36:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:36:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:36:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:36:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:36:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:36:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:36:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:36:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:36:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:36:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:36:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:36:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:36:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:36:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:36:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:36:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:36:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:36:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:36:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:36:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:36:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:36:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:36:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:36:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:36:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:36:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:36:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:36:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:36:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:36:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:36:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:36:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:36:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:36:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:36:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:36:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:36:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:36:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:36:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:36:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:36:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:36:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:36:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:36:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:36:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:36:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:36:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:36:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:36:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:36:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:36:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:36:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:36:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:36:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:36:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:36:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:36:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:36:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:36:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:36:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:36:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:36:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:36:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:36:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:36:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:36:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:36:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:36:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:36:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:36:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:36:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:36:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:36:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:36:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:36:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:36:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:36:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:36:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:36:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:36:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:36:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:36:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:36:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:36:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:36:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:36:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:36:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:36:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:36:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:36:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:36:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:36:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:36:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:36:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:36:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:36:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:36:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:36:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:36:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:36:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:36:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:36:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:36:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:36:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:36:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:36:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:36:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:36:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:36:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:36:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:36:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:36:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:36:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:36:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:36:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:36:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:36:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:36:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:36:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:36:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:36:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:36:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:36:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:36:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:36:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:36:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:36:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:36:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:36:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:36:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:36:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:36:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:36:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:36:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:36:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:36:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:36:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:36:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:36:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:36:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:36:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:36:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:36:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:36:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:36:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:36:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:36:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:36:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:36:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:36:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:36:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:36:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:36:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:36:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:36:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:36:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:36:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:36:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:36:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:36:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:36:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:36:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:36:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:36:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:36:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:36:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:36:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:36:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:36:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:36:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:36:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:36:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:36:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:36:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:36:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:36:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:36:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:36:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:36:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:36:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:36:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:36:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:36:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:36:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:36:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:36:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:36:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:36:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:36:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:36:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:36:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:36:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:36:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:36:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:36:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:36:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:36:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:36:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:36:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:36:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:36:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:36:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:36:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:36:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:36:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:36:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:36:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:37:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:37:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:37:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:37:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:37:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:37:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:37:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:37:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:37:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:37:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:37:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:37:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:37:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:37:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:37:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:37:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:37:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:37:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:37:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:37:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:37:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:37:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:37:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:37:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:37:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:37:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:37:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:37:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:37:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:37:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:37:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:37:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:37:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:37:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:37:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:37:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:37:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:37:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:37:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:37:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:37:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:37:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:37:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:37:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:37:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:37:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:37:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:37:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:37:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:37:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:37:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:37:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:37:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:37:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:37:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:37:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:37:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:37:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:37:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:37:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:37:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:37:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:37:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:37:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:37:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:37:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:37:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:37:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:37:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:37:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:37:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:37:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:37:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:37:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:37:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:37:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:37:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:37:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:37:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:37:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:37:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:37:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:37:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:37:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:37:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:37:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:37:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:37:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:37:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:37:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:37:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:37:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:37:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:37:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:37:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:37:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:37:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:37:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:37:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:37:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:37:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:37:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:37:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:37:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:37:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:37:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:37:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:37:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:37:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:37:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:37:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:37:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:37:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:37:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:37:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:37:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:37:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:37:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:37:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:37:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:37:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:37:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:37:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:37:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:37:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:37:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:37:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:37:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:37:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:37:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:37:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:37:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:37:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:37:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:37:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:37:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:37:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:37:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:37:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:37:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:37:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:37:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:37:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:37:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:37:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:37:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:37:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:37:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:37:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:37:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:37:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:37:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:37:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:37:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:37:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:37:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:37:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:37:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:37:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:37:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:37:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:37:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:37:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:37:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:37:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:37:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:37:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:37:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:37:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:37:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:37:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:37:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:37:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:37:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:37:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:37:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:37:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:37:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:37:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:37:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:37:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:37:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:37:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:37:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:37:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:37:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:37:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:37:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:37:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:37:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:37:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:37:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:37:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:37:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:37:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:37:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:37:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:37:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:38:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:38:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:38:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:38:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:38:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:38:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:38:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:38:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:38:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:38:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:38:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:38:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:38:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:38:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:38:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:38:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:38:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:38:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:38:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:38:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:38:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:38:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:38:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:38:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:38:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:38:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:38:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:38:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:38:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:38:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:38:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:38:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:38:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:38:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:38:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:38:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:38:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:38:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:38:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:38:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:38:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:38:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:38:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:38:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:38:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:38:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:38:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:38:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:38:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:38:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:38:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:38:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:38:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:38:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:38:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:38:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:38:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:38:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:38:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:38:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:38:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:38:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:38:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:38:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:38:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:38:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:38:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:38:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:38:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:38:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:38:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:38:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:38:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:38:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:38:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:38:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:38:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:38:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:38:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:38:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:38:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:38:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:38:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:38:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:38:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:38:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:38:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:38:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:38:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:38:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:38:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:38:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:38:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:38:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:38:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:38:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:38:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:38:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:38:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:38:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:38:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:38:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:38:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:38:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:38:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:38:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:38:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:38:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:38:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:38:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:38:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:38:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:38:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:38:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:38:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:38:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:38:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:38:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:38:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:38:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:38:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:38:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:38:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:38:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:38:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:38:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:38:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:38:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:38:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:38:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:38:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:38:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:38:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:38:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:38:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:38:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:38:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:38:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:38:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:38:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:38:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:38:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:38:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:38:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:38:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:38:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:38:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:38:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:38:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:38:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:38:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:38:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:38:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:38:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:38:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:38:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:38:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:38:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:38:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:38:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:38:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:38:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:38:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:38:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:38:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:38:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:38:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:38:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:38:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:38:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:38:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:38:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:38:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:38:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:38:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:38:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:38:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:38:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:38:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:38:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:38:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:38:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:38:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:38:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:38:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:38:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:38:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:38:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:38:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:38:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:38:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:38:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:38:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:38:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:38:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:38:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:38:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:38:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:38:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:38:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:38:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:38:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:38:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:38:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:38:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:38:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:38:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:38:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:38:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:38:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:38:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:38:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:38:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:38:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:38:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:38:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:39:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:39:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:39:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:39:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:39:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:39:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:39:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:39:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:39:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:39:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:39:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:39:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:39:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:39:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:39:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:39:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:39:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:39:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:39:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:39:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:39:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:39:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:39:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:39:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:39:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:39:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:39:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:39:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:39:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:39:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:39:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:39:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:39:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:39:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:39:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:39:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:41:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:41:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:41:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:41:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:41:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:41:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:41:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:41:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:41:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:41:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:41:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:41:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:41:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:41:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:41:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:41:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:41:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:41:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:41:15 rdma06 rsyslogd[1041]: imjournal: 5294 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 19 03:41:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:41:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:41:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:41:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:41:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:41:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:41:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:41:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:41:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:41:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:41:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:41:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:41:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:41:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:41:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:41:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:41:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:41:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:41:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:41:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:41:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:41:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:41:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:41:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:41:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:41:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:41:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:41:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:41:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:41:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:41:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:41:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:41:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:41:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:41:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:41:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:41:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:41:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:41:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:41:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:41:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:41:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:41:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:41:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:41:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:41:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:41:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:41:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:41:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:41:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:41:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:41:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:41:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:41:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:41:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:41:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:41:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:41:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:41:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:41:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:41:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:41:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:41:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:41:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:41:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:41:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:41:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:41:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:41:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:41:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:41:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:41:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:41:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:41:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:41:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:41:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:41:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:41:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:41:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:41:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:41:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:41:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:41:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:41:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:41:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:41:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:41:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:41:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:41:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:41:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:41:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:41:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:41:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:41:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:41:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:41:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:41:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:41:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:41:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:41:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:41:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:41:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:41:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:41:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:41:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:41:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:41:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:41:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:41:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:41:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:41:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:41:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:41:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:41:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:41:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:41:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:41:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:41:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:41:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:41:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:41:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:41:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:41:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:41:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:41:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:41:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:41:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:41:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:41:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:41:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:41:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:41:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:41:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:41:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:41:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:41:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:41:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:41:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:41:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:41:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:41:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:41:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:41:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:41:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:42:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:42:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:42:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:42:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:42:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:42:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:42:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:42:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:42:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:42:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:42:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:42:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:42:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:42:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:42:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:42:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:42:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:42:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:42:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:42:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:42:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:42:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:42:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:42:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:42:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:42:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:42:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:42:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:42:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:42:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:42:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:42:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:42:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:42:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:42:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:42:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:42:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:42:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:42:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:42:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:42:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:42:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:42:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:42:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:42:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:42:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:42:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:42:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:42:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:42:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:42:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:42:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:42:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:42:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:42:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:42:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:42:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:42:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:42:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:42:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:42:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:42:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:42:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:42:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:42:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:42:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:42:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:42:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:42:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:42:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:42:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:42:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:42:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:42:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:42:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:42:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:42:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:42:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:42:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:42:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:42:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:42:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:42:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:42:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:42:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:42:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:42:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:42:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:42:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:42:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:42:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:42:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:42:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:42:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:42:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:42:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:42:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:42:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:42:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:42:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:42:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:42:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:42:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:42:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:42:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:42:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:42:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:42:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:42:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:42:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:42:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:42:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:42:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:42:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:42:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:42:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:42:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:42:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:42:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:42:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:42:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:42:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:42:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:42:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:42:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:42:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:42:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:42:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:42:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:42:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:42:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:42:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:42:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:42:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:42:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:42:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:42:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:42:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:42:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:42:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:42:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:42:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:42:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:42:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:42:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:42:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:42:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:42:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:42:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:42:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:42:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:42:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:42:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:42:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:42:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:42:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:42:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:42:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:42:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:42:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:42:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:42:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:42:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:42:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:42:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:42:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:42:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:42:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:42:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:42:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:42:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:42:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:42:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:42:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:42:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:42:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:42:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:42:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:42:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:42:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:42:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:42:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:42:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:42:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:42:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:42:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:42:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:42:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:42:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:42:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:42:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:42:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:42:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:42:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:42:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:42:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:42:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:42:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:42:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:42:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:42:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:42:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:42:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:42:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:42:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:42:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:42:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:42:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:42:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:42:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:42:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:42:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:42:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:42:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:42:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:42:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:43:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:43:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:43:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:43:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:43:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:43:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:43:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:43:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:43:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:43:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:43:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:43:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:43:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:43:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:43:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:43:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:43:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:43:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:43:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:43:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:43:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:43:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:43:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:43:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:43:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:43:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:43:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:43:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:43:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:43:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:43:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:43:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:43:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:43:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:43:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:43:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:43:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:43:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:43:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:43:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:43:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:43:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:43:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:43:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:43:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:43:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:43:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:43:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:43:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:43:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:43:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:43:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:43:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:43:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:43:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:43:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:43:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:43:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:43:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:43:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:43:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:43:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:43:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:43:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:43:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:43:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:43:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:43:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:43:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:43:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:43:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:43:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:43:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:43:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:43:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:43:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:43:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:43:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:43:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:43:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:43:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:43:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:43:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:43:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:43:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:43:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:43:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:43:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:43:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:43:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:43:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:43:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:43:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:43:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:43:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:43:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:43:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:43:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:43:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:43:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:43:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:43:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:43:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:43:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:43:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:43:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:43:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:43:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:43:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:43:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:43:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:43:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:43:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:43:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:43:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:43:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:43:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:43:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:43:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:43:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:43:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:43:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:43:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:43:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:43:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:43:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:43:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:43:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:43:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:43:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:43:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:43:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:43:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:43:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:43:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:43:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:43:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:43:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:43:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:43:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:43:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:43:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:43:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:43:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:43:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:43:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:43:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:43:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:43:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:43:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:43:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:43:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:43:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:43:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:43:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:43:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:43:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:43:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:43:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:43:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:43:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:43:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:43:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:43:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:43:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:43:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:43:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:43:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:43:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:43:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:43:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:43:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:43:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:43:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:43:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:43:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:43:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:43:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:43:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:43:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:43:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:43:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:43:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:43:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:43:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:43:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:43:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:43:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:43:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:43:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:43:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:43:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:43:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:43:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:43:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:43:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:43:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:43:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:43:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:43:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:43:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:43:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:43:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:43:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:43:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:43:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:43:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:43:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:43:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:43:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:43:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:43:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:43:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:43:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:43:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:43:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:44:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:44:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:44:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:44:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:44:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:44:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:44:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:44:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:44:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:44:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:44:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:44:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:44:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:44:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:44:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:44:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:44:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:44:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:44:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:44:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:44:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:44:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:44:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:44:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:44:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:44:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:44:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:44:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:44:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:44:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:44:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:44:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:44:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:44:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:44:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:44:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:44:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:44:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:44:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:44:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:44:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:44:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:44:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:44:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:44:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:44:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:44:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:44:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:44:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:44:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:44:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:44:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:44:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:44:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:44:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:44:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:44:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:44:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:44:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:44:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:44:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:44:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:44:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:44:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:44:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:44:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:44:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:44:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:44:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:44:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:44:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:44:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:44:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:44:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:44:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:44:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:44:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:44:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:44:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:44:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:44:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:44:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:44:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:44:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:44:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:44:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:44:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:44:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:44:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:44:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:44:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:44:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:44:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:44:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:44:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:44:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:44:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:44:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:44:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:44:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:44:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:44:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:44:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:44:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:44:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:44:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:44:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:44:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:44:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:44:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:44:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:44:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:44:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:44:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:44:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:44:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:44:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:44:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:44:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:44:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:44:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:44:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:44:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:44:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:44:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:44:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:44:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:44:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:44:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:44:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:44:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:44:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:44:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:44:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:44:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:44:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:44:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:44:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:44:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:44:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:44:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:44:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:44:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:44:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:44:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:44:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:44:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:44:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:44:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:44:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:44:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:44:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:44:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:44:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:44:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:44:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:44:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:44:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:44:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:44:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:44:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:44:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:44:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:44:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:44:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:44:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:44:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:44:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:44:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:44:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:44:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:44:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:44:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:44:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:44:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:44:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:44:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:44:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:44:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:44:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:44:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:44:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:44:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:44:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:44:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:44:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:44:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:44:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:44:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:44:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:44:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:44:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:44:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:44:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:44:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:44:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:44:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:44:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:44:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:44:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:44:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:44:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:44:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:44:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:44:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:44:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:44:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:44:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:44:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:44:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:44:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:44:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:44:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:44:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:44:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:44:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:45:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:45:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:45:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:45:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:45:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:45:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:45:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:45:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:45:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:45:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:45:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:45:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:45:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:45:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:45:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:45:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:45:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:45:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:45:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:45:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:45:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:45:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:45:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:45:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:45:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:45:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:45:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:45:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:45:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:45:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:45:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:45:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:45:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:45:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:45:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:45:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:45:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:45:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:45:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:45:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:45:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:45:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:45:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:45:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:45:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:45:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:45:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:45:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:45:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:45:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:45:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:45:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:45:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:45:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:45:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:45:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:45:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:45:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:45:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:45:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:45:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:45:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:45:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:45:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:45:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:45:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:45:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:45:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:45:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:45:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:45:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:45:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:45:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:45:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:45:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:45:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:45:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:45:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:45:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:45:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:45:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:45:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:45:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:45:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:45:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:45:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:45:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:45:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:45:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:45:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:45:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:45:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:45:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:45:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:45:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:45:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:45:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:45:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:45:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:45:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:45:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:45:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:45:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:45:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:45:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:45:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:45:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:45:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:45:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:45:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:45:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:45:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:45:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:45:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:45:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:45:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:45:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:45:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:45:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:45:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:45:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:45:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:45:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:45:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:45:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:45:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:45:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:45:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:45:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:45:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:45:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:45:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:45:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:45:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:45:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:45:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:45:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:45:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:45:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:45:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:45:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:45:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:45:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:45:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:45:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:45:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:45:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:45:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:45:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:45:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:45:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:45:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:45:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:45:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:45:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:45:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:45:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:45:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:45:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:45:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:45:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:45:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:45:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:45:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:45:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:45:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:45:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:45:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:45:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:45:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:45:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:45:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:45:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:45:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:45:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:45:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:45:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:45:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:45:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:45:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:45:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:45:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:45:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:45:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:45:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:45:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:45:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:45:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:45:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:45:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:45:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:45:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:45:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:45:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:45:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:45:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:45:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:45:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:45:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:45:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:45:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:45:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:45:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:45:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:45:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:45:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:45:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:45:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:45:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:45:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:45:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:45:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:45:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:45:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:45:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:45:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:46:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:46:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:46:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:46:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:46:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:46:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:46:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:46:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:46:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:46:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:46:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:46:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:46:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:46:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:46:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:46:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:46:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:46:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:46:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:46:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:46:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:46:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:46:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:46:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:46:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:46:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:46:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:46:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:46:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:46:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:46:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:46:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:46:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:46:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:46:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:46:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:46:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:46:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:46:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:46:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:46:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:46:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:46:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:46:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:46:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:46:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:46:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:46:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:46:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:46:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:46:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:46:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:46:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:46:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:46:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:46:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:46:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:46:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:46:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:46:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:46:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:46:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:46:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:46:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:46:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:46:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:46:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:46:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:46:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:46:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:46:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:46:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:46:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:46:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:46:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:46:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:46:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:46:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:46:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:46:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:46:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:46:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:46:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:46:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:46:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:46:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:46:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:46:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:46:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:46:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:46:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:46:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:46:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:46:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:46:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:46:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:46:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:46:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:46:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:46:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:46:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:46:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:46:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:46:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:46:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:46:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:46:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:46:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:46:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:46:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:46:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:46:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:46:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:46:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:46:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:46:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:46:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:46:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:46:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:46:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:46:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:46:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:46:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:46:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:46:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:46:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:46:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:46:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:46:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:46:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:46:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:46:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:46:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:46:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:46:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:46:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:46:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:46:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:46:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:46:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:46:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:46:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:46:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:46:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:46:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:46:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:46:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:46:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:46:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:46:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:46:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:46:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:46:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:46:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:46:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:46:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:46:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:46:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:46:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:46:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:46:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:46:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:46:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:46:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:46:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:46:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:46:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:46:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:46:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:46:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:46:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:46:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:46:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:46:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:46:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:46:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:46:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:46:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:46:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:46:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:46:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:46:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:46:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:46:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:46:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:46:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:46:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:46:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:46:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:46:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:46:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:46:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:46:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:46:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:46:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:46:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:46:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:46:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:46:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:46:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:46:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:46:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:46:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:46:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:46:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:46:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:46:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:46:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:46:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:46:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:46:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:46:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:46:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:46:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:46:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:46:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:47:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:47:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:47:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:47:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:47:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:47:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:47:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:47:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:47:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:47:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:47:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:47:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:47:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:47:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:47:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:47:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:47:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:47:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:47:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:47:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:47:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:47:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:47:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:47:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:47:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:47:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:47:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:47:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:47:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:47:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:47:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:47:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:47:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:47:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:47:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:47:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:47:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:47:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:47:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:47:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:47:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:47:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:47:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:47:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:47:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:47:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:47:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:47:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:47:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:47:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:47:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:47:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:47:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:47:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:47:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:47:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:47:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:47:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:47:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:47:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:47:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:47:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:47:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:47:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:47:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:47:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:47:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:47:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:47:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:47:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:47:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:47:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:47:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:47:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:47:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:47:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:47:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:47:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:47:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:47:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:47:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:47:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:47:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:47:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:47:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:47:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:47:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:47:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:47:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:47:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:47:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:47:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:47:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:47:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:47:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:47:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:47:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:47:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:47:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:47:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:47:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:47:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:47:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:47:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:47:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:47:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:47:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:47:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:47:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:47:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:47:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:47:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:47:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:47:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:47:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:47:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:47:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:47:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:47:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:47:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:47:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:47:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:47:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:47:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:47:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:47:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:47:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:47:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:47:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:47:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:47:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:47:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:47:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:47:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:47:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:47:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:47:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:47:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:47:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:47:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:47:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:47:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:47:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:47:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:47:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:47:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:47:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:47:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:47:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:47:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:47:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:47:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:47:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:47:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:47:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:47:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:47:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:47:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:47:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:47:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:47:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:47:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:47:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:47:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:47:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:47:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:47:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:47:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:47:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:47:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:47:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:47:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:47:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:47:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:47:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:47:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:47:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:47:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:47:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:47:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:47:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:47:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:47:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:47:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:47:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:47:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:47:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:47:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:47:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:47:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:47:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:47:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:47:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:47:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:47:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:47:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:47:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:47:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:47:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:47:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:47:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:47:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:47:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:47:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:47:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:47:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:47:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:47:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:47:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:47:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:47:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:47:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:47:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:47:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:47:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:47:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:48:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:48:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:48:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:48:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:48:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:48:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:48:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:48:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:48:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:48:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:48:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:48:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:48:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:48:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:48:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:48:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:48:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:48:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:48:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:48:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:48:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:48:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:48:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:48:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:48:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:48:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:48:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:48:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:48:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:48:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:48:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:48:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:48:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:48:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:48:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:48:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:48:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:48:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:48:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:48:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:48:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:48:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:48:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:48:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:48:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:48:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:48:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:48:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:48:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:48:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:48:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:48:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:48:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:48:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:48:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:48:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:48:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:48:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:48:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:48:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:48:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:48:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:48:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:48:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:48:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:48:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:48:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:48:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:48:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:48:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:48:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:48:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:48:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:48:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:48:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:48:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:48:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:48:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:48:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:48:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:48:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:48:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:48:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:48:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:48:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:48:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:48:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:48:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:48:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:48:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:48:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:48:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:48:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:48:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:48:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:48:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:48:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:48:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:48:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:48:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:48:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:48:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:48:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:48:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:48:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:48:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:48:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:48:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:48:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:48:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:48:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:48:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:48:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:48:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:48:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:48:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:48:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:48:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:48:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:48:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:48:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:48:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:48:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:48:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:48:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:48:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:48:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:48:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:48:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:48:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:48:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:48:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:48:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:48:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:48:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:48:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:48:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:48:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:48:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:48:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:48:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:48:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:48:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:48:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:48:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:48:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:48:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:48:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:48:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:48:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:48:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:48:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:48:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:48:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:48:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:48:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:48:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:48:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:48:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:48:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:48:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:48:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:48:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:48:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:48:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:48:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:48:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:48:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:48:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:48:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:48:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:48:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:48:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:48:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:48:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:48:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:48:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:48:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:48:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:48:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:48:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:48:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:48:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:48:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:48:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:48:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:48:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:48:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:48:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:48:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:48:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:48:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:48:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:48:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:48:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:48:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:48:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:48:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:48:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:48:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:48:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:48:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:48:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:48:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:48:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:48:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:48:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:48:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:48:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:48:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:48:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:48:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:48:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:48:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:48:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:48:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:49:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:49:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:49:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:49:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:49:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:49:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:49:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:49:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:49:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:49:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:49:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:49:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:49:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:49:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:49:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:49:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:49:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:49:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:49:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:49:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:49:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:49:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:49:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:49:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:49:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:49:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:49:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:49:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:49:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:49:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:49:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:49:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:49:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:49:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:49:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:49:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:49:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:49:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:49:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:49:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:49:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:49:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:49:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:49:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:49:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:49:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:49:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:49:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:49:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:49:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:49:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:49:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:49:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:49:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:51:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:51:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:51:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:51:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:51:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:51:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:51:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:51:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:51:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:51:20 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 19 03:51:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:51:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:51:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:51:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:51:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:51:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:51:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:51:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:51:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:51:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:51:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:51:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:51:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:51:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:51:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:51:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:51:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:51:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:51:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:51:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:51:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:51:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:51:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:51:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:51:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:51:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:51:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:51:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:51:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:51:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:51:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:51:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:51:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:51:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:51:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:51:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:51:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:51:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:51:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:51:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:51:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:51:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:51:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:51:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:51:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:51:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:51:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:51:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:51:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:51:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:51:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:51:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:51:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:51:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:51:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:51:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:51:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:51:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:51:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:51:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:51:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:51:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:51:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:51:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:51:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:51:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:51:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:51:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:51:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:51:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:51:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:51:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:51:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:51:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:51:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:51:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:51:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:51:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:51:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:51:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:51:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:51:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:51:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:51:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:51:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:51:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:51:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:51:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:51:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:51:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:51:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:51:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:51:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:51:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:51:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:51:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:51:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:51:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:51:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:51:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:51:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:51:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:51:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:51:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:51:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:51:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:51:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:51:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:52:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:52:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:52:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:52:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:52:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:52:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:52:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:52:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:52:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:52:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:52:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:52:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:52:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:52:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:52:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:52:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:52:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:52:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:52:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:52:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:52:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:52:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:52:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:52:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:52:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:52:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:52:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:52:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:52:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:52:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:52:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:52:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:52:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:52:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:52:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:52:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:52:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:52:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:52:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:52:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:52:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:52:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:52:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:52:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:52:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:52:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:52:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:52:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:52:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:52:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:52:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:52:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:52:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:52:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:52:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:52:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:52:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:52:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:52:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:52:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:52:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:52:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:52:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:52:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:52:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:52:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:52:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:52:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:52:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:52:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:52:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:52:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:52:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:52:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:52:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:52:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:52:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:52:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:52:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:52:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:52:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:52:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:52:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:52:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:52:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:52:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:52:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:52:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:52:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:52:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:52:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:52:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:52:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:52:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:52:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:52:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:52:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:52:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:52:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:52:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:52:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:52:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:52:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:52:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:52:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:52:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:52:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:52:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:52:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:52:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:52:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:52:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:52:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:52:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:52:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:52:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:52:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:52:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:52:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:52:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:52:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:52:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:52:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:52:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:52:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:52:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:52:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:52:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:52:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:52:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:52:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:52:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:52:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:52:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:52:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:52:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:52:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:52:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:52:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:52:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:52:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:52:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:52:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:52:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:52:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:52:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:52:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:52:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:52:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:52:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:52:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:52:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:52:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:52:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:52:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:52:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:52:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:52:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:52:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:52:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:52:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:52:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:52:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:52:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:52:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:52:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:52:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:52:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:52:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:52:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:52:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:52:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:52:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:52:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:52:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:52:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:52:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:52:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:52:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:52:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:52:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:52:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:52:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:52:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:52:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:52:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:52:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:52:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:52:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:52:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:52:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:52:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:52:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:52:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:52:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:52:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:52:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:52:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:52:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:52:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:52:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:52:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:52:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:52:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:52:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:52:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:52:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:52:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:52:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:52:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:52:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:52:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:52:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:52:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:52:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:52:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:53:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:53:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:53:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:53:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:53:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:53:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:53:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:53:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:53:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:53:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:53:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:53:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:53:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:53:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:53:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:53:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:53:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:53:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:53:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:53:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:53:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:53:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:53:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:53:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:53:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:53:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:53:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:53:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:53:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:53:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:53:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:53:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:53:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:53:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:53:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:53:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:53:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:53:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:53:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:53:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:53:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:53:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:53:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:53:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:53:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:53:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:53:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:53:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:53:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:53:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:53:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:53:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:53:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:53:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:53:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:53:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:53:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:53:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:53:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:53:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:53:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:53:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:53:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:53:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:53:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:53:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:53:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:53:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:53:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:53:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:53:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:53:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:53:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:53:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:53:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:53:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:53:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:53:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:53:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:53:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:53:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:53:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:53:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:53:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:53:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:53:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:53:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:53:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:53:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:53:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:53:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:53:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:53:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:53:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:53:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:53:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:53:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:53:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:53:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:53:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:53:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:53:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:53:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:53:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:53:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:53:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:53:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:53:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:53:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:53:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:53:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:53:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:53:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:53:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:53:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:53:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:53:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:53:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:53:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:53:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:53:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:53:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:53:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:53:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:53:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:53:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:53:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:53:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:53:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:53:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:53:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:53:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:53:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:53:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:53:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:53:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:53:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:53:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:53:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:53:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:53:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:53:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:53:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:53:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:53:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:53:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:53:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:53:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:53:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:53:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:53:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:53:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:53:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:53:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:53:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:53:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:53:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:53:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:53:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:53:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:53:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:53:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:53:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:53:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:53:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:53:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:53:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:53:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:53:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:53:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:53:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:53:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:53:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:53:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:53:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:53:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:53:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:53:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:53:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:53:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:53:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:53:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:53:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:53:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:53:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:53:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:53:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:53:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:53:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:53:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:53:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:53:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:53:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:53:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:53:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:53:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:53:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:53:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:53:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:53:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:53:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:53:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:53:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:53:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:53:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:53:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:53:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:53:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:53:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:53:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:53:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:53:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:53:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:53:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:53:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:53:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:54:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:54:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:54:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:54:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:54:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:54:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:54:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:54:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:54:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:54:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:54:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:54:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:54:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:54:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:54:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:54:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:54:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:54:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:54:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:54:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:54:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:54:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:54:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:54:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:54:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:54:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:54:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:54:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:54:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:54:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:54:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:54:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:54:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:54:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:54:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:54:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:54:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:54:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:54:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:54:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:54:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:54:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:54:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:54:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:54:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:54:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:54:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:54:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:54:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:54:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:54:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:54:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:54:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:54:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:54:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:54:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:54:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:54:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:54:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:54:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:54:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:54:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:54:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:54:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:54:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:54:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:54:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:54:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:54:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:54:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:54:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:54:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:54:15 rdma06 systemd-journald[790]: Data hash table of /run/log/journal/95379d2f723847b68d9e976dee8967a9/system.journal has a fill level at 75.0 (163750 of 218332 items, 75497472 file size, 461 bytes per hash table item), suggesting rotation.
Apr 19 03:54:15 rdma06 systemd-journald[790]: /run/log/journal/95379d2f723847b68d9e976dee8967a9/system.journal: Journal header limits reached or header out-of-date, rotating.
Apr 19 03:54:15 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 19 03:54:16 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 19 03:54:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:54:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:54:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:54:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:54:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:54:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:54:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:54:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:54:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:54:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:54:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:54:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:54:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:54:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:54:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:54:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:54:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:54:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:54:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:54:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:54:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:54:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:54:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:54:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:54:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:54:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:54:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:54:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:54:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:54:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:54:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:54:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:54:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:54:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:54:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:54:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:54:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:54:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:54:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:54:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:54:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:54:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:54:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:54:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:54:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:54:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:54:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:54:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:54:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:54:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:54:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:54:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:54:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:54:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:54:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:54:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:54:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:54:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:54:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:54:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:54:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:54:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:54:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:54:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:54:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:54:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:54:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:54:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:54:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:54:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:54:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:54:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:54:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:54:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:54:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:54:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:54:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:54:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:54:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:54:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:54:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:54:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:54:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:54:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:54:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:54:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:54:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:54:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:54:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:54:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:54:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:54:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:54:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:54:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:54:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:54:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:54:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:54:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:54:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:54:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:54:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:54:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:54:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:54:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:54:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:54:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:54:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:54:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:54:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:54:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:54:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:54:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:54:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:54:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:54:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:54:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:54:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:54:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:54:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:54:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:54:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:54:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:54:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:54:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:54:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:54:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:54:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:54:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:54:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:54:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:54:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:54:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:54:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:54:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:54:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:54:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:54:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:54:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:54:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:54:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:54:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:54:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:54:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:54:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:55:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:55:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:55:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:55:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:55:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:55:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:55:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:55:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:55:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:55:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:55:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:55:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:55:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:55:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:55:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:55:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:55:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:55:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:55:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:55:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:55:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:55:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:55:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:55:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:55:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:55:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:55:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:55:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:55:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:55:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:55:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:55:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:55:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:55:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:55:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:55:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:55:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:55:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:55:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:55:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:55:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:55:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:55:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:55:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:55:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:55:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:55:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:55:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:55:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:55:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:55:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:55:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:55:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:55:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:55:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:55:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:55:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:55:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:55:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:55:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:55:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:55:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:55:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:55:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:55:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:55:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:55:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:55:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:55:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:55:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:55:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:55:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:55:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:55:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:55:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:55:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:55:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:55:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:55:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:55:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:55:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:55:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:55:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:55:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:55:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:55:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:55:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:55:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:55:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:55:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:55:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:55:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:55:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:55:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:55:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:55:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:55:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:55:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:55:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:55:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:55:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:55:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:55:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:55:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:55:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:55:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:55:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:55:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:55:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:55:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:55:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:55:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:55:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:55:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:55:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:55:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:55:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:55:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:55:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:55:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:55:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:55:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:55:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:55:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:55:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:55:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:55:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:55:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:55:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:55:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:55:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:55:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:55:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:55:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:55:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:55:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:55:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:55:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:55:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:55:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:55:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:55:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:55:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:55:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:55:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:55:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:55:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:55:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:55:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:55:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:55:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:55:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:55:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:55:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:55:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:55:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:55:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:55:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:55:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:55:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:55:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:55:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:55:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:55:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:55:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:55:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:55:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:55:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:55:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:55:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:55:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:55:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:55:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:55:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:55:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:55:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:55:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:55:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:55:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:55:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:55:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:55:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:55:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:55:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:55:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:55:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:55:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:55:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:55:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:55:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:55:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:55:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:55:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:55:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:55:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:55:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:55:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:55:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:55:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:55:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:55:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:55:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:55:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:55:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:55:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:55:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:55:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:55:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:55:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:55:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:55:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:55:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:55:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:55:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:55:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:55:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:56:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:56:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:56:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:56:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:56:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:56:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:56:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:56:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:56:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:56:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:56:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:56:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:56:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:56:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:56:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:56:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:56:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:56:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:56:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:56:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:56:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:56:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:56:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:56:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:56:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:56:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:56:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:56:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:56:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:56:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:56:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:56:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:56:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:56:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:56:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:56:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:56:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:56:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:56:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:56:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:56:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:56:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:56:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:56:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:56:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:56:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:56:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:56:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:56:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:56:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:56:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:56:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:56:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:56:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:56:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:56:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:56:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:56:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:56:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:56:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:56:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:56:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:56:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:56:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:56:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:56:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:56:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:56:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:56:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:56:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:56:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:56:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:56:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:56:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:56:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:56:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:56:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:56:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:56:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:56:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:56:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:56:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:56:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:56:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:56:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:56:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:56:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:56:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:56:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:56:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:56:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:56:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:56:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:56:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:56:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:56:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:56:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:56:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:56:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:56:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:56:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:56:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:56:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:56:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:56:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:56:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:56:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:56:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:56:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:56:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:56:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:56:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:56:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:56:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:56:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:56:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:56:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:56:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:56:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:56:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:56:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:56:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:56:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:56:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:56:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:56:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:56:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:56:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:56:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:56:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:56:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:56:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:56:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:56:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:56:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:56:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:56:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:56:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:56:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:56:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:56:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:56:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:56:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:56:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:56:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:56:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:56:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:56:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:56:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:56:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:56:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:56:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:56:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:56:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:56:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:56:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:56:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:56:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:56:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:56:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:56:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:56:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:56:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:56:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:56:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:56:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:56:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:56:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:56:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:56:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:56:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:56:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:56:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:56:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:56:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:56:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:56:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:56:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:56:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:56:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:56:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:56:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:56:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:56:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:56:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:56:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:56:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:56:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:56:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:56:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:56:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:56:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:56:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:56:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:56:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:56:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:56:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:56:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:56:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:56:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:56:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:56:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:56:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:56:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:56:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:56:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:56:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:56:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:56:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:56:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:56:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:56:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:56:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:56:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:56:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:56:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:57:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:57:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:57:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:57:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:57:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:57:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:57:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:57:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:57:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:57:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:57:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:57:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:57:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:57:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:57:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:57:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:57:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:57:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:57:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:57:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:57:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:57:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:57:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:57:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:57:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:57:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:57:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:57:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:57:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:57:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:57:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:57:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:57:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:57:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:57:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:57:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:57:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:57:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:57:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:57:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:57:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:57:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:57:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:57:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:57:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:57:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:57:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:57:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:57:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:57:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:57:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:57:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:57:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:57:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:57:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:57:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:57:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:57:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:57:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:57:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:57:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:57:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:57:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:57:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:57:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:57:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:57:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:57:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:57:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:57:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:57:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:57:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:57:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:57:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:57:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:57:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:57:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:57:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:57:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:57:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:57:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:57:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:57:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:57:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:57:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:57:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:57:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:57:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:57:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:57:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:57:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:57:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:57:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:57:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:57:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:57:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:57:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:57:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:57:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:57:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:57:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:57:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:57:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:57:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:57:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:57:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:57:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:57:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:57:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:57:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:57:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:57:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:57:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:57:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:57:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:57:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:57:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:57:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:57:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:57:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:57:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:57:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:57:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:57:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:57:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:57:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:57:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:57:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:57:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:57:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:57:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:57:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:57:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:57:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:57:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:57:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:57:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:57:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:57:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:57:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:57:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:57:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:57:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:57:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:57:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:57:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:57:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:57:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:57:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:57:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:57:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:57:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:57:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:57:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:57:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:57:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:57:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:57:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:57:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:57:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:57:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:57:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:57:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:57:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:57:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:57:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:57:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:57:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:57:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:57:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:57:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:57:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:57:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:57:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:57:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:57:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:57:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:57:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:57:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:57:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:57:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:57:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:57:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:57:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:57:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:57:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:57:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:57:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:57:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:57:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:57:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:57:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:57:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:57:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:57:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:57:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:57:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:57:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:57:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:57:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:57:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:57:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:57:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:57:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:57:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:57:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:57:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:57:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:57:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:57:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:57:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:57:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:57:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:57:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:57:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:57:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:58:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:58:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:58:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:58:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:58:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:58:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:58:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:58:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:58:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:58:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:58:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:58:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:58:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:58:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:58:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:58:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:58:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:58:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:58:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:58:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:58:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:58:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:58:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:58:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:58:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:58:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:58:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:58:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:58:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:58:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:58:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:58:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:58:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:58:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:58:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:58:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:58:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:58:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:58:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:58:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:58:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:58:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:58:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:58:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:58:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:58:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:58:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:58:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:58:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:58:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:58:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:58:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:58:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:58:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:58:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:58:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:58:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:58:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:58:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:58:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:58:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:58:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:58:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:58:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:58:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:58:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:58:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:58:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:58:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:58:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:58:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:58:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:58:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:58:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:58:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:58:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:58:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:58:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:58:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:58:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:58:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:58:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:58:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:58:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:58:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:58:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:58:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:58:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:58:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:58:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:58:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:58:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:58:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:58:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:58:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:58:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:58:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:58:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:58:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:58:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:58:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:58:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:58:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:58:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:58:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:58:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:58:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:58:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:58:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:58:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:58:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:58:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:58:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:58:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:58:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:58:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:58:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:58:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:58:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:58:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:58:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:58:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:58:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:58:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:58:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:58:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:58:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:58:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:58:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:58:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:58:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:58:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:58:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:58:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:58:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:58:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:58:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:58:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:58:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:58:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:58:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:58:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:58:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:58:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:58:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:58:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:58:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:58:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:58:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:58:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:58:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:58:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:58:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:58:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:58:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:58:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:58:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:58:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:58:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:58:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:58:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:58:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:58:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:58:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:58:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:58:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:58:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:58:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:58:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:58:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:58:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:58:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:58:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:58:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:58:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:58:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:58:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:58:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:58:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:58:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:58:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:58:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:58:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:58:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:58:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:58:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:58:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:58:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:58:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:58:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:58:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:58:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:58:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:58:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:58:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:58:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:58:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:58:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:58:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:58:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:58:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:58:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:58:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:58:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:58:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:58:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:58:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:58:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:58:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:58:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:58:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:58:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:58:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:58:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:58:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:58:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:59:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:59:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:59:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:59:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:59:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:59:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:59:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:59:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:59:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:59:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:59:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:59:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:59:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:59:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:59:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:59:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:59:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:59:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:59:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:59:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:59:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:59:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:59:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:59:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:59:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:59:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:59:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:59:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:59:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:59:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:59:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:59:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:59:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:59:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:59:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:59:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:59:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:59:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:59:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:59:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:59:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:59:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:59:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:59:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:59:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:59:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:59:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:59:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:59:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:59:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:59:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:59:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:59:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:59:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 03:59:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 03:59:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:59:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:59:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:59:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 03:59:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 03:59:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 03:59:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 03:59:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 03:59:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 03:59:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 03:59:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 03:59:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 03:59:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 03:59:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 03:59:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 03:59:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 03:59:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:01:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:01:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:01:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:01:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:01:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:01:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:01:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:01:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:01:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:01:21 rdma06 rsyslogd[1041]: imjournal: 5090 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 19 04:01:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:01:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:01:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:01:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:01:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:01:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:01:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:01:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:01:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:01:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:01:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:01:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:01:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:01:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:01:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:01:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:01:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:01:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:01:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:01:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:01:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:01:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:01:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:01:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:01:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:01:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:01:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:01:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:01:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:01:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:01:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:01:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:01:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:01:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:01:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:01:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:01:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:01:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:01:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:01:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:01:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:01:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:01:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:01:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:01:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:01:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:01:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:01:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:01:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:01:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:01:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:01:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:01:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:01:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:01:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:01:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:01:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:01:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:01:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:01:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:01:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:01:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:01:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:01:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:01:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:01:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:01:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:01:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:01:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:01:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:01:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:01:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:01:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:01:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:01:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:01:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:01:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:01:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:01:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:01:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:01:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:02:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:02:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:02:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:02:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:02:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:02:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:02:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:02:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:02:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:02:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:02:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:02:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:02:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:02:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:02:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:02:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:02:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:02:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:02:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:02:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:02:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:02:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:02:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:02:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:02:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:02:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:02:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:02:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:02:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:02:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:02:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:02:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:02:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:02:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:02:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:02:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:02:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:02:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:02:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:02:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:02:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:02:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:02:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:02:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:02:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:02:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:02:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:02:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:02:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:02:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:02:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:02:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:02:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:02:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:02:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:02:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:02:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:02:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:02:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:02:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:02:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:02:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:02:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:02:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:02:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:02:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:02:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:02:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:02:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:02:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:02:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:02:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:02:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:02:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:02:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:02:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:02:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:02:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:02:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:02:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:02:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:02:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:02:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:02:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:02:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:02:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:02:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:02:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:02:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:02:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:02:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:02:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:02:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:02:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:02:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:02:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:02:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:02:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:02:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:02:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:02:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:02:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:02:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:02:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:02:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:02:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:02:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:02:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:02:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:02:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:02:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:02:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:02:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:02:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:02:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:02:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:02:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:02:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:02:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:02:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:02:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:02:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:02:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:02:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:02:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:02:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:02:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:02:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:02:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:02:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:02:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:02:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:02:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:02:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:02:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:02:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:02:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:02:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:02:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:02:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:02:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:02:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:02:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:02:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:02:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:02:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:02:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:02:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:02:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:02:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:02:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:02:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:02:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:02:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:02:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:02:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:02:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:02:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:02:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:02:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:02:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:02:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:02:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:02:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:02:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:02:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:02:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:02:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:02:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:02:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:02:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:02:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:02:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:02:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:02:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:02:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:02:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:02:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:02:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:02:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:02:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:02:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:02:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:02:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:02:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:02:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:02:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:02:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:02:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:02:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:02:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:02:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:02:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:02:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:02:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:02:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:02:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:02:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:02:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:02:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:02:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:02:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:02:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:02:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:02:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:02:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:02:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:02:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:02:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:02:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:02:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:02:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:02:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:02:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:02:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:02:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:03:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:03:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:03:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:03:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:03:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:03:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:03:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:03:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:03:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:03:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:03:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:03:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:03:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:03:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:03:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:03:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:03:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:03:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:03:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:03:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:03:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:03:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:03:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:03:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:03:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:03:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:03:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:03:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:03:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:03:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:03:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:03:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:03:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:03:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:03:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:03:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:03:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:03:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:03:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:03:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:03:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:03:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:03:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:03:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:03:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:03:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:03:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:03:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:03:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:03:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:03:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:03:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:03:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:03:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:03:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:03:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:03:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:03:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:03:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:03:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:03:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:03:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:03:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:03:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:03:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:03:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:03:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:03:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:03:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:03:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:03:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:03:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:03:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:03:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:03:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:03:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:03:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:03:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:03:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:03:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:03:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:03:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:03:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:03:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:03:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:03:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:03:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:03:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:03:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:03:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:03:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:03:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:03:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:03:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:03:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:03:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:03:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:03:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:03:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:03:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:03:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:03:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:03:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:03:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:03:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:03:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:03:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:03:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:03:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:03:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:03:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:03:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:03:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:03:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:03:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:03:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:03:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:03:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:03:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:03:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:03:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:03:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:03:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:03:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:03:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:03:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:03:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:03:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:03:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:03:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:03:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:03:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:03:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:03:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:03:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:03:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:03:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:03:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:03:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:03:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:03:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:03:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:03:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:03:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:03:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:03:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:03:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:03:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:03:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:03:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:03:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:03:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:03:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:03:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:03:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:03:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:03:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:03:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:03:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:03:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:03:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:03:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:03:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:03:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:03:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:03:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:03:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:03:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:03:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:03:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:03:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:03:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:03:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:03:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:03:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:03:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:03:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:03:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:03:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:03:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:03:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:03:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:03:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:03:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:03:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:03:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:03:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:03:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:03:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:03:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:03:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:03:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:03:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:03:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:03:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:03:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:03:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:03:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:03:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:03:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:03:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:03:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:03:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:03:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:03:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:03:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:03:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:03:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:03:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:03:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:03:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:03:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:03:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:03:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:03:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:03:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:04:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:04:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:04:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:04:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:04:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:04:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:04:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:04:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:04:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:04:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:04:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:04:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:04:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:04:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:04:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:04:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:04:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:04:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:04:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:04:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:04:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:04:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:04:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:04:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:04:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:04:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:04:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:04:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:04:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:04:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:04:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:04:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:04:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:04:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:04:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:04:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:04:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:04:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:04:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:04:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:04:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:04:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:04:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:04:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:04:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:04:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:04:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:04:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:04:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:04:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:04:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:04:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:04:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:04:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:04:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:04:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:04:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:04:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:04:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:04:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:04:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:04:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:04:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:04:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:04:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:04:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:04:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:04:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:04:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:04:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:04:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:04:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:04:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:04:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:04:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:04:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:04:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:04:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:04:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:04:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:04:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:04:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:04:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:04:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:04:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:04:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:04:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:04:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:04:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:04:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:04:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:04:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:04:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:04:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:04:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:04:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:04:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:04:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:04:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:04:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:04:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:04:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:04:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:04:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:04:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:04:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:04:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:04:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:04:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:04:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:04:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:04:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:04:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:04:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:04:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:04:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:04:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:04:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:04:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:04:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:04:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:04:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:04:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:04:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:04:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:04:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:04:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:04:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:04:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:04:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:04:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:04:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:04:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:04:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:04:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:04:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:04:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:04:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:04:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:04:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:04:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:04:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:04:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:04:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:04:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:04:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:04:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:04:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:04:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:04:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:04:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:04:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:04:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:04:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:04:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:04:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:04:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:04:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:04:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:04:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:04:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:04:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:04:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:04:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:04:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:04:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:04:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:04:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:04:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:04:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:04:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:04:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:04:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:04:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:04:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:04:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:04:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:04:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:04:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:04:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:04:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:04:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:04:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:04:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:04:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:04:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:04:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:04:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:04:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:04:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:04:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:04:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:04:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:04:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:04:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:04:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:04:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:04:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:04:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:04:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:04:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:04:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:04:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:04:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:04:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:04:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:04:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:04:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:04:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:04:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:04:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:04:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:04:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:04:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:04:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:04:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:05:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:05:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:05:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:05:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:05:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:05:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:05:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:05:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:05:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:05:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:05:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:05:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:05:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:05:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:05:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:05:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:05:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:05:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:05:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:05:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:05:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:05:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:05:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:05:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:05:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:05:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:05:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:05:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:05:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:05:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:05:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:05:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:05:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:05:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:05:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:05:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:05:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:05:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:05:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:05:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:05:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:05:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:05:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:05:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:05:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:05:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:05:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:05:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:05:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:05:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:05:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:05:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:05:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:05:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:05:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:05:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:05:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:05:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:05:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:05:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:05:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:05:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:05:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:05:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:05:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:05:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:05:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:05:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:05:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:05:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:05:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:05:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:05:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:05:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:05:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:05:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:05:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:05:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:05:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:05:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:05:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:05:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:05:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:05:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:05:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:05:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:05:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:05:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:05:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:05:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:05:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:05:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:05:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:05:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:05:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:05:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:05:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:05:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:05:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:05:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:05:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:05:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:05:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:05:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:05:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:05:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:05:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:05:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:05:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:05:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:05:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:05:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:05:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:05:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:05:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:05:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:05:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:05:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:05:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:05:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:05:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:05:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:05:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:05:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:05:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:05:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:05:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:05:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:05:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:05:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:05:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:05:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:05:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:05:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:05:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:05:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:05:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:05:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:05:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:05:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:05:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:05:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:05:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:05:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:05:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:05:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:05:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:05:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:05:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:05:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:05:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:05:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:05:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:05:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:05:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:05:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:05:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:05:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:05:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:05:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:05:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:05:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:05:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:05:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:05:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:05:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:05:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:05:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:05:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:05:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:05:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:05:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:05:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:05:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:05:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:05:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:05:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:05:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:05:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:05:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:05:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:05:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:05:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:05:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:05:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:05:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:05:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:05:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:05:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:05:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:05:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:05:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:05:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:05:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:05:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:05:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:05:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:05:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:05:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:05:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:05:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:05:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:05:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:05:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:05:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:05:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:05:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:05:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:05:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:05:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:05:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:05:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:05:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:05:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:05:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:05:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:06:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:06:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:06:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:06:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:06:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:06:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:06:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:06:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:06:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:06:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:06:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:06:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:06:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:06:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:06:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:06:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:06:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:06:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:06:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:06:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:06:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:06:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:06:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:06:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:06:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:06:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:06:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:06:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:06:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:06:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:06:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:06:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:06:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:06:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:06:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:06:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:06:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:06:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:06:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:06:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:06:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:06:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:06:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:06:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:06:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:06:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:06:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:06:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:06:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:06:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:06:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:06:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:06:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:06:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:06:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:06:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:06:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:06:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:06:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:06:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:06:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:06:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:06:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:06:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:06:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:06:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:06:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:06:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:06:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:06:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:06:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:06:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:06:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:06:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:06:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:06:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:06:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:06:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:06:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:06:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:06:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:06:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:06:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:06:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:06:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:06:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:06:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:06:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:06:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:06:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:06:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:06:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:06:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:06:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:06:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:06:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:06:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:06:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:06:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:06:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:06:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:06:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:06:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:06:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:06:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:06:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:06:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:06:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:06:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:06:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:06:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:06:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:06:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:06:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:06:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:06:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:06:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:06:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:06:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:06:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:06:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:06:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:06:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:06:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:06:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:06:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:06:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:06:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:06:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:06:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:06:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:06:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:06:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:06:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:06:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:06:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:06:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:06:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:06:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:06:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:06:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:06:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:06:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:06:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:06:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:06:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:06:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:06:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:06:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:06:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:06:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:06:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:06:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:06:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:06:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:06:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:06:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:06:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:06:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:06:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:06:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:06:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:06:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:06:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:06:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:06:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:06:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:06:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:06:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:06:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:06:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:06:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:06:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:06:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:06:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:06:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:06:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:06:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:06:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:06:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:06:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:06:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:06:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:06:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:06:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:06:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:06:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:06:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:06:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:06:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:06:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:06:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:06:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:06:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:06:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:06:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:06:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:06:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:06:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:06:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:06:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:06:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:06:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:06:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:06:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:06:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:06:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:06:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:06:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:06:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:06:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:06:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:06:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:06:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:06:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:06:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:07:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:07:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:07:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:07:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:07:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:07:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:07:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:07:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:07:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:07:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:07:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:07:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:07:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:07:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:07:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:07:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:07:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:07:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:07:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:07:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:07:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:07:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:07:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:07:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:07:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:07:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:07:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:07:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:07:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:07:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:07:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:07:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:07:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:07:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:07:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:07:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:07:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:07:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:07:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:07:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:07:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:07:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:07:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:07:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:07:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:07:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:07:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:07:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:07:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:07:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:07:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:07:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:07:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:07:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:07:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:07:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:07:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:07:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:07:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:07:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:07:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:07:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:07:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:07:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:07:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:07:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:07:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:07:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:07:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:07:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:07:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:07:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:07:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:07:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:07:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:07:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:07:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:07:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:07:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:07:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:07:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:07:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:07:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:07:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:07:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:07:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:07:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:07:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:07:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:07:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:07:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:07:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:07:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:07:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:07:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:07:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:07:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:07:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:07:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:07:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:07:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:07:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:07:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:07:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:07:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:07:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:07:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:07:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:07:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:07:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:07:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:07:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:07:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:07:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:07:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:07:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:07:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:07:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:07:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:07:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:07:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:07:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:07:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:07:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:07:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:07:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:07:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:07:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:07:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:07:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:07:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:07:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:07:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:07:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:07:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:07:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:07:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:07:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:07:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:07:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:07:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:07:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:07:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:07:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:07:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:07:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:07:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:07:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:07:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:07:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:07:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:07:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:07:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:07:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:07:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:07:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:07:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:07:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:07:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:07:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:07:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:07:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:07:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:07:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:07:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:07:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:07:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:07:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:07:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:07:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:07:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:07:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:07:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:07:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:07:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:07:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:07:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:07:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:07:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:07:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:07:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:07:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:07:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:07:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:07:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:07:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:07:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:07:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:07:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:07:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:07:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:07:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:07:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:07:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:07:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:07:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:07:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:07:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:07:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:07:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:07:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:07:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:07:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:07:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:07:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:07:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:07:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:07:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:07:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:07:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:07:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:07:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:07:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:07:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:07:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:07:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:08:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:08:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:08:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:08:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:08:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:08:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:08:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:08:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:08:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:08:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:08:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:08:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:08:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:08:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:08:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:08:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:08:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:08:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:08:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:08:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:08:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:08:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:08:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:08:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:08:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:08:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:08:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:08:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:08:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:08:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:08:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:08:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:08:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:08:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:08:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:08:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:08:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:08:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:08:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:08:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:08:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:08:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:08:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:08:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:08:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:08:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:08:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:08:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:08:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:08:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:08:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:08:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:08:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:08:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:08:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:08:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:08:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:08:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:08:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:08:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:08:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:08:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:08:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:08:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:08:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:08:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:08:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:08:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:08:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:08:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:08:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:08:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:08:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:08:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:08:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:08:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:08:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:08:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:08:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:08:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:08:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:08:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:08:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:08:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:08:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:08:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:08:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:08:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:08:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:08:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:08:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:08:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:08:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:08:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:08:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:08:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:08:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:08:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:08:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:08:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:08:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:08:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:08:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:08:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:08:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:08:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:08:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:08:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:08:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:08:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:08:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:08:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:08:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:08:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:08:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:08:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:08:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:08:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:08:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:08:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:08:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:08:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:08:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:08:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:08:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:08:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:08:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:08:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:08:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:08:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:08:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:08:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:08:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:08:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:08:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:08:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:08:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:08:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:08:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:08:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:08:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:08:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:08:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:08:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:08:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:08:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:08:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:08:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:08:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:08:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:08:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:08:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:08:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:08:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:08:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:08:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:08:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:08:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:08:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:08:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:08:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:08:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:08:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:08:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:08:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:08:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:08:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:08:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:08:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:08:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:08:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:08:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:08:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:08:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:08:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:08:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:08:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:08:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:08:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:08:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:08:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:08:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:08:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:08:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:08:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:08:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:08:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:08:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:08:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:08:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:08:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:08:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:08:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:08:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:08:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:08:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:08:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:08:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:08:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:08:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:08:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:08:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:08:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:08:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:08:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:08:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:08:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:08:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:08:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:08:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:08:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:08:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:08:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:08:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:08:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:08:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:09:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:09:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:09:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:09:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:09:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:09:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:09:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:09:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:09:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:09:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:09:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:09:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:09:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:09:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:09:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:09:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:09:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:09:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:09:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:09:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:09:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:09:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:09:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:09:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:09:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:09:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:09:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:09:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:09:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:09:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:09:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:09:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:09:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:09:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:09:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:09:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:09:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:09:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:09:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:09:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:09:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:09:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:09:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:09:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:09:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:09:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:09:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:09:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:09:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:09:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:09:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:09:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:09:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:09:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:09:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:09:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:09:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:09:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:09:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:09:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:09:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:09:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:09:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:09:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:09:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:09:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:09:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:09:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:09:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:09:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:09:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:09:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:11:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:11:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:11:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:11:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:11:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:11:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:11:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:11:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:11:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:11:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:11:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:11:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:11:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:11:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:11:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:11:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:11:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:11:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:11:26 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 19 04:11:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:11:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:11:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:11:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:11:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:11:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:11:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:11:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:11:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:11:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:11:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:11:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:11:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:11:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:11:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:11:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:11:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:11:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:11:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:11:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:11:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:11:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:11:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:11:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:11:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:11:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:11:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:11:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:11:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:11:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:11:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:11:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:11:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:11:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:11:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:11:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:11:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:11:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:11:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:11:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:11:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:11:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:11:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:11:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:11:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:11:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:11:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:11:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:11:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:11:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:11:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:11:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:11:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:11:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:11:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:11:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:11:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:11:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:11:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:11:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:11:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:11:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:11:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:11:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:11:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:11:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:11:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:11:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:11:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:11:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:11:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:11:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:11:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:11:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:11:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:11:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:11:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:11:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:11:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:11:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:11:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:11:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:11:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:11:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:11:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:11:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:11:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:11:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:11:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:11:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:11:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:11:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:11:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:11:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:11:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:11:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:11:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:11:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:11:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:11:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:11:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:11:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:11:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:11:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:11:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:11:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:11:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:11:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:12:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:12:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:12:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:12:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:12:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:12:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:12:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:12:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:12:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:12:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:12:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:12:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:12:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:12:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:12:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:12:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:12:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:12:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:12:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:12:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:12:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:12:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:12:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:12:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:12:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:12:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:12:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:12:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:12:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:12:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:12:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:12:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:12:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:12:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:12:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:12:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:12:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:12:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:12:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:12:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:12:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:12:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:12:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:12:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:12:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:12:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:12:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:12:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:12:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:12:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:12:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:12:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:12:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:12:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:12:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:12:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:12:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:12:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:12:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:12:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:12:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:12:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:12:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:12:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:12:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:12:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:12:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:12:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:12:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:12:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:12:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:12:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:12:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:12:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:12:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:12:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:12:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:12:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:12:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:12:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:12:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:12:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:12:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:12:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:12:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:12:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:12:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:12:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:12:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:12:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:12:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:12:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:12:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:12:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:12:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:12:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:12:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:12:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:12:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:12:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:12:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:12:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:12:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:12:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:12:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:12:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:12:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:12:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:12:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:12:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:12:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:12:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:12:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:12:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:12:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:12:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:12:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:12:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:12:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:12:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:12:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:12:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:12:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:12:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:12:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:12:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:12:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:12:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:12:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:12:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:12:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:12:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:12:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:12:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:12:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:12:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:12:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:12:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:12:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:12:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:12:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:12:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:12:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:12:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:12:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:12:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:12:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:12:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:12:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:12:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:12:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:12:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:12:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:12:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:12:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:12:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:12:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:12:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:12:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:12:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:12:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:12:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:12:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:12:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:12:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:12:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:12:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:12:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:12:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:12:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:12:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:12:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:12:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:12:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:12:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:12:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:12:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:12:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:12:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:12:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:12:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:12:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:12:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:12:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:12:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:12:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:12:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:12:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:12:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:12:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:12:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:12:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:12:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:12:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:12:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:12:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:12:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:12:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:12:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:12:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:12:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:12:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:12:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:12:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:12:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:12:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:12:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:12:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:12:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:12:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:12:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:12:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:12:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:12:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:12:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:12:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:13:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:13:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:13:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:13:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:13:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:13:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:13:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:13:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:13:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:13:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:13:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:13:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:13:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:13:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:13:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:13:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:13:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:13:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:13:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:13:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:13:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:13:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:13:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:13:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:13:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:13:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:13:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:13:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:13:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:13:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:13:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:13:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:13:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:13:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:13:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:13:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:13:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:13:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:13:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:13:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:13:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:13:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:13:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:13:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:13:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:13:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:13:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:13:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:13:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:13:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:13:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:13:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:13:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:13:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:13:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:13:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:13:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:13:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:13:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:13:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:13:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:13:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:13:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:13:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:13:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:13:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:13:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:13:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:13:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:13:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:13:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:13:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:13:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:13:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:13:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:13:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:13:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:13:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:13:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:13:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:13:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:13:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:13:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:13:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:13:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:13:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:13:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:13:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:13:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:13:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:13:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:13:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:13:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:13:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:13:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:13:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:13:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:13:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:13:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:13:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:13:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:13:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:13:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:13:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:13:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:13:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:13:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:13:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:13:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:13:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:13:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:13:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:13:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:13:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:13:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:13:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:13:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:13:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:13:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:13:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:13:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:13:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:13:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:13:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:13:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:13:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:13:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:13:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:13:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:13:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:13:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:13:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:13:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:13:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:13:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:13:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:13:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:13:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:13:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:13:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:13:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:13:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:13:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:13:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:13:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:13:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:13:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:13:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:13:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:13:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:13:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:13:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:13:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:13:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:13:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:13:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:13:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:13:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:13:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:13:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:13:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:13:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:13:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:13:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:13:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:13:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:13:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:13:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:13:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:13:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:13:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:13:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:13:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:13:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:13:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:13:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:13:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:13:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:13:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:13:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:13:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:13:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:13:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:13:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:13:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:13:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:13:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:13:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:13:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:13:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:13:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:13:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:13:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:13:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:13:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:13:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:13:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:13:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:13:54 rdma06 sshd[1213881]: main: sshd: ssh-rsa algorithm is disabled
Apr 19 04:13:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:13:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:13:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:13:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:13:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:13:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:13:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:13:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:13:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:13:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:13:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:13:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:13:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:13:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:13:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:13:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:13:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:13:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:13:58 rdma06 systemd-logind[1064]: New session 18 of user root.
Apr 19 04:13:58 rdma06 systemd[1]: Started Session 18 of User root.
Apr 19 04:13:59 rdma06 systemd[1]: Starting Hostname Service...
Apr 19 04:13:59 rdma06 systemd[1]: Started Hostname Service.
Apr 19 04:14:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:14:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:14:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:14:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:14:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:14:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:14:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:14:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:14:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:14:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:14:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:14:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:14:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:14:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:14:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:14:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:14:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:14:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:14:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:14:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:14:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:14:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:14:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:14:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:14:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:14:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:14:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:14:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:14:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:14:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:14:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:14:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:14:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:14:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:14:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:14:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:14:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:14:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:14:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:14:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:14:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:14:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:14:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:14:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:14:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:14:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:14:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:14:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:14:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:14:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:14:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:14:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:14:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:14:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:14:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:14:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:14:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:14:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:14:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:14:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:14:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:14:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:14:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:14:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:14:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:14:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:14:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:14:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:14:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:14:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:14:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:14:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:14:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:14:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:14:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:14:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:14:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:14:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:14:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:14:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:14:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:14:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:14:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:14:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:14:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:14:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:14:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:14:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:14:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:14:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:14:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:14:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:14:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:14:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:14:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:14:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:14:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:14:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:14:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:14:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:14:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:14:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:14:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:14:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:14:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:14:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:14:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:14:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:14:29 rdma06 systemd[1]: systemd-hostnamed.service: Deactivated successfully.
Apr 19 04:14:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:14:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:14:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:14:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:14:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:14:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:14:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:14:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:14:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:14:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:14:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:14:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:14:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:14:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:14:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:14:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:14:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:14:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:14:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:14:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:14:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:14:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:14:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:14:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:14:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:14:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:14:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:14:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:14:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:14:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:14:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:14:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:14:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:14:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:14:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:14:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:14:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:14:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:14:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:14:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:14:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:14:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:14:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:14:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:14:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:14:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:14:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:14:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:14:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:14:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:14:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:14:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:14:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:14:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:14:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:14:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:14:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:14:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:14:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:14:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:14:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:14:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:14:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:14:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:14:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:14:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:14:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:14:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:14:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:14:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:14:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:14:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:14:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:14:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:14:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:14:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:14:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:14:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:14:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:14:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:14:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:14:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:14:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:14:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:14:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:14:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:14:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:14:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:14:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:14:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:14:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:14:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:14:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:14:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:14:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:14:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:14:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:14:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:14:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:14:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:14:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:14:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:14:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:14:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:14:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:14:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:14:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:14:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:15:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:15:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:15:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:15:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:15:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:15:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:15:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:15:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:15:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:15:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:15:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:15:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:15:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:15:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:15:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:15:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:15:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:15:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:15:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:15:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:15:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:15:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:15:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:15:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:15:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:15:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:15:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:15:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:15:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:15:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:15:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:15:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:15:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:15:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:15:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:15:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:15:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:15:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:15:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:15:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:15:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:15:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:15:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:15:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:15:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:15:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:15:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:15:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:15:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:15:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:15:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:15:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:15:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:15:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:15:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:15:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:15:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:15:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:15:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:15:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:15:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:15:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:15:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:15:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:15:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:15:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:15:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:15:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:15:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:15:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:15:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:15:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:15:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:15:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:15:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:15:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:15:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:15:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:15:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:15:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:15:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:15:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:15:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:15:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:15:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:15:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:15:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:15:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:15:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:15:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:15:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:15:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:15:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:15:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:15:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:15:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:15:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:15:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:15:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:15:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:15:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:15:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:15:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:15:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:15:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:15:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:15:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:15:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:15:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:15:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:15:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:15:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:15:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:15:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:15:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:15:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:15:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:15:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:15:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:15:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:15:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:15:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:15:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:15:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:15:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:15:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:15:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:15:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:15:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:15:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:15:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:15:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:15:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:15:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:15:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:15:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:15:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:15:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:15:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:15:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:15:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:15:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:15:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:15:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:15:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:15:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:15:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:15:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:15:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:15:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:15:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:15:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:15:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:15:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:15:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:15:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:15:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:15:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:15:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:15:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:15:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:15:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:15:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:15:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:15:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:15:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:15:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:15:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:15:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:15:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:15:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:15:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:15:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:15:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:15:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:15:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:15:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:15:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:15:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:15:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:15:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:15:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:15:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:15:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:15:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:15:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:15:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:15:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:15:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:15:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:15:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:15:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:15:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:15:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:15:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:15:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:15:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:15:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:15:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:15:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:15:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:15:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:15:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:15:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:15:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:15:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:15:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:15:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:15:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:15:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:15:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:15:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:15:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:15:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:15:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:15:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:16:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:16:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:16:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:16:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:16:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:16:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:16:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:16:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:16:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:16:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:16:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:16:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:16:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:16:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:16:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:16:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:16:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:16:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:16:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:16:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:16:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:16:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:16:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:16:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:16:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:16:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:16:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:16:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:16:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:16:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:16:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:16:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:16:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:16:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:16:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:16:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:16:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:16:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:16:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:16:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:16:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:16:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:16:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:16:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:16:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:16:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:16:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:16:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:16:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:16:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:16:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:16:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:16:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:16:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:16:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:16:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:16:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:16:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:16:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:16:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:16:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:16:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:16:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:16:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:16:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:16:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:16:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:16:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:16:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:16:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:16:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:16:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:16:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:16:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:16:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:16:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:16:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:16:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:16:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:16:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:16:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:16:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:16:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:16:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:16:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:16:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:16:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:16:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:16:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:16:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:16:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:16:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:16:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:16:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:16:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:16:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:16:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:16:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:16:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:16:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:16:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:16:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:16:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:16:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:16:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:16:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:16:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:16:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:16:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:16:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:16:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:16:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:16:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:16:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:16:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:16:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:16:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:16:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:16:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:16:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:16:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:16:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:16:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:16:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:16:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:16:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:16:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:16:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:16:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:16:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:16:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:16:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:16:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:16:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:16:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:16:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:16:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:16:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:16:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:16:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:16:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:16:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:16:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:16:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:16:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:16:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:16:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:16:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:16:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:16:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:16:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:16:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:16:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:16:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:16:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:16:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:16:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:16:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:16:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:16:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:16:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:16:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:16:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:16:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:16:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:16:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:16:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:16:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:16:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:16:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:16:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:16:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:16:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:16:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:16:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:16:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:16:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:16:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:16:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:16:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:16:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:16:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:16:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:16:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:16:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:16:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:16:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:16:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:16:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:16:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:16:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:16:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:16:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:16:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:16:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:16:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:16:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:16:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:16:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:16:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:16:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:16:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:16:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:16:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:16:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:16:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:16:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:16:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:16:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:16:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:16:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:16:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:16:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:16:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:16:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:16:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:17:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:17:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:17:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:17:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:17:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:17:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:17:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:17:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:17:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:17:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:17:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:17:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:17:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:17:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:17:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:17:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:17:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:17:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:17:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:17:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:17:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:17:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:17:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:17:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:17:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:17:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:17:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:17:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:17:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:17:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:17:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:17:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:17:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:17:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:17:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:17:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:17:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:17:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:17:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:17:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:17:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:17:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:17:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:17:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:17:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:17:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:17:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:17:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:17:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:17:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:17:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:17:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:17:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:17:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:17:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:17:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:17:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:17:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:17:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:17:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:17:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:17:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:17:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:17:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:17:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:17:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:17:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:17:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:17:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:17:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:17:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:17:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:17:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:17:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:17:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:17:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:17:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:17:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:17:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:17:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:17:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:17:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:17:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:17:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:17:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:17:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:17:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:17:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:17:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:17:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:17:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:17:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:17:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:17:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:17:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:17:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:17:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:17:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:17:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:17:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:17:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:17:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:17:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:17:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:17:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:17:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:17:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:17:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:17:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:17:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:17:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:17:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:17:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:17:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:17:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:17:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:17:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:17:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:17:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:17:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:17:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:17:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:17:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:17:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:17:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:17:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:17:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:17:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:17:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:17:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:17:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:17:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:17:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:17:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:17:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:17:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:17:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:17:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:17:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:17:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:17:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:17:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:17:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:17:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:17:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:17:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:17:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:17:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:17:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:17:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:17:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:17:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:17:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:17:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:17:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:17:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:17:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:17:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:17:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:17:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:17:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:17:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:17:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:17:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:17:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:17:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:17:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:17:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:17:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:17:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:17:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:17:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:17:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:17:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:17:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:17:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:17:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:17:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:17:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:17:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:17:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:17:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:17:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:17:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:17:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:17:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:17:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:17:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:17:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:17:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:17:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:17:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:17:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:17:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:17:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:17:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:17:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:17:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:17:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:17:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:17:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:17:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:17:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:17:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:17:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:17:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:17:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:17:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:17:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:17:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:17:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:17:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:17:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:17:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:17:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:17:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:18:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:18:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:18:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:18:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:18:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:18:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:18:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:18:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:18:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:18:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:18:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:18:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:18:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:18:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:18:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:18:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:18:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:18:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:18:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:18:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:18:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:18:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:18:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:18:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:18:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:18:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:18:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:18:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:18:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:18:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:18:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:18:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:18:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:18:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:18:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:18:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:18:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:18:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:18:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:18:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:18:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:18:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:18:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:18:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:18:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:18:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:18:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:18:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:18:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:18:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:18:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:18:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:18:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:18:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:18:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:18:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:18:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:18:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:18:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:18:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:18:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:18:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:18:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:18:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:18:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:18:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:18:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:18:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:18:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:18:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:18:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:18:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:18:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:18:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:18:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:18:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:18:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:18:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:18:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:18:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:18:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:18:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:18:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:18:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:18:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:18:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:18:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:18:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:18:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:18:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:18:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:18:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:18:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:18:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:18:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:18:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:18:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:18:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:18:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:18:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:18:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:18:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:18:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:18:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:18:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:18:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:18:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:18:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:18:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:18:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:18:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:18:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:18:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:18:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:18:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:18:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:18:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:18:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:18:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:18:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:18:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:18:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:18:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:18:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:18:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:18:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:18:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:18:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:18:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:18:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:18:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:18:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:18:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:18:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:18:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:18:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:18:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:18:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:18:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:18:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:18:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:18:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:18:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:18:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:18:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:18:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:18:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:18:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:18:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:18:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:18:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:18:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:18:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:18:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:18:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:18:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:18:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:18:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:18:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:18:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:18:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:18:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:18:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:18:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:18:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:18:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:18:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:18:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:18:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:18:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:18:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:18:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:18:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:18:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:18:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:18:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:18:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:18:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:18:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:18:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:18:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:18:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:18:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:18:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:18:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:18:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:18:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:18:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:18:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:18:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:18:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:18:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:18:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:18:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:18:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:18:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:18:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:18:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:18:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:18:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:18:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:18:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:18:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:18:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:18:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:18:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:18:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:18:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:18:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:18:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:18:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:18:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:18:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:18:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:18:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:18:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:19:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:19:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:19:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:19:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:19:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:19:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:19:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:19:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:19:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:19:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:19:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:19:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:19:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:19:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:19:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:19:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:19:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:19:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:19:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:19:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:19:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:19:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:19:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:19:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:19:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:19:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:19:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:19:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:19:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:19:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:19:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:19:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:19:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:19:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:19:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:19:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:19:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:19:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:19:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:19:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:19:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:19:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:19:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:19:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:19:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:19:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:19:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:19:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:19:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:19:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:19:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:19:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:19:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:19:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:19:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:19:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:19:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:19:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:19:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:19:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:19:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:19:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:19:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:19:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:19:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:19:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:19:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:19:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:19:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:19:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:19:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:19:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:19:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:19:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:19:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:19:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:19:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:19:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:19:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:19:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:19:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:19:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:19:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:19:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:19:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:19:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:19:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:19:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:19:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:19:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:21:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:21:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:21:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:21:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:21:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:21:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:21:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:21:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:21:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:21:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:21:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:21:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:21:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:21:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:21:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:21:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:21:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:21:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:21:31 rdma06 rsyslogd[1041]: imjournal: 5300 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 19 04:21:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:21:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:21:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:21:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:21:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:21:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:21:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:21:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:21:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:21:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:21:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:21:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:21:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:21:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:21:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:21:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:21:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:21:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:21:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:21:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:21:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:21:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:21:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:21:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:21:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:21:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:21:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:21:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:21:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:21:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:21:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:21:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:21:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:21:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:21:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:21:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:21:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:21:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:21:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:21:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:21:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:21:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:21:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:21:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:21:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:21:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:21:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:21:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:21:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:21:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:21:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:21:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:21:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:21:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:21:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:21:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:21:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:21:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:21:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:21:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:21:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:21:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:21:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:21:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:21:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:21:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:21:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:21:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:21:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:21:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:21:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:21:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:21:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:21:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:21:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:21:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:21:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:21:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:21:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:21:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:21:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:21:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:21:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:21:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:21:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:21:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:21:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:21:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:21:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:21:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:22:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:22:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:22:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:22:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:22:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:22:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:22:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:22:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:22:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:22:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:22:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:22:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:22:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:22:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:22:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:22:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:22:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:22:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:22:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:22:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:22:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:22:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:22:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:22:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:22:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:22:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:22:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:22:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:22:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:22:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:22:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:22:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:22:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:22:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:22:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:22:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:22:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:22:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:22:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:22:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:22:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:22:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:22:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:22:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:22:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:22:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:22:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:22:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:22:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:22:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:22:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:22:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:22:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:22:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:22:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:22:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:22:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:22:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:22:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:22:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:22:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:22:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:22:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:22:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:22:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:22:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:22:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:22:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:22:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:22:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:22:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:22:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:22:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:22:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:22:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:22:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:22:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:22:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:22:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:22:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:22:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:22:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:22:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:22:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:22:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:22:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:22:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:22:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:22:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:22:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:22:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:22:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:22:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:22:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:22:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:22:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:22:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:22:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:22:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:22:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:22:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:22:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:22:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:22:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:22:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:22:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:22:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:22:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:22:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:22:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:22:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:22:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:22:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:22:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:22:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:22:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:22:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:22:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:22:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:22:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:22:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:22:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:22:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:22:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:22:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:22:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:22:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:22:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:22:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:22:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:22:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:22:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:22:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:22:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:22:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:22:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:22:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:22:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:22:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:22:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:22:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:22:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:22:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:22:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:22:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:22:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:22:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:22:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:22:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:22:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:22:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:22:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:22:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:22:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:22:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:22:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:22:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:22:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:22:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:22:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:22:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:22:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:22:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:22:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:22:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:22:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:22:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:22:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:22:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:22:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:22:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:22:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:22:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:22:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:22:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:22:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:22:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:22:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:22:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:22:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:22:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:22:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:22:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:22:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:22:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:22:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:22:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:22:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:22:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:22:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:22:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:22:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:22:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:22:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:22:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:22:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:22:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:22:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:22:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:22:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:22:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:22:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:22:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:22:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:22:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:22:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:22:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:22:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:22:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:22:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:22:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:22:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:22:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:22:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:22:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:22:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:23:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:23:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:23:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:23:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:23:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:23:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:23:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:23:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:23:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:23:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:23:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:23:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:23:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:23:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:23:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:23:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:23:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:23:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:23:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:23:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:23:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:23:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:23:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:23:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:23:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:23:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:23:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:23:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:23:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:23:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:23:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:23:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:23:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:23:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:23:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:23:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:23:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:23:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:23:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:23:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:23:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:23:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:23:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:23:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:23:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:23:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:23:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:23:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:23:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:23:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:23:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:23:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:23:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:23:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:23:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:23:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:23:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:23:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:23:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:23:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:23:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:23:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:23:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:23:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:23:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:23:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:23:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:23:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:23:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:23:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:23:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:23:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:23:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:23:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:23:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:23:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:23:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:23:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:23:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:23:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:23:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:23:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:23:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:23:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:23:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:23:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:23:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:23:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:23:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:23:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:23:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:23:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:23:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:23:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:23:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:23:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:23:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:23:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:23:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:23:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:23:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:23:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:23:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:23:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:23:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:23:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:23:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:23:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:23:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:23:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:23:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:23:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:23:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:23:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:23:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:23:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:23:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:23:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:23:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:23:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:23:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:23:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:23:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:23:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:23:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:23:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:23:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:23:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:23:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:23:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:23:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:23:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:23:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:23:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:23:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:23:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:23:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:23:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:23:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:23:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:23:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:23:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:23:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:23:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:23:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:23:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:23:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:23:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:23:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:23:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:23:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:23:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:23:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:23:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:23:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:23:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:23:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:23:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:23:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:23:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:23:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:23:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:23:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:23:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:23:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:23:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:23:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:23:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:23:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:23:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:23:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:23:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:23:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:23:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:23:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:23:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:23:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:23:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:23:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:23:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:23:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:23:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:23:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:23:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:23:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:23:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:23:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:23:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:23:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:23:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:23:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:23:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:23:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:23:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:23:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:23:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:23:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:23:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:23:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:23:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:23:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:23:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:23:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:23:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:23:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:23:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:23:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:23:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:23:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:23:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:23:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:23:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:23:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:23:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:23:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:23:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:24:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:24:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:24:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:24:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:24:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:24:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:24:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:24:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:24:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:24:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:24:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:24:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:24:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:24:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:24:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:24:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:24:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:24:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:24:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:24:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:24:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:24:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:24:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:24:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:24:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:24:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:24:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:24:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:24:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:24:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:24:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:24:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:24:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:24:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:24:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:24:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:24:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:24:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:24:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:24:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:24:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:24:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:24:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:24:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:24:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:24:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:24:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:24:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:24:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:24:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:24:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:24:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:24:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:24:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:24:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:24:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:24:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:24:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:24:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:24:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:24:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:24:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:24:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:24:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:24:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:24:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:24:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:24:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:24:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:24:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:24:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:24:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:24:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:24:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:24:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:24:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:24:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:24:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:24:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:24:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:24:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:24:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:24:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:24:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:24:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:24:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:24:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:24:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:24:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:24:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:24:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:24:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:24:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:24:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:24:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:24:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:24:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:24:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:24:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:24:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:24:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:24:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:24:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:24:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:24:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:24:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:24:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:24:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:24:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:24:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:24:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:24:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:24:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:24:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:24:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:24:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:24:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:24:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:24:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:24:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:24:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:24:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:24:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:24:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:24:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:24:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:24:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:24:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:24:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:24:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:24:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:24:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:24:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:24:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:24:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:24:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:24:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:24:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:24:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:24:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:24:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:24:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:24:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:24:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:24:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:24:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:24:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:24:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:24:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:24:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:24:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:24:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:24:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:24:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:24:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:24:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:24:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:24:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:24:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:24:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:24:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:24:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:24:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:24:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:24:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:24:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:24:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:24:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:24:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:24:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:24:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:24:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:24:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:24:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:24:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:24:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:24:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:24:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:24:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:24:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:24:46 rdma06 systemd[1]: Starting dnf makecache...
Apr 19 04:24:47 rdma06 dnf[1214184]: Updating Subscription Management repositories.
Apr 19 04:24:47 rdma06 dnf[1214184]: Unable to read consumer identity
Apr 19 04:24:47 rdma06 dnf[1214184]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 19 04:24:47 rdma06 dnf[1214184]: Metadata cache refreshed recently.
Apr 19 04:24:47 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 19 04:24:47 rdma06 systemd[1]: Finished dnf makecache.
Apr 19 04:24:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:24:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:24:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:24:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:24:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:24:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:24:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:24:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:24:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:24:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:24:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:24:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:24:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:24:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:24:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:24:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:24:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:24:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:24:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:24:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:24:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:24:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:24:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:24:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:24:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:24:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:24:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:24:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:24:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:24:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:24:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:24:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:24:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:24:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:24:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:24:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:25:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:25:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:25:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:25:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:25:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:25:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:25:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:25:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:25:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:25:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:25:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:25:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:25:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:25:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:25:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:25:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:25:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:25:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:25:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:25:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:25:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:25:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:25:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:25:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:25:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:25:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:25:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:25:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:25:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:25:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:25:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:25:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:25:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:25:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:25:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:25:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:25:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:25:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:25:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:25:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:25:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:25:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:25:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:25:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:25:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:25:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:25:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:25:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:25:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:25:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:25:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:25:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:25:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:25:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:25:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:25:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:25:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:25:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:25:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:25:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:25:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:25:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:25:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:25:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:25:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:25:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:25:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:25:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:25:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:25:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:25:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:25:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:25:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:25:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:25:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:25:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:25:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:25:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:25:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:25:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:25:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:25:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:25:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:25:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:25:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:25:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:25:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:25:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:25:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:25:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:25:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:25:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:25:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:25:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:25:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:25:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:25:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:25:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:25:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:25:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:25:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:25:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:25:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:25:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:25:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:25:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:25:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:25:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:25:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:25:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:25:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:25:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:25:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:25:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:25:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:25:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:25:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:25:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:25:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:25:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:25:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:25:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:25:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:25:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:25:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:25:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:25:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:25:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:25:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:25:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:25:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:25:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:25:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:25:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:25:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:25:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:25:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:25:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:25:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:25:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:25:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:25:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:25:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:25:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:25:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:25:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:25:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:25:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:25:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:25:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:25:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:25:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:25:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:25:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:25:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:25:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:25:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:25:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:25:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:25:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:25:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:25:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:25:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:25:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:25:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:25:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:25:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:25:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:25:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:25:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:25:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:25:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:25:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:25:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:25:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:25:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:25:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:25:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:25:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:25:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:25:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:25:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:25:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:25:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:25:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:25:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:25:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:25:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:25:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:25:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:25:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:25:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:25:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:25:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:25:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:25:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:25:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:25:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:25:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:25:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:25:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:25:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:25:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:25:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:25:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:25:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:25:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:25:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:25:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:25:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:25:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:25:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:25:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:25:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:25:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:25:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:26:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:26:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:26:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:26:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:26:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:26:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:26:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:26:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:26:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:26:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:26:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:26:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:26:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:26:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:26:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:26:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:26:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:26:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:26:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:26:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:26:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:26:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:26:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:26:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:26:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:26:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:26:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:26:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:26:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:26:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:26:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:26:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:26:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:26:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:26:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:26:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:26:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:26:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:26:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:26:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:26:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:26:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:26:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:26:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:26:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:26:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:26:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:26:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:26:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:26:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:26:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:26:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:26:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:26:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:26:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:26:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:26:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:26:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:26:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:26:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:26:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:26:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:26:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:26:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:26:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:26:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:26:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:26:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:26:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:26:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:26:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:26:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:26:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:26:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:26:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:26:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:26:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:26:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:26:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:26:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:26:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:26:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:26:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:26:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:26:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:26:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:26:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:26:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:26:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:26:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:26:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:26:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:26:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:26:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:26:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:26:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:26:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:26:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:26:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:26:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:26:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:26:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:26:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:26:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:26:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:26:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:26:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:26:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:26:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:26:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:26:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:26:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:26:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:26:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:26:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:26:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:26:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:26:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:26:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:26:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:26:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:26:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:26:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:26:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:26:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:26:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:26:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:26:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:26:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:26:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:26:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:26:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:26:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:26:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:26:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:26:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:26:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:26:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:26:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:26:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:26:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:26:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:26:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:26:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:26:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:26:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:26:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:26:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:26:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:26:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:26:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:26:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:26:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:26:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:26:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:26:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:26:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:26:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:26:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:26:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:26:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:26:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:26:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:26:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:26:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:26:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:26:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:26:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:26:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:26:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:26:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:26:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:26:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:26:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:26:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:26:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:26:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:26:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:26:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:26:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:26:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:26:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:26:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:26:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:26:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:26:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:26:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:26:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:26:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:26:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:26:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:26:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:26:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:26:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:26:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:26:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:26:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:26:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:26:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:26:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:26:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:26:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:26:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:26:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:26:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:26:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:26:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:26:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:26:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:26:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:26:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:26:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:26:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:26:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:26:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:26:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:27:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:27:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:27:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:27:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:27:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:27:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:27:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:27:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:27:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:27:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:27:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:27:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:27:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:27:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:27:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:27:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:27:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:27:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:27:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:27:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:27:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:27:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:27:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:27:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:27:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:27:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:27:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:27:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:27:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:27:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:27:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:27:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:27:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:27:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:27:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:27:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:27:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:27:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:27:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:27:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:27:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:27:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:27:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:27:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:27:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:27:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:27:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:27:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:27:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:27:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:27:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:27:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:27:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:27:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:27:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:27:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:27:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:27:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:27:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:27:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:27:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:27:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:27:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:27:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:27:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:27:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:27:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:27:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:27:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:27:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:27:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:27:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:27:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:27:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:27:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:27:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:27:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:27:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:27:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:27:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:27:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:27:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:27:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:27:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:27:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:27:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:27:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:27:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:27:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:27:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:27:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:27:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:27:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:27:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:27:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:27:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:27:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:27:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:27:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:27:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:27:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:27:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:27:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:27:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:27:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:27:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:27:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:27:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:27:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:27:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:27:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:27:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:27:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:27:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:27:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:27:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:27:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:27:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:27:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:27:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:27:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:27:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:27:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:27:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:27:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:27:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:27:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:27:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:27:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:27:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:27:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:27:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:27:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:27:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:27:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:27:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:27:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:27:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:27:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:27:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:27:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:27:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:27:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:27:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:27:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:27:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:27:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:27:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:27:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:27:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:27:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:27:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:27:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:27:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:27:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:27:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:27:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:27:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:27:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:27:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:27:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:27:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:27:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:27:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:27:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:27:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:27:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:27:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:27:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:27:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:27:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:27:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:27:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:27:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:27:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:27:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:27:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:27:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:27:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:27:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:27:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:27:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:27:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:27:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:27:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:27:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:27:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:27:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:27:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:27:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:27:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:27:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:27:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:27:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:27:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:27:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:27:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:27:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:27:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:27:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:27:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:27:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:27:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:27:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:27:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:27:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:27:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:27:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:27:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:27:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:27:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:27:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:27:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:27:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:27:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:27:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:28:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:28:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:28:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:28:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:28:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:28:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:28:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:28:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:28:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:28:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:28:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:28:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:28:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:28:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:28:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:28:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:28:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:28:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:28:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:28:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:28:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:28:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:28:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:28:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:28:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:28:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:28:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:28:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:28:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:28:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:28:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:28:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:28:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:28:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:28:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:28:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:28:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:28:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:28:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:28:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:28:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:28:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:28:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:28:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:28:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:28:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:28:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:28:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:28:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:28:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:28:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:28:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:28:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:28:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:28:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:28:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:28:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:28:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:28:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:28:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:28:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:28:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:28:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:28:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:28:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:28:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:28:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:28:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:28:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:28:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:28:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:28:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:28:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:28:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:28:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:28:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:28:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:28:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:28:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:28:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:28:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:28:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:28:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:28:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:28:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:28:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:28:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:28:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:28:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:28:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:28:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:28:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:28:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:28:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:28:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:28:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:28:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:28:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:28:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:28:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:28:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:28:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:28:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:28:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:28:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:28:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:28:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:28:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:28:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:28:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:28:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:28:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:28:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:28:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:28:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:28:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:28:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:28:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:28:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:28:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:28:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:28:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:28:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:28:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:28:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:28:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:28:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:28:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:28:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:28:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:28:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:28:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:28:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:28:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:28:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:28:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:28:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:28:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:28:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:28:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:28:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:28:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:28:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:28:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:28:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:28:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:28:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:28:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:28:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:28:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:28:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:28:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:28:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:28:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:28:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:28:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:28:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:28:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:28:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:28:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:28:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:28:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:28:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:28:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:28:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:28:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:28:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:28:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:28:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:28:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:28:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:28:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:28:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:28:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:28:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:28:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:28:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:28:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:28:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:28:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:28:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:28:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:28:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:28:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:28:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:28:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:28:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:28:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:28:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:28:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:28:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:28:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:28:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:28:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:28:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:28:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:28:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:28:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:28:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:28:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:28:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:28:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:28:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:28:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:28:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:28:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:28:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:28:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:28:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:28:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:28:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:28:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:28:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:28:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:28:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:28:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:29:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:29:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:29:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:29:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:29:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:29:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:29:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:29:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:29:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:29:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:29:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:29:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:29:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:29:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:29:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:29:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:29:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:29:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:29:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:29:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:29:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:29:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:29:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:29:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:29:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:29:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:29:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:29:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:29:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:29:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:29:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:29:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:29:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:29:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:29:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:29:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:29:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:29:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:29:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:29:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:29:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:29:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:29:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:29:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:29:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:29:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:29:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:29:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:29:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:29:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:29:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:29:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:29:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:29:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:29:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:29:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:29:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:29:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:29:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:29:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:29:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:29:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:29:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:29:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:29:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:29:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:29:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:29:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:29:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:29:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:29:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:29:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:29:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:29:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:29:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:29:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:29:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:29:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:29:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:29:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:29:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:29:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:29:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:29:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:29:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:29:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:29:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:29:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:29:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:29:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:29:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:29:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:29:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:29:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:29:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:29:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:29:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:29:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:29:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:29:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:29:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:29:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:29:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:29:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:29:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:29:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:29:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:29:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:31:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:31:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:31:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:31:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:31:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:31:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:31:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:31:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:31:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:31:32 rdma06 rsyslogd[1041]: imjournal: 5090 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 19 04:31:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:31:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:31:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:31:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:31:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:31:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:31:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:31:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:31:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:31:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:31:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:31:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:31:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:31:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:31:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:31:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:31:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:31:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:31:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:31:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:31:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:31:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:31:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:31:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:31:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:31:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:31:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:31:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:31:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:31:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:31:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:31:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:31:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:31:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:31:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:31:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:31:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:31:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:31:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:31:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:31:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:31:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:31:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:31:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:31:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:31:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:31:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:31:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:31:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:31:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:31:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:31:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:31:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:31:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:32:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:32:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:32:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:32:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:32:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:32:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:32:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:32:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:32:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:32:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:32:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:32:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:32:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:32:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:32:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:32:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:32:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:32:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:32:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:32:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:32:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:32:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:32:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:32:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:32:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:32:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:32:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:32:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:32:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:32:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:32:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:32:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:32:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:32:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:32:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:32:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:32:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:32:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:32:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:32:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:32:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:32:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:32:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:32:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:32:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:32:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:32:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:32:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:32:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:32:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:32:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:32:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:32:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:32:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:32:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:32:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:32:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:32:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:32:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:32:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:32:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:32:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:32:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:32:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:32:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:32:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:32:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:32:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:32:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:32:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:32:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:32:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:32:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:32:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:32:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:32:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:32:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:32:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:32:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:32:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:32:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:32:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:32:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:32:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:32:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:32:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:32:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:32:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:32:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:32:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:32:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:32:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:32:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:32:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:32:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:32:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:32:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:32:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:32:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:32:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:32:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:32:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:32:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:32:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:32:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:32:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:32:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:32:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:32:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:32:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:32:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:32:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:32:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:32:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:32:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:32:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:32:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:32:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:32:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:32:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:32:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:32:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:32:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:32:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:32:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:32:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:32:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:32:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:32:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:32:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:32:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:32:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:32:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:32:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:32:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:32:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:32:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:32:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:32:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:32:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:32:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:32:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:32:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:32:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:32:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:32:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:32:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:32:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:32:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:32:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:32:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:32:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:32:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:32:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:32:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:32:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:32:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:32:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:32:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:32:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:32:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:32:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:32:45 rdma06 journal[2657]: Adjustment to local watchdog ignored since 'no_localwatchdog' metadata is set
Apr 19 04:32:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:32:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:32:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:32:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:32:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:32:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:32:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:32:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:32:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:32:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:32:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:32:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:32:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:32:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:32:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:32:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:32:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:32:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:32:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:32:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:32:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:32:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:32:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:32:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:32:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:32:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:32:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:32:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:32:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:32:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:32:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:32:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:32:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:32:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:32:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:32:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:32:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:32:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:32:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:32:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:32:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:32:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:32:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:32:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:32:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:32:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:32:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:32:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:32:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:32:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:32:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:32:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:32:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:32:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:33:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:33:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:33:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:33:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:33:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:33:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:33:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:33:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:33:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:33:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:33:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:33:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:33:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:33:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:33:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:33:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:33:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:33:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:33:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:33:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:33:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:33:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:33:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:33:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:33:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:33:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:33:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:33:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:33:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:33:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:33:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:33:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:33:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:33:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:33:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:33:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:33:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:33:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:33:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:33:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:33:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:33:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:33:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:33:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:33:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:33:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:33:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:33:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:33:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:33:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:33:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:33:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:33:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:33:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:33:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:33:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:33:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:33:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:33:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:33:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:33:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:33:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:33:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:33:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:33:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:33:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:33:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:33:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:33:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:33:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:33:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:33:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:33:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:33:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:33:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:33:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:33:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:33:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:33:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:33:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:33:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:33:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:33:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:33:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:33:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:33:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:33:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:33:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:33:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:33:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:33:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:33:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:33:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:33:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:33:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:33:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:33:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:33:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:33:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:33:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:33:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:33:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:33:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:33:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:33:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:33:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:33:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:33:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:33:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:33:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:33:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:33:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:33:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:33:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:33:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:33:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:33:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:33:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:33:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:33:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:33:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:33:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:33:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:33:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:33:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:33:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:33:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:33:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:33:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:33:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:33:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:33:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:33:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:33:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:33:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:33:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:33:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:33:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:33:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:33:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:33:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:33:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:33:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:33:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:33:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:33:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:33:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:33:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:33:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:33:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:33:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:33:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:33:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:33:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:33:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:33:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:33:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:33:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:33:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:33:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:33:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:33:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:33:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:33:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:33:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:33:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:33:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:33:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:33:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:33:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:33:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:33:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:33:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:33:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:33:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:33:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:33:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:33:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:33:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:33:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:33:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:33:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:33:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:33:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:33:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:33:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:33:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:33:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:33:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:33:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:33:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:33:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:33:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:33:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:33:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:33:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:33:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:33:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:33:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:33:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:33:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:33:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:33:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:33:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:33:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:33:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:33:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:33:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:33:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:33:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:33:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:33:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:33:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:33:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:33:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:33:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:34:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:34:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:34:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:34:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:34:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:34:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:34:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:34:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:34:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:34:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:34:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:34:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:34:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:34:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:34:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:34:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:34:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:34:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:34:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:34:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:34:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:34:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:34:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:34:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:34:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:34:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:34:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:34:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:34:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:34:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:34:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:34:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:34:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:34:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:34:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:34:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:34:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:34:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:34:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:34:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:34:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:34:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:34:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:34:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:34:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:34:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:34:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:34:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:34:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:34:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:34:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:34:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:34:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:34:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:34:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:34:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:34:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:34:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:34:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:34:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:34:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:34:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:34:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:34:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:34:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:34:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:34:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:34:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:34:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:34:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:34:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:34:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:34:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:34:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:34:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:34:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:34:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:34:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:34:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:34:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:34:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:34:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:34:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:34:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:34:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:34:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:34:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:34:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:34:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:34:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:34:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:34:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:34:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:34:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:34:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:34:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:34:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:34:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:34:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:34:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:34:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:34:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:34:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:34:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:34:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:34:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:34:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:34:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:34:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:34:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:34:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:34:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:34:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:34:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:34:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:34:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:34:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:34:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:34:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:34:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:34:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:34:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:34:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:34:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:34:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:34:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:34:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:34:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:34:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:34:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:34:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:34:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:34:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:34:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:34:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:34:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:34:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:34:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:34:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:34:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:34:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:34:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:34:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:34:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:34:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:34:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:34:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:34:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:34:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:34:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:34:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:34:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:34:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:34:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:34:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:34:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:34:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:34:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:34:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:34:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:34:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:34:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:34:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:34:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:34:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:34:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:34:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:34:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:34:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:34:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:34:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:34:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:34:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:34:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:34:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:34:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:34:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:34:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:34:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:34:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:34:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:34:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:34:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:34:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:34:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:34:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:34:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:34:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:34:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:34:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:34:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:34:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:34:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:34:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:34:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:34:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:34:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:34:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:34:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:34:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:34:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:34:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:34:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:34:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:34:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:34:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:34:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:34:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:34:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:34:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:34:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:34:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:34:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:34:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:34:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:34:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:35:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:35:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:35:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:35:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:35:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:35:01 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:35:01 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:35:01 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:35:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:35:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:35:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:35:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:35:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:35:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:35:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:35:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:35:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:35:01 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:35:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:35:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:35:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:35:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:35:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:35:06 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:35:06 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:35:06 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:35:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:35:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:35:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:35:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:35:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:35:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:35:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:35:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:35:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:35:06 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:35:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:35:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:35:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:35:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:35:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:35:11 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:35:11 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:35:11 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:35:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:35:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:35:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:35:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:35:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:35:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:35:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:35:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:35:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:35:11 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:35:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:35:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:35:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:35:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:35:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:35:16 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:35:16 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:35:16 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:35:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:35:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:35:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:35:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:35:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:35:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:35:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:35:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:35:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:35:16 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:35:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:35:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:35:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:35:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:35:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:35:21 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:35:21 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:35:21 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:35:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:35:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:35:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:35:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:35:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:35:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:35:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:35:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:35:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:35:21 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:35:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:35:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:35:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:35:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:35:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:35:26 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:35:26 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:35:26 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:35:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:35:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:35:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:35:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:35:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:35:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:35:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:35:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:35:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:35:26 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:35:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:35:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:35:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:35:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:35:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:35:31 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:35:31 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:35:31 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:35:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:35:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:35:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:35:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:35:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:35:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:35:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:35:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:35:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:35:31 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:35:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:35:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:35:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:35:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:35:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:35:36 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:35:36 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:35:36 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:35:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:35:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:35:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:35:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:35:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:35:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:35:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:35:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:35:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:35:36 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:35:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:35:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:35:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:35:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:35:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:35:41 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:35:41 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:35:41 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:35:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:35:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:35:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:35:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:35:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:35:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:35:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:35:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:35:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:35:41 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:35:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:35:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:35:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:35:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:35:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:35:46 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:35:46 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:35:46 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:35:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:35:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:35:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:35:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:35:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:35:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:35:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:35:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:35:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:35:46 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:35:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:35:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:35:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:35:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:35:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:35:51 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:35:51 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:35:51 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:35:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:35:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:35:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:35:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:35:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:35:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:35:51 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:35:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:35:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:35:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:35:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:35:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:35:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:35:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:35:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:35:56 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:35:56 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:35:56 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:35:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:35:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:35:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:35:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:35:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:35:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:35:56 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:35:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:35:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:35:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:36:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:36:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:36:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:36:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:36:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:36:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:36:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:36:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:36:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:36:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:36:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:36:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:36:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:36:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:36:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:36:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:36:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:36:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:36:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:36:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:36:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:36:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:36:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:36:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:36:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:36:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:36:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:36:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:36:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:36:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:36:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:36:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:36:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:36:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:36:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:36:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:36:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:36:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:36:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:36:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:36:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:36:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:36:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:36:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:36:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:36:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:36:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:36:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:36:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:36:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:36:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:36:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:36:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:36:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:36:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:36:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:36:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:36:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:36:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:36:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:36:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:36:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:36:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:36:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:36:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:36:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:36:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:36:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:36:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:36:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:36:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:36:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:36:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:36:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:36:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:36:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:36:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:36:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:36:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:36:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:36:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:36:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:36:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:36:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:36:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:36:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:36:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:36:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:36:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:36:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:36:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:36:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:36:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:36:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:36:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:36:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:36:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:36:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:36:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:36:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:36:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:36:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:36:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:36:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:36:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:36:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:36:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:36:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:36:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:36:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:36:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:36:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:36:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:36:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:36:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:36:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:36:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:36:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:36:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:36:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:36:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:36:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:36:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:36:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:36:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:36:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:36:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:36:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:36:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:36:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:36:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:36:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:36:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:36:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:36:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:36:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:36:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:36:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:36:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:36:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:36:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:36:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:36:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:36:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:36:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:36:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:36:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:36:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:36:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:36:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:36:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:36:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:36:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:36:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:36:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:36:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:36:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:36:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:36:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:36:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:36:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:36:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:36:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:36:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:36:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:36:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:36:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:36:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:36:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:36:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:36:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:36:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:36:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:36:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:36:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:36:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:36:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:36:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:36:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:36:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:36:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:36:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:36:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:36:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:36:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:36:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:36:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:36:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:36:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:36:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:36:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:36:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:36:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:36:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:36:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:36:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:36:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:36:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:36:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:36:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:36:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:36:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:36:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:36:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:36:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:36:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:36:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:36:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:36:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:36:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:36:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:36:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:36:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:36:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:36:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:36:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:37:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:37:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:37:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:37:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:37:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:37:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:37:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:37:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:37:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:37:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:37:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:37:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:37:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:37:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:37:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:37:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:37:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:37:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:37:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:37:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:37:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:37:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:37:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:37:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:37:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:37:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:37:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:37:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:37:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:37:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:37:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:37:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:37:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:37:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:37:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:37:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:37:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:37:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:37:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:37:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:37:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:37:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:37:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:37:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:37:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:37:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:37:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:37:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:37:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:37:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:37:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:37:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:37:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:37:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:37:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:37:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:37:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:37:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:37:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:37:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:37:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:37:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:37:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:37:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:37:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:37:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:37:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:37:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:37:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:37:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:37:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:37:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:37:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:37:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:37:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:37:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:37:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:37:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:37:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:37:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:37:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:37:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:37:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:37:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:37:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:37:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:37:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:37:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:37:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:37:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:37:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:37:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:37:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:37:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:37:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:37:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:37:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:37:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:37:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:37:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:37:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:37:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:37:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:37:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:37:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:37:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:37:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:37:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:37:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:37:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:37:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:37:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:37:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:37:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:37:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:37:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:37:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:37:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:37:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:37:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:37:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:37:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:37:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:37:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:37:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:37:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:37:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:37:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:37:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:37:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:37:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:37:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:37:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:37:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:37:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:37:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:37:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:37:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:37:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:37:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:37:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:37:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:37:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:37:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:37:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:37:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:37:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:37:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:37:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:37:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:37:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:37:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:37:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:37:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:37:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:37:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:37:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:37:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:37:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:37:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:37:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:37:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:37:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:37:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:37:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:37:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:37:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:37:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:37:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:37:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:37:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:37:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:37:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:37:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:37:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:37:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:37:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:37:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:37:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:37:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:37:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:37:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:37:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:37:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:37:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:37:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:37:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:37:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:37:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:37:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:37:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:37:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:37:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:37:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:37:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:37:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:37:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:37:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:37:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:37:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:37:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:37:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:37:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:37:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:37:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:37:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:37:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:37:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:37:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:37:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:37:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:37:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:37:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:37:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:37:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:37:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:38:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:38:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:38:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:38:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:38:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:38:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:38:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:38:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:38:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:38:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:38:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:38:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:38:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:38:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:38:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:38:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:38:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:38:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:38:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:38:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:38:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:38:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:38:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:38:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:38:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:38:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:38:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:38:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:38:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:38:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:38:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:38:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:38:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:38:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:38:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:38:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:38:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:38:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:38:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:38:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:38:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:38:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:38:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:38:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:38:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:38:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:38:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:38:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:38:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:38:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:38:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:38:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:38:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:38:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:38:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:38:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:38:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:38:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:38:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:38:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:38:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:38:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:38:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:38:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:38:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:38:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:38:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:38:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:38:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:38:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:38:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:38:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:38:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:38:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:38:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:38:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:38:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:38:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:38:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:38:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:38:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:38:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:38:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:38:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:38:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:38:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:38:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:38:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:38:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:38:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:38:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:38:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:38:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:38:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:38:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:38:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:38:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:38:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:38:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:38:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:38:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:38:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:38:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:38:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:38:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:38:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:38:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:38:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:38:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:38:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:38:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:38:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:38:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:38:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:38:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:38:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:38:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:38:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:38:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:38:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:38:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:38:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:38:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:38:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:38:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:38:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:38:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:38:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:38:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:38:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:38:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:38:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:38:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:38:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:38:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:38:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:38:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:38:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:38:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:38:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:38:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:38:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:38:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:38:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:38:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:38:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:38:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:38:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:38:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:38:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:38:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:38:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:38:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:38:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:38:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:38:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:38:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:38:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:38:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:38:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:38:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:38:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:38:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:38:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:38:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:38:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:38:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:38:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:38:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:38:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:38:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:38:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:38:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:38:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:38:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:38:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:38:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:38:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:38:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:38:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:38:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:38:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:38:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:38:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:38:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:38:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:38:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:38:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:38:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:38:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:38:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:38:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:38:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:38:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:38:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:38:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:38:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:38:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:38:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:38:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:38:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:38:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:38:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:38:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:38:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:38:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:38:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:38:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:38:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:38:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:38:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:38:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:38:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:38:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:38:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:38:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:39:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:39:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:39:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:39:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:39:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:39:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:39:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:39:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:39:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:39:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:39:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:39:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:39:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:39:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:39:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:39:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:39:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:39:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:39:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:39:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:39:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:39:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:39:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:39:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:39:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:39:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:39:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:39:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:39:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:39:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:39:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:39:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:39:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:39:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:39:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:39:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:39:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:39:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:39:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:39:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:39:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:39:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:39:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:39:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:39:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:39:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:39:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:39:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:39:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:39:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:39:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:39:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:39:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:39:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:39:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:39:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:39:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:39:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:39:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:39:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:39:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:39:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:39:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:39:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:39:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:39:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:39:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:39:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:39:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:39:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:39:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:39:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:39:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:39:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:39:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:39:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:39:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:39:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:39:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:39:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:39:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:39:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:39:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:39:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:39:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:39:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:39:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:39:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:39:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:39:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:39:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:39:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:39:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:39:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:39:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:39:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:39:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:39:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:39:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:39:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:39:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:39:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:39:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:39:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:39:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:39:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:39:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:39:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:41:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:41:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:41:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:41:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:41:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:41:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:41:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:41:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:41:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:41:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:41:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:41:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:41:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:41:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:41:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:41:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:41:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:41:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:41:37 rdma06 rsyslogd[1041]: imjournal: 5292 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 19 04:41:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:41:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:41:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:41:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:41:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:41:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:41:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:41:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:41:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:41:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:41:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:41:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:41:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:41:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:41:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:41:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:41:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:41:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:41:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:41:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:41:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:41:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:41:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:41:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:41:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:41:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:41:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:41:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:41:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:41:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:41:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:41:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:41:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:41:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:41:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:41:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:41:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:41:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:41:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:41:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:41:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:41:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:41:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:41:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:41:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:41:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:41:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:41:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:41:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:41:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:41:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:41:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:41:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:41:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:41:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:41:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:41:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:41:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:41:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:41:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:41:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:41:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:41:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:41:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:41:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:41:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:41:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:41:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:41:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:41:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:41:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:41:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:42:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:42:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:42:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:42:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:42:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:42:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:42:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:42:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:42:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:42:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:42:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:42:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:42:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:42:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:42:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:42:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:42:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:42:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:42:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:42:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:42:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:42:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:42:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:42:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:42:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:42:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:42:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:42:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:42:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:42:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:42:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:42:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:42:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:42:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:42:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:42:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:42:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:42:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:42:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:42:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:42:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:42:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:42:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:42:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:42:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:42:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:42:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:42:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:42:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:42:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:42:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:42:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:42:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:42:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:42:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:42:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:42:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:42:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:42:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:42:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:42:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:42:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:42:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:42:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:42:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:42:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:42:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:42:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:42:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:42:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:42:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:42:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:42:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:42:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:42:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:42:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:42:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:42:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:42:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:42:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:42:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:42:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:42:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:42:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:42:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:42:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:42:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:42:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:42:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:42:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:42:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:42:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:42:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:42:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:42:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:42:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:42:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:42:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:42:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:42:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:42:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:42:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:42:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:42:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:42:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:42:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:42:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:42:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:42:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:42:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:42:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:42:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:42:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:42:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:42:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:42:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:42:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:42:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:42:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:42:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:42:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:42:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:42:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:42:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:42:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:42:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:42:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:42:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:42:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:42:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:42:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:42:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:42:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:42:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:42:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:42:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:42:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:42:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:42:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:42:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:42:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:42:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:42:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:42:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:42:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:42:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:42:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:42:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:42:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:42:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:42:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:42:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:42:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:42:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:42:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:42:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:42:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:42:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:42:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:42:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:42:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:42:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:42:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:42:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:42:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:42:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:42:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:42:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:42:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:42:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:42:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:42:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:42:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:42:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:42:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:42:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:42:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:42:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:42:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:42:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:42:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:42:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:42:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:42:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:42:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:42:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:42:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:42:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:42:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:42:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:42:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:42:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:42:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:42:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:42:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:42:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:42:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:42:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:42:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:42:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:42:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:42:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:42:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:42:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:42:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:42:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:42:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:42:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:42:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:42:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:42:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:42:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:42:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:42:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:42:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:42:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:43:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:43:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:43:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:43:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:43:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:43:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:43:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:43:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:43:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:43:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:43:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:43:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:43:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:43:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:43:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:43:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:43:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:43:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:43:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:43:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:43:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:43:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:43:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:43:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:43:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:43:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:43:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:43:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:43:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:43:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:43:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:43:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:43:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:43:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:43:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:43:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:43:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:43:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:43:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:43:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:43:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:43:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:43:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:43:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:43:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:43:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:43:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:43:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:43:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:43:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:43:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:43:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:43:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:43:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:43:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:43:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:43:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:43:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:43:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:43:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:43:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:43:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:43:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:43:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:43:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:43:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:43:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:43:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:43:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:43:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:43:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:43:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:43:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:43:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:43:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:43:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:43:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:43:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:43:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:43:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:43:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:43:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:43:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:43:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:43:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:43:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:43:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:43:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:43:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:43:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:43:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:43:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:43:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:43:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:43:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:43:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:43:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:43:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:43:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:43:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:43:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:43:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:43:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:43:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:43:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:43:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:43:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:43:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:43:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:43:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:43:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:43:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:43:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:43:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:43:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:43:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:43:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:43:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:43:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:43:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:43:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:43:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:43:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:43:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:43:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:43:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:43:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:43:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:43:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:43:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:43:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:43:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:43:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:43:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:43:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:43:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:43:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:43:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:43:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:43:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:43:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:43:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:43:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:43:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:43:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:43:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:43:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:43:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:43:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:43:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:43:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:43:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:43:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:43:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:43:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:43:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:43:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:43:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:43:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:43:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:43:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:43:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:43:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:43:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:43:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:43:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:43:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:43:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:43:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:43:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:43:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:43:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:43:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:43:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:43:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:43:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:43:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:43:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:43:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:43:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:43:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:43:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:43:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:43:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:43:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:43:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:43:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:43:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:43:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:43:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:43:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:43:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:43:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:43:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:43:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:43:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:43:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:43:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:43:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:43:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:43:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:43:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:43:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:43:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:43:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:43:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:43:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:43:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:43:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:43:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:43:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:43:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:43:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:43:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:43:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:43:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:44:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:44:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:44:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:44:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:44:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:44:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:44:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:44:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:44:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:44:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:44:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:44:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:44:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:44:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:44:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:44:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:44:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:44:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:44:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:44:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:44:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:44:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:44:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:44:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:44:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:44:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:44:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:44:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:44:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:44:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:44:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:44:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:44:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:44:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:44:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:44:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:44:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:44:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:44:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:44:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:44:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:44:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:44:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:44:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:44:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:44:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:44:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:44:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:44:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:44:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:44:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:44:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:44:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:44:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:44:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:44:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:44:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:44:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:44:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:44:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:44:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:44:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:44:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:44:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:44:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:44:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:44:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:44:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:44:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:44:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:44:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:44:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:44:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:44:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:44:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:44:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:44:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:44:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:44:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:44:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:44:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:44:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:44:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:44:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:44:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:44:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:44:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:44:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:44:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:44:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:44:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:44:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:44:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:44:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:44:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:44:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:44:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:44:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:44:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:44:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:44:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:44:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:44:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:44:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:44:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:44:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:44:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:44:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:44:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:44:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:44:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:44:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:44:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:44:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:44:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:44:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:44:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:44:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:44:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:44:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:44:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:44:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:44:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:44:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:44:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:44:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:44:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:44:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:44:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:44:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:44:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:44:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:44:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:44:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:44:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:44:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:44:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:44:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:44:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:44:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:44:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:44:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:44:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:44:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:44:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:44:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:44:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:44:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:44:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:44:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:44:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:44:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:44:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:44:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:44:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:44:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:44:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:44:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:44:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:44:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:44:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:44:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:44:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:44:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:44:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:44:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:44:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:44:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:44:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:44:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:44:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:44:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:44:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:44:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:44:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:44:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:44:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:44:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:44:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:44:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:44:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:44:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:44:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:44:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:44:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:44:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:44:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:44:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:44:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:44:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:44:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:44:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:44:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:44:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:44:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:44:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:44:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:44:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:44:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:44:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:44:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:44:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:44:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:44:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:44:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:44:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:44:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:44:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:44:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:44:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:44:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:44:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:44:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:44:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:44:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:44:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:45:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:45:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:45:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:45:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:45:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:45:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:45:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:45:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:45:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:45:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:45:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:45:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:45:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:45:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:45:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:45:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:45:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:45:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:45:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:45:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:45:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:45:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:45:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:45:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:45:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:45:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:45:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:45:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:45:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:45:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:45:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:45:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:45:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:45:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:45:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:45:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:45:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:45:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:45:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:45:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:45:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:45:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:45:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:45:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:45:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:45:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:45:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:45:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:45:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:45:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:45:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:45:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:45:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:45:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:45:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:45:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:45:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:45:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:45:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:45:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:45:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:45:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:45:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:45:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:45:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:45:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:45:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:45:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:45:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:45:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:45:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:45:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:45:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:45:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:45:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:45:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:45:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:45:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:45:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:45:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:45:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:45:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:45:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:45:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:45:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:45:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:45:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:45:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:45:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:45:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:45:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:45:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:45:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:45:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:45:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:45:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:45:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:45:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:45:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:45:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:45:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:45:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:45:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:45:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:45:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:45:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:45:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:45:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:45:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:45:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:45:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:45:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:45:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:45:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:45:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:45:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:45:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:45:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:45:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:45:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:45:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:45:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:45:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:45:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:45:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:45:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:45:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:45:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:45:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:45:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:45:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:45:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:45:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:45:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:45:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:45:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:45:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:45:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:45:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:45:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:45:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:45:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:45:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:45:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:45:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:45:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:45:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:45:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:45:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:45:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:45:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:45:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:45:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:45:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:45:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:45:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:45:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:45:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:45:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:45:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:45:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:45:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:45:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:45:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:45:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:45:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:45:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:45:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:45:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:45:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:45:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:45:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:45:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:45:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:45:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:45:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:45:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:45:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:45:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:45:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:45:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:45:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:45:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:45:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:45:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:45:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:45:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:45:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:45:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:45:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:45:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:45:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:45:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:45:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:45:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:45:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:45:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:45:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:45:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:45:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:45:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:45:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:45:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:45:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:45:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:45:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:45:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:45:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:45:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:45:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:45:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:45:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:45:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:45:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:45:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:45:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:46:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:46:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:46:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:46:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:46:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:46:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:46:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:46:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:46:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:46:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:46:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:46:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:46:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:46:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:46:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:46:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:46:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:46:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:46:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:46:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:46:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:46:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:46:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:46:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:46:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:46:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:46:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:46:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:46:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:46:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:46:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:46:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:46:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:46:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:46:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:46:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:46:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:46:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:46:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:46:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:46:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:46:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:46:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:46:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:46:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:46:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:46:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:46:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:46:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:46:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:46:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:46:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:46:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:46:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:46:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:46:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:46:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:46:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:46:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:46:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:46:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:46:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:46:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:46:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:46:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:46:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:46:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:46:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:46:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:46:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:46:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:46:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:46:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:46:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:46:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:46:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:46:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:46:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:46:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:46:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:46:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:46:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:46:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:46:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:46:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:46:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:46:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:46:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:46:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:46:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:46:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:46:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:46:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:46:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:46:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:46:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:46:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:46:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:46:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:46:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:46:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:46:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:46:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:46:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:46:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:46:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:46:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:46:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:46:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:46:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:46:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:46:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:46:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:46:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:46:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:46:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:46:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:46:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:46:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:46:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:46:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:46:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:46:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:46:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:46:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:46:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:46:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:46:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:46:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:46:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:46:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:46:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:46:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:46:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:46:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:46:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:46:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:46:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:46:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:46:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:46:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:46:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:46:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:46:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:46:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:46:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:46:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:46:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:46:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:46:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:46:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:46:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:46:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:46:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:46:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:46:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:46:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:46:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:46:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:46:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:46:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:46:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:46:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:46:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:46:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:46:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:46:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:46:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:46:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:46:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:46:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:46:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:46:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:46:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:46:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:46:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:46:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:46:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:46:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:46:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:46:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:46:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:46:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:46:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:46:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:46:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:46:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:46:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:46:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:46:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:46:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:46:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:46:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:46:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:46:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:46:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:46:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:46:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:46:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:46:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:46:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:46:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:46:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:46:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:46:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:46:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:46:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:46:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:46:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:46:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:46:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:46:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:46:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:46:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:46:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:46:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:47:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:47:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:47:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:47:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:47:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:47:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:47:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:47:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:47:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:47:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:47:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:47:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:47:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:47:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:47:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:47:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:47:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:47:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:47:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:47:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:47:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:47:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:47:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:47:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:47:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:47:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:47:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:47:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:47:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:47:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:47:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:47:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:47:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:47:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:47:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:47:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:47:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:47:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:47:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:47:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:47:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:47:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:47:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:47:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:47:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:47:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:47:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:47:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:47:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:47:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:47:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:47:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:47:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:47:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:47:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:47:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:47:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:47:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:47:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:47:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:47:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:47:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:47:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:47:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:47:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:47:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:47:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:47:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:47:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:47:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:47:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:47:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:47:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:47:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:47:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:47:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:47:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:47:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:47:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:47:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:47:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:47:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:47:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:47:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:47:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:47:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:47:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:47:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:47:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:47:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:47:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:47:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:47:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:47:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:47:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:47:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:47:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:47:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:47:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:47:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:47:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:47:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:47:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:47:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:47:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:47:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:47:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:47:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:47:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:47:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:47:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:47:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:47:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:47:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:47:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:47:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:47:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:47:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:47:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:47:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:47:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:47:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:47:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:47:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:47:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:47:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:47:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:47:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:47:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:47:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:47:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:47:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:47:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:47:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:47:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:47:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:47:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:47:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:47:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:47:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:47:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:47:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:47:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:47:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:47:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:47:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:47:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:47:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:47:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:47:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:47:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:47:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:47:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:47:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:47:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:47:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:47:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:47:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:47:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:47:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:47:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:47:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:47:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:47:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:47:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:47:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:47:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:47:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:47:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:47:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:47:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:47:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:47:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:47:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:47:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:47:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:47:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:47:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:47:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:47:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:47:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:47:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:47:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:47:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:47:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:47:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:47:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:47:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:47:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:47:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:47:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:47:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:47:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:47:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:47:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:47:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:47:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:47:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:47:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:47:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:47:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:47:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:47:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:47:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:47:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:47:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:47:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:47:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:47:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:47:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:47:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:47:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:47:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:47:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:47:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:47:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:48:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:48:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:48:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:48:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:48:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:48:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:48:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:48:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:48:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:48:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:48:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:48:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:48:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:48:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:48:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:48:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:48:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:48:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:48:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:48:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:48:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:48:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:48:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:48:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:48:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:48:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:48:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:48:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:48:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:48:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:48:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:48:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:48:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:48:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:48:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:48:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:48:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:48:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:48:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:48:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:48:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:48:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:48:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:48:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:48:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:48:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:48:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:48:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:48:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:48:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:48:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:48:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:48:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:48:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:48:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:48:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:48:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:48:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:48:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:48:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:48:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:48:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:48:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:48:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:48:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:48:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:48:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:48:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:48:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:48:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:48:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:48:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:48:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:48:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:48:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:48:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:48:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:48:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:48:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:48:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:48:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:48:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:48:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:48:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:48:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:48:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:48:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:48:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:48:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:48:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:48:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:48:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:48:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:48:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:48:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:48:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:48:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:48:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:48:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:48:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:48:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:48:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:48:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:48:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:48:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:48:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:48:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:48:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:48:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:48:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:48:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:48:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:48:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:48:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:48:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:48:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:48:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:48:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:48:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:48:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:48:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:48:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:48:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:48:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:48:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:48:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:48:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:48:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:48:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:48:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:48:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:48:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:48:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:48:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:48:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:48:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:48:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:48:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:48:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:48:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:48:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:48:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:48:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:48:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:48:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:48:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:48:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:48:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:48:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:48:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:48:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:48:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:48:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:48:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:48:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:48:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:48:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:48:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:48:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:48:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:48:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:48:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:48:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:48:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:48:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:48:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:48:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:48:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:48:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:48:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:48:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:48:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:48:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:48:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:48:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:48:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:48:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:48:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:48:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:48:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:48:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:48:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:48:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:48:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:48:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:48:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:48:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:48:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:48:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:48:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:48:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:48:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:48:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:48:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:48:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:48:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:48:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:48:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:48:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:48:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:48:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:48:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:48:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:48:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:48:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:48:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:48:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:48:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:48:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:48:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:48:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:48:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:48:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:48:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:48:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:48:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:49:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:49:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:49:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:49:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:49:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:49:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:49:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:49:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:49:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:49:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:49:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:49:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:49:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:49:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:49:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:49:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:49:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:49:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:49:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:49:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:49:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:49:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:49:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:49:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:49:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:49:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:49:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:49:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:49:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:49:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:49:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:49:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:49:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:49:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:49:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:49:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:49:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:49:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:49:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:49:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:49:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:49:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:49:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:49:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:49:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:49:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:49:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:49:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:49:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:49:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:49:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:49:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:49:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:49:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:49:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:49:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:49:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:49:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:49:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:49:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:49:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:49:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:49:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:49:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:49:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:49:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:49:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:49:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:49:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:49:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:49:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:49:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:49:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:49:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:49:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:49:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:49:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:49:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:49:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:49:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:49:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:49:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:49:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:49:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:49:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:49:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:49:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:49:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:49:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:49:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:49:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:49:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:49:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:49:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:49:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:49:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:49:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:49:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:49:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:49:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:49:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:49:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:49:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:49:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:49:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:49:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:49:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:49:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:49:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:49:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:49:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:49:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:49:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:49:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:49:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:49:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:49:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:49:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:49:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:49:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:49:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:49:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:49:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:49:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:49:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:49:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:51:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:51:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:51:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:51:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:51:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:51:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:51:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:51:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:51:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:51:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:51:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:51:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:51:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:51:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:51:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:51:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:51:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:51:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:51:42 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 19 04:51:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:51:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:51:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:51:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:51:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:51:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:51:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:51:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:51:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:51:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:51:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:51:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:51:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:51:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:51:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:51:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:51:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:51:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:51:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:51:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:51:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:51:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:51:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:51:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:51:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:51:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:51:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:51:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:51:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:51:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:51:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:51:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:51:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:51:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:51:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:51:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:51:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:51:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:51:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:51:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:51:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:51:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:51:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:51:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:51:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:51:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:51:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:51:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:51:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:51:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:51:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:51:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:51:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:51:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:52:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:52:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:52:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:52:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:52:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:52:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:52:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:52:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:52:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:52:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:52:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:52:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:52:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:52:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:52:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:52:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:52:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:52:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:52:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:52:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:52:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:52:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:52:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:52:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:52:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:52:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:52:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:52:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:52:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:52:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:52:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:52:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:52:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:52:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:52:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:52:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:52:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:52:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:52:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:52:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:52:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:52:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:52:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:52:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:52:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:52:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:52:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:52:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:52:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:52:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:52:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:52:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:52:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:52:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:52:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:52:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:52:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:52:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:52:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:52:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:52:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:52:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:52:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:52:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:52:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:52:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:52:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:52:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:52:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:52:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:52:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:52:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:52:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:52:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:52:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:52:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:52:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:52:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:52:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:52:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:52:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:52:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:52:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:52:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:52:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:52:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:52:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:52:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:52:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:52:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:52:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:52:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:52:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:52:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:52:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:52:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:52:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:52:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:52:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:52:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:52:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:52:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:52:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:52:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:52:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:52:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:52:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:52:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:52:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:52:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:52:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:52:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:52:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:52:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:52:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:52:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:52:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:52:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:52:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:52:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:52:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:52:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:52:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:52:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:52:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:52:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:52:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:52:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:52:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:52:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:52:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:52:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:52:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:52:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:52:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:52:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:52:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:52:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:52:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:52:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:52:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:52:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:52:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:52:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:52:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:52:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:52:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:52:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:52:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:52:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:52:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:52:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:52:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:52:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:52:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:52:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:52:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:52:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:52:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:52:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:52:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:52:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:52:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:52:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:52:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:52:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:52:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:52:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:52:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:52:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:52:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:52:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:52:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:52:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:52:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:52:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:52:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:52:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:52:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:52:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:52:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:52:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:52:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:52:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:52:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:52:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:52:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:52:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:52:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:52:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:52:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:52:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:52:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:52:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:52:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:52:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:52:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:52:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:52:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:52:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:52:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:52:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:52:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:52:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:52:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:52:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:52:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:52:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:52:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:52:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:52:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:52:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:52:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:52:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:52:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:52:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:53:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:53:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:53:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:53:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:53:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:53:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:53:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:53:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:53:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:53:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:53:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:53:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:53:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:53:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:53:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:53:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:53:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:53:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:53:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:53:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:53:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:53:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:53:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:53:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:53:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:53:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:53:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:53:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:53:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:53:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:53:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:53:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:53:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:53:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:53:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:53:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:53:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:53:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:53:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:53:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:53:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:53:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:53:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:53:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:53:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:53:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:53:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:53:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:53:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:53:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:53:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:53:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:53:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:53:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:53:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:53:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:53:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:53:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:53:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:53:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:53:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:53:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:53:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:53:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:53:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:53:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:53:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:53:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:53:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:53:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:53:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:53:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:53:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:53:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:53:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:53:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:53:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:53:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:53:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:53:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:53:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:53:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:53:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:53:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:53:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:53:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:53:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:53:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:53:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:53:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:53:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:53:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:53:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:53:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:53:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:53:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:53:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:53:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:53:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:53:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:53:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:53:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:53:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:53:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:53:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:53:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:53:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:53:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:53:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:53:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:53:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:53:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:53:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:53:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:53:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:53:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:53:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:53:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:53:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:53:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:53:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:53:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:53:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:53:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:53:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:53:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:53:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:53:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:53:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:53:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:53:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:53:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:53:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:53:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:53:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:53:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:53:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:53:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:53:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:53:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:53:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:53:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:53:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:53:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:53:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:53:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:53:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:53:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:53:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:53:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:53:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:53:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:53:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:53:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:53:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:53:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:53:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:53:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:53:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:53:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:53:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:53:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:53:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:53:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:53:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:53:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:53:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:53:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:53:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:53:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:53:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:53:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:53:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:53:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:53:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:53:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:53:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:53:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:53:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:53:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:53:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:53:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:53:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:53:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:53:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:53:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:53:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:53:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:53:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:53:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:53:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:53:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:53:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:53:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:53:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:53:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:53:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:53:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:53:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:53:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:53:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:53:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:53:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:53:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:53:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:53:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:53:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:53:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:53:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:53:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:53:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:53:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:53:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:53:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:53:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:53:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:54:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:54:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:54:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:54:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:54:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:54:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:54:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:54:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:54:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:54:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:54:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:54:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:54:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:54:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:54:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:54:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:54:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:54:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:54:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:54:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:54:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:54:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:54:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:54:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:54:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:54:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:54:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:54:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:54:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:54:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:54:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:54:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:54:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:54:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:54:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:54:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:54:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:54:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:54:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:54:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:54:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:54:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:54:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:54:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:54:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:54:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:54:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:54:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:54:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:54:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:54:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:54:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:54:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:54:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:54:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:54:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:54:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:54:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:54:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:54:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:54:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:54:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:54:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:54:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:54:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:54:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:54:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:54:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:54:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:54:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:54:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:54:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:54:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:54:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:54:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:54:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:54:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:54:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:54:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:54:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:54:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:54:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:54:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:54:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:54:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:54:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:54:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:54:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:54:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:54:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:54:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:54:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:54:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:54:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:54:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:54:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:54:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:54:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:54:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:54:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:54:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:54:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:54:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:54:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:54:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:54:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:54:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:54:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:54:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:54:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:54:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:54:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:54:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:54:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:54:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:54:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:54:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:54:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:54:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:54:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:54:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:54:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:54:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:54:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:54:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:54:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:54:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:54:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:54:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:54:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:54:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:54:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:54:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:54:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:54:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:54:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:54:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:54:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:54:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:54:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:54:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:54:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:54:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:54:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:54:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:54:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:54:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:54:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:54:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:54:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:54:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:54:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:54:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:54:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:54:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:54:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:54:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:54:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:54:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:54:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:54:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:54:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:54:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:54:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:54:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:54:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:54:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:54:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:54:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:54:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:54:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:54:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:54:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:54:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:54:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:54:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:54:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:54:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:54:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:54:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:54:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:54:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:54:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:54:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:54:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:54:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:54:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:54:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:54:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:54:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:54:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:54:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:54:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:54:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:54:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:54:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:54:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:54:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:54:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:54:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:54:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:54:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:54:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:54:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:54:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:54:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:54:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:54:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:54:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:54:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:54:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:54:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:54:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:54:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:54:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:54:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:55:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:55:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:55:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:55:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:55:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:55:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:55:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:55:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:55:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:55:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:55:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:55:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:55:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:55:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:55:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:55:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:55:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:55:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:55:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:55:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:55:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:55:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:55:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:55:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:55:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:55:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:55:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:55:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:55:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:55:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:55:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:55:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:55:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:55:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:55:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:55:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:55:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:55:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:55:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:55:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:55:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:55:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:55:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:55:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:55:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:55:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:55:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:55:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:55:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:55:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:55:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:55:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:55:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:55:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:55:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:55:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:55:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:55:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:55:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:55:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:55:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:55:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:55:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:55:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:55:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:55:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:55:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:55:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:55:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:55:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:55:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:55:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:55:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:55:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:55:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:55:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:55:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:55:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:55:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:55:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:55:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:55:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:55:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:55:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:55:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:55:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:55:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:55:22 rdma06 systemd-journald[790]: Data hash table of /run/log/journal/95379d2f723847b68d9e976dee8967a9/system.journal has a fill level at 75.0 (163750 of 218332 items, 75497472 file size, 461 bytes per hash table item), suggesting rotation.
Apr 19 04:55:22 rdma06 systemd-journald[790]: /run/log/journal/95379d2f723847b68d9e976dee8967a9/system.journal: Journal header limits reached or header out-of-date, rotating.
Apr 19 04:55:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:55:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:55:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:55:22 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 19 04:55:22 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 19 04:55:22 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 19 04:55:23 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 19 04:55:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:55:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:55:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:55:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:55:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:55:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:55:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:55:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:55:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:55:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:55:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:55:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:55:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:55:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:55:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:55:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:55:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:55:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:55:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:55:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:55:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:55:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:55:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:55:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:55:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:55:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:55:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:55:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:55:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:55:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:55:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:55:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:55:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:55:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:55:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:55:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:55:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:55:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:55:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:55:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:55:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:55:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:55:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:55:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:55:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:55:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:55:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:55:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:55:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:55:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:55:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:55:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:55:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:55:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:55:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:55:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:55:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:55:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:55:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:55:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:55:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:55:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:55:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:55:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:55:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:55:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:55:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:55:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:55:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:55:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:55:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:55:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:55:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:55:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:55:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:55:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:55:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:55:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:55:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:55:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:55:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:55:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:55:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:55:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:55:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:55:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:55:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:55:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:55:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:55:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:55:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:55:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:55:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:55:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:55:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:55:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:55:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:55:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:55:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:55:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:55:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:55:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:55:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:55:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:55:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:55:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:55:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:55:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:55:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:55:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:55:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:55:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:55:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:55:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:55:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:55:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:55:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:55:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:55:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:55:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:55:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:55:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:55:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:55:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:55:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:55:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:56:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:56:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:56:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:56:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:56:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:56:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:56:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:56:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:56:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:56:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:56:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:56:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:56:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:56:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:56:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:56:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:56:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:56:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:56:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:56:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:56:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:56:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:56:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:56:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:56:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:56:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:56:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:56:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:56:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:56:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:56:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:56:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:56:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:56:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:56:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:56:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:56:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:56:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:56:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:56:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:56:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:56:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:56:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:56:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:56:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:56:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:56:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:56:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:56:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:56:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:56:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:56:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:56:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:56:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:56:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:56:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:56:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:56:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:56:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:56:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:56:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:56:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:56:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:56:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:56:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:56:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:56:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:56:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:56:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:56:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:56:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:56:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:56:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:56:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:56:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:56:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:56:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:56:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:56:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:56:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:56:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:56:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:56:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:56:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:56:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:56:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:56:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:56:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:56:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:56:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:56:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:56:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:56:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:56:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:56:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:56:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:56:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:56:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:56:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:56:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:56:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:56:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:56:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:56:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:56:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:56:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:56:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:56:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:56:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:56:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:56:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:56:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:56:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:56:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:56:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:56:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:56:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:56:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:56:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:56:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:56:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:56:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:56:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:56:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:56:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:56:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:56:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:56:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:56:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:56:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:56:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:56:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:56:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:56:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:56:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:56:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:56:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:56:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:56:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:56:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:56:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:56:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:56:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:56:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:56:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:56:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:56:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:56:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:56:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:56:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:56:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:56:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:56:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:56:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:56:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:56:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:56:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:56:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:56:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:56:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:56:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:56:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:56:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:56:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:56:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:56:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:56:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:56:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:56:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:56:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:56:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:56:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:56:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:56:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:56:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:56:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:56:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:56:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:56:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:56:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:56:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:56:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:56:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:56:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:56:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:56:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:56:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:56:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:56:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:56:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:56:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:56:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:56:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:56:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:56:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:56:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:56:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:56:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:56:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:56:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:56:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:56:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:56:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:56:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:56:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:56:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:56:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:56:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:56:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:56:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:56:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:56:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:56:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:56:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:56:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:56:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:57:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:57:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:57:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:57:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:57:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:57:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:57:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:57:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:57:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:57:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:57:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:57:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:57:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:57:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:57:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:57:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:57:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:57:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:57:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:57:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:57:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:57:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:57:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:57:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:57:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:57:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:57:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:57:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:57:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:57:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:57:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:57:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:57:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:57:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:57:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:57:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:57:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:57:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:57:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:57:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:57:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:57:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:57:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:57:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:57:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:57:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:57:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:57:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:57:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:57:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:57:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:57:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:57:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:57:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:57:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:57:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:57:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:57:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:57:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:57:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:57:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:57:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:57:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:57:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:57:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:57:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:57:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:57:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:57:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:57:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:57:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:57:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:57:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:57:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:57:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:57:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:57:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:57:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:57:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:57:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:57:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:57:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:57:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:57:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:57:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:57:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:57:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:57:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:57:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:57:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:57:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:57:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:57:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:57:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:57:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:57:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:57:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:57:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:57:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:57:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:57:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:57:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:57:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:57:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:57:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:57:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:57:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:57:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:57:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:57:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:57:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:57:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:57:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:57:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:57:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:57:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:57:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:57:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:57:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:57:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:57:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:57:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:57:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:57:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:57:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:57:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:57:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:57:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:57:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:57:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:57:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:57:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:57:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:57:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:57:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:57:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:57:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:57:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:57:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:57:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:57:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:57:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:57:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:57:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:57:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:57:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:57:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:57:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:57:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:57:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:57:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:57:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:57:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:57:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:57:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:57:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:57:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:57:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:57:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:57:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:57:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:57:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:57:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:57:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:57:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:57:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:57:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:57:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:57:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:57:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:57:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:57:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:57:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:57:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:57:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:57:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:57:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:57:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:57:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:57:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:57:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:57:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:57:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:57:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:57:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:57:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:57:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:57:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:57:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:57:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:57:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:57:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:57:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:57:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:57:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:57:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:57:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:57:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:57:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:57:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:57:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:57:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:57:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:57:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:57:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:57:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:57:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:57:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:57:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:57:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:57:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:57:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:57:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:57:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:57:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:57:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:58:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:58:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:58:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:58:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:58:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:58:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:58:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:58:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:58:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:58:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:58:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:58:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:58:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:58:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:58:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:58:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:58:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:58:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:58:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:58:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:58:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:58:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:58:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:58:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:58:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:58:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:58:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:58:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:58:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:58:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:58:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:58:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:58:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:58:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:58:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:58:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:58:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:58:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:58:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:58:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:58:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:58:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:58:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:58:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:58:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:58:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:58:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:58:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:58:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:58:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:58:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:58:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:58:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:58:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:58:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:58:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:58:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:58:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:58:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:58:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:58:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:58:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:58:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:58:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:58:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:58:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:58:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:58:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:58:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:58:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:58:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:58:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:58:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:58:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:58:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:58:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:58:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:58:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:58:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:58:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:58:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:58:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:58:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:58:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:58:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:58:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:58:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:58:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:58:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:58:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:58:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:58:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:58:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:58:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:58:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:58:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:58:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:58:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:58:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:58:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:58:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:58:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:58:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:58:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:58:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:58:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:58:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:58:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:58:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:58:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:58:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:58:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:58:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:58:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:58:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:58:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:58:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:58:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:58:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:58:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:58:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:58:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:58:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:58:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:58:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:58:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:58:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:58:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:58:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:58:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:58:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:58:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:58:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:58:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:58:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:58:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:58:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:58:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:58:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:58:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:58:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:58:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:58:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:58:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:58:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:58:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:58:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:58:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:58:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:58:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:58:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:58:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:58:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:58:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:58:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:58:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:58:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:58:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:58:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:58:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:58:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:58:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:58:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:58:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:58:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:58:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:58:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:58:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:58:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:58:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:58:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:58:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:58:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:58:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:58:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:58:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:58:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:58:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:58:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:58:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:58:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:58:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:58:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:58:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:58:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:58:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:58:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:58:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:58:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:58:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:58:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:58:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:58:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:58:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:58:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:58:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:58:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:58:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:58:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:58:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:58:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:58:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:58:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:58:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:58:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:58:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:58:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:58:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:58:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:58:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:58:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:58:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:58:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:58:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:58:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:58:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:59:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:59:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:59:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:59:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:59:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:59:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:59:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:59:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:59:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:59:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:59:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:59:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:59:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:59:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:59:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:59:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:59:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:59:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:59:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:59:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:59:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:59:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:59:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:59:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:59:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:59:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:59:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:59:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:59:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:59:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:59:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:59:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:59:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:59:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:59:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:59:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:59:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:59:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:59:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:59:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:59:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:59:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:59:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:59:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:59:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:59:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:59:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:59:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:59:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:59:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:59:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:59:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:59:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:59:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:59:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:59:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:59:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:59:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:59:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:59:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:59:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:59:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:59:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:59:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:59:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:59:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:59:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:59:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:59:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:59:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:59:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:59:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:59:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:59:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:59:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:59:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:59:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:59:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:59:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:59:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:59:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:59:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:59:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:59:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:59:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:59:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:59:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:59:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:59:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:59:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:59:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:59:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:59:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:59:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:59:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:59:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:59:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:59:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:59:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:59:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:59:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:59:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:59:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:59:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:59:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:59:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:59:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:59:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:59:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:59:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:59:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:59:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:59:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:59:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:59:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:59:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:59:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:59:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:59:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:59:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:59:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:59:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:59:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:59:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:59:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:59:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 04:59:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 04:59:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:59:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:59:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:59:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 04:59:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 04:59:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 04:59:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 04:59:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 04:59:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 04:59:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 04:59:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 04:59:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 04:59:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 04:59:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 04:59:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 04:59:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 04:59:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:01:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:01:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:01:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:01:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:01:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:01:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:01:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:01:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:01:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:01:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:01:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:01:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:01:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:01:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:01:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:01:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:01:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:01:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:01:43 rdma06 rsyslogd[1041]: imjournal: 5092 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 19 05:01:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:01:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:01:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:01:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:01:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:01:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:01:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:01:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:01:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:01:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:01:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:01:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:01:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:01:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:01:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:01:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:01:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:01:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:01:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:01:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:01:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:01:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:01:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:01:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:01:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:01:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:01:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:01:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:01:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:01:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:01:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:01:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:01:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:01:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:01:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:01:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:01:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:01:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:01:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:01:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:01:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:01:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:01:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:01:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:01:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:01:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:01:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:01:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:01:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:01:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:01:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:01:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:01:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:01:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:02:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:02:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:02:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:02:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:02:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:02:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:02:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:02:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:02:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:02:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:02:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:02:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:02:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:02:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:02:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:02:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:02:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:02:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:02:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:02:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:02:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:02:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:02:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:02:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:02:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:02:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:02:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:02:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:02:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:02:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:02:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:02:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:02:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:02:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:02:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:02:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:02:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:02:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:02:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:02:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:02:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:02:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:02:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:02:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:02:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:02:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:02:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:02:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:02:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:02:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:02:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:02:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:02:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:02:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:02:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:02:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:02:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:02:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:02:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:02:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:02:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:02:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:02:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:02:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:02:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:02:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:02:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:02:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:02:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:02:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:02:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:02:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:02:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:02:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:02:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:02:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:02:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:02:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:02:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:02:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:02:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:02:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:02:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:02:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:02:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:02:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:02:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:02:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:02:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:02:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:02:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:02:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:02:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:02:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:02:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:02:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:02:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:02:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:02:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:02:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:02:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:02:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:02:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:02:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:02:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:02:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:02:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:02:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:02:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:02:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:02:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:02:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:02:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:02:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:02:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:02:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:02:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:02:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:02:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:02:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:02:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:02:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:02:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:02:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:02:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:02:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:02:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:02:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:02:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:02:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:02:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:02:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:02:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:02:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:02:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:02:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:02:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:02:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:02:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:02:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:02:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:02:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:02:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:02:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:02:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:02:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:02:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:02:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:02:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:02:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:02:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:02:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:02:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:02:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:02:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:02:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:02:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:02:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:02:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:02:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:02:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:02:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:02:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:02:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:02:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:02:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:02:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:02:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:02:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:02:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:02:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:02:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:02:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:02:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:02:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:02:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:02:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:02:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:02:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:02:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:02:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:02:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:02:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:02:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:02:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:02:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:02:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:02:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:02:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:02:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:02:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:02:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:02:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:02:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:02:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:02:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:02:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:02:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:02:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:02:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:02:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:02:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:02:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:02:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:02:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:02:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:02:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:02:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:02:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:02:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:02:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:02:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:02:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:02:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:02:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:02:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:03:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:03:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:03:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:03:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:03:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:03:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:03:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:03:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:03:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:03:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:03:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:03:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:03:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:03:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:03:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:03:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:03:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:03:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:03:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:03:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:03:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:03:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:03:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:03:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:03:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:03:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:03:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:03:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:03:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:03:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:03:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:03:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:03:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:03:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:03:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:03:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:03:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:03:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:03:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:03:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:03:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:03:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:03:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:03:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:03:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:03:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:03:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:03:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:03:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:03:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:03:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:03:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:03:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:03:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:03:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:03:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:03:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:03:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:03:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:03:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:03:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:03:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:03:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:03:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:03:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:03:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:03:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:03:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:03:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:03:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:03:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:03:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:03:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:03:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:03:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:03:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:03:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:03:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:03:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:03:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:03:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:03:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:03:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:03:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:03:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:03:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:03:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:03:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:03:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:03:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:03:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:03:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:03:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:03:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:03:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:03:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:03:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:03:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:03:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:03:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:03:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:03:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:03:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:03:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:03:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:03:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:03:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:03:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:03:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:03:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:03:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:03:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:03:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:03:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:03:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:03:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:03:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:03:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:03:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:03:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:03:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:03:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:03:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:03:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:03:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:03:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:03:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:03:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:03:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:03:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:03:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:03:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:03:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:03:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:03:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:03:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:03:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:03:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:03:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:03:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:03:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:03:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:03:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:03:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:03:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:03:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:03:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:03:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:03:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:03:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:03:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:03:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:03:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:03:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:03:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:03:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:03:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:03:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:03:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:03:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:03:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:03:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:03:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:03:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:03:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:03:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:03:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:03:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:03:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:03:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:03:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:03:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:03:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:03:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:03:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:03:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:03:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:03:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:03:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:03:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:03:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:03:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:03:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:03:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:03:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:03:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:03:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:03:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:03:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:03:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:03:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:03:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:03:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:03:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:03:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:03:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:03:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:03:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:03:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:03:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:03:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:03:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:03:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:03:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:03:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:03:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:03:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:03:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:03:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:03:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:03:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:03:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:03:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:03:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:03:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:03:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:04:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:04:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:04:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:04:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:04:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:04:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:04:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:04:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:04:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:04:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:04:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:04:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:04:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:04:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:04:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:04:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:04:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:04:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:04:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:04:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:04:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:04:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:04:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:04:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:04:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:04:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:04:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:04:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:04:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:04:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:04:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:04:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:04:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:04:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:04:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:04:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:04:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:04:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:04:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:04:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:04:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:04:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:04:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:04:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:04:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:04:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:04:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:04:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:04:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:04:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:04:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:04:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:04:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:04:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:04:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:04:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:04:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:04:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:04:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:04:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:04:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:04:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:04:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:04:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:04:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:04:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:04:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:04:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:04:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:04:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:04:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:04:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:04:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:04:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:04:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:04:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:04:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:04:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:04:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:04:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:04:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:04:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:04:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:04:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:04:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:04:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:04:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:04:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:04:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:04:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:04:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:04:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:04:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:04:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:04:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:04:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:04:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:04:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:04:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:04:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:04:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:04:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:04:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:04:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:04:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:04:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:04:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:04:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:04:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:04:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:04:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:04:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:04:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:04:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:04:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:04:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:04:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:04:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:04:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:04:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:04:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:04:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:04:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:04:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:04:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:04:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:04:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:04:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:04:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:04:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:04:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:04:37 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:04:37 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:04:37 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:04:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:04:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:04:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:04:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:04:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:04:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:04:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:04:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:04:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:04:37 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:04:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:04:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:04:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:04:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:04:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:04:42 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:04:42 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:04:42 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:04:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:04:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:04:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:04:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:04:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:04:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:04:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:04:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:04:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:04:42 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:04:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:04:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:04:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:04:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:04:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:04:47 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:04:47 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:04:47 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:04:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:04:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:04:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:04:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:04:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:04:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:04:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:04:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:04:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:04:47 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:04:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:04:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:04:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:04:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:04:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:04:52 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:04:52 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:04:52 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:04:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:04:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:04:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:04:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:04:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:04:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:04:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:04:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:04:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:04:52 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:04:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:04:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:04:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:04:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:04:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:04:57 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:04:57 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:04:57 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:04:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:04:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:04:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:04:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:04:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:04:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:04:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:04:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:04:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:04:57 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:05:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:05:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:05:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:05:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:05:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:05:02 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:05:02 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:05:02 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:05:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:05:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:05:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:05:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:05:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:05:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:05:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:05:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:05:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:05:02 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:05:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:05:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:05:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:05:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:05:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:05:07 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:05:07 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:05:07 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:05:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:05:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:05:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:05:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:05:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:05:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:05:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:05:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:05:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:05:07 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:05:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:05:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:05:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:05:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:05:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:05:12 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:05:12 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:05:12 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:05:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:05:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:05:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:05:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:05:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:05:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:05:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:05:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:05:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:05:12 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:05:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:05:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:05:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:05:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:05:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:05:17 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:05:17 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:05:17 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:05:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:05:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:05:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:05:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:05:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:05:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:05:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:05:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:05:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:05:17 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:05:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:05:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:05:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:05:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:05:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:05:22 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:05:22 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:05:22 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:05:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:05:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:05:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:05:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:05:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:05:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:05:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:05:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:05:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:05:22 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:05:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:05:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:05:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:05:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:05:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:05:27 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:05:27 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:05:27 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:05:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:05:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:05:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:05:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:05:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:05:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:05:27 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:05:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:05:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:05:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:05:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:05:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:05:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:05:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:05:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:05:32 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:05:32 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:05:32 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:05:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:05:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:05:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:05:32 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:05:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:05:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:05:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:05:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:05:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:05:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:05:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:05:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:05:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:05:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:05:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:05:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:05:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:05:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:05:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:05:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:05:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:05:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:05:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:05:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:05:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:05:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:05:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:05:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:05:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:05:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:05:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:05:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:05:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:05:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:05:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:05:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:05:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:05:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:05:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:05:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:05:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:05:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:05:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:05:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:05:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:05:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:05:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:05:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:05:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:05:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:05:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:05:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:05:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:05:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:05:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:05:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:05:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:05:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:05:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:05:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:05:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:05:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:05:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:05:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:05:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:05:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:05:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:05:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:05:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:05:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:05:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:05:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:05:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:05:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:05:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:05:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:05:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:05:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:05:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:05:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:05:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:05:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:05:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:05:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:05:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:05:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:05:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:05:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:05:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:05:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:05:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:05:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:05:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:05:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:05:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:05:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:05:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:05:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:05:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:05:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:06:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:06:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:06:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:06:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:06:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:06:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:06:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:06:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:06:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:06:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:06:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:06:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:06:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:06:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:06:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:06:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:06:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:06:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:06:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:06:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:06:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:06:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:06:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:06:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:06:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:06:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:06:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:06:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:06:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:06:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:06:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:06:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:06:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:06:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:06:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:06:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:06:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:06:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:06:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:06:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:06:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:06:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:06:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:06:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:06:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:06:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:06:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:06:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:06:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:06:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:06:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:06:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:06:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:06:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:06:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:06:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:06:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:06:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:06:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:06:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:06:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:06:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:06:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:06:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:06:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:06:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:06:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:06:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:06:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:06:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:06:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:06:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:06:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:06:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:06:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:06:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:06:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:06:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:06:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:06:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:06:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:06:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:06:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:06:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:06:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:06:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:06:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:06:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:06:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:06:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:06:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:06:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:06:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:06:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:06:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:06:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:06:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:06:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:06:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:06:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:06:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:06:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:06:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:06:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:06:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:06:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:06:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:06:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:06:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:06:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:06:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:06:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:06:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:06:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:06:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:06:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:06:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:06:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:06:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:06:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:06:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:06:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:06:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:06:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:06:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:06:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:06:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:06:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:06:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:06:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:06:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:06:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:06:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:06:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:06:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:06:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:06:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:06:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:06:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:06:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:06:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:06:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:06:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:06:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:06:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:06:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:06:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:06:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:06:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:06:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:06:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:06:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:06:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:06:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:06:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:06:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:06:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:06:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:06:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:06:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:06:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:06:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:06:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:06:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:06:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:06:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:06:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:06:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:06:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:06:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:06:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:06:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:06:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:06:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:06:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:06:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:06:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:06:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:06:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:06:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:06:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:06:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:06:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:06:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:06:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:06:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:06:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:06:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:06:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:06:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:06:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:06:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:06:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:06:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:06:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:06:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:06:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:06:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:06:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:06:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:06:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:06:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:06:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:06:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:06:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:06:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:06:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:06:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:06:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:06:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:06:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:06:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:06:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:06:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:06:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:06:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:07:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:07:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:07:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:07:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:07:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:07:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:07:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:07:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:07:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:07:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:07:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:07:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:07:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:07:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:07:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:07:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:07:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:07:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:07:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:07:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:07:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:07:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:07:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:07:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:07:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:07:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:07:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:07:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:07:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:07:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:07:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:07:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:07:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:07:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:07:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:07:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:07:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:07:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:07:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:07:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:07:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:07:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:07:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:07:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:07:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:07:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:07:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:07:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:07:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:07:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:07:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:07:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:07:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:07:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:07:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:07:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:07:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:07:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:07:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:07:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:07:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:07:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:07:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:07:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:07:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:07:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:07:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:07:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:07:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:07:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:07:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:07:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:07:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:07:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:07:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:07:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:07:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:07:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:07:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:07:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:07:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:07:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:07:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:07:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:07:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:07:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:07:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:07:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:07:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:07:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:07:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:07:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:07:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:07:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:07:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:07:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:07:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:07:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:07:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:07:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:07:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:07:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:07:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:07:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:07:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:07:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:07:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:07:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:07:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:07:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:07:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:07:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:07:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:07:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:07:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:07:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:07:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:07:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:07:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:07:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:07:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:07:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:07:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:07:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:07:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:07:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:07:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:07:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:07:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:07:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:07:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:07:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:07:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:07:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:07:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:07:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:07:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:07:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:07:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:07:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:07:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:07:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:07:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:07:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:07:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:07:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:07:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:07:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:07:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:07:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:07:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:07:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:07:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:07:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:07:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:07:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:07:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:07:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:07:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:07:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:07:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:07:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:07:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:07:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:07:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:07:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:07:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:07:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:07:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:07:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:07:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:07:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:07:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:07:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:07:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:07:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:07:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:07:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:07:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:07:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:07:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:07:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:07:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:07:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:07:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:07:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:07:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:07:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:07:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:07:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:07:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:07:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:07:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:07:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:07:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:07:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:07:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:07:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:07:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:07:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:07:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:07:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:07:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:07:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:07:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:07:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:07:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:07:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:07:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:07:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:07:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:07:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:07:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:07:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:07:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:07:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:08:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:08:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:08:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:08:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:08:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:08:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:08:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:08:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:08:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:08:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:08:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:08:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:08:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:08:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:08:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:08:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:08:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:08:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:08:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:08:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:08:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:08:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:08:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:08:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:08:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:08:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:08:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:08:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:08:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:08:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:08:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:08:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:08:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:08:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:08:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:08:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:08:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:08:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:08:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:08:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:08:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:08:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:08:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:08:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:08:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:08:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:08:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:08:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:08:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:08:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:08:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:08:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:08:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:08:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:08:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:08:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:08:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:08:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:08:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:08:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:08:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:08:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:08:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:08:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:08:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:08:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:08:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:08:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:08:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:08:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:08:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:08:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:08:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:08:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:08:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:08:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:08:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:08:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:08:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:08:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:08:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:08:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:08:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:08:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:08:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:08:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:08:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:08:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:08:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:08:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:08:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:08:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:08:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:08:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:08:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:08:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:08:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:08:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:08:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:08:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:08:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:08:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:08:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:08:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:08:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:08:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:08:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:08:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:08:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:08:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:08:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:08:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:08:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:08:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:08:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:08:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:08:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:08:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:08:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:08:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:08:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:08:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:08:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:08:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:08:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:08:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:08:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:08:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:08:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:08:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:08:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:08:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:08:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:08:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:08:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:08:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:08:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:08:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:08:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:08:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:08:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:08:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:08:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:08:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:08:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:08:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:08:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:08:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:08:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:08:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:08:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:08:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:08:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:08:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:08:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:08:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:08:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:08:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:08:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:08:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:08:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:08:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:08:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:08:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:08:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:08:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:08:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:08:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:08:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:08:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:08:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:08:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:08:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:08:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:08:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:08:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:08:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:08:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:08:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:08:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:08:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:08:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:08:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:08:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:08:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:08:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:08:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:08:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:08:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:08:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:08:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:08:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:08:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:08:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:08:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:08:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:08:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:08:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:08:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:08:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:08:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:08:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:08:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:08:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:08:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:08:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:08:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:08:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:08:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:08:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:08:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:08:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:08:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:08:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:08:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:08:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:09:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:09:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:09:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:09:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:09:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:09:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:09:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:09:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:09:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:09:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:09:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:09:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:09:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:09:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:09:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:09:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:09:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:09:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:09:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:09:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:09:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:09:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:09:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:09:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:09:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:09:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:09:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:09:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:09:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:09:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:09:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:09:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:09:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:09:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:09:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:09:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:09:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:09:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:09:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:09:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:09:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:09:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:09:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:09:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:09:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:09:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:09:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:09:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:09:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:09:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:09:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:09:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:09:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:09:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:09:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:09:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:09:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:09:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:09:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:09:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:09:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:09:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:09:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:09:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:09:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:09:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:09:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:09:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:09:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:09:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:09:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:09:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:09:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:09:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:09:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:09:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:09:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:09:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:09:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:09:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:09:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:09:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:09:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:09:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:09:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:09:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:09:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:09:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:09:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:09:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:09:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:09:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:09:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:09:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:09:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:09:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:09:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:09:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:09:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:09:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:09:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:09:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:09:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:09:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:09:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:09:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:09:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:09:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:09:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:09:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:09:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:09:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:09:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:09:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:09:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:09:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:09:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:09:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:09:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:09:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:09:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:09:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:09:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:09:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:09:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:09:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:09:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:09:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:09:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:09:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:09:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:09:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:09:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:09:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:09:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:09:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:09:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:09:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:09:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:09:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:09:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:09:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:09:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:09:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:11:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:11:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:11:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:11:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:11:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:11:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:11:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:11:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:11:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:11:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:11:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:11:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:11:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:11:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:11:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:11:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:11:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:11:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:11:48 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 19 05:11:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:11:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:11:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:11:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:11:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:11:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:11:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:11:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:11:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:11:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:11:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:11:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:11:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:11:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:11:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:11:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:11:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:11:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:11:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:11:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:11:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:11:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:11:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:11:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:11:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:11:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:11:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:11:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:11:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:11:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:11:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:11:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:11:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:11:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:11:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:11:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:12:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:12:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:12:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:12:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:12:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:12:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:12:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:12:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:12:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:12:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:12:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:12:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:12:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:12:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:12:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:12:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:12:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:12:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:12:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:12:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:12:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:12:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:12:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:12:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:12:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:12:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:12:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:12:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:12:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:12:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:12:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:12:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:12:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:12:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:12:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:12:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:12:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:12:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:12:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:12:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:12:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:12:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:12:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:12:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:12:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:12:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:12:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:12:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:12:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:12:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:12:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:12:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:12:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:12:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:12:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:12:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:12:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:12:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:12:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:12:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:12:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:12:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:12:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:12:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:12:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:12:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:12:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:12:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:12:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:12:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:12:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:12:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:12:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:12:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:12:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:12:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:12:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:12:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:12:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:12:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:12:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:12:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:12:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:12:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:12:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:12:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:12:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:12:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:12:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:12:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:12:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:12:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:12:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:12:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:12:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:12:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:12:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:12:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:12:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:12:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:12:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:12:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:12:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:12:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:12:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:12:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:12:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:12:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:12:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:12:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:12:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:12:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:12:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:12:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:12:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:12:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:12:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:12:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:12:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:12:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:12:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:12:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:12:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:12:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:12:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:12:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:12:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:12:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:12:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:12:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:12:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:12:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:12:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:12:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:12:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:12:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:12:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:12:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:12:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:12:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:12:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:12:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:12:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:12:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:12:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:12:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:12:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:12:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:12:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:12:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:12:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:12:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:12:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:12:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:12:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:12:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:12:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:12:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:12:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:12:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:12:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:12:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:12:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:12:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:12:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:12:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:12:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:12:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:12:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:12:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:12:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:12:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:12:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:12:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:12:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:12:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:12:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:12:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:12:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:12:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:12:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:12:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:12:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:12:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:12:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:12:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:12:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:12:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:12:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:12:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:12:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:12:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:12:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:12:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:12:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:12:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:12:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:12:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:12:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:12:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:12:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:12:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:12:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:12:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:12:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:12:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:12:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:12:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:12:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:12:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:12:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:12:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:12:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:12:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:12:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:12:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:13:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:13:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:13:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:13:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:13:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:13:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:13:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:13:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:13:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:13:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:13:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:13:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:13:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:13:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:13:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:13:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:13:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:13:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:13:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:13:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:13:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:13:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:13:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:13:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:13:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:13:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:13:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:13:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:13:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:13:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:13:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:13:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:13:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:13:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:13:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:13:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:13:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:13:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:13:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:13:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:13:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:13:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:13:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:13:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:13:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:13:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:13:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:13:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:13:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:13:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:13:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:13:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:13:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:13:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:13:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:13:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:13:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:13:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:13:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:13:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:13:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:13:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:13:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:13:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:13:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:13:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:13:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:13:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:13:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:13:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:13:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:13:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:13:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:13:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:13:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:13:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:13:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:13:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:13:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:13:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:13:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:13:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:13:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:13:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:13:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:13:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:13:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:13:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:13:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:13:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:13:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:13:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:13:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:13:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:13:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:13:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:13:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:13:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:13:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:13:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:13:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:13:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:13:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:13:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:13:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:13:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:13:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:13:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:13:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:13:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:13:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:13:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:13:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:13:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:13:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:13:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:13:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:13:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:13:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:13:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:13:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:13:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:13:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:13:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:13:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:13:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:13:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:13:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:13:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:13:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:13:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:13:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:13:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:13:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:13:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:13:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:13:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:13:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:13:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:13:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:13:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:13:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:13:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:13:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:13:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:13:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:13:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:13:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:13:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:13:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:13:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:13:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:13:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:13:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:13:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:13:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:13:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:13:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:13:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:13:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:13:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:13:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:13:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:13:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:13:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:13:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:13:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:13:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:13:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:13:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:13:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:13:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:13:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:13:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:13:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:13:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:13:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:13:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:13:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:13:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:13:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:13:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:13:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:13:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:13:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:13:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:13:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:13:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:13:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:13:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:13:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:13:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:13:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:13:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:13:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:13:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:13:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:13:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:13:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:13:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:13:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:13:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:13:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:13:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:13:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:13:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:13:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:13:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:13:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:13:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:13:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:13:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:13:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:13:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:13:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:13:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:14:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:14:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:14:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:14:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:14:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:14:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:14:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:14:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:14:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:14:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:14:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:14:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:14:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:14:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:14:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:14:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:14:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:14:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:14:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:14:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:14:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:14:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:14:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:14:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:14:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:14:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:14:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:14:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:14:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:14:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:14:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:14:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:14:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:14:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:14:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:14:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:14:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:14:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:14:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:14:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:14:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:14:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:14:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:14:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:14:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:14:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:14:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:14:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:14:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:14:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:14:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:14:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:14:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:14:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:14:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:14:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:14:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:14:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:14:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:14:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:14:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:14:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:14:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:14:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:14:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:14:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:14:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:14:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:14:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:14:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:14:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:14:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:14:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:14:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:14:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:14:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:14:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:14:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:14:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:14:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:14:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:14:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:14:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:14:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:14:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:14:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:14:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:14:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:14:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:14:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:14:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:14:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:14:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:14:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:14:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:14:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:14:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:14:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:14:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:14:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:14:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:14:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:14:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:14:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:14:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:14:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:14:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:14:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:14:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:14:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:14:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:14:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:14:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:14:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:14:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:14:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:14:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:14:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:14:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:14:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:14:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:14:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:14:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:14:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:14:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:14:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:14:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:14:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:14:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:14:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:14:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:14:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:14:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:14:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:14:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:14:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:14:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:14:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:14:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:14:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:14:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:14:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:14:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:14:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:14:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:14:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:14:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:14:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:14:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:14:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:14:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:14:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:14:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:14:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:14:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:14:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:14:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:14:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:14:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:14:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:14:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:14:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:14:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:14:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:14:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:14:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:14:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:14:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:14:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:14:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:14:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:14:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:14:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:14:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:14:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:14:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:14:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:14:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:14:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:14:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:14:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:14:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:14:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:14:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:14:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:14:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:14:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:14:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:14:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:14:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:14:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:14:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:14:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:14:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:14:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:14:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:14:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:14:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:14:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:14:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:14:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:14:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:14:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:14:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:14:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:14:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:14:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:14:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:14:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:14:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:14:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:14:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:14:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:14:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:14:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:14:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:15:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:15:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:15:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:15:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:15:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:15:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:15:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:15:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:15:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:15:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:15:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:15:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:15:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:15:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:15:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:15:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:15:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:15:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:15:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:15:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:15:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:15:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:15:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:15:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:15:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:15:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:15:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:15:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:15:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:15:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:15:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:15:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:15:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:15:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:15:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:15:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:15:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:15:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:15:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:15:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:15:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:15:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:15:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:15:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:15:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:15:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:15:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:15:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:15:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:15:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:15:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:15:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:15:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:15:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:15:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:15:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:15:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:15:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:15:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:15:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:15:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:15:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:15:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:15:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:15:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:15:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:15:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:15:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:15:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:15:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:15:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:15:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:15:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:15:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:15:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:15:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:15:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:15:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:15:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:15:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:15:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:15:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:15:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:15:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:15:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:15:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:15:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:15:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:15:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:15:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:15:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:15:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:15:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:15:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:15:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:15:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:15:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:15:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:15:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:15:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:15:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:15:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:15:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:15:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:15:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:15:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:15:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:15:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:15:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:15:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:15:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:15:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:15:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:15:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:15:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:15:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:15:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:15:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:15:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:15:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:15:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:15:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:15:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:15:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:15:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:15:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:15:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:15:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:15:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:15:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:15:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:15:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:15:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:15:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:15:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:15:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:15:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:15:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:15:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:15:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:15:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:15:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:15:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:15:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:15:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:15:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:15:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:15:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:15:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:15:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:15:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:15:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:15:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:15:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:15:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:15:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:15:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:15:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:15:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:15:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:15:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:15:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:15:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:15:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:15:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:15:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:15:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:15:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:15:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:15:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:15:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:15:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:15:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:15:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:15:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:15:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:15:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:15:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:15:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:15:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:15:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:15:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:15:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:15:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:15:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:15:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:15:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:15:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:15:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:15:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:15:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:15:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:15:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:15:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:15:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:15:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:15:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:15:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:15:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:15:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:15:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:15:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:15:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:15:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:15:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:15:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:15:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:15:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:15:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:15:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:15:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:15:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:15:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:15:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:15:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:15:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:16:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:16:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:16:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:16:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:16:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:16:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:16:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:16:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:16:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:16:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:16:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:16:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:16:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:16:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:16:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:16:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:16:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:16:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:16:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:16:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:16:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:16:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:16:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:16:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:16:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:16:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:16:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:16:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:16:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:16:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:16:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:16:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:16:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:16:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:16:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:16:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:16:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:16:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:16:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:16:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:16:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:16:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:16:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:16:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:16:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:16:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:16:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:16:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:16:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:16:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:16:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:16:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:16:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:16:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:16:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:16:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:16:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:16:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:16:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:16:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:16:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:16:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:16:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:16:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:16:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:16:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:16:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:16:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:16:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:16:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:16:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:16:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:16:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:16:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:16:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:16:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:16:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:16:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:16:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:16:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:16:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:16:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:16:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:16:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:16:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:16:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:16:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:16:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:16:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:16:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:16:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:16:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:16:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:16:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:16:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:16:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:16:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:16:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:16:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:16:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:16:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:16:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:16:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:16:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:16:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:16:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:16:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:16:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:16:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:16:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:16:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:16:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:16:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:16:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:16:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:16:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:16:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:16:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:16:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:16:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:16:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:16:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:16:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:16:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:16:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:16:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:16:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:16:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:16:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:16:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:16:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:16:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:16:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:16:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:16:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:16:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:16:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:16:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:16:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:16:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:16:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:16:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:16:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:16:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:16:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:16:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:16:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:16:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:16:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:16:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:16:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:16:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:16:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:16:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:16:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:16:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:16:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:16:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:16:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:16:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:16:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:16:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:16:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:16:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:16:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:16:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:16:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:16:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:16:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:16:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:16:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:16:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:16:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:16:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:16:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:16:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:16:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:16:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:16:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:16:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:16:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:16:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:16:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:16:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:16:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:16:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:16:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:16:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:16:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:16:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:16:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:16:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:16:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:16:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:16:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:16:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:16:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:16:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:16:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:16:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:16:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:16:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:16:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:16:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:16:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:16:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:16:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:16:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:16:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:16:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:16:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:16:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:16:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:16:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:16:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:16:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:17:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:17:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:17:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:17:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:17:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:17:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:17:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:17:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:17:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:17:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:17:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:17:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:17:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:17:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:17:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:17:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:17:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:17:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:17:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:17:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:17:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:17:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:17:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:17:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:17:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:17:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:17:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:17:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:17:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:17:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:17:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:17:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:17:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:17:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:17:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:17:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:17:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:17:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:17:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:17:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:17:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:17:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:17:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:17:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:17:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:17:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:17:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:17:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:17:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:17:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:17:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:17:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:17:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:17:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:17:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:17:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:17:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:17:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:17:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:17:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:17:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:17:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:17:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:17:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:17:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:17:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:17:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:17:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:17:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:17:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:17:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:17:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:17:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:17:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:17:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:17:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:17:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:17:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:17:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:17:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:17:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:17:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:17:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:17:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:17:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:17:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:17:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:17:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:17:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:17:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:17:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:17:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:17:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:17:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:17:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:17:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:17:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:17:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:17:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:17:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:17:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:17:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:17:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:17:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:17:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:17:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:17:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:17:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:17:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:17:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:17:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:17:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:17:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:17:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:17:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:17:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:17:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:17:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:17:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:17:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:17:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:17:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:17:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:17:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:17:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:17:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:17:37 rdma06 opafmd[1140340]: RX GETTBL(DeviceTreeMemberRecord):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:           SA RX DUPLICATE REQUESTS:          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]: SA RX DEAD RMPP TRANSACTION PACKET:          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:                SA DROPPED REQUESTS:          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:           SA NO AVAILABLE CONTEXTS:          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:          SA RX GETMULTI() Non-RMPP:          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:          SA RX GETMULTI RMPP Abort:          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:          SA TX GETMULTI RMPP Abort:          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:         SA TX GETMULTI Ack Retries:          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:                 SA TX RMPP Retries:          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:                    SA TX RMPP STOP:          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:          SA TX RMMP ABORT(TIMEOUT):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:    SA TX RMMP ABORT(BAD FIRST SEG):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:         SA TX RMMP ABORT(BAD TYPE):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]: SA TX RMMP ABORT(WINDOW TOO SMALL):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:   SA TX RMMP ABORT(SEGNUM TOO BIG):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:      SA TX RMMP ABORT(BAD VERSION):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]: SA TX RMMP ABORT(TOO MANY RETRIES):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:          SA TX RMMP ABORT(UNKNOWN):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:                  SA TX STATUS BUSY:          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:              SA TX STATUS REDIRECT:          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:              SA TX STATUS BADCLASS:          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:             SA TX STATUS BADMETHOD:          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:         SA TX STATUS BADMETHODATTR:          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:              SA TX STATUS BADFIELD:          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:          SA TX STATUS NO RESOURCES:          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:           SA TX STATUS INVALID REQ:          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:            SA TX STATUS NO RECORDS:          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:      SA TX STATUS TOO MANY RECORDS:          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:               SA TX STATUS BAD GID:          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:        SA TX STATUS TOO FEW FIELDS:          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:               SA TX STATUS UNKNOWN:          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:                DBSYNC Send Command:          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:        DBSYNC Send Command Failure:          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:                DBSYNC Receive Data:          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:        DBSYNC Receive Data Failure:          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:                       DBSYNC Reply:          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:               DBSYNC Reply Failure:          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:         SA Maximum Contexts In Use:          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:           SA Maximum Contexts Free:          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:        Maximum SM Sweep Time in ms:          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:         Seconds since last cleared: 182456
Apr 19 05:17:37 rdma06 opafmd[1140340]:                  Time last cleared: Mon Apr 17 02:36:41 2023
Apr 19 05:17:37 rdma06 opafmd[1140340]:                            COUNTER: THIS SWEEP LAST SWEEP      TOTAL
Apr 19 05:17:37 rdma06 opafmd[1140340]: ------------------------------------ ---------- ---------- ----------
Apr 19 05:17:37 rdma06 opafmd[1140340]:   SM State transition to DISCOVERY:          1          0          1
Apr 19 05:17:37 rdma06 opafmd[1140340]:      SM State transition to MASTER:          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:     SM State transition to STANDBY:          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:    SM State transition to INACTIVE:          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:      Total transmitted SMA Packets:      36472          0      36472
Apr 19 05:17:37 rdma06 opafmd[1140340]:          Direct Routed SMA Packets:      36471          0      36471
Apr 19 05:17:37 rdma06 opafmd[1140340]:             LID Routed SMA Packets:          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:              SMA Query Retransmits:          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:    SMA Query Retransmits Exhausted:          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:                  SM TX GET(Notice):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:                  SM TX SET(Notice):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:           Total SM RX TRAP(Notice):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:                 SM RX TRAP(PortUp):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:               SM RX TRAP(PortDown):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:            SM RX TRAP(McGrpCreate):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:            SM RX TRAP(McGrpDelete):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:                 SM RX TRAP(UnPath):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:                 SM RX TRAP(RePath):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:           SM RX TRAP(PortStateChg):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:          SM RX TRAP(LinkIntegrity):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:          SM RX TRAP(BufferOverrun):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:            SM RX TRAP(FlowControl):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:            SM RX TRAP(LocalChange):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:            SM RX TRAP(SysImageChg):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:                SM RX TRAP(BadMKey):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:                SM RX TRAP(BadPkey):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:                SM RX TRAP(BadQkey):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:          SM RX TRAP(BadPkeySwPort):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:     SM RX TRAP(LinkWidthDowngrade):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:       SM RX TRAP(CostMatrixChange):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:          SM TX TRAPREPRESS(Notice):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:         SM TX GET(NodeDescription):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:                SM TX GET(NodeInfo):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:              SM TX GET(SwitchInfo):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:              SM TX SET(SwitchInfo):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:                SM TX GET(PortInfo):      36471          0      36471
Apr 19 05:17:37 rdma06 opafmd[1140340]:                SM TX SET(PortInfo):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:           SM TX GET(PortStateInfo):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:           SM TX SET(PortStateInfo):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:               SM TX GET(PKeyTable):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:               SM TX SET(PKeyTable):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:       SM TX GET(SL2SCMappingTable):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:       SM TX SET(SL2SCMappingTable):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:       SM TX GET(SC2SLMappingTable):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:       SM TX SET(SC2SLMappingTable):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:      SM TX GET(SC2VLtMappingTable):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:      SM TX SET(SC2VLtMappingTable):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:     SM TX GET(SC2VLntMappingTable):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:     SM TX SET(SC2VLntMappingTable):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:      SM TX GET(SC2VLrMappingTable):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:      SM TX SET(SC2VLrMappingTable):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:       SM TX GET(SC2SCMappingTable):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:       SM TX SET(SC2SCMappingTable):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:           SM TX SET(SC2SCMultiSet):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:      SM TX GET(VLArbitrationTable):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:      SM TX SET(VLArbitrationTable):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:                     SM TX GET(LFT):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:                     SM TX SET(LFT):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:                     SM TX GET(MFT):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:                     SM TX SET(MFT):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:               SM TX GET(AGGREGATE):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:               SM TX SET(AGGREGATE):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:                 SM TX GET(LedInfo):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:                 SM TX SET(LedInfo):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:               SM TX GET(CableInfo):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:                  SM RX GET(SmInfo):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:                  SM TX GET(SmInfo):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:                  SM RX SET(SmInfo):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:                  SM TX SET(SmInfo):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:              SM TX GETRESP(SmInfo):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:               SM TX GET(PortGroup):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:               SM TX SET(PortGroup):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:                    SM TX GET(PGFT):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:                    SM TX SET(PGFT):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:      SM TX GET(BufferControlTable):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:      SM TX SET(BufferControlTable):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:               SM TX GET(ARLidMask):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:               SM TX SET(ARLidMask):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:          SM RX GET(CongestionInfo):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:       SM RX GET(HfiCongestSetting):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:       SM RX SET(HfiCongestSetting):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:       SM RX GET(HfiCongestControl):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:       SM RX SET(HfiCongestControl):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:        SM RX GET(SwCongestSetting):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:        SM RX SET(SwCongestSetting):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:  SM RX GET(SwPortCongestionSetting:          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:                   SM RX GETRESP(*):      72942          0      72942
Apr 19 05:17:37 rdma06 opafmd[1140340]:                  SM RX STATUS BUSY:          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:              SM RX STATUS REDIRECT:          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:              SM RX STATUS BADCLASS:          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:             SM RX STATUS BADMETHOD:          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:         SM RX STATUS BADMETHODATTR:          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:              SM RX STATUS BADFIELD:          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:               SM RX STATUS UNKNOWN:          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:           SA RX GET(ClassPortInfo):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:               SA TX REPORT(Notice):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:              SA RX SET(InformInfo):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:              SA RX GET(NodeRecord):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:           SA RX GETTBL(NodeRecord):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:                SA RX GET(PortInfo):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:             SA RX GETTBL(PortInfo):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:                SA RX GET(SC2SCMAP):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:             SA RX GETTBL(SC2SCMap):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:                SA RX GET(SL2SCMAP):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:             SA RX GETTBL(SL2SCMap):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:                SA RX GET(SC2SLMap):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:             SA RX GETTBL(SC2SLMap):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:               SA RX GET(SC2VLtMap):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:            SA RX GETTBL(SC2VLtMap):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:              SA RX GET(SC2VLntMap):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:           SA RX GETTBL(SC2VLntMap):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:               SA RX GET(SC2VLrMap):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:            SA RX GETTBL(SC2VLrMap):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:              SA RX GET(SwitchInfo):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:          SA RX GETTBLE(SwitchInfo):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:                     SA RX GET(LFT):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:                  SA RX GETTBL(LFT):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:                     SA RX GET(MFT):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:                  SA RX GETTBL(MFT):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:           SA RX GET(VLArbitration):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:        SA RX GETTBL(VLArbitration):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:                  SA RX GET(SmInfo):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:               SA RX GETTBL(SmInfo):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:              SA RX GET(InformInfo):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:           SA RX GETTBL(InformInfo):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:              SA RX GET(LinkRecord):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:           SA RX GETTBL(LinkRecord):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:           SA RX GET(ServiceRecord):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:           SA RX SET(ServiceRecord):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:        SA RX GETTBL(ServiceRecord):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:        SA RX DELETE(ServiceRecord):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:              SA RX GET(PKeyRecord):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:           SA RX GETTBL(PKeyRecord):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:              SA RX GET(PathRecord):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:           SA RX GETTBL(PathRecord):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:          SA RX GET(McMemberRecord):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:          SA RX SET(McMemberRecord):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:       SA RX GETTBL(McMemberRecord):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:       SA RX DELETE(McMemberRecord):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:        SA RX GETTRACE(TraceRecord):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:    SA RX GETMULTI(MultiPathRecord):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:           SA RX GET(vFabricRecord):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:        SA RX GETTBL(vFabricRecord):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:        SA RX GET(FabricInfoRecord):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:   SA RX GET(QuarantinedNodeRecord):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]: SA RX GETTBL(QuarantinedNodeRecord):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:          SA RX GET(CongInfoRecord):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:       SA RX GETTBL(CongInfoRecord):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:        SA RX GET(SwitchCongRecord):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:     SA RX GETTBL(SwitchCongRecord):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:    SA RX GET(SwitchPortCongRecord):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]: SA RX GETTBL(SwitchPortCongRecord):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:           SA RX GET(HFICongRecord):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:           SA RX GET(HFICongRecord):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:       SA RX GET(HFICongCtrlRecord):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:       SA RX GET(HFICongCtrlRecord):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:           SA RX GET(BfrCtrlRecord):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:        SA RX GETTBL(BfrCtrlRecord):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:               SA RX GET(CableInfo):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:            SA RX GETTBL(CableInfo):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:         SA RX GET(PortGroupRecord):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:      SA RX GETTBL(PortGroupRecord):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:      SA RX GET(PortGroupFwdRecord):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:   SA RX GETTBL(PortGroupFwdRecord):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:                             (null):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:                             (null):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:           SA RX REPORTRESP(Notice):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]: SA RX GET(DeviceGroupMemberRecord):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]: SA RX GETTBL(DeviceGroupMemberRecord):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:   SA RX GET(DeviceGroupNameRecord):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]: SA RX GETTBL(DeviceGroupNameRecord):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:  SA RX GET(DeviceTreeMemberRecord):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]: SA RX GETTBL(DeviceTreeMemberRecord):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:           SA RX DUPLICATE REQUESTS:          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]: SA RX DEAD RMPP TRANSACTION PACKET:          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:                SA DROPPED REQUESTS:          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:           SA NO AVAILABLE CONTEXTS:          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:          SA RX GETMULTI() Non-RMPP:          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:          SA RX GETMULTI RMPP Abort:          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:          SA TX GETMULTI RMPP Abort:          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:         SA TX GETMULTI Ack Retries:          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:                 SA TX RMPP Retries:          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:                    SA TX RMPP STOP:          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:          SA TX RMMP ABORT(TIMEOUT):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:    SA TX RMMP ABORT(BAD FIRST SEG):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:         SA TX RMMP ABORT(BAD TYPE):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]: SA TX RMMP ABORT(WINDOW TOO SMALL):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:   SA TX RMMP ABORT(SEGNUM TOO BIG):          0          0          0
Apr 19 05:17:37 rdma06 opafmd[1140340]:      SA TX RMMP ABORT(BAD VERSION):          0          0          0
Apr 19 05:17:37 rdma06 fm0_sm[1140340]: [RT-Conf server]: Setting SM LogLevel to 7
Apr 19 05:17:38 rdma06 opafmd[1140340]: SA TX RMMP ABORT(TOO MANY RETRIES)ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:17:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:17:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:17:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:17:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:17:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:17:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:17:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:17:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:17:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:17:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:17:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:17:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:17:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:17:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:17:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:17:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:17:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:17:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:17:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:17:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:17:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:17:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:17:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:17:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:17:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:17:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:17:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:17:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:17:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:17:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:17:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:17:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:17:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:17:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:17:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:17:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:17:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:17:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:17:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:17:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:17:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:17:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:17:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:17:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:17:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:17:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:17:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:17:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:17:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:17:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:17:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:17:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:17:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:17:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:17:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:17:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:17:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:17:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:17:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:17:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:17:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:17:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:17:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:17:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:17:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:17:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:17:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:17:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:17:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:17:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:17:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:17:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:17:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:17:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:17:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:17:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:17:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:17:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:17:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:17:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:17:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:17:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:17:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:17:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:17:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:17:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:17:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:17:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:17:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:18:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:18:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:18:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:18:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:18:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:18:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:18:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:18:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:18:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:18:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:18:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:18:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:18:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:18:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:18:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:18:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:18:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:18:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:18:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:18:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:18:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:18:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:18:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:18:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:18:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:18:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:18:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:18:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:18:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:18:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:18:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:18:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:18:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:18:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:18:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:18:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:18:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:18:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:18:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:18:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:18:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:18:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:18:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:18:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:18:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:18:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:18:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:18:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:18:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:18:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:18:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:18:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:18:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:18:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:18:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:18:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:18:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:18:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:18:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:18:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:18:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:18:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:18:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:18:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:18:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:18:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:18:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:18:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:18:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:18:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:18:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:18:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:18:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:18:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:18:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:18:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:18:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:18:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:18:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:18:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:18:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:18:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:18:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:18:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:18:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:18:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:18:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:18:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:18:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:18:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:18:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:18:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:18:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:18:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:18:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:18:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:18:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:18:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:18:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:18:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:18:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:18:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:18:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:18:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:18:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:18:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:18:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:18:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:18:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:18:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:18:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:18:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:18:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:18:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:18:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:18:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:18:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:18:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:18:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:18:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:18:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:18:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:18:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:18:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:18:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:18:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:18:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:18:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:18:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:18:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:18:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:18:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:18:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:18:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:18:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:18:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:18:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:18:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:18:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:18:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:18:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:18:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:18:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:18:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:18:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:18:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:18:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:18:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:18:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:18:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:18:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:18:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:18:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:18:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:18:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:18:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:18:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:18:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:18:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:18:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:18:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:18:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:18:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:18:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:18:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:18:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:18:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:18:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:18:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:18:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:18:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:18:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:18:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:18:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:18:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:18:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:18:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:18:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:18:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:18:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:18:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:18:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:18:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:18:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:18:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:18:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:18:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:18:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:18:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:18:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:18:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:18:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:18:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:18:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:18:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:18:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:18:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:18:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:18:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:18:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:18:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:18:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:18:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:18:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:18:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:18:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:18:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:18:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:18:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:18:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:18:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:18:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:18:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:18:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:18:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:18:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:19:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:19:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:19:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:19:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:19:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:19:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:19:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:19:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:19:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:19:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:19:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:19:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:19:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:19:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:19:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:19:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:19:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:19:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:19:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:19:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:19:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:19:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:19:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:19:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:19:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:19:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:19:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:19:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:19:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:19:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:19:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:19:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:19:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:19:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:19:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:19:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:19:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:19:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:19:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:19:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:19:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:19:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:19:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:19:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:19:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:19:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:19:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:19:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:19:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:19:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:19:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:19:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:19:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:19:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:19:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:19:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:19:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:19:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:19:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:19:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:19:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:19:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:19:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:19:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:19:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:19:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:19:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:19:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:19:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:19:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:19:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:19:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:19:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:19:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:19:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:19:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:19:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:19:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:19:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:19:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:19:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:19:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:19:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:19:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:19:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:19:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:19:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:19:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:19:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:19:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:19:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:19:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:19:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:19:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:19:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:19:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:19:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:19:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:19:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:19:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:19:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:19:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:19:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:19:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:19:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:19:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:19:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:19:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:19:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:19:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:19:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:19:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:19:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:19:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:19:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:19:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:19:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:19:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:19:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:19:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:19:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:19:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:19:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:19:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:19:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:19:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:19:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:19:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:19:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:19:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:19:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:19:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:19:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:19:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:19:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:19:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:19:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:19:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:19:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:19:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:19:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:19:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:19:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:19:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:21:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:21:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:21:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:21:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:21:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:21:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:21:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:21:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:21:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:21:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:21:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:21:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:21:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:21:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:21:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:21:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:21:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:21:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:21:49 rdma06 rsyslogd[1041]: imjournal: 5317 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 19 05:21:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:21:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:21:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:21:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:21:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:21:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:21:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:21:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:21:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:21:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:21:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:21:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:21:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:21:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:21:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:21:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:21:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:21:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:21:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:21:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:21:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:21:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:21:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:21:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:21:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:21:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:21:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:21:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:21:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:21:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:21:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:21:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:21:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:21:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:21:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:21:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:22:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:22:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:22:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:22:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:22:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:22:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:22:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:22:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:22:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:22:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:22:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:22:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:22:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:22:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:22:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:22:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:22:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:22:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:22:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:22:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:22:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:22:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:22:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:22:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:22:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:22:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:22:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:22:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:22:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:22:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:22:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:22:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:22:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:22:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:22:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:22:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:22:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:22:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:22:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:22:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:22:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:22:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:22:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:22:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:22:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:22:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:22:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:22:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:22:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:22:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:22:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:22:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:22:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:22:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:22:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:22:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:22:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:22:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:22:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:22:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:22:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:22:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:22:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:22:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:22:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:22:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:22:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:22:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:22:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:22:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:22:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:22:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:22:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:22:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:22:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:22:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:22:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:22:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:22:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:22:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:22:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:22:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:22:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:22:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:22:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:22:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:22:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:22:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:22:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:22:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:22:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:22:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:22:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:22:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:22:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:22:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:22:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:22:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:22:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:22:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:22:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:22:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:22:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:22:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:22:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:22:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:22:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:22:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:22:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:22:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:22:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:22:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:22:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:22:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:22:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:22:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:22:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:22:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:22:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:22:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:22:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:22:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:22:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:22:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:22:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:22:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:22:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:22:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:22:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:22:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:22:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:22:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:22:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:22:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:22:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:22:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:22:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:22:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:22:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:22:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:22:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:22:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:22:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:22:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:22:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:22:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:22:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:22:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:22:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:22:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:22:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:22:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:22:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:22:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:22:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:22:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:22:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:22:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:22:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:22:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:22:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:22:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:22:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:22:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:22:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:22:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:22:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:22:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:22:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:22:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:22:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:22:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:22:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:22:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:22:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:22:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:22:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:22:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:22:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:22:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:22:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:22:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:22:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:22:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:22:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:22:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:22:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:22:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:22:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:22:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:22:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:22:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:22:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:22:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:22:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:22:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:22:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:22:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:22:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:22:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:22:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:22:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:22:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:22:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:22:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:22:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:22:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:22:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:22:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:22:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:22:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:22:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:22:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:22:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:22:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:22:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:23:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:23:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:23:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:23:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:23:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:23:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:23:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:23:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:23:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:23:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:23:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:23:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:23:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:23:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:23:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:23:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:23:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:23:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:23:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:23:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:23:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:23:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:23:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:23:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:23:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:23:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:23:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:23:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:23:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:23:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:23:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:23:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:23:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:23:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:23:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:23:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:23:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:23:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:23:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:23:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:23:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:23:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:23:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:23:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:23:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:23:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:23:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:23:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:23:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:23:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:23:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:23:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:23:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:23:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:23:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:23:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:23:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:23:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:23:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:23:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:23:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:23:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:23:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:23:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:23:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:23:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:23:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:23:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:23:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:23:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:23:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:23:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:23:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:23:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:23:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:23:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:23:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:23:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:23:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:23:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:23:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:23:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:23:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:23:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:23:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:23:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:23:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:23:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:23:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:23:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:23:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:23:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:23:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:23:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:23:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:23:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:23:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:23:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:23:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:23:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:23:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:23:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:23:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:23:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:23:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:23:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:23:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:23:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:23:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:23:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:23:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:23:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:23:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:23:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:23:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:23:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:23:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:23:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:23:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:23:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:23:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:23:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:23:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:23:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:23:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:23:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:23:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:23:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:23:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:23:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:23:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:23:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:23:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:23:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:23:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:23:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:23:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:23:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:23:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:23:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:23:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:23:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:23:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:23:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:23:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:23:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:23:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:23:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:23:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:23:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:23:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:23:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:23:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:23:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:23:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:23:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:23:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:23:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:23:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:23:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:23:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:23:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:23:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:23:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:23:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:23:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:23:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:23:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:23:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:23:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:23:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:24:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:24:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:24:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:24:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:24:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:24:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:24:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:24:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:24:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:24:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:24:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:24:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:24:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:24:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:24:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:24:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:24:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:24:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:24:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:24:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:24:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:24:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:24:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:24:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:24:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:24:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:24:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:24:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:24:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:24:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:24:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:24:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:24:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:24:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:24:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:24:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:24:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:24:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:24:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:24:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:24:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:24:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:24:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:24:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:24:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:24:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:24:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:24:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:24:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:24:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:24:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:24:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:24:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:24:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:24:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:24:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:24:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:24:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:24:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:24:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:24:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:24:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:24:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:24:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:24:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:24:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:24:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:24:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:24:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:24:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:24:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:24:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:24:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:24:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:24:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:24:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:24:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:24:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:24:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:24:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:24:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:24:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:24:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:24:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:24:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:24:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:24:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:24:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:24:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:24:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:24:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:24:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:24:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:24:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:24:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:24:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:24:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:24:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:24:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:24:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:24:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:24:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:24:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:24:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:24:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:24:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:24:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:24:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:24:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:24:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:24:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:24:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:24:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:24:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:24:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:24:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:24:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:24:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:24:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:24:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:24:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:24:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:24:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:24:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:24:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:24:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:24:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:24:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:24:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:24:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:24:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:24:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:24:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:24:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:24:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:24:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:24:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:24:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:24:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:24:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:24:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:24:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:24:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:24:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:24:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:24:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:24:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:24:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:24:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:24:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:24:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:24:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:24:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:24:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:24:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:24:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:24:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:24:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:24:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:24:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:24:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:24:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:24:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:24:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:24:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:24:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:24:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:24:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:24:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:24:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:24:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:24:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:24:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:24:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:24:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:24:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:24:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:24:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:24:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:24:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:24:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:24:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:24:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:24:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:24:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:24:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:24:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:24:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:24:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:24:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:24:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:24:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:24:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:24:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:24:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:24:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:24:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:24:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:25:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:25:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:25:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:25:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:25:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:25:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:25:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:25:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:25:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:25:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:25:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:25:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:25:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:25:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:25:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:25:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:25:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:25:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:25:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:25:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:25:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:25:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:25:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:25:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:25:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:25:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:25:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:25:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:25:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:25:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:25:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:25:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:25:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:25:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:25:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:25:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:25:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:25:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:25:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:25:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:25:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:25:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:25:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:25:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:25:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:25:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:25:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:25:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:25:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:25:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:25:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:25:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:25:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:25:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:25:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:25:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:25:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:25:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:25:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:25:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:25:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:25:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:25:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:25:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:25:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:25:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:25:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:25:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:25:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:25:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:25:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:25:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:25:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:25:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:25:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:25:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:25:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:25:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:25:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:25:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:25:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:25:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:25:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:25:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:25:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:25:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:25:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:25:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:25:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:25:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:25:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:25:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:25:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:25:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:25:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:25:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:25:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:25:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:25:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:25:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:25:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:25:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:25:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:25:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:25:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:25:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:25:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:25:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:25:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:25:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:25:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:25:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:25:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:25:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:25:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:25:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:25:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:25:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:25:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:25:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:25:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:25:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:25:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:25:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:25:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:25:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:25:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:25:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:25:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:25:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:25:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:25:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:25:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:25:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:25:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:25:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:25:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:25:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:25:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:25:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:25:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:25:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:25:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:25:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:25:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:25:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:25:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:25:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:25:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:25:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:25:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:25:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:25:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:25:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:25:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:25:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:25:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:25:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:25:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:25:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:25:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:25:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:25:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:25:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:25:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:25:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:25:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:25:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:25:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:25:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:25:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:26:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:26:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:26:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:26:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:26:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:26:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:26:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:26:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:26:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:26:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:26:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:26:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:26:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:26:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:26:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:26:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:26:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:26:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:26:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:26:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:26:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:26:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:26:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:26:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:26:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:26:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:26:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:26:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:26:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:26:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:26:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:26:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:26:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:26:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:26:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:26:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:26:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:26:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:26:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:26:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:26:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:26:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:26:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:26:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:26:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:26:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:26:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:26:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:26:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:26:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:26:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:26:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:26:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:26:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:26:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:26:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:26:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:26:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:26:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:26:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:26:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:26:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:26:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:26:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:26:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:26:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:26:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:26:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:26:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:26:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:26:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:26:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:26:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:26:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:26:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:26:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:26:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:26:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:26:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:26:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:26:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:26:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:26:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:26:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:26:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:26:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:26:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:26:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:26:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:26:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:26:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:26:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:26:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:26:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:26:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:26:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:26:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:26:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:26:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:26:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:26:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:26:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:26:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:26:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:26:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:26:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:26:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:26:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:26:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:26:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:26:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:26:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:26:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:26:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:26:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:26:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:26:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:26:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:26:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:26:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:26:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:26:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:26:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:26:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:26:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:26:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:26:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:26:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:26:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:26:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:26:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:26:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:26:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:26:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:26:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:26:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:26:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:26:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:26:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:26:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:26:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:26:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:26:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:26:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:26:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:26:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:26:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:26:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:26:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:26:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:26:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:26:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:26:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:26:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:26:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:26:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:26:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:26:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:26:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:26:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:26:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:26:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:26:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:26:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:26:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:26:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:26:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:26:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:26:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:26:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:26:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:26:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:26:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:26:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:26:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:26:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:26:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:26:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:26:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:26:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:26:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:26:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:26:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:26:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:26:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:26:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:26:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:26:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:26:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:26:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:26:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:26:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:26:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:26:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:26:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:26:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:26:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:26:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:26:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:26:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:26:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:26:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:26:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:26:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:26:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:26:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:26:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:26:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:26:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:26:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:26:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:26:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:26:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:26:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:26:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:26:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:27:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:27:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:27:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:27:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:27:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:27:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:27:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:27:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:27:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:27:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:27:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:27:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:27:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:27:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:27:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:27:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:27:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:27:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:27:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:27:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:27:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:27:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:27:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:27:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:27:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:27:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:27:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:27:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:27:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:27:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:27:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:27:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:27:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:27:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:27:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:27:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:27:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:27:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:27:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:27:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:27:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:27:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:27:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:27:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:27:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:27:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:27:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:27:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:27:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:27:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:27:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:27:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:27:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:27:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:27:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:27:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:27:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:27:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:27:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:27:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:27:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:27:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:27:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:27:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:27:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:27:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:27:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:27:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:27:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:27:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:27:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:27:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:27:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:27:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:27:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:27:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:27:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:27:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:27:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:27:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:27:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:27:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:27:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:27:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:27:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:27:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:27:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:27:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:27:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:27:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:27:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:27:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:27:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:27:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:27:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:27:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:27:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:27:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:27:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:27:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:27:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:27:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:27:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:27:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:27:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:27:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:27:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:27:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:27:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:27:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:27:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:27:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:27:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:27:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:27:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:27:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:27:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:27:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:27:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:27:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:27:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:27:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:27:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:27:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:27:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:27:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:27:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:27:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:27:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:27:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:27:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:27:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:27:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:27:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:27:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:27:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:27:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:27:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:27:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:27:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:27:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:27:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:27:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:27:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:27:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:27:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:27:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:27:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:27:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:27:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:27:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:27:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:27:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:27:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:27:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:27:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:27:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:27:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:27:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:27:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:27:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:27:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:27:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:27:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:27:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:27:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:27:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:27:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:27:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:27:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:27:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:27:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:27:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:27:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:27:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:27:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:27:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:27:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:27:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:27:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:27:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:27:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:27:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:27:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:27:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:27:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:27:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:27:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:27:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:27:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:27:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:27:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:27:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:27:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:27:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:27:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:27:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:27:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:27:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:27:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:27:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:27:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:27:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:27:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:27:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:27:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:27:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:27:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:27:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:27:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:27:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:27:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:27:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:27:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:27:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:27:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:28:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:28:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:28:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:28:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:28:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:28:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:28:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:28:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:28:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:28:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:28:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:28:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:28:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:28:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:28:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:28:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:28:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:28:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:28:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:28:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:28:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:28:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:28:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:28:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:28:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:28:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:28:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:28:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:28:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:28:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:28:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:28:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:28:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:28:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:28:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:28:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:28:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:28:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:28:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:28:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:28:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:28:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:28:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:28:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:28:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:28:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:28:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:28:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:28:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:28:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:28:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:28:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:28:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:28:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:28:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:28:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:28:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:28:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:28:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:28:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:28:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:28:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:28:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:28:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:28:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:28:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:28:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:28:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:28:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:28:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:28:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:28:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:28:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:28:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:28:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:28:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:28:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:28:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:28:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:28:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:28:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:28:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:28:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:28:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:28:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:28:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:28:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:28:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:28:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:28:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:28:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:28:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:28:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:28:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:28:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:28:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:28:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:28:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:28:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:28:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:28:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:28:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:28:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:28:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:28:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:28:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:28:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:28:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:28:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:28:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:28:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:28:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:28:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:28:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:28:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:28:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:28:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:28:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:28:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:28:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:28:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:28:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:28:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:28:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:28:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:28:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:28:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:28:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:28:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:28:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:28:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:28:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:28:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:28:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:28:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:28:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:28:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:28:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:28:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:28:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:28:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:28:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:28:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:28:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:28:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:28:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:28:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:28:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:28:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:28:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:28:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:28:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:28:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:28:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:28:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:28:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:28:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:28:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:28:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:28:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:28:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:28:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:28:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:28:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:28:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:28:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:28:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:28:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:28:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:28:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:28:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:28:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:28:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:28:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:28:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:28:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:28:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:28:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:28:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:28:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:28:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:28:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:28:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:28:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:28:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:28:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:28:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:28:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:28:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:28:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:28:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:28:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:28:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:28:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:28:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:28:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:28:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:28:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:28:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:28:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:28:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:28:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:28:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:28:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:28:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:28:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:28:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:28:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:28:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:28:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:28:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:28:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:28:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:28:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:28:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:28:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:29:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:29:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:29:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:29:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:29:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:29:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:29:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:29:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:29:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:29:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:29:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:29:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:29:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:29:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:29:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:29:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:29:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:29:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:29:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:29:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:29:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:29:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:29:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:29:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:29:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:29:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:29:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:29:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:29:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:29:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:29:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:29:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:29:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:29:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:29:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:29:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:29:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:29:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:29:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:29:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:29:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:29:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:29:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:29:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:29:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:29:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:29:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:29:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:29:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:29:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:29:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:29:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:29:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:29:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:29:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:29:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:29:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:29:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:29:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:29:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:29:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:29:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:29:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:29:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:29:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:29:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:29:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:29:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:29:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:29:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:29:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:29:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:29:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:29:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:29:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:29:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:29:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:29:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:29:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:29:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:29:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:29:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:29:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:29:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:29:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:29:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:29:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:29:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:29:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:29:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:29:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:29:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:29:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:29:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:29:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:29:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:29:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:29:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:29:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:29:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:29:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:29:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:29:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:29:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:29:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:29:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:29:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:29:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:29:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:29:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:29:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:29:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:29:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:29:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:29:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:29:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:29:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:29:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:29:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:29:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:29:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:29:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:29:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:29:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:29:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:29:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:29:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:29:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:29:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:29:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:29:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:29:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:29:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:29:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:29:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:29:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:29:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:29:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:29:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:29:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:29:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:29:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:29:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:29:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:29:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:29:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:29:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:29:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:29:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:29:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:29:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:29:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:29:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:29:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:29:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:29:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:29:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:29:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:29:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:29:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:29:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:29:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:31:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:31:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:31:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:31:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:31:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:31:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:31:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:31:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:31:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:31:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:31:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:31:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:31:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:31:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:31:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:31:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:31:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:31:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:31:54 rdma06 rsyslogd[1041]: imjournal: 5291 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 19 05:31:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:31:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:31:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:31:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:31:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:31:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:31:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:31:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:31:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:31:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:31:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:31:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:31:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:31:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:31:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:31:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:31:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:31:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:32:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:32:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:32:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:32:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:32:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:32:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:32:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:32:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:32:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:32:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:32:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:32:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:32:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:32:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:32:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:32:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:32:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:32:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:32:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:32:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:32:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:32:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:32:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:32:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:32:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:32:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:32:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:32:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:32:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:32:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:32:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:32:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:32:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:32:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:32:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:32:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:32:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:32:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:32:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:32:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:32:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:32:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:32:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:32:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:32:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:32:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:32:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:32:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:32:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:32:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:32:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:32:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:32:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:32:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:32:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:32:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:32:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:32:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:32:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:32:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:32:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:32:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:32:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:32:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:32:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:32:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:32:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:32:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:32:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:32:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:32:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:32:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:32:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:32:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:32:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:32:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:32:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:32:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:32:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:32:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:32:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:32:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:32:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:32:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:32:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:32:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:32:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:32:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:32:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:32:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:32:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:32:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:32:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:32:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:32:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:32:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:32:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:32:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:32:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:32:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:32:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:32:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:32:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:32:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:32:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:32:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:32:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:32:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:32:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:32:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:32:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:32:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:32:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:32:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:32:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:32:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:32:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:32:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:32:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:32:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:32:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:32:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:32:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:32:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:32:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:32:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:32:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:32:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:32:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:32:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:32:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:32:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:32:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:32:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:32:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:32:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:32:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:32:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:32:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:32:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:32:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:32:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:32:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:32:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:32:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:32:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:32:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:32:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:32:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:32:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:32:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:32:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:32:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:32:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:32:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:32:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:32:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:32:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:32:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:32:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:32:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:32:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:32:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:32:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:32:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:32:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:32:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:32:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:32:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:32:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:32:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:32:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:32:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:32:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:32:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:32:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:32:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:32:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:32:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:32:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:32:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:32:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:32:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:32:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:32:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:32:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:32:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:32:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:32:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:32:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:32:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:32:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:32:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:32:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:32:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:32:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:32:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:32:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:32:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:32:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:32:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:32:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:32:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:32:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:32:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:32:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:32:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:32:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:32:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:32:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:32:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:32:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:32:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:32:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:32:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:32:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:33:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:33:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:33:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:33:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:33:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:33:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:33:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:33:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:33:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:33:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:33:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:33:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:33:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:33:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:33:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:33:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:33:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:33:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:33:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:33:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:33:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:33:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:33:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:33:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:33:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:33:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:33:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:33:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:33:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:33:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:33:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:33:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:33:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:33:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:33:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:33:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:33:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:33:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:33:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:33:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:33:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:33:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:33:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:33:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:33:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:33:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:33:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:33:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:33:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:33:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:33:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:33:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:33:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:33:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:33:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:33:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:33:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:33:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:33:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:33:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:33:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:33:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:33:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:33:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:33:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:33:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:33:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:33:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:33:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:33:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:33:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:33:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:33:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:33:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:33:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:33:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:33:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:33:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:33:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:33:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:33:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:33:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:33:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:33:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:33:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:33:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:33:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:33:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:33:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:33:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:33:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:33:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:33:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:33:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:33:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:33:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:33:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:33:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:33:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:33:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:33:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:33:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:33:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:33:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:33:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:33:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:33:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:33:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:33:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:33:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:33:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:33:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:33:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:33:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:33:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:33:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:33:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:33:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:33:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:33:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:33:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:33:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:33:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:33:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:33:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:33:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:33:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:33:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:33:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:33:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:33:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:33:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:33:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:33:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:33:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:33:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:33:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:33:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:33:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:33:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:33:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:33:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:33:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:33:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:33:40 rdma06 fm0_sm[1140340]: [RT-Conf server]: Setting SM LogLevel to 7
Apr 19 05:33:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:33:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:33:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:33:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:33:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:33:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:33:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:33:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:33:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:33:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:33:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:33:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:33:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:33:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:33:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:33:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:33:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:33:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:33:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:33:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:33:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:33:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:33:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:33:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:33:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:33:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:33:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:33:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:33:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:33:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:33:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:33:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:33:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:33:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:33:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:33:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:33:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:33:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:33:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:33:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:33:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:33:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:33:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:33:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:33:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:33:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:33:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:33:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:33:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:33:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:33:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:33:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:33:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:33:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:33:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:33:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:33:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:33:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:33:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:33:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:33:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:33:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:33:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:33:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:33:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:33:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:33:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:33:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:33:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:33:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:33:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:33:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:34:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:34:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:34:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:34:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:34:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:34:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:34:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:34:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:34:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:34:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:34:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:34:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:34:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:34:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:34:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:34:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:34:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:34:03 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:34:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:34:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:34:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:34:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:34:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:34:08 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:34:08 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:34:08 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:34:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:34:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:34:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:34:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:34:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:34:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:34:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:34:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:34:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:34:08 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:34:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:34:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:34:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:34:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:34:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:34:13 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:34:13 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:34:13 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:34:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:34:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:34:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:34:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:34:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:34:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:34:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:34:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:34:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:34:13 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:34:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:34:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:34:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:34:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:34:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:34:18 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:34:18 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:34:18 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:34:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:34:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:34:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:34:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:34:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:34:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:34:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:34:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:34:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:34:18 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:34:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:34:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:34:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:34:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:34:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:34:23 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:34:23 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:34:23 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:34:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:34:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:34:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:34:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:34:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:34:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:34:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:34:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:34:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:34:23 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:34:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:34:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:34:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:34:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:34:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:34:28 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:34:28 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:34:28 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:34:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:34:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:34:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:34:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:34:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:34:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:34:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:34:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:34:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:34:28 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:34:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:34:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:34:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:34:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:34:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:34:33 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:34:33 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:34:33 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:34:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:34:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:34:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:34:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:34:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:34:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:34:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:34:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:34:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:34:33 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:34:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:34:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:34:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:34:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:34:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:34:38 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:34:38 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:34:38 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:34:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:34:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:34:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:34:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:34:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:34:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:34:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:34:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:34:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:34:38 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:34:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:34:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:34:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:34:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:34:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:34:43 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:34:43 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:34:43 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:34:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:34:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:34:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:34:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:34:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:34:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:34:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:34:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:34:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:34:43 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:34:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:34:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:34:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:34:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:34:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:34:48 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:34:48 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:34:48 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:34:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:34:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:34:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:34:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:34:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:34:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:34:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:34:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:34:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:34:48 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:34:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:34:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:34:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:34:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:34:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:34:53 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:34:53 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:34:53 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:34:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:34:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:34:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:34:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:34:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:34:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:34:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:34:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:34:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:34:53 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:34:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:34:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:34:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:34:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:34:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:34:58 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:34:58 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:34:58 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:34:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:34:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:34:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:34:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:34:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:34:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:34:58 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:34:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:34:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:34:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:35:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:35:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:35:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:35:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:35:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:35:03 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:35:03 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:35:03 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:35:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:35:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:35:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:35:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:35:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:35:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:35:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:35:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:35:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:35:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:35:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:35:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:35:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:35:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:35:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:35:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:35:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:35:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:35:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:35:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:35:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:35:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:35:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:35:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:35:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:35:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:35:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:35:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:35:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:35:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:35:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:35:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:35:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:35:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:35:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:35:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:35:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:35:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:35:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:35:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:35:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:35:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:35:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:35:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:35:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:35:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:35:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:35:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:35:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:35:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:35:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:35:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:35:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:35:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:35:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:35:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:35:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:35:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:35:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:35:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:35:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:35:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:35:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:35:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:35:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:35:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:35:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:35:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:35:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:35:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:35:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:35:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:35:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:35:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:35:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:35:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:35:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:35:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:35:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:35:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:35:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:35:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:35:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:35:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:35:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:35:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:35:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:35:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:35:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:35:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:35:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:35:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:35:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:35:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:35:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:35:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:35:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:35:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:35:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:35:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:35:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:35:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:35:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:35:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:35:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:35:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:35:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:35:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:35:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:35:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:35:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:35:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:35:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:35:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:35:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:35:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:35:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:35:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:35:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:35:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:35:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:35:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:35:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:35:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:35:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:35:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:35:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:35:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:35:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:35:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:35:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:35:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:35:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:35:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:35:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:35:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:35:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:35:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:35:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:35:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:35:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:35:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:35:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:35:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:35:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:35:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:35:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:35:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:35:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:35:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:35:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:35:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:35:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:35:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:35:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:35:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:35:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:35:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:35:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:35:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:35:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:35:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:35:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:35:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:35:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:35:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:35:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:35:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:35:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:35:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:35:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:35:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:35:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:35:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:35:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:35:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:35:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:35:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:35:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:35:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:35:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:35:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:35:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:35:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:35:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:35:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:35:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:35:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:35:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:35:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:35:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:35:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:35:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:35:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:35:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:35:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:35:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:35:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:35:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:35:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:35:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:35:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:35:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:35:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:35:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:35:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:35:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:35:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:36:00 rdma06 systemd[1]: Starting dnf makecache...
Apr 19 05:36:01 rdma06 dnf[1216578]: Updating Subscription Management repositories.
Apr 19 05:36:01 rdma06 dnf[1216578]: Unable to read consumer identity
Apr 19 05:36:01 rdma06 dnf[1216578]: This system is not registered with an entitlement server. You can use subscription-manager to register.
Apr 19 05:36:01 rdma06 dnf[1216578]: Metadata cache refreshed recently.
Apr 19 05:36:01 rdma06 systemd[1]: dnf-makecache.service: Deactivated successfully.
Apr 19 05:36:01 rdma06 systemd[1]: Finished dnf makecache.
Apr 19 05:36:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:36:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:36:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:36:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:36:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:36:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:36:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:36:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:36:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:36:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:36:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:36:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:36:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:36:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:36:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:36:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:36:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:36:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:36:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:36:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:36:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:36:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:36:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:36:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:36:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:36:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:36:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:36:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:36:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:36:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:36:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:36:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:36:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:36:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:36:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:36:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:36:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:36:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:36:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:36:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:36:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:36:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:36:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:36:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:36:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:36:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:36:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:36:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:36:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:36:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:36:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:36:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:36:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:36:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:36:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:36:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:36:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:36:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:36:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:36:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:36:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:36:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:36:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:36:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:36:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:36:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:36:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:36:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:36:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:36:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:36:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:36:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:36:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:36:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:36:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:36:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:36:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:36:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:36:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:36:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:36:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:36:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:36:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:36:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:36:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:36:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:36:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:36:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:36:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:36:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:36:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:36:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:36:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:36:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:36:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:36:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:36:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:36:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:36:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:36:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:36:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:36:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:36:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:36:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:36:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:36:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:36:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:36:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:36:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:36:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:36:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:36:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:36:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:36:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:36:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:36:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:36:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:36:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:36:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:36:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:36:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:36:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:36:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:36:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:36:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:36:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:36:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:36:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:36:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:36:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:36:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:36:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:36:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:36:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:36:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:36:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:36:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:36:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:36:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:36:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:36:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:36:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:36:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:36:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:36:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:36:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:36:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:36:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:36:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:36:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:36:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:36:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:36:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:36:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:36:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:36:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:36:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:36:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:36:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:36:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:36:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:36:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:36:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:36:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:36:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:36:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:36:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:36:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:36:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:36:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:36:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:36:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:36:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:36:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:36:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:36:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:36:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:36:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:36:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:36:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:36:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:36:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:36:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:36:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:36:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:36:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:36:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:36:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:36:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:36:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:36:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:36:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:36:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:36:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:36:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:36:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:36:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:36:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:36:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:36:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:36:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:36:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:36:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:36:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:36:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:36:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:36:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:36:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:36:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:36:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:36:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:36:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:36:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:36:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:36:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:36:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:37:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:37:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:37:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:37:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:37:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:37:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:37:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:37:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:37:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:37:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:37:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:37:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:37:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:37:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:37:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:37:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:37:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:37:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:37:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:37:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:37:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:37:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:37:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:37:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:37:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:37:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:37:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:37:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:37:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:37:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:37:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:37:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:37:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:37:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:37:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:37:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:37:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:37:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:37:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:37:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:37:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:37:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:37:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:37:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:37:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:37:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:37:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:37:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:37:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:37:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:37:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:37:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:37:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:37:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:37:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:37:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:37:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:37:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:37:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:37:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:37:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:37:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:37:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:37:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:37:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:37:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:37:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:37:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:37:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:37:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:37:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:37:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:37:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:37:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:37:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:37:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:37:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:37:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:37:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:37:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:37:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:37:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:37:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:37:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:37:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:37:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:37:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:37:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:37:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:37:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:37:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:37:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:37:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:37:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:37:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:37:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:37:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:37:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:37:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:37:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:37:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:37:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:37:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:37:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:37:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:37:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:37:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:37:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:37:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:37:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:37:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:37:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:37:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:37:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:37:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:37:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:37:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:37:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:37:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:37:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:37:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:37:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:37:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:37:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:37:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:37:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:37:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:37:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:37:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:37:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:37:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:37:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:37:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:37:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:37:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:37:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:37:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:37:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:37:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:37:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:37:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:37:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:37:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:37:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:37:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:37:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:37:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:37:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:37:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:37:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:37:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:37:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:37:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:37:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:37:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:37:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:37:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:37:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:37:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:37:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:37:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:37:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:37:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:37:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:37:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:37:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:37:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:37:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:37:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:37:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:37:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:37:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:37:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:37:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:37:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:37:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:37:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:37:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:37:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:37:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:37:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:37:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:37:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:37:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:37:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:37:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:37:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:37:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:37:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:37:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:37:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:37:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:37:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:37:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:37:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:37:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:37:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:37:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:37:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:37:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:37:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:37:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:37:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:37:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:37:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:37:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:37:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:37:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:37:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:37:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:37:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:37:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:37:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:37:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:37:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:37:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:38:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:38:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:38:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:38:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:38:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:38:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:38:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:38:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:38:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:38:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:38:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:38:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:38:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:38:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:38:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:38:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:38:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:38:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:38:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:38:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:38:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:38:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:38:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:38:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:38:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:38:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:38:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:38:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:38:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:38:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:38:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:38:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:38:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:38:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:38:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:38:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:38:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:38:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:38:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:38:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:38:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:38:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:38:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:38:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:38:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:38:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:38:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:38:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:38:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:38:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:38:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:38:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:38:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:38:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:38:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:38:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:38:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:38:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:38:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:38:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:38:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:38:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:38:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:38:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:38:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:38:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:38:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:38:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:38:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:38:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:38:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:38:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:38:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:38:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:38:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:38:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:38:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:38:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:38:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:38:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:38:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:38:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:38:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:38:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:38:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:38:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:38:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:38:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:38:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:38:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:38:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:38:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:38:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:38:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:38:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:38:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:38:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:38:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:38:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:38:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:38:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:38:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:38:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:38:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:38:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:38:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:38:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:38:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:38:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:38:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:38:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:38:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:38:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:38:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:38:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:38:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:38:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:38:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:38:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:38:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:38:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:38:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:38:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:38:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:38:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:38:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:38:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:38:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:38:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:38:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:38:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:38:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:38:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:38:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:38:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:38:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:38:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:38:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:38:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:38:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:38:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:38:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:38:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:38:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:38:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:38:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:38:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:38:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:38:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:38:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:38:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:38:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:38:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:38:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:38:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:38:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:38:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:38:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:38:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:38:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:38:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:38:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:38:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:38:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:38:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:38:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:38:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:38:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:38:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:38:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:38:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:38:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:38:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:38:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:38:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:38:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:38:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:38:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:38:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:38:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:38:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:38:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:38:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:38:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:38:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:38:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:38:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:38:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:38:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:38:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:38:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:38:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:38:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:38:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:38:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:38:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:38:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:38:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:38:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:38:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:38:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:38:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:38:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:38:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:38:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:38:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:38:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:38:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:38:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:38:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:38:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:38:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:38:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:38:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:38:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:38:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:39:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:39:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:39:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:39:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:39:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:39:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:39:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:39:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:39:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:39:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:39:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:39:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:39:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:39:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:39:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:39:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:39:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:39:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:39:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:39:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:39:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:39:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:39:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:39:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:39:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:39:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:39:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:39:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:39:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:39:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:39:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:39:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:39:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:39:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:39:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:39:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:39:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:39:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:39:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:39:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:39:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:39:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:39:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:39:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:39:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:39:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:39:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:39:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:39:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:39:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:39:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:39:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:39:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:39:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:39:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:39:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:39:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:39:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:39:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:39:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:39:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:39:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:39:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:39:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:39:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:39:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:39:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:39:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:39:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:39:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:39:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:39:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:39:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:39:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:39:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:39:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:39:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:39:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:39:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:39:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:39:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:39:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:39:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:39:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:39:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:39:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:39:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:39:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:39:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:39:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:39:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:39:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:39:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:39:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:39:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:39:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:39:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:39:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:39:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:39:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:39:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:39:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:39:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:39:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:39:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:39:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:39:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:39:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:39:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:39:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:39:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:39:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:39:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:39:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:39:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:39:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:39:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:39:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:39:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:39:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:39:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:39:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:39:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:39:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:39:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:39:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:41:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:41:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:41:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:41:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:41:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:41:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:41:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:41:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:41:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:41:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:41:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:41:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:41:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:41:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:41:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:41:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:41:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:41:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:41:59 rdma06 rsyslogd[1041]: imjournal: 5299 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 19 05:42:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:42:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:42:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:42:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:42:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:42:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:42:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:42:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:42:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:42:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:42:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:42:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:42:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:42:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:42:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:42:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:42:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:42:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:42:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:42:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:42:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:42:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:42:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:42:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:42:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:42:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:42:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:42:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:42:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:42:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:42:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:42:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:42:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:42:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:42:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:42:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:42:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:42:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:42:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:42:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:42:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:42:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:42:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:42:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:42:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:42:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:42:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:42:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:42:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:42:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:42:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:42:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:42:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:42:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:42:18 rdma06 systemd[1]: proc-sys-fs-binfmt_misc.automount: Got automount request for /proc/sys/fs/binfmt_misc, triggered by 1217020 (sysctl)
Apr 19 05:42:18 rdma06 systemd[1]: Mounting Arbitrary Executable File Formats File System...
Apr 19 05:42:18 rdma06 systemd[1]: Mounted Arbitrary Executable File Formats File System.
Apr 19 05:42:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:42:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:42:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:42:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:42:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:42:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:42:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:42:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:42:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:42:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:42:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:42:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:42:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:42:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:42:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:42:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:42:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:42:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:42:21 rdma06 kernel: seq_file: buggy .next function _ctx_stats_seq_next [hfi1] did not update position index
Apr 19 05:42:21 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: INVALID_CSR_ADDR,SEQ_CRC_ERR,RST_FOR_LINK_TIMEOUT,HOLD_REINIT,NEG_EDGE_LINK_TRANSFER_ACTIVE
Apr 19 05:42:21 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: INVALID_CSR_ADDR,HOLD_REINIT
Apr 19 05:42:21 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 05:42:21 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 05:42:21 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 05:42:21 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 05:42:21 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 05:42:21 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 05:42:21 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 05:42:21 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 05:42:21 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 05:42:21 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 05:42:21 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 05:42:21 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 05:42:21 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 05:42:21 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 05:42:21 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 05:42:21 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 05:42:21 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 05:42:21 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 05:42:21 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 05:42:21 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: Repeating LCB Err bits 0x8000000 - masking
Apr 19 05:42:21 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 05:42:21 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 05:42:21 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 05:42:21 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 05:42:21 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 05:42:21 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 05:42:21 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 05:42:21 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 05:42:21 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 05:42:21 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 05:42:21 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 05:42:21 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 05:42:21 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 05:42:21 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 05:42:21 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 05:42:21 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 05:42:21 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 05:42:21 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 05:42:21 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 05:42:21 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 05:42:21 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 05:42:21 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: Repeating LCB Err bits 0x8000000 - masking
Apr 19 05:42:24 rdma06 opafmd[1140340]: :          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:          SA TX RMMP ABORT(UNKNOWN):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:                  SA TX STATUS BUSY:          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:              SA TX STATUS REDIRECT:          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:              SA TX STATUS BADCLASS:          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:             SA TX STATUS BADMETHOD:          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:         SA TX STATUS BADMETHODATTR:          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:              SA TX STATUS BADFIELD:          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:          SA TX STATUS NO RESOURCES:          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:           SA TX STATUS INVALID REQ:          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:            SA TX STATUS NO RECORDS:          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:      SA TX STATUS TOO MANY RECORDS:          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:               SA TX STATUS BAD GID:          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:        SA TX STATUS TOO FEW FIELDS:          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:               SA TX STATUS UNKNOWN:          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:                DBSYNC Send Command:          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:        DBSYNC Send Command Failure:          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:                DBSYNC Receive Data:          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:        DBSYNC Receive Data Failure:          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:                       DBSYNC Reply:          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:               DBSYNC Reply Failure:          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:         SA Maximum Contexts In Use:          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:           SA Maximum Contexts Free:          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:        Maximum SM Sweep Time in ms:          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:         Seconds since last cleared: 183943
Apr 19 05:42:24 rdma06 opafmd[1140340]:                  Time last cleared: Mon Apr 17 02:36:41 2023
Apr 19 05:42:24 rdma06 opafmd[1140340]:                            COUNTER: THIS SWEEP LAST SWEEP      TOTAL
Apr 19 05:42:24 rdma06 opafmd[1140340]: ------------------------------------ ---------- ---------- ----------
Apr 19 05:42:24 rdma06 opafmd[1140340]:   SM State transition to DISCOVERY:          1          0          1
Apr 19 05:42:24 rdma06 opafmd[1140340]:      SM State transition to MASTER:          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:     SM State transition to STANDBY:          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:    SM State transition to INACTIVE:          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:      Total transmitted SMA Packets:      36769          0      36769
Apr 19 05:42:24 rdma06 opafmd[1140340]:          Direct Routed SMA Packets:      36768          0      36768
Apr 19 05:42:24 rdma06 opafmd[1140340]:             LID Routed SMA Packets:          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:              SMA Query Retransmits:          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:    SMA Query Retransmits Exhausted:          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:                  SM TX GET(Notice):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:                  SM TX SET(Notice):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:           Total SM RX TRAP(Notice):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:                 SM RX TRAP(PortUp):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:               SM RX TRAP(PortDown):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:            SM RX TRAP(McGrpCreate):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:            SM RX TRAP(McGrpDelete):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:                 SM RX TRAP(UnPath):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:                 SM RX TRAP(RePath):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:           SM RX TRAP(PortStateChg):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:          SM RX TRAP(LinkIntegrity):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:          SM RX TRAP(BufferOverrun):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:            SM RX TRAP(FlowControl):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:            SM RX TRAP(LocalChange):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:            SM RX TRAP(SysImageChg):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:                SM RX TRAP(BadMKey):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:                SM RX TRAP(BadPkey):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:                SM RX TRAP(BadQkey):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:          SM RX TRAP(BadPkeySwPort):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:     SM RX TRAP(LinkWidthDowngrade):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:       SM RX TRAP(CostMatrixChange):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:          SM TX TRAPREPRESS(Notice):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:         SM TX GET(NodeDescription):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:                SM TX GET(NodeInfo):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:              SM TX GET(SwitchInfo):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:              SM TX SET(SwitchInfo):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:                SM TX GET(PortInfo):      36768          0      36768
Apr 19 05:42:24 rdma06 opafmd[1140340]:                SM TX SET(PortInfo):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:           SM TX GET(PortStateInfo):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:           SM TX SET(PortStateInfo):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:               SM TX GET(PKeyTable):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:               SM TX SET(PKeyTable):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:       SM TX GET(SL2SCMappingTable):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:       SM TX SET(SL2SCMappingTable):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:       SM TX GET(SC2SLMappingTable):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:       SM TX SET(SC2SLMappingTable):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:      SM TX GET(SC2VLtMappingTable):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:      SM TX SET(SC2VLtMappingTable):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:     SM TX GET(SC2VLntMappingTable):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:     SM TX SET(SC2VLntMappingTable):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:      SM TX GET(SC2VLrMappingTable):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:      SM TX SET(SC2VLrMappingTable):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:       SM TX GET(SC2SCMappingTable):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:       SM TX SET(SC2SCMappingTable):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:           SM TX SET(SC2SCMultiSet):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:      SM TX GET(VLArbitrationTable):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:      SM TX SET(VLArbitrationTable):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:                     SM TX GET(LFT):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:                     SM TX SET(LFT):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:                     SM TX GET(MFT):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:                     SM TX SET(MFT):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:               SM TX GET(AGGREGATE):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:               SM TX SET(AGGREGATE):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:                 SM TX GET(LedInfo):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:                 SM TX SET(LedInfo):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:               SM TX GET(CableInfo):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:                  SM RX GET(SmInfo):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:                  SM TX GET(SmInfo):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:                  SM RX SET(SmInfo):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:                  SM TX SET(SmInfo):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:              SM TX GETRESP(SmInfo):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:               SM TX GET(PortGroup):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:               SM TX SET(PortGroup):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:                    SM TX GET(PGFT):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:                    SM TX SET(PGFT):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:      SM TX GET(BufferControlTable):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:      SM TX SET(BufferControlTable):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:               SM TX GET(ARLidMask):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:               SM TX SET(ARLidMask):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:          SM RX GET(CongestionInfo):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:       SM RX GET(HfiCongestSetting):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:       SM RX SET(HfiCongestSetting):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:       SM RX GET(HfiCongestControl):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:       SM RX SET(HfiCongestControl):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:        SM RX GET(SwCongestSetting):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:        SM RX SET(SwCongestSetting):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:  SM RX GET(SwPortCongestionSetting:          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:                   SM RX GETRESP(*):      73536          0      73536
Apr 19 05:42:24 rdma06 opafmd[1140340]:                  SM RX STATUS BUSY:          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:              SM RX STATUS REDIRECT:          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:              SM RX STATUS BADCLASS:          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:             SM RX STATUS BADMETHOD:          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:         SM RX STATUS BADMETHODATTR:          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:              SM RX STATUS BADFIELD:          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:               SM RX STATUS UNKNOWN:          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:           SA RX GET(ClassPortInfo):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:               SA TX REPORT(Notice):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:              SA RX SET(InformInfo):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:              SA RX GET(NodeRecord):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:           SA RX GETTBL(NodeRecord):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:                SA RX GET(PortInfo):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:             SA RX GETTBL(PortInfo):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:                SA RX GET(SC2SCMAP):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:             SA RX GETTBL(SC2SCMap):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:                SA RX GET(SL2SCMAP):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:             SA RX GETTBL(SL2SCMap):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:                SA RX GET(SC2SLMap):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:             SA RX GETTBL(SC2SLMap):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:               SA RX GET(SC2VLtMap):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:            SA RX GETTBL(SC2VLtMap):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:              SA RX GET(SC2VLntMap):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:           SA RX GETTBL(SC2VLntMap):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:               SA RX GET(SC2VLrMap):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:            SA RX GETTBL(SC2VLrMap):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:              SA RX GET(SwitchInfo):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:          SA RX GETTBLE(SwitchInfo):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:                     SA RX GET(LFT):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:                  SA RX GETTBL(LFT):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:                     SA RX GET(MFT):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:                  SA RX GETTBL(MFT):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:           SA RX GET(VLArbitration):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:        SA RX GETTBL(VLArbitration):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:                  SA RX GET(SmInfo):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:               SA RX GETTBL(SmInfo):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:              SA RX GET(InformInfo):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:           SA RX GETTBL(InformInfo):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:              SA RX GET(LinkRecord):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:           SA RX GETTBL(LinkRecord):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:           SA RX GET(ServiceRecord):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:           SA RX SET(ServiceRecord):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:        SA RX GETTBL(ServiceRecord):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:        SA RX DELETE(ServiceRecord):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:              SA RX GET(PKeyRecord):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:           SA RX GETTBL(PKeyRecord):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:              SA RX GET(PathRecord):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:           SA RX GETTBL(PathRecord):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:          SA RX GET(McMemberRecord):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:          SA RX SET(McMemberRecord):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:       SA RX GETTBL(McMemberRecord):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:       SA RX DELETE(McMemberRecord):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:        SA RX GETTRACE(TraceRecord):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:    SA RX GETMULTI(MultiPathRecord):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:           SA RX GET(vFabricRecord):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:        SA RX GETTBL(vFabricRecord):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:        SA RX GET(FabricInfoRecord):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:   SA RX GET(QuarantinedNodeRecord):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]: SA RX GETTBL(QuarantinedNodeRecord):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:          SA RX GET(CongInfoRecord):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:       SA RX GETTBL(CongInfoRecord):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:        SA RX GET(SwitchCongRecord):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:     SA RX GETTBL(SwitchCongRecord):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:    SA RX GET(SwitchPortCongRecord):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]: SA RX GETTBL(SwitchPortCongRecord):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:           SA RX GET(HFICongRecord):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:           SA RX GET(HFICongRecord):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:       SA RX GET(HFICongCtrlRecord):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:       SA RX GET(HFICongCtrlRecord):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:           SA RX GET(BfrCtrlRecord):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:        SA RX GETTBL(BfrCtrlRecord):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:               SA RX GET(CableInfo):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:            SA RX GETTBL(CableInfo):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:         SA RX GET(PortGroupRecord):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:      SA RX GETTBL(PortGroupRecord):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:      SA RX GET(PortGroupFwdRecord):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:   SA RX GETTBL(PortGroupFwdRecord):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:                             (null):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:                             (null):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:           SA RX REPORTRESP(Notice):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]: SA RX GET(DeviceGroupMemberRecord):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]: SA RX GETTBL(DeviceGroupMemberRecord):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:   SA RX GET(DeviceGroupNameRecord):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]: SA RX GETTBL(DeviceGroupNameRecord):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:  SA RX GET(DeviceTreeMemberRecord):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]: SA RX GETTBL(DeviceTreeMemberRecord):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:           SA RX DUPLICATE REQUESTS:          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]: SA RX DEAD RMPP TRANSACTION PACKET:          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:                SA DROPPED REQUESTS:          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:           SA NO AVAILABLE CONTEXTS:          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:          SA RX GETMULTI() Non-RMPP:          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:          SA RX GETMULTI RMPP Abort:          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:          SA TX GETMULTI RMPP Abort:          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:         SA TX GETMULTI Ack Retries:          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:                 SA TX RMPP Retries:          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:                    SA TX RMPP STOP:          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:          SA TX RMMP ABORT(TIMEOUT):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:    SA TX RMMP ABORT(BAD FIRST SEG):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:         SA TX RMMP ABORT(BAD TYPE):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]: SA TX RMMP ABORT(WINDOW TOO SMALL):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:   SA TX RMMP ABORT(SEGNUM TOO BIG):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:      SA TX RMMP ABORT(BAD VERSION):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]: SA TX RMMP ABORT(TOO MANY RETRIES):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:          SA TX RMMP ABORT(UNKNOWN):          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:                  SA TX STATUS BUSY:          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:              SA TX STATUS REDIRECT:          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:              SA TX STATUS BADCLASS:          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:             SA TX STATUS BADMETHOD:          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:         SA TX STATUS BADMETHODATTR:          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:              SA TX STATUS BADFIELD:          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:          SA TX STATUS NO RESOURCES:          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:           SA TX STATUS INVALID REQ:          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:            SA TX STATUS NO RECORDS:          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:      SA TX STATUS TOO MANY RECORDS:          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:               SA TX STATUS BAD GID:          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:        SA TX STATUS TOO FEW FIELDS:          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:               SA TX STATUS UNKNOWN:          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:                DBSYNC Send Command:          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:        DBSYNC Send Command Failure:          0          0          0
Apr 19 05:42:24 rdma06 opafmd[1140340]:                DBSYNC Receive Data:          0          0          0ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:42:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:42:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:42:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:42:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:42:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:42:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:42:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:42:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:42:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:42:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:42:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:42:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:42:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:42:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:42:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:42:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:42:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:42:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:42:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:42:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:42:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:42:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:42:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:42:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:42:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:42:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:42:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:42:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:42:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:42:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:42:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:42:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:42:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:42:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:42:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:42:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:42:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:42:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:42:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:42:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:42:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:42:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:42:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:42:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:42:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:42:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:42:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:42:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:42:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:42:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:42:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:42:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:42:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:42:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:42:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:42:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:42:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:42:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:42:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:42:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:42:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:42:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:42:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:42:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:42:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:42:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:42:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:42:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:42:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:42:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:42:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:42:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:42:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:42:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:42:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:42:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:42:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:42:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:42:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:42:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:42:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:42:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:42:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:42:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:42:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:42:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:42:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:42:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:42:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:42:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:42:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:42:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:42:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:42:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:42:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:42:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:42:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:42:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:42:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:42:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:42:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:42:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:42:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:42:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:42:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:42:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:42:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:42:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:42:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:42:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:42:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:42:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:42:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:42:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:42:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:42:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:42:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:42:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:42:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:42:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:42:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:42:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:42:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:42:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:42:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:42:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:42:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:42:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:42:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:42:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:42:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:42:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:42:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:42:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:42:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:42:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:42:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:42:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:42:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:42:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:42:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:42:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:42:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:43:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:43:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:43:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:43:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:43:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:43:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:43:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:43:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:43:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:43:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:43:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:43:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:43:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:43:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:43:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:43:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:43:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:43:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:43:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:43:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:43:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:43:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:43:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:43:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:43:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:43:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:43:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:43:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:43:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:43:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:43:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:43:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:43:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:43:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:43:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:43:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:43:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:43:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:43:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:43:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:43:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:43:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:43:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:43:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:43:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:43:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:43:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:43:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:43:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:43:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:43:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:43:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:43:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:43:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:43:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:43:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:43:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:43:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:43:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:43:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:43:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:43:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:43:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:43:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:43:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:43:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:43:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:43:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:43:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:43:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:43:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:43:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:43:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:43:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:43:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:43:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:43:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:43:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:43:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:43:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:43:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:43:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:43:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:43:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:43:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:43:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:43:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:43:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:43:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:43:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:43:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:43:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:43:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:43:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:43:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:43:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:43:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:43:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:43:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:43:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:43:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:43:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:43:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:43:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:43:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:43:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:43:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:43:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:43:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:43:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:43:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:43:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:43:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:43:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:43:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:43:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:43:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:43:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:43:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:43:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:43:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:43:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:43:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:43:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:43:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:43:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:43:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:43:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:43:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:43:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:43:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:43:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:43:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:43:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:43:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:43:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:43:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:43:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:43:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:43:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:43:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:43:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:43:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:43:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:43:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:43:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:43:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:43:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:43:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:43:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:43:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:43:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:43:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:43:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:43:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:43:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:43:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:43:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:43:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:43:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:43:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:43:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:43:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:43:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:43:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:43:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:43:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:43:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:43:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:43:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:43:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:43:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:43:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:43:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:43:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:43:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:43:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:43:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:43:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:43:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:43:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:43:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:43:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:43:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:43:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:43:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:43:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:43:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:43:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:43:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:43:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:43:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:43:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:43:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:43:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:43:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:43:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:43:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:43:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:43:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:43:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:43:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:43:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:43:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:43:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:43:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:43:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:43:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:43:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:43:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:43:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:43:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:43:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:43:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:43:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:43:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:44:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:44:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:44:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:44:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:44:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:44:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:44:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:44:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:44:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:44:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:44:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:44:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:44:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:44:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:44:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:44:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:44:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:44:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:44:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:44:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:44:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:44:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:44:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:44:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:44:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:44:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:44:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:44:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:44:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:44:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:44:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:44:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:44:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:44:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:44:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:44:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:44:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:44:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:44:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:44:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:44:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:44:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:44:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:44:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:44:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:44:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:44:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:44:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:44:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:44:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:44:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:44:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:44:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:44:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:44:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:44:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:44:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:44:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:44:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:44:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:44:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:44:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:44:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:44:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:44:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:44:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:44:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:44:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:44:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:44:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:44:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:44:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:44:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:44:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:44:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:44:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:44:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:44:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:44:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:44:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:44:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:44:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:44:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:44:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:44:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:44:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:44:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:44:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:44:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:44:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:44:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:44:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:44:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:44:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:44:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:44:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:44:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:44:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:44:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:44:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:44:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:44:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:44:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:44:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:44:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:44:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:44:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:44:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:44:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:44:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:44:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:44:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:44:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:44:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:44:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:44:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:44:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:44:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:44:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:44:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:44:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:44:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:44:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:44:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:44:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:44:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:44:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:44:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:44:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:44:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:44:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:44:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:44:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:44:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:44:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:44:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:44:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:44:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:44:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:44:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:44:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:44:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:44:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:44:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:44:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:44:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:44:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:44:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:44:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:44:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:44:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:44:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:44:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:44:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:44:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:44:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:44:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:44:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:44:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:44:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:44:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:44:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:44:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:44:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:44:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:44:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:44:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:44:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:44:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:44:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:44:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:44:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:44:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:44:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:44:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:44:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:44:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:44:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:44:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:44:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:44:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:44:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:44:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:44:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:44:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:44:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:44:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:44:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:44:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:44:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:44:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:44:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:44:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:44:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:44:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:44:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:44:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:44:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:44:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:44:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:44:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:44:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:44:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:44:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:44:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:44:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:44:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:44:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:44:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:44:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:44:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:44:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:44:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:44:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:44:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:44:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:45:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:45:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:45:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:45:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:45:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:45:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:45:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:45:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:45:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:45:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:45:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:45:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:45:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:45:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:45:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:45:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:45:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:45:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:45:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:45:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:45:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:45:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:45:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:45:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:45:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:45:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:45:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:45:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:45:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:45:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:45:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:45:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:45:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:45:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:45:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:45:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:45:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:45:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:45:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:45:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:45:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:45:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:45:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:45:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:45:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:45:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:45:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:45:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:45:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:45:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:45:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:45:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:45:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:45:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:45:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:45:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:45:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:45:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:45:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:45:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:45:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:45:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:45:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:45:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:45:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:45:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:45:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:45:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:45:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:45:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:45:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:45:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:45:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:45:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:45:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:45:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:45:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:45:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:45:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:45:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:45:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:45:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:45:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:45:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:45:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:45:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:45:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:45:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:45:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:45:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:45:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:45:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:45:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:45:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:45:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:45:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:45:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:45:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:45:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:45:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:45:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:45:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:45:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:45:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:45:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:45:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:45:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:45:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:45:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:45:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:45:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:45:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:45:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:45:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:45:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:45:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:45:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:45:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:45:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:45:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:45:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:45:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:45:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:45:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:45:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:45:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:45:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:45:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:45:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:45:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:45:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:45:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:45:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:45:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:45:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:45:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:45:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:45:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:45:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:45:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:45:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:45:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:45:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:45:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:45:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:45:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:45:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:45:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:45:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:45:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:45:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:45:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:45:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:45:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:45:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:45:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:45:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:45:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:45:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:45:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:45:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:45:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:45:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:45:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:45:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:45:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:45:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:45:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:45:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:45:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:45:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:45:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:45:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:45:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:45:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:45:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:45:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:45:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:45:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:45:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:45:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:45:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:45:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:45:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:45:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:45:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:45:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:45:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:45:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:45:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:45:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:45:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:45:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:45:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:45:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:45:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:45:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:45:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:45:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:45:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:45:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:45:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:45:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:45:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:45:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:45:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:45:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:45:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:45:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:45:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:45:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:45:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:45:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:45:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:45:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:45:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:46:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:46:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:46:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:46:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:46:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:46:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:46:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:46:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:46:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:46:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:46:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:46:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:46:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:46:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:46:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:46:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:46:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:46:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:46:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:46:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:46:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:46:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:46:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:46:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:46:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:46:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:46:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:46:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:46:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:46:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:46:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:46:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:46:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:46:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:46:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:46:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:46:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:46:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:46:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:46:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:46:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:46:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:46:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:46:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:46:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:46:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:46:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:46:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:46:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:46:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:46:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:46:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:46:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:46:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:46:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:46:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:46:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:46:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:46:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:46:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:46:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:46:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:46:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:46:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:46:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:46:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:46:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:46:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:46:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:46:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:46:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:46:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:46:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:46:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:46:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:46:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:46:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:46:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:46:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:46:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:46:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:46:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:46:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:46:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:46:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:46:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:46:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:46:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:46:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:46:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:46:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:46:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:46:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:46:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:46:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:46:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:46:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:46:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:46:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:46:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:46:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:46:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:46:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:46:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:46:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:46:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:46:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:46:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:46:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:46:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:46:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:46:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:46:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:46:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:46:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:46:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:46:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:46:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:46:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:46:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:46:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:46:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:46:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:46:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:46:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:46:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:46:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:46:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:46:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:46:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:46:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:46:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:46:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:46:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:46:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:46:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:46:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:46:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:46:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:46:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:46:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:46:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:46:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:46:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:46:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:46:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:46:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:46:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:46:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:46:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:46:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:46:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:46:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:46:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:46:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:46:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:46:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:46:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:46:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:46:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:46:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:46:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:46:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:46:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:46:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:46:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:46:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:46:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:46:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:46:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:46:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:46:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:46:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:46:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:46:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:46:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:46:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:46:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:46:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:46:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:46:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:46:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:46:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:46:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:46:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:46:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:46:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:46:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:46:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:46:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:46:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:46:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:46:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:46:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:46:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:46:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:46:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:46:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:46:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:46:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:46:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:46:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:46:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:46:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:46:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:46:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:46:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:46:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:46:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:46:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:46:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:46:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:46:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:46:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:46:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:46:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:47:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:47:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:47:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:47:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:47:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:47:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:47:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:47:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:47:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:47:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:47:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:47:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:47:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:47:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:47:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:47:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:47:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:47:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:47:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:47:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:47:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:47:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:47:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:47:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:47:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:47:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:47:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:47:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:47:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:47:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:47:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:47:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:47:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:47:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:47:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:47:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:47:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:47:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:47:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:47:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:47:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:47:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:47:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:47:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:47:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:47:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:47:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:47:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:47:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:47:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:47:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:47:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:47:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:47:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:47:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:47:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:47:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:47:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:47:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:47:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:47:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:47:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:47:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:47:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:47:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:47:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:47:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:47:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:47:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:47:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:47:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:47:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:47:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:47:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:47:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:47:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:47:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:47:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:47:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:47:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:47:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:47:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:47:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:47:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:47:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:47:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:47:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:47:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:47:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:47:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:47:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:47:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:47:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:47:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:47:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:47:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:47:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:47:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:47:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:47:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:47:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:47:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:47:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:47:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:47:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:47:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:47:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:47:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:47:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:47:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:47:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:47:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:47:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:47:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:47:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:47:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:47:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:47:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:47:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:47:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:47:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:47:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:47:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:47:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:47:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:47:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:47:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:47:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:47:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:47:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:47:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:47:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:47:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:47:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:47:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:47:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:47:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:47:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:47:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:47:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:47:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:47:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:47:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:47:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:47:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:47:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:47:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:47:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:47:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:47:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:47:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:47:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:47:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:47:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:47:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:47:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:47:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:47:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:47:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:47:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:47:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:47:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:47:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:47:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:47:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:47:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:47:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:47:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:47:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:47:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:47:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:47:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:47:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:47:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:47:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:47:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:47:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:47:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:47:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:47:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:48:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:48:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:48:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:48:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:48:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:48:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:48:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:48:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:48:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:48:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:48:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:48:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:48:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:48:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:48:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:48:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:48:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:48:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:48:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:48:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:48:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:48:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:48:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:48:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:48:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:48:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:48:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:48:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:48:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:48:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:48:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:48:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:48:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:48:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:48:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:48:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:48:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:48:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:48:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:48:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:48:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:48:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:48:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:48:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:48:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:48:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:48:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:48:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:48:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:48:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:48:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:48:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:48:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:48:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:48:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:48:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:48:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:48:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:48:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:48:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:48:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:48:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:48:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:48:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:48:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:48:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:48:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:48:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:48:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:48:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:48:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:48:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:48:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:48:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:48:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:48:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:48:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:48:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:48:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:48:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:48:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:48:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:48:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:48:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:48:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:48:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:48:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:48:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:48:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:48:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:48:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:48:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:48:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:48:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:48:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:48:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:48:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:48:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:48:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:48:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:48:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:48:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:48:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:48:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:48:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:48:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:48:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:48:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:48:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:48:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:48:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:48:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:48:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:48:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:48:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:48:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:48:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:48:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:48:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:48:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:48:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:48:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:48:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:48:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:48:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:48:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:48:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:48:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:48:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:48:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:48:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:48:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:48:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:48:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:48:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:48:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:48:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:48:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:48:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:48:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:48:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:48:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:48:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:48:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:48:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:48:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:48:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:48:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:48:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:48:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:48:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:48:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:48:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:48:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:48:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:48:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:48:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:48:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:48:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:48:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:48:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:48:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:48:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:48:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:48:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:48:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:48:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:48:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:48:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:48:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:48:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:48:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:48:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:48:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:48:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:48:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:48:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:48:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:48:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:48:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:48:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:48:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:48:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:48:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:48:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:48:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:48:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:48:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:48:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:48:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:48:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:48:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:48:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:48:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:48:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:48:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:48:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:48:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:48:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:48:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:48:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:48:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:48:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:48:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:48:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:48:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:48:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:48:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:48:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:48:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:48:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:48:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:48:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:48:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:48:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:48:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:49:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:49:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:49:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:49:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:49:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:49:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:49:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:49:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:49:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:49:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:49:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:49:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:49:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:49:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:49:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:49:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:49:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:49:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:49:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:49:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:49:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:49:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:49:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:49:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:49:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:49:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:49:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:49:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:49:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:49:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:49:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:49:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:49:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:49:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:49:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:49:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:49:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:49:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:49:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:49:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:49:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:49:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:49:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:49:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:49:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:49:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:49:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:49:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:49:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:49:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:49:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:49:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:49:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:49:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:49:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:49:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:49:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:49:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:49:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:49:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:49:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:49:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:49:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:49:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:49:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:49:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:49:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:49:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:49:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:49:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:49:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:49:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:49:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:49:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:49:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:49:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:49:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:49:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:49:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:49:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:49:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:49:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:49:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:49:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:49:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:49:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:49:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:49:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:49:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:49:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:49:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:49:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:49:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:49:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:49:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:49:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:49:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:49:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:49:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:49:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:49:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:49:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:49:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:49:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:49:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:49:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:49:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:49:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:49:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:49:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:49:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:49:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:49:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:49:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:49:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:49:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:49:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:49:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:49:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:49:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:49:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:49:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:49:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:49:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:49:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:49:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:49:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:49:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:49:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:49:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:49:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:49:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:49:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:49:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:49:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:49:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:49:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:49:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:49:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:49:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:49:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:49:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:49:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:49:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:49:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:49:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:49:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:49:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:49:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:49:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:49:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:49:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:49:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:49:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:49:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:49:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:49:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:49:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:49:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:49:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:49:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:49:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:49:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:49:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:49:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:49:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:49:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:49:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:51:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:51:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:51:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:51:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:51:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:51:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:51:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:51:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:51:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:51:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:51:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:51:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:51:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:51:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:51:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:51:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:51:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:51:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:52:00 rdma06 rsyslogd[1041]: imjournal: 5585 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 19 05:52:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:52:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:52:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:52:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:52:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:52:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:52:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:52:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:52:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:52:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:52:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:52:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:52:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:52:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:52:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:52:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:52:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:52:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:52:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:52:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:52:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:52:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:52:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:52:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:52:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:52:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:52:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:52:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:52:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:52:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:52:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:52:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:52:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:52:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:52:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:52:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:52:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:52:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:52:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:52:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:52:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:52:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:52:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:52:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:52:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:52:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:52:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:52:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:52:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:52:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:52:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:52:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:52:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:52:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:52:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:52:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:52:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:52:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:52:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:52:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:52:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:52:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:52:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:52:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:52:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:52:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:52:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:52:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:52:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:52:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:52:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:52:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:52:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:52:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:52:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:52:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:52:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:52:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:52:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:52:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:52:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:52:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:52:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:52:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:52:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:52:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:52:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:52:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:52:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:52:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:52:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:52:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:52:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:52:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:52:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:52:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:52:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:52:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:52:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:52:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:52:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:52:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:52:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:52:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:52:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:52:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:52:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:52:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:52:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:52:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:52:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:52:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:52:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:52:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:52:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:52:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:52:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:52:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:52:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:52:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:52:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:52:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:52:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:52:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:52:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:52:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:52:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:52:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:52:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:52:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:52:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:52:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:52:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:52:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:52:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:52:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:52:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:52:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:52:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:52:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:52:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:52:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:52:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:52:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:52:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:52:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:52:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:52:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:52:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:52:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:52:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:52:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:52:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:52:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:52:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:52:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:52:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:52:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:52:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:52:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:52:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:52:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:52:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:52:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:52:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:52:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:52:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:52:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:52:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:52:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:52:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:52:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:52:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:52:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:52:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:52:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:52:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:52:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:52:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:52:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:52:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:52:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:52:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:52:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:52:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:52:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:52:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:52:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:52:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:52:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:52:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:52:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:52:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:52:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:52:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:52:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:52:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:52:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:52:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:52:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:52:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:52:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:52:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:52:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:52:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:52:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:52:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:52:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:52:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:52:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:52:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:52:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:52:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:52:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:52:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:52:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:53:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:53:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:53:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:53:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:53:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:53:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:53:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:53:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:53:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:53:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:53:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:53:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:53:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:53:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:53:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:53:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:53:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:53:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:53:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:53:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:53:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:53:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:53:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:53:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:53:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:53:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:53:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:53:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:53:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:53:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:53:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:53:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:53:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:53:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:53:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:53:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:53:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:53:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:53:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:53:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:53:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:53:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:53:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:53:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:53:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:53:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:53:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:53:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:53:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:53:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:53:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:53:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:53:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:53:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:53:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:53:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:53:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:53:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:53:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:53:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:53:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:53:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:53:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:53:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:53:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:53:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:53:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:53:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:53:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:53:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:53:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:53:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:53:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:53:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:53:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:53:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:53:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:53:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:53:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:53:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:53:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:53:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:53:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:53:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:53:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:53:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:53:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:53:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:53:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:53:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:53:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:53:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:53:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:53:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:53:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:53:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:53:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:53:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:53:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:53:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:53:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:53:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:53:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:53:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:53:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:53:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:53:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:53:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:53:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:53:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:53:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:53:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:53:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:53:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:53:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:53:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:53:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:53:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:53:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:53:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:53:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:53:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:53:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:53:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:53:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:53:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:53:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:53:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:53:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:53:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:53:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:53:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:53:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:53:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:53:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:53:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:53:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:53:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:53:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:53:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:53:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:53:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:53:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:53:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:54:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:54:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:54:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:54:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:54:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:54:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:54:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:54:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:54:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:54:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:54:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:54:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:54:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:54:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:54:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:54:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:54:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:54:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:54:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:54:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:54:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:54:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:54:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:54:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:54:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:54:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:54:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:54:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:54:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:54:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:54:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:54:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:54:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:54:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:54:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:54:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:54:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:54:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:54:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:54:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:54:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:54:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:54:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:54:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:54:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:54:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:54:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:54:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:54:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:54:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:54:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:54:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:54:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:54:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:54:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:54:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:54:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:54:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:54:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:54:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:54:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:54:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:54:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:54:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:54:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:54:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:54:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:54:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:54:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:54:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:54:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:54:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:54:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:54:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:54:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:54:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:54:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:54:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:54:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:54:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:54:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:54:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:54:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:54:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:54:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:54:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:54:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:54:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:54:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:54:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:54:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:54:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:54:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:54:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:54:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:54:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:54:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:54:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:54:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:54:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:54:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:54:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:54:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:54:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:54:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:54:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:54:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:54:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:54:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:54:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:54:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:54:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:54:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:54:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:54:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:54:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:54:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:54:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:54:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:54:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:54:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:54:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:54:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:54:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:54:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:54:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:54:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:54:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:54:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:54:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:54:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:54:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:54:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:54:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:54:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:54:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:54:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:54:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:54:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:54:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:54:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:54:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:54:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:54:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:54:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:54:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:54:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:54:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:54:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:54:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:54:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:54:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:54:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:54:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:54:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:54:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:54:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:54:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:54:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:54:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:54:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:54:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:54:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:54:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:54:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:54:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:54:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:54:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:54:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:54:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:54:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:54:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:54:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:54:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:54:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:54:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:54:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:54:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:54:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:54:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:55:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:55:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:55:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:55:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:55:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:55:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:55:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:55:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:55:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:55:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:55:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:55:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:55:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:55:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:55:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:55:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:55:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:55:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:55:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:55:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:55:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:55:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:55:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:55:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:55:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:55:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:55:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:55:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:55:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:55:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:55:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:55:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:55:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:55:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:55:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:55:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:55:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:55:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:55:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:55:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:55:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:55:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:55:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:55:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:55:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:55:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:55:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:55:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:55:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:55:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:55:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:55:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:55:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:55:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:55:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:55:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:55:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:55:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:55:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:55:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:55:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:55:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:55:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:55:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:55:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:55:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:55:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:55:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:55:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:55:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:55:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:55:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:55:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:55:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:55:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:55:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:55:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:55:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:55:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:55:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:55:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:55:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:55:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:55:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:55:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:55:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:55:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:55:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:55:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:55:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:55:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:55:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:55:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:55:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:55:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:55:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:55:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:55:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:55:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:55:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:55:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:55:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:55:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:55:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:55:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:55:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:55:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:55:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:55:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:55:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:55:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:55:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:55:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:55:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:55:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:55:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:55:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:55:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:55:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:55:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:55:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:55:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:55:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:55:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:55:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:55:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:55:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:55:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:55:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:55:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:55:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:55:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:55:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:55:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:55:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:55:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:55:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:55:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:55:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:55:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:55:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:55:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:55:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:55:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:55:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:55:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:55:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:55:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:55:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:55:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:55:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:55:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:55:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:55:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:55:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:55:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:55:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:55:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:55:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:55:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:55:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:55:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:55:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:55:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:55:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:55:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:55:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:55:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:55:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:55:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:55:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:55:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:55:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:55:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:55:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:55:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:55:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:55:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:55:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:55:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:55:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:55:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:55:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:55:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:55:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:55:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:55:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:55:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:55:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:55:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:55:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:55:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:55:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:55:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:55:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:55:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:55:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:55:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:55:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:55:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:55:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:55:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:55:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:55:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:55:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:55:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:55:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:55:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:55:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:55:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:55:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:55:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:55:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:55:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:55:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:55:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:56:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:56:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:56:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:56:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:56:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:56:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:56:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:56:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:56:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:56:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:56:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:56:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:56:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:56:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:56:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:56:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:56:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:56:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:56:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:56:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:56:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:56:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:56:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:56:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:56:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:56:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:56:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:56:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:56:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:56:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:56:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:56:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:56:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:56:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:56:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:56:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:56:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:56:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:56:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:56:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:56:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:56:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:56:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:56:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:56:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:56:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:56:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:56:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:56:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:56:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:56:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:56:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:56:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:56:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:56:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:56:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:56:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:56:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:56:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:56:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:56:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:56:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:56:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:56:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:56:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:56:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:56:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:56:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:56:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:56:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:56:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:56:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:56:19 rdma06 systemd-journald[790]: Data hash table of /run/log/journal/95379d2f723847b68d9e976dee8967a9/system.journal has a fill level at 75.0 (163750 of 218332 items, 75497472 file size, 461 bytes per hash table item), suggesting rotation.
Apr 19 05:56:19 rdma06 systemd-journald[790]: /run/log/journal/95379d2f723847b68d9e976dee8967a9/system.journal: Journal header limits reached or header out-of-date, rotating.
Apr 19 05:56:19 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 19 05:56:19 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 19 05:56:20 rdma06 rsyslogd[1041]: imjournal: journal files changed, reloading...  [v8.2102.0-113.el9_2 try https://www.rsyslog.com/e/0 ]
Apr 19 05:56:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:56:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:56:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:56:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:56:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:56:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:56:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:56:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:56:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:56:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:56:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:56:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:56:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:56:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:56:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:56:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:56:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:56:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:56:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:56:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:56:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:56:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:56:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:56:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:56:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:56:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:56:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:56:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:56:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:56:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:56:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:56:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:56:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:56:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:56:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:56:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:56:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:56:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:56:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:56:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:56:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:56:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:56:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:56:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:56:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:56:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:56:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:56:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:56:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:56:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:56:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:56:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:56:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:56:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:56:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:56:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:56:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:56:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:56:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:56:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:56:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:56:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:56:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:56:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:56:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:56:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:56:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:56:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:56:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:56:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:56:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:56:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:56:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:56:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:56:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:56:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:56:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:56:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:56:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:56:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:56:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:56:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:56:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:56:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:56:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:56:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:56:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:56:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:56:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:56:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:56:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:56:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:56:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:56:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:56:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:56:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:56:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:56:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:56:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:56:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:56:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:56:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:56:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:56:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:56:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:56:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:56:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:56:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:56:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:56:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:56:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:56:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:56:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:56:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:56:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:56:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:56:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:56:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:56:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:56:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:56:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:56:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:56:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:56:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:56:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:56:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:56:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:56:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:56:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:56:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:56:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:56:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:56:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:56:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:56:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:56:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:56:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:56:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:56:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:56:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:56:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:56:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:56:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:56:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:57:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:57:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:57:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:57:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:57:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:57:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:57:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:57:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:57:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:57:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:57:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:57:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:57:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:57:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:57:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:57:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:57:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:57:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:57:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:57:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:57:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:57:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:57:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:57:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:57:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:57:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:57:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:57:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:57:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:57:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:57:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:57:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:57:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:57:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:57:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:57:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:57:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:57:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:57:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:57:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:57:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:57:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:57:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:57:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:57:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:57:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:57:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:57:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:57:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:57:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:57:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:57:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:57:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:57:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:57:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:57:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:57:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:57:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:57:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:57:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:57:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:57:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:57:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:57:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:57:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:57:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:57:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:57:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:57:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:57:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:57:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:57:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:57:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:57:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:57:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:57:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:57:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:57:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:57:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:57:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:57:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:57:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:57:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:57:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:57:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:57:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:57:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:57:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:57:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:57:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:57:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:57:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:57:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:57:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:57:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:57:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:57:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:57:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:57:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:57:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:57:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:57:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:57:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:57:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:57:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:57:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:57:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:57:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:57:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:57:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:57:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:57:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:57:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:57:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:57:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:57:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:57:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:57:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:57:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:57:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:57:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:57:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:57:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:57:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:57:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:57:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:57:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:57:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:57:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:57:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:57:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:57:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:57:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:57:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:57:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:57:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:57:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:57:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:57:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:57:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:57:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:57:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:57:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:57:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:57:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:57:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:57:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:57:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:57:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:57:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:57:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:57:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:57:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:57:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:57:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:57:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:57:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:57:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:57:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:57:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:57:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:57:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:57:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:57:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:57:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:57:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:57:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:57:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:57:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:57:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:57:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:57:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:57:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:57:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:57:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:57:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:57:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:57:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:57:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:57:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:57:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:57:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:57:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:57:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:57:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:57:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:57:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:57:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:57:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:57:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:57:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:57:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:57:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:57:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:57:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:57:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:57:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:57:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:57:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:57:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:57:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:57:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:57:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:57:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:57:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:57:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:57:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:57:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:57:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:57:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:57:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:57:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:57:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:57:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:57:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:57:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:58:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:58:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:58:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:58:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:58:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:58:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:58:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:58:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:58:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:58:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:58:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:58:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:58:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:58:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:58:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:58:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:58:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:58:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:58:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:58:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:58:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:58:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:58:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:58:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:58:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:58:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:58:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:58:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:58:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:58:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:58:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:58:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:58:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:58:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:58:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:58:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:58:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:58:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:58:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:58:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:58:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:58:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:58:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:58:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:58:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:58:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:58:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:58:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:58:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:58:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:58:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:58:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:58:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:58:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:58:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:58:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:58:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:58:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:58:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:58:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:58:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:58:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:58:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:58:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:58:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:58:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:58:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:58:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:58:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:58:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:58:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:58:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:58:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:58:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:58:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:58:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:58:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:58:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:58:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:58:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:58:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:58:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:58:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:58:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:58:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:58:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:58:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:58:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:58:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:58:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:58:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:58:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:58:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:58:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:58:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:58:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:58:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:58:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:58:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:58:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:58:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:58:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:58:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:58:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:58:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:58:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:58:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:58:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:58:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:58:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:58:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:58:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:58:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:58:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:58:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:58:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:58:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:58:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:58:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:58:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:58:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:58:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:58:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:58:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:58:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:58:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:58:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:58:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:58:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:58:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:58:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:58:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:58:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:58:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:58:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:58:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:58:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:58:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:58:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:58:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:58:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:58:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:58:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:58:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:58:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:58:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:58:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:58:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:58:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:58:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:58:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:58:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:58:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:58:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:58:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:58:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:58:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:58:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:58:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:58:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:58:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:58:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:58:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:58:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:58:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:58:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:58:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:58:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:58:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:58:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:58:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:58:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:58:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:58:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:58:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:58:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:58:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:58:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:58:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:58:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:58:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:58:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:58:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:58:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:58:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:58:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:58:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:58:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:58:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:58:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:58:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:58:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:58:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:58:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:58:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:58:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:58:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:58:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:58:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:58:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:58:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:58:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:58:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:58:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:58:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:58:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:58:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:58:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:58:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:58:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:58:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:58:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:58:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:58:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:58:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:58:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:59:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:59:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:59:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:59:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:59:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:59:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:59:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:59:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:59:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:59:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:59:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:59:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:59:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:59:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:59:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:59:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:59:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:59:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:59:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:59:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:59:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:59:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:59:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:59:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:59:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:59:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:59:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:59:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:59:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:59:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:59:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:59:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:59:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:59:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:59:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:59:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:59:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:59:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:59:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:59:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:59:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:59:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:59:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:59:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:59:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:59:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:59:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:59:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:59:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:59:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:59:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:59:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:59:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:59:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:59:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:59:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:59:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:59:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:59:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:59:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:59:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:59:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:59:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:59:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:59:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:59:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:59:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:59:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:59:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:59:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:59:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:59:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:59:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:59:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:59:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:59:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:59:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:59:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:59:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:59:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:59:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:59:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:59:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:59:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:59:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:59:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:59:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:59:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:59:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:59:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:59:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:59:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:59:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:59:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:59:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:59:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:59:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:59:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:59:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:59:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:59:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:59:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:59:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:59:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:59:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:59:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:59:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:59:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:59:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:59:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:59:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:59:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:59:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:59:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:59:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:59:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:59:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:59:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:59:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:59:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:59:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:59:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:59:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:59:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:59:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:59:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:59:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:59:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:59:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:59:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:59:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:59:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:59:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:59:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:59:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 05:59:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 05:59:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:59:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:59:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:59:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 05:59:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 05:59:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 05:59:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 05:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 05:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 05:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 05:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 05:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 05:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 05:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 05:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 05:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 05:59:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 06:02:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 06:02:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:02:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:02:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:02:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 06:02:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:02:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:02:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:02:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 06:02:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 06:02:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 06:02:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 06:02:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 06:02:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 06:02:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 06:02:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 06:02:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 06:02:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 06:02:05 rdma06 rsyslogd[1041]: imjournal: 5300 messages lost due to rate-limiting (20000 allowed within 600 seconds)
Apr 19 06:02:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 06:02:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:02:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:02:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:02:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 06:02:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:02:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:02:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:02:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 06:02:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 06:02:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 06:02:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 06:02:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 06:02:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 06:02:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 06:02:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 06:02:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 06:02:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 06:02:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 06:02:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:02:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:02:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:02:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 06:02:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:02:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:02:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:02:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 06:02:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 06:02:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 06:02:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 06:02:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 06:02:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 06:02:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 06:02:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 06:02:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 06:02:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 06:02:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 06:02:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:02:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:02:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:02:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 06:02:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:02:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:02:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:02:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 06:02:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 06:02:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 06:02:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 06:02:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 06:02:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 06:02:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 06:02:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 06:02:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 06:02:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 06:02:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 06:02:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:02:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:02:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:02:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 06:02:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:02:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:02:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:02:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 06:02:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 06:02:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 06:02:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 06:02:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 06:02:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 06:02:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 06:02:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 06:02:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 06:02:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 06:02:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 06:02:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:02:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:02:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:02:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 06:02:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:02:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:02:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:02:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 06:02:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 06:02:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 06:02:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 06:02:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 06:02:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 06:02:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 06:02:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 06:02:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 06:02:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 06:02:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 06:02:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:02:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:02:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:02:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 06:02:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:02:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:02:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:02:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 06:02:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 06:02:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 06:02:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 06:02:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 06:02:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 06:02:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 06:02:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 06:02:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 06:02:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 06:02:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 06:02:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:02:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:02:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:02:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 06:02:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:02:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:02:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:02:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 06:02:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 06:02:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 06:02:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 06:02:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 06:02:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 06:02:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 06:02:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 06:02:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 06:02:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 06:02:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 06:02:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:02:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:02:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:02:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 06:02:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:02:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:02:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:02:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 06:02:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 06:02:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 06:02:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 06:02:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 06:02:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 06:02:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 06:02:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 06:02:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 06:02:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 06:02:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 06:02:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:02:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:02:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:02:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 06:02:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:02:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:02:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:02:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 06:02:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 06:02:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 06:02:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 06:02:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 06:02:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 06:02:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 06:02:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 06:02:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 06:02:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 06:02:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 06:02:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:02:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:02:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:02:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 06:02:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:02:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:02:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:02:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 06:02:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 06:02:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 06:02:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 06:02:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 06:02:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 06:02:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 06:02:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 06:02:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 06:02:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 06:02:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 06:02:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:02:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:02:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:02:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 06:02:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:02:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:02:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:02:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 06:02:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 06:02:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 06:02:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 06:02:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 06:02:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 06:02:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 06:02:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 06:02:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 06:02:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 06:03:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 06:03:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:03:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:03:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:03:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 06:03:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:03:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:03:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:03:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 06:03:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 06:03:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 06:03:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 06:03:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 06:03:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 06:03:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 06:03:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 06:03:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 06:03:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 06:03:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 06:03:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:03:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:03:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:03:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 06:03:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:03:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:03:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:03:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 06:03:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 06:03:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 06:03:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 06:03:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 06:03:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 06:03:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 06:03:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 06:03:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 06:03:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 06:03:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 06:03:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:03:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:03:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:03:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 06:03:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:03:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:03:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:03:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 06:03:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 06:03:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 06:03:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 06:03:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 06:03:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 06:03:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 06:03:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 06:03:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 06:03:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 06:03:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 06:03:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:03:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:03:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:03:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 06:03:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:03:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:03:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:03:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 06:03:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 06:03:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 06:03:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 06:03:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 06:03:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 06:03:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 06:03:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 06:03:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 06:03:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 06:03:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 06:03:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:03:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:03:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:03:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 06:03:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:03:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:03:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:03:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 06:03:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 06:03:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 06:03:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 06:03:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 06:03:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 06:03:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 06:03:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 06:03:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 06:03:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 06:03:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 06:03:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:03:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:03:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:03:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 06:03:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:03:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:03:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:03:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 06:03:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 06:03:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 06:03:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 06:03:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 06:03:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 06:03:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 06:03:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 06:03:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 06:03:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 06:03:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 06:03:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:03:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:03:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:03:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 06:03:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:03:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:03:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:03:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 06:03:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 06:03:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 06:03:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 06:03:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 06:03:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 06:03:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 06:03:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 06:03:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 06:03:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 06:03:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 06:03:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:03:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:03:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:03:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 06:03:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:03:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:03:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:03:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 06:03:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 06:03:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 06:03:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 06:03:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 06:03:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 06:03:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 06:03:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 06:03:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 06:03:39 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 06:03:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 06:03:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:03:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:03:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:03:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 06:03:44 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:03:44 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:03:44 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:03:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 06:03:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 06:03:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 06:03:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 06:03:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 06:03:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 06:03:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 06:03:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 06:03:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 06:03:44 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 06:03:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 06:03:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:03:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:03:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:03:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 06:03:49 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:03:49 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:03:49 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:03:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 06:03:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 06:03:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 06:03:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 06:03:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 06:03:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 06:03:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 06:03:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 06:03:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 06:03:49 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 06:03:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 06:03:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:03:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:03:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:03:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 06:03:54 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:03:54 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:03:54 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:03:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 06:03:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 06:03:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 06:03:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 06:03:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 06:03:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 06:03:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 06:03:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 06:03:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 06:03:54 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 06:03:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 06:03:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:03:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:03:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:03:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 06:03:59 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:03:59 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:03:59 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:03:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 06:03:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 06:03:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 06:03:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 06:03:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 06:03:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 06:03:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 06:03:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 06:03:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 06:03:59 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 06:04:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 06:04:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:04:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:04:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:04:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 06:04:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 06:04:04 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:04:04 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:04:04 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:04:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 06:04:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 06:04:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 06:04:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 06:04:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 06:04:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 06:04:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 06:04:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 06:04:04 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 06:04:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 06:04:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:04:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:04:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:04:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 06:04:09 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:04:09 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:04:09 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:04:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 06:04:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 06:04:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 06:04:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 06:04:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 06:04:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 06:04:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 06:04:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 06:04:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 06:04:09 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 06:04:10 rdma06 kernel: seq_file: buggy .next function _ctx_stats_seq_next [hfi1] did not update position index
Apr 19 06:04:10 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: INVALID_CSR_ADDR,HOLD_REINIT
Apr 19 06:04:10 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: INVALID_CSR_ADDR,HOLD_REINIT
Apr 19 06:04:10 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 06:04:10 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 06:04:10 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 06:04:10 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 06:04:10 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 06:04:10 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 06:04:10 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 06:04:10 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 06:04:10 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 06:04:10 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 06:04:10 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 06:04:10 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 06:04:10 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 06:04:10 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 06:04:10 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 06:04:10 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 06:04:10 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 06:04:10 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 06:04:10 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 06:04:10 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: Repeating LCB Err bits 0x8000000 - masking
Apr 19 06:04:10 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 06:04:10 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 06:04:10 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 06:04:10 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 06:04:10 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 06:04:10 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 06:04:10 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 06:04:10 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 06:04:10 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 06:04:10 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 06:04:10 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 06:04:10 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 06:04:10 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 06:04:10 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 06:04:10 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 06:04:10 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 06:04:10 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 06:04:10 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 06:04:10 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 06:04:10 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 06:04:10 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 06:04:11 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: Repeating LCB Err bits 0x8000000 - masking
Apr 19 06:04:13 rdma06 opafmd[1140340]:           SA RX REPORTRESP(Notice):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]: SA RX GET(DeviceGroupMemberRecord):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]: SA RX GETTBL(DeviceGroupMemberRecord):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:   SA RX GET(DeviceGroupNameRecord):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]: SA RX GETTBL(DeviceGroupNameRecord):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:  SA RX GET(DeviceTreeMemberRecord):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]: SA RX GETTBL(DeviceTreeMemberRecord):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:           SA RX DUPLICATE REQUESTS:          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]: SA RX DEAD RMPP TRANSACTION PACKET:          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:                SA DROPPED REQUESTS:          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:           SA NO AVAILABLE CONTEXTS:          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:          SA RX GETMULTI() Non-RMPP:          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:          SA RX GETMULTI RMPP Abort:          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:          SA TX GETMULTI RMPP Abort:          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:         SA TX GETMULTI Ack Retries:          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:                 SA TX RMPP Retries:          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:                    SA TX RMPP STOP:          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:          SA TX RMMP ABORT(TIMEOUT):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:    SA TX RMMP ABORT(BAD FIRST SEG):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:         SA TX RMMP ABORT(BAD TYPE):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]: SA TX RMMP ABORT(WINDOW TOO SMALL):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:   SA TX RMMP ABORT(SEGNUM TOO BIG):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:      SA TX RMMP ABORT(BAD VERSION):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]: SA TX RMMP ABORT(TOO MANY RETRIES):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:          SA TX RMMP ABORT(UNKNOWN):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:                  SA TX STATUS BUSY:          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:              SA TX STATUS REDIRECT:          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:              SA TX STATUS BADCLASS:          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:             SA TX STATUS BADMETHOD:          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:         SA TX STATUS BADMETHODATTR:          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:              SA TX STATUS BADFIELD:          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:          SA TX STATUS NO RESOURCES:          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:           SA TX STATUS INVALID REQ:          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:            SA TX STATUS NO RECORDS:          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:      SA TX STATUS TOO MANY RECORDS:          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:               SA TX STATUS BAD GID:          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:        SA TX STATUS TOO FEW FIELDS:          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:               SA TX STATUS UNKNOWN:          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:                DBSYNC Send Command:          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:        DBSYNC Send Command Failure:          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:                DBSYNC Receive Data:          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:        DBSYNC Receive Data Failure:          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:                       DBSYNC Reply:          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:               DBSYNC Reply Failure:          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:         SA Maximum Contexts In Use:          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:           SA Maximum Contexts Free:          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:        Maximum SM Sweep Time in ms:          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:         Seconds since last cleared: 185252
Apr 19 06:04:13 rdma06 opafmd[1140340]:                  Time last cleared: Mon Apr 17 02:36:41 2023
Apr 19 06:04:13 rdma06 opafmd[1140340]:                            COUNTER: THIS SWEEP LAST SWEEP      TOTAL
Apr 19 06:04:13 rdma06 opafmd[1140340]: ------------------------------------ ---------- ---------- ----------
Apr 19 06:04:13 rdma06 opafmd[1140340]:   SM State transition to DISCOVERY:          1          0          1
Apr 19 06:04:13 rdma06 opafmd[1140340]:      SM State transition to MASTER:          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:     SM State transition to STANDBY:          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:    SM State transition to INACTIVE:          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:      Total transmitted SMA Packets:      37031          0      37031
Apr 19 06:04:13 rdma06 opafmd[1140340]:          Direct Routed SMA Packets:      37030          0      37030
Apr 19 06:04:13 rdma06 opafmd[1140340]:             LID Routed SMA Packets:          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:              SMA Query Retransmits:          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:    SMA Query Retransmits Exhausted:          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:                  SM TX GET(Notice):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:                  SM TX SET(Notice):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:           Total SM RX TRAP(Notice):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:                 SM RX TRAP(PortUp):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:               SM RX TRAP(PortDown):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:            SM RX TRAP(McGrpCreate):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:            SM RX TRAP(McGrpDelete):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:                 SM RX TRAP(UnPath):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:                 SM RX TRAP(RePath):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:           SM RX TRAP(PortStateChg):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:          SM RX TRAP(LinkIntegrity):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:          SM RX TRAP(BufferOverrun):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:            SM RX TRAP(FlowControl):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:            SM RX TRAP(LocalChange):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:            SM RX TRAP(SysImageChg):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:                SM RX TRAP(BadMKey):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:                SM RX TRAP(BadPkey):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:                SM RX TRAP(BadQkey):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:          SM RX TRAP(BadPkeySwPort):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:     SM RX TRAP(LinkWidthDowngrade):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:       SM RX TRAP(CostMatrixChange):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:          SM TX TRAPREPRESS(Notice):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:         SM TX GET(NodeDescription):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:                SM TX GET(NodeInfo):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:              SM TX GET(SwitchInfo):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:              SM TX SET(SwitchInfo):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:                SM TX GET(PortInfo):      37030          0      37030
Apr 19 06:04:13 rdma06 opafmd[1140340]:                SM TX SET(PortInfo):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:           SM TX GET(PortStateInfo):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:           SM TX SET(PortStateInfo):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:               SM TX GET(PKeyTable):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:               SM TX SET(PKeyTable):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:       SM TX GET(SL2SCMappingTable):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:       SM TX SET(SL2SCMappingTable):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:       SM TX GET(SC2SLMappingTable):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:       SM TX SET(SC2SLMappingTable):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:      SM TX GET(SC2VLtMappingTable):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:      SM TX SET(SC2VLtMappingTable):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:     SM TX GET(SC2VLntMappingTable):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:     SM TX SET(SC2VLntMappingTable):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:      SM TX GET(SC2VLrMappingTable):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:      SM TX SET(SC2VLrMappingTable):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:       SM TX GET(SC2SCMappingTable):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:       SM TX SET(SC2SCMappingTable):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:           SM TX SET(SC2SCMultiSet):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:      SM TX GET(VLArbitrationTable):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:      SM TX SET(VLArbitrationTable):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:                     SM TX GET(LFT):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:                     SM TX SET(LFT):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:                     SM TX GET(MFT):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:                     SM TX SET(MFT):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:               SM TX GET(AGGREGATE):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:               SM TX SET(AGGREGATE):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:                 SM TX GET(LedInfo):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:                 SM TX SET(LedInfo):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:               SM TX GET(CableInfo):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:                  SM RX GET(SmInfo):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:                  SM TX GET(SmInfo):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:                  SM RX SET(SmInfo):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:                  SM TX SET(SmInfo):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:              SM TX GETRESP(SmInfo):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:               SM TX GET(PortGroup):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:               SM TX SET(PortGroup):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:                    SM TX GET(PGFT):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:                    SM TX SET(PGFT):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:      SM TX GET(BufferControlTable):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:      SM TX SET(BufferControlTable):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:               SM TX GET(ARLidMask):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:               SM TX SET(ARLidMask):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:          SM RX GET(CongestionInfo):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:       SM RX GET(HfiCongestSetting):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:       SM RX SET(HfiCongestSetting):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:       SM RX GET(HfiCongestControl):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:       SM RX SET(HfiCongestControl):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:        SM RX GET(SwCongestSetting):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:        SM RX SET(SwCongestSetting):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:  SM RX GET(SwPortCongestionSetting:          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:                   SM RX GETRESP(*):      74060          0      74060
Apr 19 06:04:13 rdma06 opafmd[1140340]:                  SM RX STATUS BUSY:          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:              SM RX STATUS REDIRECT:          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:              SM RX STATUS BADCLASS:          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:             SM RX STATUS BADMETHOD:          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:         SM RX STATUS BADMETHODATTR:          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:              SM RX STATUS BADFIELD:          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:               SM RX STATUS UNKNOWN:          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:           SA RX GET(ClassPortInfo):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:               SA TX REPORT(Notice):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:              SA RX SET(InformInfo):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:              SA RX GET(NodeRecord):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:           SA RX GETTBL(NodeRecord):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:                SA RX GET(PortInfo):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:             SA RX GETTBL(PortInfo):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:                SA RX GET(SC2SCMAP):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:             SA RX GETTBL(SC2SCMap):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:                SA RX GET(SL2SCMAP):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:             SA RX GETTBL(SL2SCMap):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:                SA RX GET(SC2SLMap):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:             SA RX GETTBL(SC2SLMap):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:               SA RX GET(SC2VLtMap):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:            SA RX GETTBL(SC2VLtMap):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:              SA RX GET(SC2VLntMap):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:           SA RX GETTBL(SC2VLntMap):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:               SA RX GET(SC2VLrMap):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:            SA RX GETTBL(SC2VLrMap):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:              SA RX GET(SwitchInfo):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:          SA RX GETTBLE(SwitchInfo):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:                     SA RX GET(LFT):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:                  SA RX GETTBL(LFT):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:                     SA RX GET(MFT):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:                  SA RX GETTBL(MFT):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:           SA RX GET(VLArbitration):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:        SA RX GETTBL(VLArbitration):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:                  SA RX GET(SmInfo):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:               SA RX GETTBL(SmInfo):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:              SA RX GET(InformInfo):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:           SA RX GETTBL(InformInfo):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:              SA RX GET(LinkRecord):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:           SA RX GETTBL(LinkRecord):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:           SA RX GET(ServiceRecord):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:           SA RX SET(ServiceRecord):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:        SA RX GETTBL(ServiceRecord):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:        SA RX DELETE(ServiceRecord):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:              SA RX GET(PKeyRecord):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:           SA RX GETTBL(PKeyRecord):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:              SA RX GET(PathRecord):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:           SA RX GETTBL(PathRecord):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:          SA RX GET(McMemberRecord):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:          SA RX SET(McMemberRecord):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:       SA RX GETTBL(McMemberRecord):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:       SA RX DELETE(McMemberRecord):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:        SA RX GETTRACE(TraceRecord):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:    SA RX GETMULTI(MultiPathRecord):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:           SA RX GET(vFabricRecord):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:        SA RX GETTBL(vFabricRecord):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:        SA RX GET(FabricInfoRecord):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:   SA RX GET(QuarantinedNodeRecord):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]: SA RX GETTBL(QuarantinedNodeRecord):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:          SA RX GET(CongInfoRecord):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:       SA RX GETTBL(CongInfoRecord):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:        SA RX GET(SwitchCongRecord):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:     SA RX GETTBL(SwitchCongRecord):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:    SA RX GET(SwitchPortCongRecord):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]: SA RX GETTBL(SwitchPortCongRecord):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:           SA RX GET(HFICongRecord):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:           SA RX GET(HFICongRecord):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:       SA RX GET(HFICongCtrlRecord):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:       SA RX GET(HFICongCtrlRecord):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:           SA RX GET(BfrCtrlRecord):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:        SA RX GETTBL(BfrCtrlRecord):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:               SA RX GET(CableInfo):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:            SA RX GETTBL(CableInfo):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:         SA RX GET(PortGroupRecord):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:      SA RX GETTBL(PortGroupRecord):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:      SA RX GET(PortGroupFwdRecord):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:   SA RX GETTBL(PortGroupFwdRecord):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:                             (null):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:                             (null):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:           SA RX REPORTRESP(Notice):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]: SA RX GET(DeviceGroupMemberRecord):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]: SA RX GETTBL(DeviceGroupMemberRecord):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:   SA RX GET(DeviceGroupNameRecord):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]: SA RX GETTBL(DeviceGroupNameRecord):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:  SA RX GET(DeviceTreeMemberRecord):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]: SA RX GETTBL(DeviceTreeMemberRecord):          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:           SA RX DUPLICATE REQUESTS:          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]: SA RX DEAD RMPP TRANSACTION PACKET:          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:                SA DROPPED REQUESTS:          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:           SA NO AVAILABLE CONTEXTS:          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:          SA RX GETMULTI() Non-RMPP:          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:          SA RX GETMULTI RMPP Abort:          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:          SA TX GETMULTI RMPP Abort:          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:         SA TX GETMULTI Ack Retries:          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:                 SA TX RMPP Retries:          0          0          0
Apr 19 06:04:13 rdma06 opafmd[1140340]:                    SA TX RMPP STOP:          0          0          0
Apr 19 06:04:14 rdma06 opafmd[1140340]:          SA TX RMMP ABORT(TIMEibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 06:04:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:04:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:04:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:04:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 06:04:14 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:04:14 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:04:14 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:04:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 06:04:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 06:04:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 06:04:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 06:04:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 06:04:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 06:04:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 06:04:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 06:04:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 06:04:14 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 06:04:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 06:04:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:04:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:04:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:04:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 06:04:19 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:04:19 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:04:19 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:04:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 06:04:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 06:04:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 06:04:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 06:04:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 06:04:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 06:04:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 06:04:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 06:04:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 06:04:19 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 06:04:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 06:04:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:04:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:04:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:04:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 06:04:24 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:04:24 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:04:24 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:04:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 06:04:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 06:04:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 06:04:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 06:04:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 06:04:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 06:04:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 06:04:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 06:04:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 06:04:24 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 06:04:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 06:04:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:04:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:04:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:04:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 06:04:29 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:04:29 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:04:29 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:04:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 06:04:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 06:04:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 06:04:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 06:04:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 06:04:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 06:04:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 06:04:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 06:04:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 06:04:29 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 06:04:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 06:04:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:04:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:04:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:04:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 06:04:34 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:04:34 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:04:34 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:04:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 06:04:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 06:04:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 06:04:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 06:04:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 06:04:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 06:04:34 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 06:04:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 06:04:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 06:04:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 06:04:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 06:04:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:04:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:04:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:04:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 06:04:39 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:04:39 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:04:39 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:04:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 06:04:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 06:04:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 06:04:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 06:04:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 06:04:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 06:04:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 06:04:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 06:04:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 06:04:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 06:04:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 06:04:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:04:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:04:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:04:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 06:04:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:04:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:04:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:04:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 06:04:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 06:04:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 06:04:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 06:04:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 06:04:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 06:04:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 06:04:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 06:04:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 06:04:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 06:04:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 06:04:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:04:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:04:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:04:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 06:04:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:04:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:04:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:04:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 06:04:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 06:04:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 06:04:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 06:04:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 06:04:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 06:04:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 06:04:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 06:04:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 06:04:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 06:04:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 06:04:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:04:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:04:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:04:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 06:04:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:04:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:04:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:04:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 06:04:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 06:04:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 06:04:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 06:04:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 06:04:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 06:04:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 06:04:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 06:04:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 06:04:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 06:05:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 06:05:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:05:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:05:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:05:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 06:05:00 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:05:00 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:05:00 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:05:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 06:05:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 06:05:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 06:05:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 06:05:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 06:05:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 06:05:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 06:05:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 06:05:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 06:05:00 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 06:05:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 06:05:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:05:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:05:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:05:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 06:05:05 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:05:05 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:05:05 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:05:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 06:05:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 06:05:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 06:05:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 06:05:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 06:05:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 06:05:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 06:05:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 06:05:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 06:05:05 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 06:05:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 06:05:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:05:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:05:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:05:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 06:05:10 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:05:10 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:05:10 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:05:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 06:05:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 06:05:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 06:05:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 06:05:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 06:05:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 06:05:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 06:05:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 06:05:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 06:05:10 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 06:05:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 06:05:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:05:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:05:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:05:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 06:05:15 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:05:15 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:05:15 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:05:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 06:05:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 06:05:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 06:05:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 06:05:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 06:05:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 06:05:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 06:05:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 06:05:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 06:05:15 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 06:05:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 06:05:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:05:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:05:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:05:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 06:05:20 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:05:20 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:05:20 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:05:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 06:05:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 06:05:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 06:05:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 06:05:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 06:05:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 06:05:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 06:05:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 06:05:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 06:05:20 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 06:05:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 06:05:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:05:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:05:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:05:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 06:05:25 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:05:25 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:05:25 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:05:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 06:05:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 06:05:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 06:05:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 06:05:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 06:05:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 06:05:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 06:05:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 06:05:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 06:05:25 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 06:05:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 06:05:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:05:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:05:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:05:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 06:05:30 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:05:30 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:05:30 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:05:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 06:05:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 06:05:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 06:05:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 06:05:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 06:05:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 06:05:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 06:05:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 06:05:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 06:05:30 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 06:05:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 06:05:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:05:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:05:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:05:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 06:05:35 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:05:35 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:05:35 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:05:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 06:05:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 06:05:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 06:05:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 06:05:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 06:05:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 06:05:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 06:05:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 06:05:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 06:05:35 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 06:05:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 06:05:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:05:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:05:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:05:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 06:05:40 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:05:40 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:05:40 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:05:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 06:05:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 06:05:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 06:05:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 06:05:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 06:05:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 06:05:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 06:05:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 06:05:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 06:05:40 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 06:05:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 06:05:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:05:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:05:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:05:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 06:05:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 06:05:45 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:05:45 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:05:45 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:05:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 06:05:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 06:05:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 06:05:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 06:05:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 06:05:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 06:05:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 06:05:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 06:05:45 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 06:05:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 06:05:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:05:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:05:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:05:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 06:05:50 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:05:50 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:05:50 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:05:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 06:05:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 06:05:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 06:05:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 06:05:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 06:05:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 06:05:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 06:05:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 06:05:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 06:05:50 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
Apr 19 06:05:51 rdma06 kernel: seq_file: buggy .next function _ctx_stats_seq_next [hfi1] did not update position index
Apr 19 06:05:51 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: INVALID_CSR_ADDR,HOLD_REINIT
Apr 19 06:05:51 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: INVALID_CSR_ADDR,HOLD_REINIT
Apr 19 06:05:51 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 06:05:51 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 06:05:51 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 06:05:51 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 06:05:51 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 06:05:51 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 06:05:51 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 06:05:51 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 06:05:51 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 06:05:51 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 06:05:51 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 06:05:51 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 06:05:51 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 06:05:51 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 06:05:51 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 06:05:51 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 06:05:51 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 06:05:51 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 06:05:51 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 06:05:51 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: Repeating LCB Err bits 0x8000000 - masking
Apr 19 06:05:51 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 06:05:51 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 06:05:51 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 06:05:51 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 06:05:51 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 06:05:51 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 06:05:51 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 06:05:51 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 06:05:51 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 06:05:51 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 06:05:51 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 06:05:51 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 06:05:51 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 06:05:51 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 06:05:51 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 06:05:51 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 06:05:51 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 06:05:51 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 06:05:51 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 06:05:51 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 06:05:51 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: LCB Error: HOLD_REINIT
Apr 19 06:05:51 rdma06 kernel: hfi1 0000:21:00.0: hfi1_0: Repeating LCB Err bits 0x8000000 - masking
Apr 19 06:05:54 rdma06 opafmd[1140340]: OUT):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:    SA TX RMMP ABORT(BAD FIRST SEG):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:         SA TX RMMP ABORT(BAD TYPE):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]: SA TX RMMP ABORT(WINDOW TOO SMALL):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:   SA TX RMMP ABORT(SEGNUM TOO BIG):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:      SA TX RMMP ABORT(BAD VERSION):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]: SA TX RMMP ABORT(TOO MANY RETRIES):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:          SA TX RMMP ABORT(UNKNOWN):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:                  SA TX STATUS BUSY:          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:              SA TX STATUS REDIRECT:          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:              SA TX STATUS BADCLASS:          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:             SA TX STATUS BADMETHOD:          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:         SA TX STATUS BADMETHODATTR:          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:              SA TX STATUS BADFIELD:          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:          SA TX STATUS NO RESOURCES:          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:           SA TX STATUS INVALID REQ:          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:            SA TX STATUS NO RECORDS:          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:      SA TX STATUS TOO MANY RECORDS:          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:               SA TX STATUS BAD GID:          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:        SA TX STATUS TOO FEW FIELDS:          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:               SA TX STATUS UNKNOWN:          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:                DBSYNC Send Command:          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:        DBSYNC Send Command Failure:          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:                DBSYNC Receive Data:          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:        DBSYNC Receive Data Failure:          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:                       DBSYNC Reply:          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:               DBSYNC Reply Failure:          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:         SA Maximum Contexts In Use:          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:           SA Maximum Contexts Free:          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:        Maximum SM Sweep Time in ms:          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:         Seconds since last cleared: 185353
Apr 19 06:05:54 rdma06 opafmd[1140340]:                  Time last cleared: Mon Apr 17 02:36:41 2023
Apr 19 06:05:54 rdma06 opafmd[1140340]:                            COUNTER: THIS SWEEP LAST SWEEP      TOTAL
Apr 19 06:05:54 rdma06 opafmd[1140340]: ------------------------------------ ---------- ---------- ----------
Apr 19 06:05:54 rdma06 opafmd[1140340]:   SM State transition to DISCOVERY:          1          0          1
Apr 19 06:05:54 rdma06 opafmd[1140340]:      SM State transition to MASTER:          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:     SM State transition to STANDBY:          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:    SM State transition to INACTIVE:          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:      Total transmitted SMA Packets:      37051          0      37051
Apr 19 06:05:54 rdma06 opafmd[1140340]:          Direct Routed SMA Packets:      37050          0      37050
Apr 19 06:05:54 rdma06 opafmd[1140340]:             LID Routed SMA Packets:          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:              SMA Query Retransmits:          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:    SMA Query Retransmits Exhausted:          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:                  SM TX GET(Notice):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:                  SM TX SET(Notice):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:           Total SM RX TRAP(Notice):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:                 SM RX TRAP(PortUp):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:               SM RX TRAP(PortDown):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:            SM RX TRAP(McGrpCreate):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:            SM RX TRAP(McGrpDelete):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:                 SM RX TRAP(UnPath):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:                 SM RX TRAP(RePath):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:           SM RX TRAP(PortStateChg):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:          SM RX TRAP(LinkIntegrity):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:          SM RX TRAP(BufferOverrun):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:            SM RX TRAP(FlowControl):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:            SM RX TRAP(LocalChange):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:            SM RX TRAP(SysImageChg):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:                SM RX TRAP(BadMKey):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:                SM RX TRAP(BadPkey):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:                SM RX TRAP(BadQkey):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:          SM RX TRAP(BadPkeySwPort):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:     SM RX TRAP(LinkWidthDowngrade):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:       SM RX TRAP(CostMatrixChange):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:          SM TX TRAPREPRESS(Notice):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:         SM TX GET(NodeDescription):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:                SM TX GET(NodeInfo):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:              SM TX GET(SwitchInfo):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:              SM TX SET(SwitchInfo):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:                SM TX GET(PortInfo):      37050          0      37050
Apr 19 06:05:54 rdma06 opafmd[1140340]:                SM TX SET(PortInfo):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:           SM TX GET(PortStateInfo):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:           SM TX SET(PortStateInfo):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:               SM TX GET(PKeyTable):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:               SM TX SET(PKeyTable):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:       SM TX GET(SL2SCMappingTable):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:       SM TX SET(SL2SCMappingTable):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:       SM TX GET(SC2SLMappingTable):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:       SM TX SET(SC2SLMappingTable):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:      SM TX GET(SC2VLtMappingTable):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:      SM TX SET(SC2VLtMappingTable):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:     SM TX GET(SC2VLntMappingTable):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:     SM TX SET(SC2VLntMappingTable):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:      SM TX GET(SC2VLrMappingTable):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:      SM TX SET(SC2VLrMappingTable):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:       SM TX GET(SC2SCMappingTable):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:       SM TX SET(SC2SCMappingTable):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:           SM TX SET(SC2SCMultiSet):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:      SM TX GET(VLArbitrationTable):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:      SM TX SET(VLArbitrationTable):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:                     SM TX GET(LFT):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:                     SM TX SET(LFT):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:                     SM TX GET(MFT):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:                     SM TX SET(MFT):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:               SM TX GET(AGGREGATE):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:               SM TX SET(AGGREGATE):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:                 SM TX GET(LedInfo):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:                 SM TX SET(LedInfo):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:               SM TX GET(CableInfo):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:                  SM RX GET(SmInfo):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:                  SM TX GET(SmInfo):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:                  SM RX SET(SmInfo):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:                  SM TX SET(SmInfo):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:              SM TX GETRESP(SmInfo):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:               SM TX GET(PortGroup):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:               SM TX SET(PortGroup):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:                    SM TX GET(PGFT):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:                    SM TX SET(PGFT):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:      SM TX GET(BufferControlTable):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:      SM TX SET(BufferControlTable):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:               SM TX GET(ARLidMask):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:               SM TX SET(ARLidMask):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:          SM RX GET(CongestionInfo):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:       SM RX GET(HfiCongestSetting):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:       SM RX SET(HfiCongestSetting):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:       SM RX GET(HfiCongestControl):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:       SM RX SET(HfiCongestControl):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:        SM RX GET(SwCongestSetting):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:        SM RX SET(SwCongestSetting):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:  SM RX GET(SwPortCongestionSetting:          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:                   SM RX GETRESP(*):      74100          0      74100
Apr 19 06:05:54 rdma06 opafmd[1140340]:                  SM RX STATUS BUSY:          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:              SM RX STATUS REDIRECT:          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:              SM RX STATUS BADCLASS:          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:             SM RX STATUS BADMETHOD:          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:         SM RX STATUS BADMETHODATTR:          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:              SM RX STATUS BADFIELD:          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:               SM RX STATUS UNKNOWN:          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:           SA RX GET(ClassPortInfo):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:               SA TX REPORT(Notice):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:              SA RX SET(InformInfo):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:              SA RX GET(NodeRecord):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:           SA RX GETTBL(NodeRecord):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:                SA RX GET(PortInfo):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:             SA RX GETTBL(PortInfo):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:                SA RX GET(SC2SCMAP):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:             SA RX GETTBL(SC2SCMap):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:                SA RX GET(SL2SCMAP):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:             SA RX GETTBL(SL2SCMap):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:                SA RX GET(SC2SLMap):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:             SA RX GETTBL(SC2SLMap):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:               SA RX GET(SC2VLtMap):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:            SA RX GETTBL(SC2VLtMap):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:              SA RX GET(SC2VLntMap):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:           SA RX GETTBL(SC2VLntMap):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:               SA RX GET(SC2VLrMap):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:            SA RX GETTBL(SC2VLrMap):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:              SA RX GET(SwitchInfo):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:          SA RX GETTBLE(SwitchInfo):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:                     SA RX GET(LFT):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:                  SA RX GETTBL(LFT):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:                     SA RX GET(MFT):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:                  SA RX GETTBL(MFT):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:           SA RX GET(VLArbitration):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:        SA RX GETTBL(VLArbitration):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:                  SA RX GET(SmInfo):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:               SA RX GETTBL(SmInfo):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:              SA RX GET(InformInfo):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:           SA RX GETTBL(InformInfo):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:              SA RX GET(LinkRecord):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:           SA RX GETTBL(LinkRecord):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:           SA RX GET(ServiceRecord):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:           SA RX SET(ServiceRecord):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:        SA RX GETTBL(ServiceRecord):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:        SA RX DELETE(ServiceRecord):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:              SA RX GET(PKeyRecord):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:           SA RX GETTBL(PKeyRecord):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:              SA RX GET(PathRecord):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:           SA RX GETTBL(PathRecord):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:          SA RX GET(McMemberRecord):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:          SA RX SET(McMemberRecord):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:       SA RX GETTBL(McMemberRecord):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:       SA RX DELETE(McMemberRecord):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:        SA RX GETTRACE(TraceRecord):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:    SA RX GETMULTI(MultiPathRecord):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:           SA RX GET(vFabricRecord):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:        SA RX GETTBL(vFabricRecord):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:        SA RX GET(FabricInfoRecord):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:   SA RX GET(QuarantinedNodeRecord):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]: SA RX GETTBL(QuarantinedNodeRecord):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:          SA RX GET(CongInfoRecord):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:       SA RX GETTBL(CongInfoRecord):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:        SA RX GET(SwitchCongRecord):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:     SA RX GETTBL(SwitchCongRecord):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:    SA RX GET(SwitchPortCongRecord):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]: SA RX GETTBL(SwitchPortCongRecord):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:           SA RX GET(HFICongRecord):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:           SA RX GET(HFICongRecord):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:       SA RX GET(HFICongCtrlRecord):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:       SA RX GET(HFICongCtrlRecord):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:           SA RX GET(BfrCtrlRecord):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:        SA RX GETTBL(BfrCtrlRecord):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:               SA RX GET(CableInfo):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:            SA RX GETTBL(CableInfo):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:         SA RX GET(PortGroupRecord):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:      SA RX GETTBL(PortGroupRecord):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:      SA RX GET(PortGroupFwdRecord):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:   SA RX GETTBL(PortGroupFwdRecord):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:                             (null):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:                             (null):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:           SA RX REPORTRESP(Notice):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]: SA RX GET(DeviceGroupMemberRecord):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]: SA RX GETTBL(DeviceGroupMemberRecord):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:   SA RX GET(DeviceGroupNameRecord):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]: SA RX GETTBL(DeviceGroupNameRecord):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:  SA RX GET(DeviceTreeMemberRecord):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]: SA RX GETTBL(DeviceTreeMemberRecord):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:           SA RX DUPLICATE REQUESTS:          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]: SA RX DEAD RMPP TRANSACTION PACKET:          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:                SA DROPPED REQUESTS:          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:           SA NO AVAILABLE CONTEXTS:          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:          SA RX GETMULTI() Non-RMPP:          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:          SA RX GETMULTI RMPP Abort:          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:          SA TX GETMULTI RMPP Abort:          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:         SA TX GETMULTI Ack Retries:          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:                 SA TX RMPP Retries:          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:                    SA TX RMPP STOP:          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:          SA TX RMMP ABORT(TIMEOUT):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:    SA TX RMMP ABORT(BAD FIRST SEG):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:         SA TX RMMP ABORT(BAD TYPE):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]: SA TX RMMP ABORT(WINDOW TOO SMALL):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:   SA TX RMMP ABORT(SEGNUM TOO BIG):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:      SA TX RMMP ABORT(BAD VERSION):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]: SA TX RMMP ABORT(TOO MANY RETRIES):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:          SA TX RMMP ABORT(UNKNOWN):          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:                  SA TX STATUS BUSY:          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:              SA TX STATUS REDIRECT:          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:              SA TX STATUS BADCLASS:          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:             SA TX STATUS BADMETHOD:          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:         SA TX STATUS BADMETHODATTR:          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:              SA TX STATUS BADFIELD:          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:          SA TX STATUS NO RESOURCES:          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:           SA TX STATUS INVALID REQ:          0          0          0
Apr 19 06:05:54 rdma06 opafmd[1140340]:            SA TX STATUS NO RECORDS:          0          0          0
Apr 19 06:05:55 rdma06 opafmd[1140340]:      SA TX STATUS TOO MANY RECORDS:          0          0       ibwarn: [1140340] umad_dump: agent id 0 status 0 timeout 0
Apr 19 06:05:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:05:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:05:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:05:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_dump: agent id 2 status 0 timeout 0
Apr 19 06:05:55 rdma06 opafmd[1140340]: ibwarn: [1140340] umad_addr_dump: qpn 0 qkey 0x0 lid 65535 sl 0
Apr 19 06:05:55 rdma06 opafmd[1140340]: grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x1
Apr 19 06:05:55 rdma06 opafmd[1140340]: Gid 0x00000000000000000000000000000000
Apr 19 06:05:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  Management Class 0x81 method 0x1 attrId 0x15 attrM 0x1000200
Apr 19 06:05:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2:  base_version 0x80 class_version 0x80
Apr 19 06:05:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid -1 qpn 0 qkey 0 sl 0
Apr 19 06:05:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: dlid: 0xffffffff, slid: 0x1
Apr 19 06:05:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: P_Key 0xffff not found in pkey table
Apr 19 06:05:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: ... using 0x7fff found at index 1
Apr 19 06:05:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_send_mad2: >>> sending: len 176 pktsz 240
Apr 19 06:05:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: UMAD Status: Success (0)
Apr 19 06:05:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc: Received MAD length=424, total umad size=488
Apr 19 06:05:55 rdma06 fm0_sm[1140340]: opamgt: [1140340] omgt_recv_mad_alloc:   Base_Version 0x80 Class 0x81 Method 0x81 attrId 0x1500 attr_mod 0x20001 status 0x80
